dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,171 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

72 Commits

Author SHA1 Message Date
Jim Ma 48846e2ff6
feat: peer exchange (#3141) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2024-04-09 18:29:57 +08:00
Jim Ma d18625b3e2
fix: parent peertask conductor race condition (#3154) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2024-03-29 15:44:08 +08:00
Jim Ma 09b11a280a
chore: optimize stream task id (#2983) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-12-29 16:16:06 +08:00
Gaius 5b7ea5da2a
Revert "feat: replace fmt.Sprintf with net.JoinHostPort" (#2649) 
Revert "feat: replace fmt.Sprintf with net.JoinHostPort (#2642)"

This reverts commit a6863f9b19.
 2023-08-17 17:21:50 +08:00
Gaius a6863f9b19
feat: replace fmt.Sprintf with net.JoinHostPort (#2642) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-08-16 16:45:14 +08:00
Jim Ma a64140ff72
fix: potential nil panic (#2602) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-08-07 17:55:03 +08:00
Gaius b3f23f71a3
feat: update dy7.io/api to v2 (#2558) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-07-20 23:17:57 +08:00
Jim Ma 9137707139
feat: support breakpoint resume for running tasks (#2457) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:58 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius f7a399b47a
feat: add v2 version of the idgen (#2056) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 1666a17827
feat: remove NetTopology from scheduler and manager (#2007) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:25 +08:00
Gaius e83fb85216
feat: add v2 verison of the grpc to scheduler (#1999) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:24 +08:00
Gaius 91454c8a60
feat: remove callsystem and pattern (#1925) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:17 +08:00
Jim Ma 876c2ae17e
feat: support split running tasks (#1794) 
* feat: support split running tasks

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: go tests

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: add split runnging tasks e2e config

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: actions

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: actions

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize get running tasks logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize get running tasks logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:06 +08:00
greenhandatsjtu bc8d623b3d
feat: add traffic shaper for download tasks (#1654) 
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
 2023-06-28 17:33:59 +08:00
Jim Ma c5f4ca50fa
fix: grpc download tidy file error (#1697) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:57 +08:00
Jim Ma 73de99353d
feat: empty file e2e (#1687) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:56 +08:00
Jim Ma 7224801bae
feat: refactor peertask option (#1600) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Jim Ma ecfe467585
feat: add client grpc dial timeout (#1599) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Jim Ma 717ac99ef5
feat: daemon support auto issue certificate (#1586) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Jim Ma f6506115a7
feat: support grpc recursive download (#1518) 
* feat: daemon grpc recursive download
* fix: init storage error

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:42 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:41 +08:00
Jim Ma c921c265c5
fix: init storage error (#1486) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:38 +08:00
Jim Ma cf7debf9c4
feat: back source error detail (#1437) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:34 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418) 
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:31 +08:00
Gaius 30313a4caa
feat: client add create object storage (#1409) 
* feat: add create object storage rest api

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:30 +08:00
Gaius 64f5e9357b
refactor: scheduler announce task (#1407) 
* feat: scheduler announce normal task

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove cid

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:30 +08:00
Gaius 6fe519a233
fix: seed peer reuse value 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:23 +08:00
Gaius 5763ee86bd
feat: add seed peer metrics (#1342) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:23 +08:00
Jim Ma 3c6af10433
feat: add sync piece watchdog (#1272) 
* feat: add sync piece watchdog

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:21 +08:00
Gaius 2a0314e45b
refactor: scheduler grpc (#1310) 
* refactor: scheduler grpc

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate protoc

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:20 +08:00
Jim Ma f3750b1ac0
feat: back source when receive seed request (#1309) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:20 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292) 
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma 429252a071
feat: implement pattern in client daemon (#1231) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma 49f9ebde68
feat: implement client seed mode (#1247) 
* feat: implement client seed mode

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:18 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227) 
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:13 +08:00
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165) 
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:06 +08:00
Jim Ma ac4c4bb912
feat: reuse partial completed task (#1107) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:00 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078) 
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma 5478414751
feat: add prefetch metric in client (#1068) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:58 +08:00
Jim Ma f80c75efdb
Feature: prefetch ranged requests (#1053) 
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
Jim Ma 6e636b1125
chore: optimize metrics and trace in daemon (#1022) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma a0049b17ae
chore: register to scheduler after updated running tasks (#1016) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma e9f824e0a7
chore: optimize defer and test (#1010) 
* chore: optimize defer and test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: random test failed

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma c00f07c2de
Extract storage driver into peer tasks (#998) 
* chore: extract storage instead load every time

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: gofmt

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:52 +08:00
Jim Ma c3951e0cb3
feat: trace https proxy request (#996) 
* feat: trace https proxy request

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon test

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma 6f76c74838
chore: clarify daemon interface (#991) 
1. Rename FilePeerTask to FileTask.
2. Rename StreamPeerTask to StreamTask.
3. Extract pulling piece task logic to pieceTaskPoller.

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma 17ff9c17be
fix: send piece result error not handled (#987) 
* fix: send piece result error not handled
* chore: optimize daemon cancel logic
* chore: remove redundant log and ctx.Done check
* chore: handle piece download with 404 status

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:50 +08:00