Commit Graph

36 Commits

Author SHA1 Message Date
Jim Ma c5f4ca50fa
fix: grpc download tidy file error (#1697)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:57 +08:00
Jim Ma f4d34e56ed
chore: update download rpc check (#1684)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:56 +08:00
Jim Ma 7bceca51bc
chore: update debug info (#1617)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:51 +08:00
Jim Ma 98fb1fc427
feat: add grpc mux transport (#1602)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:49 +08:00
Gaius 272f38f567
feat: grpc add ratelimit (#1572)
feat: grpc ratelimit

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:46 +08:00
Jim Ma f6506115a7
feat: support grpc recursive download (#1518)
* feat: daemon grpc recursive download
* fix: init storage error

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:42 +08:00
Jim Ma 4984cdfbbc
fix: check same peer id for sync pieces (#1525)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:42 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515)
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:41 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418)
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:31 +08:00
Gaius 64f5e9357b
refactor: scheduler announce task (#1407)
* feat: scheduler announce normal task

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove cid

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:30 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Jim Ma 1a8b293a66
fix: seed peer did not send done seed result and no content length send (#1316)
fix: seed peer did not send done seed result
fix: unknown length back source seed did not send content length to other peers

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:21 +08:00
Gaius 2a0314e45b
refactor: scheduler grpc (#1310)
* refactor: scheduler grpc

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate protoc

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:20 +08:00
Jim Ma f3750b1ac0
feat: back source when receive seed request (#1309)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:20 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292)
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
Jim Ma 429252a071
feat: implement pattern in client daemon (#1231)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
Jim Ma 49f9ebde68
feat: implement client seed mode (#1247)
* feat: implement client seed mode

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:18 +08:00
Jim Ma 41f766168b
chore: add sync pieces trace and update sync pieces logic for done task (#1263)
* chore: add sync pieces trace

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: when task is done, sync piece tasks will still receive remote request

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: go context lint

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* Update client/daemon/rpcserver/rpcserver.go

Co-authored-by: cndoit18 <cndoit18@outlook.com>
Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: record close send error

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: clean PeerPacket when peer task completed

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize sync piece completed check

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix go lint

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: update sync piece test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

Co-authored-by: cndoit18 <cndoit18@outlook.com>
2023-06-28 17:33:15 +08:00
Jim Ma 902fbb031b
chore: optimize sync pieces (#1253)
* chore: optimize sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:14 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227)
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma 73c5e3a575
fix: sync pieces hang (#1221)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:11 +08:00
Gaius 2bcdba6401
feat: add grpc health interface (#1195)
* feat: add grpc health interface

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: add dfdaemon upload server healthy interface

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:09 +08:00
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165)
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:06 +08:00
Jim Ma fc2b1f8187
chore: init url meta in rpc server (#1098)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:59 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078)
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:59 +08:00
Jim Ma 6f76c74838
chore: clarify daemon interface (#991)
1. Rename FilePeerTask to FileTask.
2. Rename StreamPeerTask to StreamTask.
3. Extract pulling piece task logic to pieceTaskPoller.

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977)
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:49 +08:00
Jim Ma 7ba341fc5c
fix: infinitely get pieces when piece num is invalid (#926)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:47 +08:00
Gaius cb4202319e
feat: make idgen package public (#931)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:47 +08:00
sunwp d79a2a6877
move RPC code definition to proto file (#829)
* move rpc code to proto
* replace Cdn to CDN

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:38 +08:00
Gaius 558782af00
feat: add idgen peer id (#800)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:35 +08:00
Gaius 466912aa48
docs: scheduler config (#698)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:25 +08:00
Jim Ma af99b88616
feat: support limit from dfget client (#578)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:39 +08:00
Jim Ma 8180a52470
chore: optimize client rpc package name and other docs (#541)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:35 +08:00