dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,653 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

63 Commits

Author SHA1 Message Date
embroede 8c905d9b8d
feat: enable configuration of some directory modes for dfdaemon (#2340) 
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:57 +08:00
Jim Ma f5aff5e27f
feat: enhance daemon health check (#2130) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:35 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius f7a399b47a
feat: add v2 version of the idgen (#2056) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 1666a17827
feat: remove NetTopology from scheduler and manager (#2007) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:25 +08:00
Gaius e83fb85216
feat: add v2 verison of the grpc to scheduler (#1999) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:24 +08:00
Jim Ma 7cb802aeb7
feat: remove legacy peers support (#1939) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:19 +08:00
Gaius 91454c8a60
feat: remove callsystem and pattern (#1925) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:17 +08:00
Jim Ma 0a8f5118c9
feat: support reregister peer task in client (#1876) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:13 +08:00
Jim Ma 55d5308b19
fix: open end range in concurrent back source (#1764) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:03 +08:00
greenhandatsjtu bc8d623b3d
feat: add traffic shaper for download tasks (#1654) 
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
 2023-06-28 17:33:59 +08:00
Jim Ma c5f4ca50fa
fix: grpc download tidy file error (#1697) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:57 +08:00
Jim Ma 40300990de
feat: support download empty file (#1686) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:56 +08:00
Jim Ma 7224801bae
feat: refactor peertask option (#1600) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Jim Ma ecfe467585
feat: add client grpc dial timeout (#1599) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Gaius 272f38f567
feat: grpc add ratelimit (#1572) 
feat: grpc ratelimit

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:46 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:41 +08:00
Gaius ed4c381383
feat: rename steal peers to candidate peers (#1476) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:37 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius a2511cb945
feat: rewrite interface{} to any (#1419) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418) 
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 15d3274224
feat: rename digest func and add new digest func (#1405) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius 5763ee86bd
feat: add seed peer metrics (#1342) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:23 +08:00
Gaius 2a0314e45b
refactor: scheduler grpc (#1310) 
* refactor: scheduler grpc

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate protoc

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:20 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292) 
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma 49f9ebde68
feat: implement client seed mode (#1247) 
* feat: implement client seed mode

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:18 +08:00
Jim Ma 3682b26647
feat: in tree plugin (#1276) 
* feat: in tree plugin

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:16 +08:00
Gaius 076ba20b51
feat: move dfnet to pkg dir (#1265) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:15 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227) 
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:13 +08:00
Gaius 3fdde9b891
fix: make generate (#1228) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:11 +08:00
Jim Ma 10da5fa0d6
feat: implement grpc client side sync pieces (#1167) 
* feat: implement grpc client side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: client unit test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: report sync piece error

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize pieceTaskSyncManager cancel

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:07 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078) 
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma f80c75efdb
Feature: prefetch ranged requests (#1053) 
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
Jim Ma a0049b17ae
chore: register to scheduler after updated running tasks (#1016) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma e9f824e0a7
chore: optimize defer and test (#1010) 
* chore: optimize defer and test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: random test failed

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma c00f07c2de
Extract storage driver into peer tasks (#998) 
* chore: extract storage instead load every time

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: gofmt

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:52 +08:00
Gaius 43d769b0b4
fix: register task with size scope (#1003) 
* feat: register task with size scope

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: fix peer manager data race

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:52 +08:00
Jim Ma 0c03af1979
fix: when scheduler is not available, replace the scheduler client (#999) 
* fix: when scheduler is not available, just replace the scheduler client in single peer task

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma c3951e0cb3
feat: trace https proxy request (#996) 
* feat: trace https proxy request

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon test

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma c87be98799
chore: parameterize tests in peer task (#994) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma 88278be464
fix: total pieces count not set cause digest invalid (#992) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma 6f76c74838
chore: clarify daemon interface (#991) 
1. Rename FilePeerTask to FileTask.
2. Rename StreamPeerTask to StreamTask.
3. Extract pulling piece task logic to pieceTaskPoller.

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977) 
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:49 +08:00
Eng Zer Jun f579d3c9ce
refactor: move from io/ioutil to io and os packages (#906) 
* chore: run `go fmt ./...`

This commit synchronizes `//go:build` lines with `// +build` lines.

Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* refactor: move from io/ioutil to io and os packages

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2023-06-28 17:32:45 +08:00
sunwp 46105b0986
unify back source interface (#877) 
* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* metaData Spelling correction

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs unit test

Signed-off-by: sunwp <244372610@qq.com>

* reset fqdn

Signed-off-by: sunwp <244372610@qq.com>

* new seed task with header

Signed-off-by: sunwp <244372610@qq.com>

* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* add unit tests & rename CheckRespCode -> CheckResponseCode & rename GetLastModifiedMillis -> GetLastModified

Signed-off-by: sunwp <244372610@qq.com>

* add request test

Signed-off-by: sunwp <244372610@qq.com>

* golang lint

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:42 +08:00
Gaius f6ce817b72
feat: move dfnet to internal (#862) 
* feat: move dfnet to internal

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: rename dfutils to util

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:41 +08:00
sunwp 44730a2e46
unify piece size (#854) 
* unify piece size

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:40 +08:00
sunwp c1d84fb56d
validate grpc model (#825) 
* base validate

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:40 +08:00
sunwp d79a2a6877
move RPC code definition to proto file (#829) 
* move rpc code to proto
* replace Cdn to CDN

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:38 +08:00