Commit Graph

34 Commits

Author SHA1 Message Date
Jim Ma 9137707139
feat: support breakpoint resume for running tasks (#2457)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:58 +08:00
embroede 8c905d9b8d
feat: enable configuration of some directory modes for dfdaemon (#2340)
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:57 +08:00
Jim Ma f5aff5e27f
feat: enhance daemon health check (#2130)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:35 +08:00
Gaius e83fb85216
feat: add v2 verison of the grpc to scheduler (#1999)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:24 +08:00
Jim Ma 7cb802aeb7
feat: remove legacy peers support (#1939)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:19 +08:00
greenhandatsjtu bc8d623b3d
feat: add traffic shaper for download tasks (#1654)
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
2023-06-28 17:33:59 +08:00
Jim Ma 7224801bae
feat: refactor peertask option (#1600)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:48 +08:00
Jim Ma ecfe467585
feat: add client grpc dial timeout (#1599)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:48 +08:00
Gaius 272f38f567
feat: grpc add ratelimit (#1572)
feat: grpc ratelimit

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:46 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515)
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:41 +08:00
Gaius ed4c381383
feat: rename steal peers to candidate peers (#1476)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:37 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418)
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius 15d3274224
feat: rename digest func and add new digest func (#1405)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius 2a0314e45b
refactor: scheduler grpc (#1310)
* refactor: scheduler grpc

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate protoc

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:20 +08:00
Jim Ma 3682b26647
feat: in tree plugin (#1276)
* feat: in tree plugin

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:16 +08:00
Gaius 076ba20b51
feat: move dfnet to pkg dir (#1265)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:15 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227)
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma 10da5fa0d6
feat: implement grpc client side sync pieces (#1167)
* feat: implement grpc client side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: client unit test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: report sync piece error

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize pieceTaskSyncManager cancel

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:07 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137)
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:05 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078)
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:59 +08:00
Jim Ma c00f07c2de
Extract storage driver into peer tasks (#998)
* chore: extract storage instead load every time

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: gofmt

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:52 +08:00
Jim Ma 88278be464
fix: total pieces count not set cause digest invalid (#992)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977)
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:49 +08:00
Jim Ma beaf4ce18d
feat: update source.Response and source client interface (#945)
* feat: update source.Response and source client interface

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
Jim Ma 7ba341fc5c
fix: infinitely get pieces when piece num is invalid (#926)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:47 +08:00
Eng Zer Jun f579d3c9ce
refactor: move from io/ioutil to io and os packages (#906)
* chore: run `go fmt ./...`

This commit synchronizes `//go:build` lines with `// +build` lines.

Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* refactor: move from io/ioutil to io and os packages

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-06-28 17:32:45 +08:00
sunwp 46105b0986
unify back source interface (#877)
* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* metaData Spelling correction

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs unit test

Signed-off-by: sunwp <244372610@qq.com>

* reset fqdn

Signed-off-by: sunwp <244372610@qq.com>

* new seed task with header

Signed-off-by: sunwp <244372610@qq.com>

* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* add unit tests & rename CheckRespCode -> CheckResponseCode & rename GetLastModifiedMillis -> GetLastModified

Signed-off-by: sunwp <244372610@qq.com>

* add request test

Signed-off-by: sunwp <244372610@qq.com>

* golang lint

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
Gaius f6ce817b72
feat: move dfnet to internal (#862)
* feat: move dfnet to internal

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: rename dfutils to util

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:41 +08:00
sunwp c1d84fb56d
validate grpc model (#825)
* base validate

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:40 +08:00
sunwp d79a2a6877
move RPC code definition to proto file (#829)
* move rpc code to proto
* replace Cdn to CDN

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:38 +08:00
Gaius e837ae9bdf
chore: add lint errcheck and fix errcheck(#766)
* feat: add lint errcheck and fix errcheck

Signed-off-by: Gaius <gaius.qi@gmail.com>

* replace assert to require

Signed-off-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>

Co-authored-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>
2023-06-28 17:32:35 +08:00
Jim Ma 806f2ef11d
feat: calculate piece metadata digest (#787)
* feat: calculate piece metadata digest
* fix: back source not work in midway
* chore: add partial back source test
* chore: optimize digest log

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:35 +08:00