776a0fef42
feat: concurrent multiple pieces back source ( #1426 )
...
* feat: concurrent multiple pieces back source
* chore: update http source client
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:33 +08:00
3b426d2c54
feat: rename client/clientutil to client/util ( #1420 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
ad36eb64a7
feat: remove github/pkg/errors package ( #1416 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:31 +08:00
30313a4caa
feat: client add create object storage ( #1409 )
...
* feat: add create object storage rest api
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:30 +08:00
64f5e9357b
refactor: scheduler announce task ( #1407 )
...
* feat: scheduler announce normal task
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: remove cid
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:30 +08:00
ad827170a4
feat: dfdaemon upload and object storage service add middlewares ( #1404 )
...
* feat: dfdaemon upload and object storage service add middlewares
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
230966cf8f
feat: dfdaemon add GetObject rest api ( #1398 )
...
* feat: dfdaemon add GetObject rest api
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:28 +08:00
171c0d54ee
feat: dfdaemon add object storage rest api ( #1390 )
...
* feat: dfdaemon add object storage rest api
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:28 +08:00
4889a37886
feat: replace gin-gonic/gin with gorilla/mux ( #1389 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:28 +08:00
3c6af10433
feat: add sync piece watchdog ( #1272 )
...
* feat: add sync piece watchdog
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:21 +08:00
605865d3a9
feat: client add tls verify config ( #1323 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:21 +08:00
999f5919c6
feat: reduce dynconfig log ( #1312 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:20 +08:00
2a0314e45b
refactor: scheduler grpc ( #1310 )
...
* refactor: scheduler grpc
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: generate protoc
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:20 +08:00
faa5e4e465
feat: add seed peer logic ( #1302 )
...
* feat: announce seed peer
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: remove cdn logic
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: remove cdn job
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: dfdaemon change host uuid to host id
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: go generate mocks
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: remove cdn compatibility
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: change docker compose
Signed-off-by: Gaius <gaius.qi@gmail.com>
* fix: reuse panic
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: compatible with v2.0.3-beta.2
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:19 +08:00
429252a071
feat: implement pattern in client daemon ( #1231 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
076ba20b51
feat: move dfnet to pkg dir ( #1265 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:15 +08:00
bab1ddec60
Fix client initialization trivial things ( #1176 )
...
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:33:08 +08:00
04bd2ea383
feat: scheduler addresses log ( #1183 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:08 +08:00
104e8359a9
fix daemon cannot exit after alivetime ( #1177 )
...
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:33:07 +08:00
21d219175f
feat: dfdaemon get available scheduler addresses in the same cluster ( #1178 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:07 +08:00
010887ab66
feat: optimize proxy performance ( #1137 )
...
* feat: optimize back source trace and logic
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: fix keepalive performance
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: optimize generate peer id and transport performance
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: io.Copy call superfluous sendfile, then increase network latency
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:05 +08:00
ebc1aff720
feat: support health probe in daemon ( #1120 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:01 +08:00
4597f74283
feat: merge ranged request storage into parent ( #1078 )
...
* feat: merge ranged request storage into parent
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:59 +08:00
f80c75efdb
Feature: prefetch ranged requests ( #1053 )
...
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:56 +08:00
38178804e8
feat: add daemon metrics support ( #960 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
b7f8ad4e20
feat: support disk usage gc in client ( #953 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
7ba341fc5c
fix: infinitely get pieces when piece num is invalid ( #926 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:47 +08:00
cb4202319e
feat: make idgen package public ( #931 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:47 +08:00
1cde964259
feat: make dfpath public ( #929 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:46 +08:00
705c5ed342
fix: plugin dir is empty ( #922 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:46 +08:00
f579d3c9ce
refactor: move from io/ioutil to io and os packages ( #906 )
...
* chore: run `go fmt ./...`
This commit synchronizes `//go:build` lines with `// +build` lines.
Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* refactor: move from io/ioutil to io and os packages
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-06-28 17:32:45 +08:00
75c7be0769
feat: add data storage directory ( #907 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:44 +08:00
c2c721bc78
refactor: dfpath pkg ( #879 )
...
* feat: dfpath
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: dfpath
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: path
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: plugin
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:43 +08:00
afe5bdb63a
fix: searcher log ( #878 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:42 +08:00
540cd233a0
feat: support customize transport in daemon ( #866 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:41 +08:00
f6ce817b72
feat: move dfnet to internal ( #862 )
...
* feat: move dfnet to internal
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: rename dfutils to util
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:41 +08:00
999097e4ed
feat: support mutli manager addrs ( #846 )
...
* feat: support mutli manager addrs (#844 )
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:39 +08:00
613abf8861
feat: add package reachable ( #822 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:37 +08:00
72213eb1d4
feat: dfdaemon get scheduler list dynamically from manager ( #812 )
...
* feat: dfdaemon get scheduler list dynamically from manager
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:37 +08:00
e837ae9bdf
chore: add lint errcheck and fix errcheck( #766 )
...
* feat: add lint errcheck and fix errcheck
Signed-off-by: Gaius <gaius.qi@gmail.com>
* replace assert to require
Signed-off-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>
Co-authored-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>
2023-06-28 17:32:35 +08:00
806f2ef11d
feat: calculate piece metadata digest ( #787 )
...
* feat: calculate piece metadata digest
* fix: back source not work in midway
* chore: add partial back source test
* chore: optimize digest log
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:35 +08:00
466912aa48
docs: scheduler config ( #698 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:25 +08:00
f46f370b32
feat: add piece download timeout ( #621 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:31:44 +08:00
37910d1fd7
feat: add sni proxy support ( #600 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:43 +08:00
8180a52470
chore: optimize client rpc package name and other docs ( #541 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:35 +08:00
1efd2f6cef
feat: Add distribute Schedule Tracer & Refactor scheduler ( #537 )
...
* check same cdn list avoid updating client too frequent
Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
2023-06-28 17:31:35 +08:00
a15ce877c7
fix: scheduler concurrent dead lock ( #509 )
...
* fix scheduler concurrent dead lock
Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
2023-06-28 17:31:33 +08:00
7b8aaf6b31
Refactor scheduler ( #369 )
...
Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
2023-06-28 17:30:29 +08:00
975e8a9058
feature: rename PeerHost to Daemon in client ( #438 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:30:26 +08:00
Jim Ma