a413dc4bdd
chore: report client back source error ( #1209 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:10 +08:00
010887ab66
feat: optimize proxy performance ( #1137 )
...
* feat: optimize back source trace and logic
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: fix keepalive performance
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: optimize generate peer id and transport performance
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: io.Copy call superfluous sendfile, then increase network latency
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:05 +08:00
f80c75efdb
Feature: prefetch ranged requests ( #1053 )
...
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:56 +08:00
c00f07c2de
Extract storage driver into peer tasks ( #998 )
...
* chore: extract storage instead load every time
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: test
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: gofmt
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:52 +08:00
5479b3a53e
feat: stream send error code ( #986 )
...
* feat: change task and peer ttl (#984 )
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: send error code
Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: add retry interval
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: dfdaemon report successful piece before end of piece
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: update submodule version
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:51 +08:00
c87be98799
chore: parameterize tests in peer task ( #994 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
88278be464
fix: total pieces count not set cause digest invalid ( #992 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
6f76c74838
chore: clarify daemon interface ( #991 )
...
1. Rename FilePeerTask to FileTask.
2. Rename StreamPeerTask to StreamTask.
3. Extract pulling piece task logic to pieceTaskPoller.
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
17ff9c17be
fix: send piece result error not handled ( #987 )
...
* fix: send piece result error not handled
* chore: optimize daemon cancel logic
* chore: remove redundant log and ctx.Done check
* chore: handle piece download with 404 status
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:50 +08:00
cb36b79dd1
feat: when write last piece, generate digest ( #982 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:50 +08:00
c38bba2612
feat: merge same tasks in daemon ( #977 )
...
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:49 +08:00
8d6e4044a4
chore: optimize back source update digest logic ( #950 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
beaf4ce18d
feat: update source.Response and source client interface ( #945 )
...
* feat: update source.Response and source client interface
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
029c4eea36
fix: total pieces not set when back source ( #908 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:44 +08:00
ff3a542fde
chore: update UnknownSourceFileLen ( #888 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:43 +08:00
5b016d92b4
fix: small size task failed due to digest error ( #886 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:43 +08:00
456e776bfd
correct metadata spell ( #884 )
...
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
46105b0986
unify back source interface ( #877 )
...
* unify back source interface
Signed-off-by: sunwp <244372610@qq.com>
* metaData Spelling correction
Signed-off-by: sunwp <244372610@qq.com>
* hdfs compile
Signed-off-by: sunwp <244372610@qq.com>
* hdfs compile
Signed-off-by: sunwp <244372610@qq.com>
* hdfs unit test
Signed-off-by: sunwp <244372610@qq.com>
* reset fqdn
Signed-off-by: sunwp <244372610@qq.com>
* new seed task with header
Signed-off-by: sunwp <244372610@qq.com>
* unify back source interface
Signed-off-by: sunwp <244372610@qq.com>
* add unit tests & rename CheckRespCode -> CheckResponseCode & rename GetLastModifiedMillis -> GetLastModified
Signed-off-by: sunwp <244372610@qq.com>
* add request test
Signed-off-by: sunwp <244372610@qq.com>
* golang lint
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
3086147df6
docs: keep alive ( #868 )
...
* docs: keep alive
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: add transport log
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:41 +08:00
540cd233a0
feat: support customize transport in daemon ( #866 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:41 +08:00
f6ce817b72
feat: move dfnet to internal ( #862 )
...
* feat: move dfnet to internal
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: rename dfutils to util
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:41 +08:00
44730a2e46
unify piece size ( #854 )
...
* unify piece size
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:40 +08:00
c1d84fb56d
validate grpc model ( #825 )
...
* base validate
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:40 +08:00
c3b1e01fda
fix: file peer task back source digest not match ( #849 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:39 +08:00
d79a2a6877
move RPC code definition to proto file ( #829 )
...
* move rpc code to proto
* replace Cdn to CDN
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:38 +08:00
e837ae9bdf
chore: add lint errcheck and fix errcheck( #766 )
...
* feat: add lint errcheck and fix errcheck
Signed-off-by: Gaius <gaius.qi@gmail.com>
* replace assert to require
Signed-off-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>
Co-authored-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>
2023-06-28 17:32:35 +08:00
806f2ef11d
feat: calculate piece metadata digest ( #787 )
...
* feat: calculate piece metadata digest
* fix: back source not work in midway
* chore: add partial back source test
* chore: optimize digest log
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:35 +08:00
359d11367e
chore: rename cdnsystem to cdn ( #626 )
...
Signed-off-by: santong <244372610@qq.com>
2023-06-28 17:31:46 +08:00
f46f370b32
feat: add piece download timeout ( #621 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:31:44 +08:00
60bee061d2
feat: client back source ( #579 )
...
* feat: client back source
Signed-off-by: santong <244372610@qq.com>
2023-06-28 17:31:43 +08:00
dbf583524c
chore: optimize peer task report function ( #543 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:35 +08:00
94e35fa9df
chore: optimize compute piece size function ( #528 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:34 +08:00
fdf2b688fa
feat: back source when no available peers or scheduler error ( #521 )
...
* feature: back source when no available peers
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:34 +08:00
777e98784e
feature: export peer.TaskManager for embedding dragonfly in custom binary ( #434 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:30:19 +08:00
6cce5a9872
Feature/code spec ( #336 )
...
* Modify the code according to https://github.com/uber-go/guide/blob/master/style.md
Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:24:44 +08:00
5b9c3033f8
Support peer task data multiplex ( #347 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:23:35 +08:00
155ee02a54
Add CDN unit-tests ( #338 )
...
* feat: refactor cdn storage module & remove unused ctx & cdn test
Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
2023-06-28 17:22:43 +08:00
f9b4d77bb2
Fix golang lint ( #249 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:22:38 +08:00
7e595898e5
Fix lint ( #244 )
...
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: lint hostIP
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: error check
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: nil pointer
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: rename taskID
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: rename peerID
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: taskID
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: licenses
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: rename
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: golangci-lint
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:22:37 +08:00
19e70ecce8
refactor: support multi digest not only md5 ( #236 )
...
Signed-off-by: zuozheng.hzz <zuozheng.hzz@alibaba-inc.com>
2023-06-28 17:22:37 +08:00
3955a7d716
daemon: update default timeout and add context for downloading piece ( #190 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:21:33 +08:00
0af8817c1f
Merge branch main-rc to branch main
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: 李玉海 <yuhai.lyh@alibaba-inc.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: santong <weipeng.swp@alibaba-inc.com>
Co-authored-by: zuozheng.hzz <zuozheng.hzz@alibaba-inc.com>
Co-authored-by: 加菲 <garfieldhu@aliyun.com>
2023-06-28 17:20:20 +08:00
Jim Ma