Commit Graph

40 Commits

Author SHA1 Message Date
1037husterljx 31032e05aa
fix [4000] digest not set error (#1465)
Signed-off-by: ljx373327 <ljx373327@alibaba-inc.com>

Co-authored-by: ljx373327 <ljx373327@alibaba-inc.com>
2023-06-28 17:33:36 +08:00
Jim Ma 1cf39f3f9b
chore: check header length before update (#1445)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:35 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius 15d3274224
feat: rename digest func and add new digest func (#1405)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius e66b567eed
refactor: digest package (#1403)
* refactor: digest package

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Jim Ma 175a27e278
fix: find partial completed overflow (#1346)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:24 +08:00
Jim Ma 1a8b293a66
fix: seed peer did not send done seed result and no content length send (#1316)
fix: seed peer did not send done seed result
fix: unknown length back source seed did not send content length to other peers

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:21 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292)
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
Jim Ma 49f9ebde68
feat: implement client seed mode (#1247)
* feat: implement client seed mode

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:18 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227)
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma 517ed6c51e
fix: client sync piece panic (#1246)
* fix: client sync piece panic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize sync piece log

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon panic when register fail

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165)
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:06 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137)
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:05 +08:00
Jim Ma ac4c4bb912
feat: reuse partial completed task (#1107)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:00 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078)
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:59 +08:00
Jim Ma f80c75efdb
Feature: prefetch ranged requests (#1053)
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:56 +08:00
Jim Ma 5027f9c998
chore: use buildx to build docker images in e2e (#1018)
* chore: build images concurrency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:54 +08:00
Jim Ma 88278be464
fix: total pieces count not set cause digest invalid (#992)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
guoguodan 3b6966c585
feat: update typo in local_storage.go (#955) 2023-06-28 17:32:51 +08:00
Jim Ma cb36b79dd1
feat: when write last piece, generate digest (#982)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:50 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977)
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:49 +08:00
Jim Ma 8d6e4044a4
chore: optimize back source update digest logic (#950)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
Jim Ma 7ba341fc5c
fix: infinitely get pieces when piece num is invalid (#926)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:47 +08:00
Eng Zer Jun f579d3c9ce
refactor: move from io/ioutil to io and os packages (#906)
* chore: run `go fmt ./...`

This commit synchronizes `//go:build` lines with `// +build` lines.

Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* refactor: move from io/ioutil to io and os packages

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-06-28 17:32:45 +08:00
Jim Ma ca2f196aff
fix: mismatch digest peer task did not mark invalid (#903)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:44 +08:00
sunwp 456e776bfd
correct metadata spell (#884)
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
sunwp c1d84fb56d
validate grpc model (#825)
* base validate

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:40 +08:00
Jim Ma 9a1b0f207e
chore: optimize client storage gc log (#790)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:35 +08:00
Jim Ma 806f2ef11d
feat: calculate piece metadata digest (#787)
* feat: calculate piece metadata digest
* fix: back source not work in midway
* chore: add partial back source test
* chore: optimize digest log

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:35 +08:00
Jim Ma 6546115359
feat: support dump http content in client for debugging (#770)
* feat: support dump http content in client for debugging

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:33 +08:00
Gaius fcde9dc9b5
Add manager metrics and logger (#676)
* feat: add manager metrics and logger

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:31:49 +08:00
Gaius 6b9b814995
feat: release fd (#668)
* feat: release fd

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:31:48 +08:00
Jim Ma 0df064202e
Fix: proxy for stress testing tool (#507)
* chore: fix proxy for stress testing tool

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:33 +08:00
Jim Ma 777e98784e
feature: export peer.TaskManager for embedding dragonfly in custom binary (#434)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:30:19 +08:00
sunwp 6cce5a9872
Feature/code spec (#336)
* Modify the code according to https://github.com/uber-go/guide/blob/master/style.md

Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:24:44 +08:00
Jim Ma 5b9c3033f8
Support peer task data multiplex (#347)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:23:35 +08:00
Jim Ma a033e5f5be
feature: remove unsafe code in client/daemon/storage (#258)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:22:38 +08:00
Jim Ma a777c8882c
Fix: use atomic to avoid data race in client (#254)
* fix: use atomic to avoid data race

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* feature: temporary disable cdnsystem&manager test for CI

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* feature: update .github/workflows/ci.yaml and codecov.yml

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* feature: update ratelimiter test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* feature: update dynconfig test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: ratelimiter and file utils test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: ratelimiter and queue test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: httputil test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* feat: remove httputils

Signed-off-by: Gaius <gaius.qi@gmail.com>

Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:22:38 +08:00
Jim Ma 0af8817c1f
Merge branch main-rc to branch main
Signed-off-by: Jim Ma <majinjing3@gmail.com>

Co-authored-by: 李玉海 <yuhai.lyh@alibaba-inc.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: santong <weipeng.swp@alibaba-inc.com>
Co-authored-by: zuozheng.hzz <zuozheng.hzz@alibaba-inc.com>
Co-authored-by: 加菲 <garfieldhu@aliyun.com>
2023-06-28 17:20:20 +08:00