Commit Graph

62 Commits

Author SHA1 Message Date
Jim Ma b23b18d113
chore: mute some reclaim check log (#3469)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2024-08-28 16:58:17 +08:00
ytq fae3613ff2
refactor: Preallocate pieceDigests slice in genMetadata for improved (#3325)
Signed-off-by: yantingqiu <1173578083@qq.com>
2024-07-03 14:47:45 +08:00
Qi 6c44d25dce
BUGFIX: hardlink (#3277) 2024-05-16 11:07:47 +08:00
Jim Ma fe033b0564
feat: storage write buffer size (#3127)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2024-03-15 17:51:10 +08:00
Jim Ma e329bdbb39
feat: back to source with piece group(multiple pieces) (#2826)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-12-27 15:54:21 +08:00
Jim Ma 3dc2e468a2
chore: split concurrentDownloadSource (#2842) 2023-10-31 16:51:46 +08:00
Gaius bd140f1104
refactor: file close error
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-08-08 12:08:15 +08:00
Jim Ma 342438eae6
chore: optimize unhandled file close error (#2599)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-08-08 11:48:29 +08:00
Gaius b3f23f71a3
feat: update dy7.io/api to v2 (#2558)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-07-20 23:17:57 +08:00
Jim Ma bdfeb48f02
feat: daemon store file exclusive (#2465)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:59 +08:00
Eryu Guan fab5609d3d
feat: don't GC task if expire time is 0 (#2102)
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-06-28 17:34:32 +08:00
Jim Ma d7a7e69916
chore: remove unused MarkInvalid in daemon (#2101)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:32 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:30 +08:00
Jim Ma ba58047871
feat: optional save list metadata to p2p (#1822)
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
Jim Ma f581d10208
chore: daemon avoid alway open metadata files (#1823)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
cuidajun 536b039462
feat: daemon call leaveHost when exit (#1788)
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>

Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
2023-06-28 17:34:06 +08:00
Jim Ma 32eaba3ffb
fix: did not call scheduler leave tasks in forceGC (#1782)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:05 +08:00
Jim Ma c9bfe9eb08
feat: unregister failed task storage (#1717)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:00 +08:00
Jim Ma 73de99353d
feat: empty file e2e (#1687)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:56 +08:00
Jim Ma 40300990de
feat: support download empty file (#1686)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:56 +08:00
Gaius 198613a357
chore: add disable seed peer action (#1653)
* chore: add disable seed peer action
* fix: concurrent back source

Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:54 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515)
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:41 +08:00
1037husterljx 31032e05aa
fix [4000] digest not set error (#1465)
Signed-off-by: ljx373327 <ljx373327@alibaba-inc.com>

Co-authored-by: ljx373327 <ljx373327@alibaba-inc.com>
2023-06-28 17:33:36 +08:00
Jim Ma 1cf39f3f9b
chore: check header length before update (#1445)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:35 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius 15d3274224
feat: rename digest func and add new digest func (#1405)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius e66b567eed
refactor: digest package (#1403)
* refactor: digest package

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Jim Ma 175a27e278
fix: find partial completed overflow (#1346)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:24 +08:00
Jim Ma 1a8b293a66
fix: seed peer did not send done seed result and no content length send (#1316)
fix: seed peer did not send done seed result
fix: unknown length back source seed did not send content length to other peers

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:21 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292)
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
Jim Ma 49f9ebde68
feat: implement client seed mode (#1247)
* feat: implement client seed mode

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:18 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227)
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma 517ed6c51e
fix: client sync piece panic (#1246)
* fix: client sync piece panic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize sync piece log

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon panic when register fail

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165)
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:06 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137)
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:05 +08:00
Jim Ma ac4c4bb912
feat: reuse partial completed task (#1107)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:00 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078)
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:59 +08:00
Jim Ma f80c75efdb
Feature: prefetch ranged requests (#1053)
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:56 +08:00
Jim Ma 5027f9c998
chore: use buildx to build docker images in e2e (#1018)
* chore: build images concurrency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:54 +08:00
Jim Ma 88278be464
fix: total pieces count not set cause digest invalid (#992)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
guoguodan 3b6966c585
feat: update typo in local_storage.go (#955) 2023-06-28 17:32:51 +08:00
Jim Ma cb36b79dd1
feat: when write last piece, generate digest (#982)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:50 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977)
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:49 +08:00
Jim Ma 8d6e4044a4
chore: optimize back source update digest logic (#950)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
Jim Ma 7ba341fc5c
fix: infinitely get pieces when piece num is invalid (#926)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:47 +08:00
Eng Zer Jun f579d3c9ce
refactor: move from io/ioutil to io and os packages (#906)
* chore: run `go fmt ./...`

This commit synchronizes `//go:build` lines with `// +build` lines.

Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* refactor: move from io/ioutil to io and os packages

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-06-28 17:32:45 +08:00
Jim Ma ca2f196aff
fix: mismatch digest peer task did not mark invalid (#903)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:44 +08:00
sunwp 456e776bfd
correct metadata spell (#884)
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
sunwp c1d84fb56d
validate grpc model (#825)
* base validate

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:40 +08:00