Commit Graph

88 Commits

Author SHA1 Message Date
Gaius 501a7be05b
feat: trigger download task by stream (#3200)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2024-04-17 10:26:34 +08:00
Jim Ma 5cee9010bf
fix: single long task keepalive (#3184)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2024-04-10 22:27:17 +08:00
Jim Ma 48846e2ff6
feat: peer exchange (#3141)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2024-04-09 18:29:57 +08:00
Jim Ma fc53dcf980
feat: optimize reload storage (#3167)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2024-04-08 18:07:10 +08:00
Qi 405ed005bf
Improve the search order for localTaskStore array (#3166)
Signed-off-by: Qi <37133148+aobt@users.noreply.github.com>
2024-04-08 14:32:32 +08:00
Qi 49ac8d52bd
fix index map in storage_manager (#3145)
Signed-off-by: Qi <37133148+aobt@users.noreply.github.com>
2024-03-25 21:07:57 +08:00
Jim Ma db3fa4fcaa
fix: storage buffer size (#3130)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2024-03-16 14:00:08 +08:00
Jim Ma fe033b0564
feat: storage write buffer size (#3127)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2024-03-15 17:51:10 +08:00
Jim Ma e329bdbb39
feat: back to source with piece group(multiple pieces) (#2826)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-12-27 15:54:21 +08:00
Jim Ma 3dc2e468a2
chore: split concurrentDownloadSource (#2842) 2023-10-31 16:51:46 +08:00
Gaius 12fc630612
fix: if scheduler has no seed peer, return error in preheating (#2835)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-10-30 16:45:39 +08:00
Jim Ma 9eb15b218d
fix: overhead gc when DiskGCThreshold not set (#2750)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-09-21 21:04:17 +08:00
Gaius 05a66805cc
fix: directories created via os.MkdirAll are not checked for permissions (#2613)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-08-09 13:52:26 +08:00
Gaius bd140f1104
refactor: file close error
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-08-08 12:08:15 +08:00
Jim Ma 342438eae6
chore: optimize unhandled file close error (#2599)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-08-08 11:48:29 +08:00
Gaius b3f23f71a3
feat: update dy7.io/api to v2 (#2558)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-07-20 23:17:57 +08:00
Jim Ma bdfeb48f02
feat: daemon store file exclusive (#2465)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:59 +08:00
embroede 8c905d9b8d
feat: enable configuration of some directory modes for dfdaemon (#2340)
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:57 +08:00
Eryu Guan fab5609d3d
feat: don't GC task if expire time is 0 (#2102)
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-06-28 17:34:32 +08:00
Jim Ma d7a7e69916
chore: remove unused MarkInvalid in daemon (#2101)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:32 +08:00
Eryu Guan 3352deff39
fix: unregister task from scheduler in storage.deleteTask (#2100)
On graceful shutdown we will call forceGC() to delete all cached tasks
if 'keepStorage' is false. And we rely on deleteTask() to delete the
given task from both local storage and scheduler.

But commit 86a6030c8f ("feat: unregister failed task storage (#1717)")
changed MarkReclaim() to MarkInvalid() in deleteTask(), and resulted in
task leak in scheduler.

Fix it by calling MarkReclaim() in deleteTask().

Fixes: 86a6030c8f ("feat: unregister failed task storage (#1717)")

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-06-28 17:34:32 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:30 +08:00
Gaius c89234e976
feat: add manager v2 api (#1990)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:23 +08:00
Jim Ma ba58047871
feat: optional save list metadata to p2p (#1822)
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
Jim Ma f581d10208
chore: daemon avoid alway open metadata files (#1823)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
cuidajun 536b039462
feat: daemon call leaveHost when exit (#1788)
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>

Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
2023-06-28 17:34:06 +08:00
Jim Ma 32eaba3ffb
fix: did not call scheduler leave tasks in forceGC (#1782)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:05 +08:00
Jim Ma c9bfe9eb08
feat: unregister failed task storage (#1717)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:00 +08:00
cuidajun 9dd9bc5c9f
chore: change disk usage debug log format to decimal (#1727)
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
2023-06-28 17:34:00 +08:00
Gaius 7cc842d64d
test: remove test main (#1710)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:58 +08:00
Jim Ma 73de99353d
feat: empty file e2e (#1687)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:56 +08:00
Jim Ma 40300990de
feat: support download empty file (#1686)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:56 +08:00
Gaius 198613a357
chore: add disable seed peer action (#1653)
* chore: add disable seed peer action
* fix: concurrent back source

Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:54 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515)
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:41 +08:00
1037husterljx 31032e05aa
fix [4000] digest not set error (#1465)
Signed-off-by: ljx373327 <ljx373327@alibaba-inc.com>

Co-authored-by: ljx373327 <ljx373327@alibaba-inc.com>
2023-06-28 17:33:36 +08:00
Jim Ma 1cf39f3f9b
chore: check header length before update (#1445)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:35 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius a2511cb945
feat: rewrite interface{} to any (#1419)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418)
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:31 +08:00
Gaius 15d3274224
feat: rename digest func and add new digest func (#1405)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius e66b567eed
refactor: digest package (#1403)
* refactor: digest package

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Jim Ma 175a27e278
fix: find partial completed overflow (#1346)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:24 +08:00
Jim Ma 3a9f160e7a
fix: reuse seed panic (#1319)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:21 +08:00
Jim Ma 1a8b293a66
fix: seed peer did not send done seed result and no content length send (#1316)
fix: seed peer did not send done seed result
fix: unknown length back source seed did not send content length to other peers

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:21 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292)
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
Jim Ma 49f9ebde68
feat: implement client seed mode (#1247)
* feat: implement client seed mode

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:18 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227)
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma 517ed6c51e
fix: client sync piece panic (#1246)
* fix: client sync piece panic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize sync piece log

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon panic when register fail

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:13 +08:00