On graceful shutdown we will call forceGC() to delete all cached tasks
if 'keepStorage' is false. And we rely on deleteTask() to delete the
given task from both local storage and scheduler.
But commit 86a6030c8f ("feat: unregister failed task storage (#1717)")
changed MarkReclaim() to MarkInvalid() in deleteTask(), and resulted in
task leak in scheduler.
Fix it by calling MarkReclaim() in deleteTask().
Fixes: 86a6030c8f ("feat: unregister failed task storage (#1717)")
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
* feat: update namely/protoc-all image version to 1.47_0
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: generate mock file
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.
For example:
- add a file into cache system
dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file
- check if a file exists in cache system
dfcache stat --cid testid --local # only check local cache
dfcache stat --cid testid # check other peers as well
- export/download a file from cache system
dfcache export --cid testid -O /path/to/output
- delete a file from cache system, both local cache and P2P network
dfcache delete -i testid -t testtag
Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
* fix: client sync piece panic
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: optimize sync piece log
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: daemon panic when register fail
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: extract storage instead load every time
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: test
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: gofmt
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: run `go fmt ./...`
This commit synchronizes `//go:build` lines with `// +build` lines.
Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* refactor: move from io/ioutil to io and os packages
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* feat: calculate piece metadata digest
* fix: back source not work in midway
* chore: add partial back source test
* chore: optimize digest log
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: use atomic to avoid data race
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feature: temporary disable cdnsystem&manager test for CI
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feature: update .github/workflows/ci.yaml and codecov.yml
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feature: update ratelimiter test
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feature: update dynconfig test
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: ratelimiter and file utils test
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: ratelimiter and queue test
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: httputil test
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feat: remove httputils
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
* feat: add disk quota gc for daemon
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: skip marked tasks when check disk quota
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Jim Ma