Commit Graph

30 Commits

Author SHA1 Message Date
Chlins Zhang d0bfdd4e77
refactor: modernize the code by new features provided by golang stand… (#3963)
refactor: modernize the code by new features provided by golang standard library

Signed-off-by: chlins <chlins.zhang@gmail.com>
2025-04-17 07:48:42 +00:00
Jim Ma ba58047871
feat: optional save list metadata to p2p (#1822)
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
Jim Ma 905f12f5f0
feat: oss get metadata (#1724)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:59 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515)
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:41 +08:00
cuidajun 8173c11d3a
fix oss client download bug; support oss recursive download; (#1448)
* feat: dfget support oss recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* refactor recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* remove URLEntry interface, use simple struct

Signed-off-by: bigerous <bigerous@qq.com>
2023-06-28 17:33:36 +08:00
Jim Ma 776a0fef42
feat: concurrent multiple pieces back source (#1426)
* feat: concurrent multiple pieces back source
* chore: update http source client

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:33 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418)
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:31 +08:00
Jim Ma 3682b26647
feat: in tree plugin (#1276)
* feat: in tree plugin

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:16 +08:00
Gaius a95edf5e0d
feat: change source client dialer config (#1115)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:01 +08:00
Gaius f16c5592ab
feat: change back-to-source timeout (#1112)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:00 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977)
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:49 +08:00
Jim Ma beaf4ce18d
feat: update source.Response and source client interface (#945)
* feat: update source.Response and source client interface

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
Jim Ma 062687e4c7
feat: support recursive download in dfget (#932)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:47 +08:00
Jim Ma 705c5ed342
fix: plugin dir is empty (#922)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:46 +08:00
sunwp eaca95d24e
Load source plugins (#894)
* unit test

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:44 +08:00
Gaius c2c721bc78
refactor: dfpath pkg (#879)
* feat: dfpath

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: dfpath

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: path

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: plugin

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:43 +08:00
sunwp 929fe7db2d
Fix load source plugins (#892)
* load source plugins

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:43 +08:00
Jim Ma ff3a542fde
chore: update UnknownSourceFileLen (#888)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:43 +08:00
sunwp 46105b0986
unify back source interface (#877)
* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* metaData Spelling correction

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs unit test

Signed-off-by: sunwp <244372610@qq.com>

* reset fqdn

Signed-off-by: sunwp <244372610@qq.com>

* new seed task with header

Signed-off-by: sunwp <244372610@qq.com>

* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* add unit tests & rename CheckRespCode -> CheckResponseCode & rename GetLastModifiedMillis -> GetLastModified

Signed-off-by: sunwp <244372610@qq.com>

* add request test

Signed-off-by: sunwp <244372610@qq.com>

* golang lint

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
Jim Ma 0ad1bf9674
Fix: source plugin not loaded (#811)
* fix: source plugin not loaded

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:36 +08:00
Jim Ma fdf2b688fa
feat: back source when no available peers or scheduler error (#521)
* feature: back source when no available peers

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:31:34 +08:00
sunwp 7b8aaf6b31
Refactor scheduler (#369)
Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
2023-06-28 17:30:29 +08:00
lengrongfu 4751a92db6
feat: CDN implementation supports HDFS type storage (#420)
Signed-off-by: lengrongfu <lengrongfu@meituan.com>
2023-06-28 17:30:27 +08:00
zzy987 402f1eb397
fix: log specification (#452)
Signed-off-by: 张哲溢 <zhangzheyi@zhangzheyideMacBook-Pro.local>

Co-authored-by: 张哲溢 <zhangzheyi@zhangzheyideMacBook-Pro.local>
2023-06-28 17:30:27 +08:00
sunwp 158ea2ec4a
fix: wrong cache header (#423)
Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
2023-06-28 17:28:03 +08:00
Jim Ma 2d825f5a96
feat: Add plugin support for resource (#291)
* feature: add plugin support for resource

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: skip link for mockgen import

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* feature: add generate in Makefile

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: remove unused package

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: remove unused test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* feature: remove unused import package

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:28:01 +08:00
Jim Ma 5175e8009a
Fix: infinite recursion in MkDirAll (#358)
* fix: infinite recursion in MkDirAll

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* disable manager test, enable cdn test

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:27:51 +08:00
sunwp 6cce5a9872
Feature/code spec (#336)
* Modify the code according to https://github.com/uber-go/guide/blob/master/style.md

Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:24:44 +08:00
sunwp 155ee02a54
Add CDN unit-tests (#338)
* feat: refactor cdn storage module & remove unused ctx & cdn test

Signed-off-by: santong <weipeng.swp@alibaba-inc.com>
2023-06-28 17:22:43 +08:00