dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,183 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

66 Commits

Author SHA1 Message Date
Jim Ma 1c24c3fc8c
fix: expire header timezone (#1840) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:10 +08:00
Jim Ma f6d8d5c6b2
chore: remove unused code (#1838) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:10 +08:00
Jim Ma fdd189e661
chore: enable cache list metadata e2e (#1829) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma ba58047871
feat: optional save list metadata to p2p (#1822) 
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma 5040af8b4f
feat: add s3 resource client and recursive e2e test (#1826) 
* feat: add s3 resource client
* chore: add minio server and recursive e2e test
* chore: add download grpc test tool

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma 2d46f6424a
feat: update source temporary error logic (#1739) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:01 +08:00
Gaius 4958967c0c
feat: http request content log (#1736) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:01 +08:00
Gaius 74e3d75f37
feat: add http request log (#1734) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:00 +08:00
cuidajun c6119d6ce0
fix: backsource temporary error judgement (#1726) 
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>

Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
 2023-06-28 17:34:00 +08:00
Jim Ma 905f12f5f0
feat: oss get metadata (#1724) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:59 +08:00
Jim Ma 1b1e9cace1
fix: hdfs not registered (#1702) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:57 +08:00
Gaius 9ccf2162de
feat: source changes ResponseHeaderTimeout and ExpectContinueTimeout (#1662) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:54 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:41 +08:00
Gaius a41a524f4c
feat: dfdaemon add Authorization and WWWAuthenticate headers (#1513) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:41 +08:00
cuidajun 1d7c87627c
fix: oss client judge directory bug (#1488) 
Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:38 +08:00
Gaius 4d71217572
feat: generate dag mock 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:36 +08:00
cuidajun 8173c11d3a
fix oss client download bug; support oss recursive download; (#1448) 
* feat: dfget support oss recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* refactor recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* remove URLEntry interface, use simple struct

Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:36 +08:00
Jim Ma cf7debf9c4
feat: back source error detail (#1437) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:34 +08:00
Gaius f68ce30f26
feat: remove errors package (#1434) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:33 +08:00
Jim Ma 776a0fef42
feat: concurrent multiple pieces back source (#1426) 
* feat: concurrent multiple pieces back source
* chore: update http source client

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:33 +08:00
Gaius a2511cb945
feat: rewrite interface{} to any (#1419) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418) 
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:31 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius f688d341bd
feat: remove cdn (#1401) 
* feat: remove cdn

Signed-off-by: Gaius <gaius.qi@gmail.com>

* docs: generate swagger

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292) 
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma 3682b26647
feat: in tree plugin (#1276) 
* feat: in tree plugin

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:16 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137) 
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:05 +08:00
Gaius a95edf5e0d
feat: change source client dialer config (#1115) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:01 +08:00
Gaius f16c5592ab
feat: change back-to-source timeout (#1112) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:00 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078) 
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977) 
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:49 +08:00
sunwp 25f8cd56dd
Fix hdfs write (#948) 
* rename reader to limitedReader

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:49 +08:00
Jim Ma beaf4ce18d
feat: update source.Response and source client interface (#945) 
* feat: update source.Response and source client interface

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:48 +08:00
sunwp 0a152e01e9
Fix HDFS File reader (#940) 
* remove unused hdfs writeTo method

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:48 +08:00
Jim Ma 062687e4c7
feat: support recursive download in dfget (#932) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:47 +08:00
Jim Ma 705c5ed342
fix: plugin dir is empty (#922) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:46 +08:00
Jim Ma e24b1621e4
docs: update plugin docs (#921) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:46 +08:00
Eng Zer Jun f579d3c9ce
refactor: move from io/ioutil to io and os packages (#906) 
* chore: run `go fmt ./...`

This commit synchronizes `//go:build` lines with `// +build` lines.

Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* refactor: move from io/ioutil to io and os packages

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2023-06-28 17:32:45 +08:00
Jim Ma 10ec7d2bbc
fix: go plugin test build error (#912) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:45 +08:00
sunwp eaca95d24e
Load source plugins (#894) 
* unit test

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:44 +08:00
sunwp 0096c613b0
Fix hdfs test (#902) 
* unit test

Signed-off-by: sunwp <244372610@qq.com>

* fix hdfs range test

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:44 +08:00
sunwp 23747be9d3
unit test (#898) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:44 +08:00
Gaius c2c721bc78
refactor: dfpath pkg (#879) 
* feat: dfpath

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: dfpath

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: path

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: plugin

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:43 +08:00
sunwp 929fe7db2d
Fix load source plugins (#892) 
* load source plugins

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:43 +08:00
Jim Ma ff3a542fde
chore: update UnknownSourceFileLen (#888) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:43 +08:00
sunwp 46105b0986
unify back source interface (#877) 
* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* metaData Spelling correction

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs unit test

Signed-off-by: sunwp <244372610@qq.com>

* reset fqdn

Signed-off-by: sunwp <244372610@qq.com>

* new seed task with header

Signed-off-by: sunwp <244372610@qq.com>

* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* add unit tests & rename CheckRespCode -> CheckResponseCode & rename GetLastModifiedMillis -> GetLastModified

Signed-off-by: sunwp <244372610@qq.com>

* add request test

Signed-off-by: sunwp <244372610@qq.com>

* golang lint

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:42 +08:00
Jim Ma 0ad1bf9674
Fix: source plugin not loaded (#811) 
* fix: source plugin not loaded

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:36 +08:00
Jim Ma 806f2ef11d
feat: calculate piece metadata digest (#787) 
* feat: calculate piece metadata digest
* fix: back source not work in midway
* chore: add partial back source test
* chore: optimize digest log

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:35 +08:00
Jim Ma 3d63ab2ca3
chore: update golang import lint (#780) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:34 +08:00