dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,702 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

80 Commits

Author SHA1 Message Date
Gaius b3f23f71a3
feat: update dy7.io/api to v2 (#2558) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-07-20 23:17:57 +08:00
Guangwen Feng 7bc0a0f71e
test: add unit test for Header.has (#2555) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-07-19 16:56:49 +08:00
Guangwen Feng 94e4b7a5a2
test: add unit test for Request.Context() (#2554) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-07-18 17:52:40 +08:00
Jim Ma b0eaa169ba
fix: e2e test dfget recursive (#2458) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:57 +08:00
Jim Ma 4c6814600f
chore: update oras error format (#2282) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:46 +08:00
Jim Ma 94a7c7639f
feat: update resource director (#2243) 
* feat: update resource director
* chore: add digest check for oras

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:44 +08:00
Jim Ma 144335ffb1
feat: add authinfo injector (#2149) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:43 +08:00
Gaius 2cda1def7f
feat: rename HostName to Hostname (#2205) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:40 +08:00
hiigao eebca2416a
feat: enable oss client download object concurrently. (#2105) 
feat: enalbe councrrent oss backsource by set contentLength

pieceManager calls DownloadSource() to download source objects.
DownloadSource() firstly connect to source and get the response from
source. Then, DownloadSource() checks the "ContentLength" value of the
response. If it larger than 0, pieceManager will download the object by
multiple threads.

Unfortunately, oss client doesn't set ContentLength value, even though
ContentLength can be set by the oss.GetObjectResult.Response.Headers.

So let's set response a ContentLength value to enable concurrent
downloads from oss.

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:33 +08:00
hiigao 0349f3cf8f
feat: oss client supports STS access (set security token in header) (#2103) 
feat: oss client supports STS access

Set security token in header and parse securityToken to init oss.Client.
Codes are in ossSourceClient getClient(header source.Header).

Fixes: #2049

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:32 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Jim Ma 7e69d8bed6
feat: support source client option (#2008) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:25 +08:00
Akash H R 7614664039
feat: oras source client (#1983) 
Signed-off-by: Akash HR <akashhr2021@gmail.com>
 2023-06-28 17:34:23 +08:00
Jim Ma 56e668bb0d
chore: optimize download log (#1944) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:19 +08:00
Jim Ma 1c24c3fc8c
fix: expire header timezone (#1840) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:10 +08:00
Jim Ma f6d8d5c6b2
chore: remove unused code (#1838) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:10 +08:00
Jim Ma fdd189e661
chore: enable cache list metadata e2e (#1829) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma ba58047871
feat: optional save list metadata to p2p (#1822) 
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma 5040af8b4f
feat: add s3 resource client and recursive e2e test (#1826) 
* feat: add s3 resource client
* chore: add minio server and recursive e2e test
* chore: add download grpc test tool

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma 2d46f6424a
feat: update source temporary error logic (#1739) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:01 +08:00
Gaius 4958967c0c
feat: http request content log (#1736) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:01 +08:00
Gaius 74e3d75f37
feat: add http request log (#1734) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:00 +08:00
cuidajun c6119d6ce0
fix: backsource temporary error judgement (#1726) 
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>

Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
 2023-06-28 17:34:00 +08:00
Jim Ma 905f12f5f0
feat: oss get metadata (#1724) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:59 +08:00
Jim Ma 1b1e9cace1
fix: hdfs not registered (#1702) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:57 +08:00
Gaius 9ccf2162de
feat: source changes ResponseHeaderTimeout and ExpectContinueTimeout (#1662) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:54 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:41 +08:00
Gaius a41a524f4c
feat: dfdaemon add Authorization and WWWAuthenticate headers (#1513) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:41 +08:00
cuidajun 1d7c87627c
fix: oss client judge directory bug (#1488) 
Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:38 +08:00
Gaius 4d71217572
feat: generate dag mock 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:36 +08:00
cuidajun 8173c11d3a
fix oss client download bug; support oss recursive download; (#1448) 
* feat: dfget support oss recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* refactor recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* remove URLEntry interface, use simple struct

Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:36 +08:00
Jim Ma cf7debf9c4
feat: back source error detail (#1437) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:34 +08:00
Gaius f68ce30f26
feat: remove errors package (#1434) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:33 +08:00
Jim Ma 776a0fef42
feat: concurrent multiple pieces back source (#1426) 
* feat: concurrent multiple pieces back source
* chore: update http source client

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:33 +08:00
Gaius a2511cb945
feat: rewrite interface{} to any (#1419) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418) 
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:31 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius f688d341bd
feat: remove cdn (#1401) 
* feat: remove cdn

Signed-off-by: Gaius <gaius.qi@gmail.com>

* docs: generate swagger

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292) 
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma 3682b26647
feat: in tree plugin (#1276) 
* feat: in tree plugin

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:16 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137) 
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:05 +08:00
Gaius a95edf5e0d
feat: change source client dialer config (#1115) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:01 +08:00
Gaius f16c5592ab
feat: change back-to-source timeout (#1112) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:00 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078) 
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977) 
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:49 +08:00
sunwp 25f8cd56dd
Fix hdfs write (#948) 
* rename reader to limitedReader

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:49 +08:00
Jim Ma beaf4ce18d
feat: update source.Response and source client interface (#945) 
* feat: update source.Response and source client interface

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:48 +08:00
sunwp 0a152e01e9
Fix HDFS File reader (#940) 
* remove unused hdfs writeTo method

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:48 +08:00
Jim Ma 062687e4c7
feat: support recursive download in dfget (#932) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:47 +08:00