dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,815 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

94 Commits

Author SHA1 Message Date
Chlins Zhang d0bfdd4e77
refactor: modernize the code by new features provided by golang stand… (#3963) 
refactor: modernize the code by new features provided by golang standard library

Signed-off-by: chlins <chlins.zhang@gmail.com>
 2025-04-17 07:48:42 +00:00
Jim Ma 7f0394c1c0
feat: support google cloud storage (#3756) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2025-01-16 21:36:39 +08:00
Gaius 58959be0c2
feat: implement delete persistent cache task in scheduler (#3619) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-10-29 22:24:18 +08:00
Jim Ma 58b612fb3a
feat: add source http metric (#3468) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2024-08-28 15:48:50 +08:00
Jim Ma 8516147df6
chore: export content range in http source (#3437) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2024-08-16 16:09:39 +08:00
Chongzhi Deng 44658cf361
chore: fix typo (#3435) 
Signed-off-by: BruceAko <chongzhi@hust.edu.cn>
 2024-08-15 17:19:22 +08:00
Gaius 77b88ad717
feat: send config of the seed peer cluster for load limit (#3370) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-07-09 22:10:44 +08:00
Gaius 12fc630612
fix: if scheduler has no seed peer, return error in preheating (#2835) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-10-30 16:45:39 +08:00
Sasha Melentyev d189e18026
chore: gofmt -w -r 'interface{} -> any' . (#2790) 
Signed-off-by: Sasha Melentyev <sasha@melentyev.io>
 2023-10-11 10:39:34 +08:00
Guangwen Feng 47e7880fc4
test: add unit test for GetLastModified (#2747) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-09-21 12:08:20 +08:00
piamo e636ac76f6
fix: dfget couldn't download s3 directory correctly (#2731) 
fix: s3 list api return empty caused by incorrect parameter 'prefix' which has leading '/'

Signed-off-by: hpan <hpan@sensetime.com>
Co-authored-by: hpan <hpan@sensetime.com>
 2023-09-18 10:50:34 +08:00
Guangwen Feng 91b9ccfed7
test: add unit test for request.WithContext (#2646) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-08-16 17:18:42 +08:00
Jim Ma 9cc9fa27a6
fix: invalid error handling (#2610) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-08-08 15:35:20 +08:00
Guangwen Feng 2bc537d306
test: add unit test for Header.get (#2568) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-07-25 16:48:14 +08:00
Gaius b3f23f71a3
feat: update dy7.io/api to v2 (#2558) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-07-20 23:17:57 +08:00
Guangwen Feng 7bc0a0f71e
test: add unit test for Header.has (#2555) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-07-19 16:56:49 +08:00
Guangwen Feng 94e4b7a5a2
test: add unit test for Request.Context() (#2554) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-07-18 17:52:40 +08:00
Jim Ma b0eaa169ba
fix: e2e test dfget recursive (#2458) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:57 +08:00
Jim Ma 4c6814600f
chore: update oras error format (#2282) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:46 +08:00
Jim Ma 94a7c7639f
feat: update resource director (#2243) 
* feat: update resource director
* chore: add digest check for oras

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:44 +08:00
Jim Ma 144335ffb1
feat: add authinfo injector (#2149) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:43 +08:00
Gaius 2cda1def7f
feat: rename HostName to Hostname (#2205) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:40 +08:00
hiigao eebca2416a
feat: enable oss client download object concurrently. (#2105) 
feat: enalbe councrrent oss backsource by set contentLength

pieceManager calls DownloadSource() to download source objects.
DownloadSource() firstly connect to source and get the response from
source. Then, DownloadSource() checks the "ContentLength" value of the
response. If it larger than 0, pieceManager will download the object by
multiple threads.

Unfortunately, oss client doesn't set ContentLength value, even though
ContentLength can be set by the oss.GetObjectResult.Response.Headers.

So let's set response a ContentLength value to enable concurrent
downloads from oss.

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:33 +08:00
hiigao 0349f3cf8f
feat: oss client supports STS access (set security token in header) (#2103) 
feat: oss client supports STS access

Set security token in header and parse securityToken to init oss.Client.
Codes are in ossSourceClient getClient(header source.Header).

Fixes: #2049

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:32 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Jim Ma 7e69d8bed6
feat: support source client option (#2008) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:25 +08:00
Akash H R 7614664039
feat: oras source client (#1983) 
Signed-off-by: Akash HR <akashhr2021@gmail.com>
 2023-06-28 17:34:23 +08:00
Jim Ma 56e668bb0d
chore: optimize download log (#1944) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:19 +08:00
Jim Ma 1c24c3fc8c
fix: expire header timezone (#1840) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:10 +08:00
Jim Ma f6d8d5c6b2
chore: remove unused code (#1838) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:10 +08:00
Jim Ma fdd189e661
chore: enable cache list metadata e2e (#1829) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma ba58047871
feat: optional save list metadata to p2p (#1822) 
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma 5040af8b4f
feat: add s3 resource client and recursive e2e test (#1826) 
* feat: add s3 resource client
* chore: add minio server and recursive e2e test
* chore: add download grpc test tool

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:09 +08:00
Jim Ma 2d46f6424a
feat: update source temporary error logic (#1739) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:01 +08:00
Gaius 4958967c0c
feat: http request content log (#1736) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:01 +08:00
Gaius 74e3d75f37
feat: add http request log (#1734) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:00 +08:00
cuidajun c6119d6ce0
fix: backsource temporary error judgement (#1726) 
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>

Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
 2023-06-28 17:34:00 +08:00
Jim Ma 905f12f5f0
feat: oss get metadata (#1724) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:59 +08:00
Jim Ma 1b1e9cace1
fix: hdfs not registered (#1702) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:57 +08:00
Gaius 9ccf2162de
feat: source changes ResponseHeaderTimeout and ExpectContinueTimeout (#1662) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:54 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:41 +08:00
Gaius a41a524f4c
feat: dfdaemon add Authorization and WWWAuthenticate headers (#1513) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:41 +08:00
cuidajun 1d7c87627c
fix: oss client judge directory bug (#1488) 
Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:38 +08:00
Gaius 4d71217572
feat: generate dag mock 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:36 +08:00
cuidajun 8173c11d3a
fix oss client download bug; support oss recursive download; (#1448) 
* feat: dfget support oss recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* refactor recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* remove URLEntry interface, use simple struct

Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:36 +08:00
Jim Ma cf7debf9c4
feat: back source error detail (#1437) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:34 +08:00
Gaius f68ce30f26
feat: remove errors package (#1434) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:33 +08:00
Jim Ma 776a0fef42
feat: concurrent multiple pieces back source (#1426) 
* feat: concurrent multiple pieces back source
* chore: update http source client

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:33 +08:00
Gaius a2511cb945
feat: rewrite interface{} to any (#1419) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418) 
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00