b0eaa169ba
fix: e2e test dfget recursive ( #2458 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:57 +08:00
4c6814600f
chore: update oras error format ( #2282 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:46 +08:00
94a7c7639f
feat: update resource director ( #2243 )
...
* feat: update resource director
* chore: add digest check for oras
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:44 +08:00
144335ffb1
feat: add authinfo injector ( #2149 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:43 +08:00
2cda1def7f
feat: rename HostName to Hostname ( #2205 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:40 +08:00
eebca2416a
feat: enable oss client download object concurrently. ( #2105 )
...
feat: enalbe councrrent oss backsource by set contentLength
pieceManager calls DownloadSource() to download source objects.
DownloadSource() firstly connect to source and get the response from
source. Then, DownloadSource() checks the "ContentLength" value of the
response. If it larger than 0, pieceManager will download the object by
multiple threads.
Unfortunately, oss client doesn't set ContentLength value, even though
ContentLength can be set by the oss.GetObjectResult.Response.Headers.
So let's set response a ContentLength value to enable concurrent
downloads from oss.
Signed-off-by: hiigao <hii_gao@hotmail.com>
2023-06-28 17:34:33 +08:00
0349f3cf8f
feat: oss client supports STS access (set security token in header) ( #2103 )
...
feat: oss client supports STS access
Set security token in header and parse securityToken to init oss.Client.
Codes are in ossSourceClient getClient(header source.Header).
Fixes : #2049
Signed-off-by: hiigao <hii_gao@hotmail.com>
2023-06-28 17:34:32 +08:00
79024c87ca
refactor: parse http range ( #2071 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:30 +08:00
7e69d8bed6
feat: support source client option ( #2008 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:25 +08:00
7614664039
feat: oras source client ( #1983 )
...
Signed-off-by: Akash HR <akashhr2021@gmail.com>
2023-06-28 17:34:23 +08:00
f6d8d5c6b2
chore: remove unused code ( #1838 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:10 +08:00
fdd189e661
chore: enable cache list metadata e2e ( #1829 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
5040af8b4f
feat: add s3 resource client and recursive e2e test ( #1826 )
...
* feat: add s3 resource client
* chore: add minio server and recursive e2e test
* chore: add download grpc test tool
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
2d46f6424a
feat: update source temporary error logic ( #1739 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:01 +08:00
4958967c0c
feat: http request content log ( #1736 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:01 +08:00
74e3d75f37
feat: add http request log ( #1734 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:00 +08:00
c6119d6ce0
fix: backsource temporary error judgement ( #1726 )
...
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
2023-06-28 17:34:00 +08:00
905f12f5f0
feat: oss get metadata ( #1724 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:59 +08:00
1b1e9cace1
fix: hdfs not registered ( #1702 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:57 +08:00
9ccf2162de
feat: source changes ResponseHeaderTimeout and ExpectContinueTimeout ( #1662 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:54 +08:00
a41a524f4c
feat: dfdaemon add Authorization and WWWAuthenticate headers ( #1513 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:41 +08:00
1d7c87627c
fix: oss client judge directory bug ( #1488 )
...
Signed-off-by: bigerous <bigerous@qq.com>
2023-06-28 17:33:38 +08:00
8173c11d3a
fix oss client download bug; support oss recursive download; ( #1448 )
...
* feat: dfget support oss recursive download
Signed-off-by: bigerous <bigerous@qq.com>
* refactor recursive download
Signed-off-by: bigerous <bigerous@qq.com>
* remove URLEntry interface, use simple struct
Signed-off-by: bigerous <bigerous@qq.com>
2023-06-28 17:33:36 +08:00
cf7debf9c4
feat: back source error detail ( #1437 )
...
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:34 +08:00
f68ce30f26
feat: remove errors package ( #1434 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:33 +08:00
776a0fef42
feat: concurrent multiple pieces back source ( #1426 )
...
* feat: concurrent multiple pieces back source
* chore: update http source client
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:33 +08:00
0222649b88
feat: update namely/protoc-all image version to 1.47_0 ( #1418 )
...
* feat: update namely/protoc-all image version to 1.47_0
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: generate mock file
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
ad36eb64a7
feat: remove github/pkg/errors package ( #1416 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:31 +08:00
7a30db3fba
refactor: pkg util ( #1402 )
...
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
f688d341bd
feat: remove cdn ( #1401 )
...
* feat: remove cdn
Signed-off-by: Gaius <gaius.qi@gmail.com>
* docs: generate swagger
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
823c722c3e
feat: support response header ( #1292 )
...
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
3682b26647
feat: in tree plugin ( #1276 )
...
* feat: in tree plugin
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:16 +08:00
Jim Ma