Commit Graph

13 Commits

Author SHA1 Message Date
hiigao eebca2416a
feat: enable oss client download object concurrently. (#2105)
feat: enalbe councrrent oss backsource by set contentLength

pieceManager calls DownloadSource() to download source objects.
DownloadSource() firstly connect to source and get the response from
source. Then, DownloadSource() checks the "ContentLength" value of the
response. If it larger than 0, pieceManager will download the object by
multiple threads.

Unfortunately, oss client doesn't set ContentLength value, even though
ContentLength can be set by the oss.GetObjectResult.Response.Headers.

So let's set response a ContentLength value to enable concurrent
downloads from oss.

Signed-off-by: hiigao <hii_gao@hotmail.com>
2023-06-28 17:34:33 +08:00
hiigao 0349f3cf8f
feat: oss client supports STS access (set security token in header) (#2103)
feat: oss client supports STS access

Set security token in header and parse securityToken to init oss.Client.
Codes are in ossSourceClient getClient(header source.Header).

Fixes: #2049

Signed-off-by: hiigao <hii_gao@hotmail.com>
2023-06-28 17:34:32 +08:00
Jim Ma 7e69d8bed6
feat: support source client option (#2008)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:25 +08:00
Jim Ma f6d8d5c6b2
chore: remove unused code (#1838)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:10 +08:00
Jim Ma 2d46f6424a
feat: update source temporary error logic (#1739)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:01 +08:00
cuidajun c6119d6ce0
fix: backsource temporary error judgement (#1726)
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>

Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
2023-06-28 17:34:00 +08:00
Jim Ma 905f12f5f0
feat: oss get metadata (#1724)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:59 +08:00
cuidajun 1d7c87627c
fix: oss client judge directory bug (#1488)
Signed-off-by: bigerous <bigerous@qq.com>
2023-06-28 17:33:38 +08:00
cuidajun 8173c11d3a
fix oss client download bug; support oss recursive download; (#1448)
* feat: dfget support oss recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* refactor recursive download

Signed-off-by: bigerous <bigerous@qq.com>

* remove URLEntry interface, use simple struct

Signed-off-by: bigerous <bigerous@qq.com>
2023-06-28 17:33:36 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:31 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius f688d341bd
feat: remove cdn (#1401)
* feat: remove cdn

Signed-off-by: Gaius <gaius.qi@gmail.com>

* docs: generate swagger

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Jim Ma 3682b26647
feat: in tree plugin (#1276)
* feat: in tree plugin

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:16 +08:00