feat: enalbe councrrent oss backsource by set contentLength
pieceManager calls DownloadSource() to download source objects.
DownloadSource() firstly connect to source and get the response from
source. Then, DownloadSource() checks the "ContentLength" value of the
response. If it larger than 0, pieceManager will download the object by
multiple threads.
Unfortunately, oss client doesn't set ContentLength value, even though
ContentLength can be set by the oss.GetObjectResult.Response.Headers.
So let's set response a ContentLength value to enable concurrent
downloads from oss.
Signed-off-by: hiigao <hii_gao@hotmail.com>
feat: oss client supports STS access
Set security token in header and parse securityToken to init oss.Client.
Codes are in ossSourceClient getClient(header source.Header).
Fixes: #2049
Signed-off-by: hiigao <hii_gao@hotmail.com>
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feat: add s3 resource client
* chore: add minio server and recursive e2e test
* chore: add download grpc test tool
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feat: update namely/protoc-all image version to 1.47_0
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: generate mock file
Signed-off-by: Gaius <gaius.qi@gmail.com>
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feat: optimize back source trace and logic
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: fix keepalive performance
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: optimize generate peer id and transport performance
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* fix: io.Copy call superfluous sendfile, then increase network latency
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata
Signed-off-by: Jim Ma <majinjing3@gmail.com>
* chore: run `go fmt ./...`
This commit synchronizes `//go:build` lines with `// +build` lines.
Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* refactor: move from io/ioutil to io and os packages
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Jim Ma