The algorithm for conditionally downloading object files is the same,
whether you are using GCP storage or an S3/Minio-compatible
bucket. The only thing that differs is how the respective clients
handle enumerating through the objects in the bucket; by implementing
just that in each provider, I can have the select-and-fetch code in
once place.
This deliberately omits the parallelised fetching that the GCP client
had, for the sake of lining the clients up. It can be reintroduced (in
the factored out code) later.
Signed-off-by: Michael Bridgen <michael@weave.works>
Added Support for Google Cloud Storage with Workload Identity as Source Provider. This enables the use of GCP without enabling S3 compatible access.
Signed-off-by: pa250194 <pa250194@ncr.com>
Michael Bridgen
pa250194