ed6c6ebc3c
This commit introduces a BucketProvider interface for fetch operations against object storage provider buckets. Allowing for easier introduction of new provider implementations. The algorithm for conditionally downloading object files is the same, whether you are using GCP storage or an S3/Minio-compatible bucket. The only thing that differs is how the respective clients handle enumerating through the objects in the bucket; by implementing just that in each provider, I can have the select-and-fetch code in once place. The client implementations do now include safe-guards to ensure the fetched object is the same as metadata has been collected for. In addition, minor changes have been made to the object fetch operation to take into account that: - Etags can change between composition of index and actual fetch, in which case the etag is now updated. - Objects can disappear between composition of index and actual fetch, in which case the item is removed from the index. Lastly, the requirement for authentication has been removed (and not referring to a Secret at all is thus allowed), to provide support for e.g. public buckets. Co-authored-by: Hidde Beydals <hello@hidde.co> Co-authored by: Michael Bridgen <michael@weave.works> Signed-off-by: pa250194 <pa250194@ncr.com> |
||
|---|---|---|
| .. | ||
| gcs | ||
| s3 | ||