Michael Bridgen
6d77ccf89f
Factor out fetching objects
...
The algorithm for conditionally downloading object files is the same,
whether you are using GCP storage or an S3/Minio-compatible
bucket. The only thing that differs is how the respective clients
handle enumerating through the objects in the bucket; by implementing
just that in each provider, I can have the select-and-fetch code in
once place.
This deliberately omits the parallelised fetching that the GCP client
had, for the sake of lining the clients up. It can be reintroduced (in
the factored out code) later.
Signed-off-by: Michael Bridgen <michael@weave.works>
2021-11-10 14:08:50 +00:00
pa250194
b3711bb5a1
Fix: errgroup provides waitgroup within the Go function
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-11-10 14:08:50 +00:00
pa250194
b4bd037e5d
General cleanup and function renaming
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-11-10 14:08:50 +00:00
pa250194
6f8209166c
Refactor Bucket Controller
...
Fix bug in bucket provider interface
Added Bucket Provider Interface
Signed-off-by: pa250194 <pa250194@ncr.com>
Fix context timeout defer issue
Signed-off-by: pa250194 <pa250194@ncr.com>
Fix GCP storage provider test
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-11-10 14:08:50 +00:00
pa250194
c4e4b3928c
Added Logger to closing GCP client
...
Signed-off-by: pa250194 <pa250194@ncr.com>
Added log for GCP provider auth error
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:57:48 -05:00
pa250194
c98130548e
Added Tests to GCP provider
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:21:15 -05:00
pa250194
99c79bffaa
Tests for GCP Bucket Provider
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:19:19 -05:00
pa250194
be1ed50ac4
Service Account Key Authentication to GCP Provider
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 14:16:53 -05:00
pa250194
38bf4d9859
Fixed spelling and capitalization
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-10-14 13:51:02 -05:00
pa250194
7c0d4c070e
Refactor comments and method names
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-23 13:43:08 -05:00
pa250194
38be5ed111
Cleanup obsolete comments
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-23 12:40:55 -05:00
pa250194
057c65e939
Removed resumable downloads
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-23 12:39:00 -05:00
pa250194
6ff5970fe1
Added more tests and cleaned up GCP provider logic
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194
c204f6a8ee
Added Tests to GCP provider
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194
eeb38bdf2c
Tests for GCP Bucket Provider
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194
0444c6e16d
Service Account Key Authentication to GCP Provider
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194
78379ddcd5
Added initial testing for new GCP provider
...
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
pa250194
7da9619b58
Feature: Add Support for Google Cloud Storage along with Workload Identity
...
Added Support for Google Cloud Storage with Workload Identity as Source Provider. This enables the use of GCP without enabling S3 compatible access.
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00