Commit Graph

37 Commits

Author SHA1 Message Date
Matt Morrison 811f7d0420
Push issuer documents to GCS 2023-10-24 07:57:21 +13:00
justinsb 27dd371c08 Refactor: Add context.Context to more VFS methods
This allows for propagation of opentelemetry traces.
2023-10-14 14:31:05 -04:00
Kubernetes Prow Robot 436732b724
Merge pull request #15395 from Codelax/vfs-remove-all
add removeAll to vfs
2023-07-15 14:55:13 -07:00
guoguangwu da8c0ca830 chore: remove refs to deprecated io/ioutil 2023-06-07 18:02:33 +08:00
Jules Casteran e27ae94daf add removeAll to vfs 2023-05-10 15:24:15 +02:00
justinsb 0b8cb933ad gce terraform: Use google_storage_object_acl
We are using the schema of google_storage_object_acl.
2023-01-22 10:15:25 -05:00
justinsb c2810a5c96 Refactor to reduce vfs -> terraformWriter dependency 2023-01-04 13:38:47 -05:00
justinsb 707758f1e9 Support multiple terraform file providers
By adding them on-demand, it is arguably a little simpler as well.

Co-authored-by: John Gardiner Myers <jgmyers@proofpoint.com>
2023-01-04 08:14:18 -05:00
justinsb 6c2edaee7e Add Context arg to vfs ReadFile
This is an "action" method, so should take a context.
2023-01-01 09:51:44 -05:00
justinsb ae074bbc6b VFS: Add context to CreateFile & WriteFile
As "request" methods, these should have context parameters.
2022-12-23 10:39:52 -05:00
justinsb 46f27168b8 vfs: avoid early initialization of GCS client
By initializing on demand, we avoid the need for some context.Context
during "build" time and better reflect the notion of (passive)
builders vs (active) requests.
2022-12-20 15:23:16 -05:00
John Gardiner Myers 8b6f2973f8 Refactor GSPath use of Literal 2022-11-26 19:24:23 -08:00
Nat Henderson 9b08c4bb51 Enable internal load balancers when running on GCP
* Add ILBs, broadly following the AWS model.  The following new
capabilities are added for clusters in GCP:
  * Cluster's spec.api.loadBalancer can be set to 'type: internal' on
    GCP.
    * Therefore, GCP can now create:
        * regional backend services
        * regional (non-legacy) healthchecks
        * firewall rules with "internal" load-balancing scheme
        * firewall rules with dot-notation-specified IP addresses
  * Cluster's spec.api.loadBalancer's 'subnets' field functions
    as in the AWS model.

A few incidental changes are included, either because this change
touched the relevant code or because my use case happened to trigger the
issues that are fixed here.

* Cluster's spec.networkID field can be prefixed by project to use
  GCP's common cross-project networking model.
    * The presumption is that all specified subnets belong to this
      network and therefore this project.

* Add missing operation wait on forwarding rule creation.

* Some Terraform output improvements:
    * Permit no-ACL files in GCS buckets in Terraform output.
    * Enable marginally better cross-resource reference in Terraform outputs
    * Add project to network + subnetwork literals in Terraform output.
    * Add terraform output to backend services and health checks.

Testing:
  * Add mocks for backend services and health checks.
  * Add minimal integration test - copied from gce_private and ilb added.
  * Add update cluster goldens.

Co-authored-by: Travis Reid <travis_reid@apple.com>
2022-04-25 13:31:47 -07:00
Tessia Piboubès 705a7b536c Use a pointer type in type assertion 2022-04-20 20:07:58 +02:00
Peter Rifel f56f98154a
Add TerraformProvider 2021-12-02 08:01:35 -06:00
Peter Rifel 675754edeb
Add Terraform support for GS Paths 2021-12-02 08:01:34 -06:00
Ciprian Hacman ea7df00719 Run hack/update-gofmt.sh 2021-12-01 22:39:50 +02:00
Peter Rifel 4d9f0128a3
Upgrade to klog2
This splits up the kubernetes 1.19 PR to make it easier to keep up to date until we get it sorted out.
2020-08-16 20:56:48 -05:00
Ciprian Hacman d54aadc89c Fix nits for removal of S3 file versions 2020-05-28 06:50:32 +03:00
Ciprian Hacman 9675692b84 Implement RemoveAll() for S3 paths 2020-05-25 07:46:32 +03:00
Justin Santa Barbara 108d1eee5d Replace deprecated x/net/context with context
It's not x-perimental any more!
2020-04-09 23:58:19 -04:00
Kubernetes Prow Robot a34ad252ff
Merge pull request #8496 from justinsb/log_acls
GCS: Log ACLs if we're writing them
2020-02-06 22:49:43 -08:00
Justin SB 9fb80f9048
GCS: Log ACLs if we're writing them
We log at V(4) because they are fairly verbose.
2020-02-06 14:46:41 -05:00
Justin SB 9e7a026332
GCS: Fix bug where around retry on GCS
We were recomputing the MD5, but we would need to rewind the stream first.
2020-02-06 14:45:39 -05:00
tanjunchen 3f9400a588 util/pkg/vfs/:staticcheck 2019-12-23 10:20:56 +08:00
mikesplain 9e55b8230a Update copyright notices
Also cleans some white spaces
2019-09-09 14:47:51 -04:00
Justin SB 3e33ac7682
Change code from glog to klog
We don't call klog.InitFlags yet, because that will cause a flag
redefinition error until we get everyone to stop using glog.  That
will happen when we update to k8s 1.13.
2019-05-06 12:54:51 -04:00
Justin Santa Barbara 49e5797bc0 Google Cloud Storage md5 decoding fix
The MD5 is presented base64 encoded; we were trying to decode it as
hex.
2018-10-09 18:16:15 -07:00
Justin Santa Barbara 412cf377c2 VFS: WriteFile takes an io.ReadSeeker
Means we don't have to buffer big files in memory, in combination with
WriteTo for reading.
2018-02-26 09:09:17 -05:00
Justin Santa Barbara 6f6bafb65e VFS: Support io.WriterTo interface
Allows us to handle much bigger files (no need to buffer in-memory)
2018-01-08 22:34:27 -05:00
chrislovecnm 8d1ee1fa16 updating files for goimports 2017-11-01 12:51:43 -06:00
Justin Santa Barbara b2bcba4a6d GCE: Use object-level permissions for files in GCS
This lets us configure cross-project permissions while ourselves needing
minimal permissions, but also gives us a nice hook for future lockdown
of object-level permissions.
2017-10-29 19:17:00 -04:00
Justin Santa Barbara dbbe3f373b GCE: Set up permissions for cross-project configurations
This ensures that the cluster can read the kops state store files, even
if the GCS bucket is in a different project.

We automatically set up an IAM access policy that grants access.
2017-10-28 03:24:18 -04:00
Justin Santa Barbara 83300fc39f GCS paths; retry on error
The AWS SDK does this for us, I think, the GS SDK does not.
2017-09-27 09:31:33 -04:00
Justin Santa Barbara d257e73b1c GCS: Don't reuse same error message
We had exactly the same error message for two code paths, which made
figuring out the cause harder.
2017-09-16 21:39:14 -04:00
Justin Santa Barbara 3d14d07616 Support cloud-config on GCE 2017-02-28 20:08:03 -05:00
Justin Santa Barbara 645f330dad Re-enable GCE support
We move everything to the models.  We feature-flag it, because we
probably want to change the names etc, and we aren't going to be able to
offer smooth upgrades until that is done.
2017-02-28 20:08:03 -05:00