Commit Graph

59 Commits

Author SHA1 Message Date
justinsb d8c449a4f8 Basic OpenTelemetry support
We initially support capturing to a file (in our own format, as it
doesn't appear a suitable format exists).  This means we don't need a
server to capture the traces, and can start capturing through prow
without a lot of infrastructure changes.

Co-authored-by: Peter Rifel <rifelpet@users.noreply.github.com>
2023-11-18 09:12:19 -05:00
justinsb 27dd371c08 Refactor: Add context.Context to more VFS methods
This allows for propagation of opentelemetry traces.
2023-10-14 14:31:05 -04:00
Mia-Cross e12f3dc07b
scaleway: terraform support (#15892)
* tasks dependencies

* FindAddresses should proceed even if LBID is empty

* refacto instance: handle several instances in IG + volumeSize in model

* refacto lb_backend: set servers' IPs in backend task rather than instance

* dns changes for tf support

* render Scaleway s3 objects + add zone to outputs

* add terraform support to documentation

* new instance unique name should be given according to actual instances names

* avoid untimely changes in tf + tag instance IPs to avoid orphaned IPs at cluster deletion

* update integration tests

* review comments

* add IPs to the resources to be deleted with the cluster

* add task + model for DNS record
2023-09-29 04:22:42 -07:00
Jules Casteran e27ae94daf add removeAll to vfs 2023-05-10 15:24:15 +02:00
srikiz efdccb2e65 incorporate review comments 2023-04-11 07:51:36 +05:30
srikiz e77e660ed9 More changes based on testing 2023-04-11 07:51:36 +05:30
justinsb c2810a5c96 Refactor to reduce vfs -> terraformWriter dependency 2023-01-04 13:38:47 -05:00
Kubernetes Prow Robot aeca5f9b54
Merge pull request #14926 from justinsb/multiple_tfproviders
Support multiple terraform file providers
2023-01-04 09:24:02 -08:00
justinsb 50556d6ab1 vfs: don't cache bucketDetails in S3Path
We cache it in the S3Context anyway, and this potentially avoids some
race conditions (albeit likely harmless race conditions).

Co-authored-by: John Gardiner Myers <jgmyers@proofpoint.com>
2023-01-04 08:26:48 -05:00
justinsb 707758f1e9 Support multiple terraform file providers
By adding them on-demand, it is arguably a little simpler as well.

Co-authored-by: John Gardiner Myers <jgmyers@proofpoint.com>
2023-01-04 08:14:18 -05:00
justinsb 6c2edaee7e Add Context arg to vfs ReadFile
This is an "action" method, so should take a context.
2023-01-01 09:51:44 -05:00
justinsb ae074bbc6b VFS: Add context to CreateFile & WriteFile
As "request" methods, these should have context parameters.
2022-12-23 10:39:52 -05:00
justinsb bba2eaf946 vfs: add context to s3 functions
We don't have the early initialization problem, but we do want to pass
the context through to the requests e.g. for tracing.
2022-12-20 15:34:42 -05:00
John Gardiner Myers 89cda928e1 Fixes for checking if S3 bucket is public 2022-12-17 11:05:38 -08:00
justinsb f4984dafab Support public buckets for serviceAccountIssuers on S3
S3 is also nudging towards bucket level permissions, so don't set an ACL when bucket is public.
2022-12-17 09:57:45 -08:00
Ciprian Hacman 8b8dd8d2dd Use aws_s3_object instead of deprecated aws_s3_bucket_object 2022-04-13 16:49:03 +03:00
Ole Markus With 2fa53989c4 Configure dualstack endpoint for s3
Use dualstack https endpoints on ipv6only cluster. Always use
dualstack endpoints through the SDK
2021-11-20 08:00:00 +01:00
Eng Zer Jun 425173ae9f
refactor: move from io/ioutil to io and os packages
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-11-12 15:37:18 +08:00
Peter Rifel 7eaa647b2a
Include the provider alias on TerraformPath resources that reference their provider 2021-09-17 18:08:59 -05:00
Peter Rifel c3a7f9f75a
Add TerraformProvider definition to TerraformPath interface 2021-09-17 18:08:58 -05:00
Ole Markus With 67b4024694 Reconcile if managedFile is public or not 2021-08-13 20:20:44 +02:00
John Gardiner Myers eb076e3713 Render managed files with Terraform 2021-06-28 12:15:15 -07:00
AkiraFukushima d52ec60c02
Fix issuer and jwks object path for IRSA 2021-06-01 23:35:21 +09:00
Peter Rifel 7c900b7fae Generate and upload keys.json + discovery.json to public store
Generate and upload keys.json + discovery.json to public store

Don't enable anonymous auth on publicjwks

Remove tests that won't work using FS VFS anymore
2021-03-19 20:03:26 +01:00
Peter Rifel 4d9f0128a3
Upgrade to klog2
This splits up the kubernetes 1.19 PR to make it easier to keep up to date until we get it sorted out.
2020-08-16 20:56:48 -05:00
Justin SB 319ddcc333 S3 DeleteAllVersions: use pagination
This way we're not limited to one page of versions.  This is likely a
purely theoretical concern, at least as we're using it today.
2020-05-31 18:21:05 -04:00
Ciprian Hacman d54aadc89c Fix nits for removal of S3 file versions 2020-05-28 06:50:32 +03:00
Ciprian Hacman 9675692b84 Implement RemoveAll() for S3 paths 2020-05-25 07:46:32 +03:00
Ciprian Hacman b565122875 Remove delete markers also from S3 bucket 2020-05-24 17:42:02 +03:00
Ciprian Hacman a48ccfa06c Return warning instead of error to hide issues during cluster teardown 2020-05-24 15:20:20 +03:00
Ciprian Hacman 1a38a3feaa Return os.ErrNotExist when no versions are found 2020-05-24 11:42:18 +03:00
Ciprian Hacman 56af880c53 Remove TODO that was not addressed for a long time 2020-05-24 10:11:56 +03:00
Ciprian Hacman 831e3f0e57 Remove all versions of a file form the S3 bucket 2020-05-24 08:38:46 +03:00
feifei.zhang@huawei.com 48ebd260d3 fix golint failures 2019-11-24 16:38:58 +08:00
mikesplain 9e55b8230a Update copyright notices
Also cleans some white spaces
2019-09-09 14:47:51 -04:00
Justin SB 3e33ac7682
Change code from glog to klog
We don't call klog.InitFlags yet, because that will cause a flag
redefinition error until we get everyone to stop using glog.  That
will happen when we update to k8s 1.13.
2019-05-06 12:54:51 -04:00
Justin Santa Barbara 85d47cd67d s3: lazy-evaluate encryption policy
Should help performance a little bit, and should be a little faster.
2018-10-11 06:46:34 -07:00
David Archer 83db56fab0
Use appropriate log level for KOPS_STATE_S3_ACL debug message 2018-08-30 09:58:23 -04:00
Justin Santa Barbara 1753423027 DigitalOcean: don't try to set SSE
We lost the p.sse check in a bad merge; restoring it here.

Fix #5519
2018-08-14 21:26:18 -04:00
Christian Jantz 6fba37ea63 Merge branch 'master' of github.com:gekart/kops into feature/s3_bucket_encryption 2018-05-23 10:49:21 +02:00
andrewsykim 2947bb1b9e allow s3 vfs scheme and sse to be configurable 2018-04-01 23:05:11 -04:00
Grischa Ekart 7c41e35bbc Implement AWS Default Bucket Encryption PR #4235 2018-03-07 23:26:28 +01:00
Justin Santa Barbara 412cf377c2 VFS: WriteFile takes an io.ReadSeeker
Means we don't have to buffer big files in memory, in combination with
WriteTo for reading.
2018-02-26 09:09:17 -05:00
Justin Santa Barbara 6f6bafb65e VFS: Support io.WriterTo interface
Allows us to handle much bigger files (no need to buffer in-memory)
2018-01-08 22:34:27 -05:00
chrislovecnm 7057aaf1bb Enabling the file assets
File assets and the SHA files are uploaded to the new location. Files
when are users uses s3 are upload public read only. The copyfile task
uses only the existing SHA value.

This PR include major refactoring of the use of URLs.  Strings are no
longer categnated, but converted into a URL struct and path.Join is
utlilized.

A new values.go file is included so that we can start refactoring more
code out of the "fi" package.

A
2017-12-17 15:26:57 -07:00
chrislovecnm 8d1ee1fa16 updating files for goimports 2017-11-01 12:51:43 -06:00
Justin Santa Barbara b2bcba4a6d GCE: Use object-level permissions for files in GCS
This lets us configure cross-project permissions while ourselves needing
minimal permissions, but also gives us a nice hook for future lockdown
of object-level permissions.
2017-10-29 19:17:00 -04:00
Justin Santa Barbara 106875115d Support for deleting tokens & keypairs
This now allows for deleting all secrets, which means we can have a
procedure for rotating all keys.
2017-09-09 01:04:45 -04:00
Miao Luo 22c49e76cb Fix user-defined s3 endpoint support.
Address review feedbacks and remove unintended space.
2017-04-19 23:46:05 -07:00
Miao Luo 76437a77d4 Support user-defined s3 endpoint. 2017-04-18 11:27:07 -07:00