Commit Graph

335 Commits

Author SHA1 Message Date
Stefan Prodan 484cbcc14c
Refactor: Move inventory helpers to internal package
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-10-11 13:45:54 +03:00
Stefan Prodan 3090eff734
Refactor: Acquire artifacts with `fluxcd/pkg/http/fetch`
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-10-10 15:30:58 +03:00
Stefan Prodan 06e91e047b
Refactor: Use impersonation from `fluxcd/pkg/runtime/client`
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-10-07 17:36:41 +03:00
Stefan Prodan 731188e45b
Refactor: Extract generator to internal package
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-10-07 12:49:36 +03:00
Stefan Prodan 06ce60f7bf
Refactor: Extract decrytor to internal package
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-10-06 19:45:12 +03:00
Max Jonas Werner c3c4cc13ac
support alternative kustomization file names
`KustomizeGenerator.WriteFile` now detects alternative kustomization
file names such as `kustomization.yml` and `Kustomization`.

closes #737

Signed-off-by: Max Jonas Werner <max@e13.dev>
2022-10-04 15:34:00 +02:00
Stefan Prodan 69509c92e2
Build with Go 1.19
- Update Go to 1.19 in CI
- Use Go 1.19 in base image
- Update controller-gen v0.8.0 (v0.7 fails with Go 1.19) and regenerate manifests

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-28 18:17:34 +03:00
Stefan Prodan 65a58330fc
Fix health checking for global objects
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-14 12:32:49 +03:00
Stefan Prodan 8feccf0aeb
Merge pull request #726 from bkreitch/try-offline-methods-first
Sort SOPS masterkeys so offline decrypt methods are tried first
2022-09-12 12:06:11 +03:00
Boris Kreitchman 5fe7910ab8 Sort masterkeys so offline decrypt methods are tried first
Signed-off-by: Boris Kreitchman <bkreitch@gmail.com>
2022-09-10 15:38:57 +03:00
Stefan Prodan 9e0930cfa1
Merge pull request #723 from pjbgf/native-fuzz
fuzz: Refactor Fuzzers based on Go native fuzzing
2022-09-10 09:47:24 +03:00
Stefan Prodan 0c6916e34a
Update Source API to v0.29.0
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-10 09:25:38 +03:00
Paulo Gomes d2af6edc56
tests: Fix subtests calling Fail from parent
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-09-07 12:27:33 +01:00
Paulo Gomes 73f005e541
fuzz: Refactor Fuzzers based on Go native fuzzing
The existing fuzzers are converted into the Go native format. This
works well with most fuzzers, apart from the one that fuzzes the
controller.

That specific test depends on funcs and vars that are defined on
suite_test.go, which is not supported when building fuzzers leveraging
Go native fuzz for oss-fuzz. That test is therefore ignored when
executed with make fuzz-native. But, all tests are covered with
make fuzz-smoketest, in order to support oss-fuzz.

Once the controller is rewritten we can revist this and ensure that
fuzzers does not require envtest nor embedded CRDs.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-09-07 12:26:43 +01:00
Stefan Prodan 928f22df5c
Reconcile Kubernetes class type objects in a dedicated stage
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-02 11:31:54 +03:00
Santosh Kaluskar e486234e4c Enable RecoverPanic option on reconciler
Signed-off-by: Santosh Kaluskar <dtshbl@gmail.com>
2022-08-13 21:05:10 +05:30
Stefan Prodan 602238d171
Add file path to sops decryption errors
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-09 12:40:08 +03:00
Stefan Prodan a7d790f1b2
Add support for OCIRepository sources
- allow `OCIRepository` to be specified in `sourceRef.kind`
- react to `OCIRepository` artifacts events
- add end-to-end test for OCI repositories

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-08 18:26:03 +03:00
Hidde Beydals 2cdc9a578d decryptor: recover from SOPS store panic
Based on user reports, there seems to be a small chance for the
underlying SOPS store implementation to panic when a user provides input
and/or output format instructions which do not actually match the type
of the file. Recover from this to ensure continuity of operations.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-07-13 15:50:54 +02:00
Stefan Prodan cfd5200fbb
Retry downloading the artifact on not found errors
- Extract the artifact operations such as download, verify, untar into a dedicated struct
- Introduce a dedicated type for artifact not found errors
- On artifact not found errors, log the error, update the ready status message and requeue the object
- Retry the artifact download at the interval set with `--requeue-dependency` (defaults to 30s)

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-07-01 13:55:47 +03:00
Somtochi Onyekwere 13044520c1 Pass polling options to impersonation client
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-06-28 15:28:56 +01:00
Somtochi Onyekwere 2ac475ebef sops/gcpkms: add key source implementation
This adds a SOPS GCP KMS key source which makes use of the latest GCP
client, and supports both injection of master key credentials and a
default client making use of environmental runtime values.

The implementation fully replaces SOPS', and is covered with
compatability tests.

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-05-24 21:51:46 +02:00
Stefan Prodan 1535282fa1
Merge pull request #657 from fluxcd/gen-test-relbase
generator: test secure build relative base dir
2022-05-24 11:48:43 +03:00
Hidde Beydals db95625609 generator: test secure build relative base dir
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-05-23 12:14:20 +02:00
Stefan Prodan e5d47c264f
Merge pull request #655 from SomtochiAma/image-digest
Set digests in image override
2022-05-22 10:38:53 +03:00
Somtochi Onyekwere 17664e8862 remove empty test
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-05-19 13:22:42 +01:00
Somtochi Onyekwere 8c246a5450 Set digests in image override
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-05-19 13:15:11 +01:00
Sanskar Jaiswal 326e20c41b update code comments and kms decryption docs
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
2022-05-19 14:34:16 +05:30
Sanskar Jaiswal 1ee85e5988 add and update tests for awskms and hcvault
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
2022-05-19 14:34:16 +05:30
Sanskar Jaiswal e5a37b7013 migrate aws kms implementation to aws-go-sdk-v2
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
2022-05-19 14:34:15 +05:30
Sanskar Jaiswal 53aded2596 add support for AWS KMS credentials using .spec.decryption
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
2022-05-19 14:33:22 +05:30
Hidde Beydals e5653b2ff3 Drop SOPS LocalClient implementation
As patch has been merged upstream, and `keyservice.LocalClient` does
now accept the `KeyServiceServer` interface.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-05-09 22:02:05 +02:00
Stefan Prodan ba03c878b0
Merge pull request #638 from fluxcd/no-remote-bases
Disallow remote bases usage in Kustomize overlays
2022-05-03 11:35:05 +03:00
Hidde Beydals 3f9999b9ee
Merge pull request #615 from nstogner/secret-ref-key 2022-04-29 20:23:53 +02:00
Nick Stogner 6aeff8c924 Support defining a KubeConfig Secret data key
Signed-off-by: Nick Stogner <nicholas.stogner@gmail.com>
2022-04-29 10:04:14 -04:00
Hidde Beydals 36df540a5d decryptor: detect format of Secret data field
This checks the base64 decoded bytes from a Secret field for any of the
marker bytes, thereby allowing data to be encrypted into any format.
Instead of the previous behavior which assumed it to either be YAML or
JSON.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-29 15:26:36 +02:00
Hidde Beydals a7639c68d3 decryptor: detect DockerConfigJsonKey as JSON out
This ensures the Secret field gets formatted back into JSON, instead of
it being detected as binary output.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-29 11:00:44 +02:00
Stefan Prodan cafbe253a1
Disallow remote bases usage in Kustomize overlays
Add an optional flag for disabling remote bases. While the `--no-remote-bases` is set to `false` by default, Flux users are encouraged to enable it on production system for security and performance reasons. Using Kustomize remote bases means that kustomize-controller must clone the remote repositories on every reconciliation instead of using the source-controller artifacts cache. Allowing remote bases on multi-tenant clusters, means platform admins have no control over which repositories make up the desired state.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-04-27 14:54:43 +03:00
Hidde Beydals 6854ff519c generator: recover from kustomize build panics
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-27 12:11:40 +02:00
Stefan Prodan d1a686235c
Add test for build panic
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-04-27 12:58:56 +03:00
Steven E. Harris e6beca11b1
Confirm format detection of file sources via tests
Signed-off-by: Steven E. Harris <seh@panix.com>
2022-04-26 11:07:17 -04:00
Steven E. Harris ee0e4c1674
Remove invalid "EnvSources" entry from test case
Signed-off-by: Steven E. Harris <seh@panix.com>
2022-04-26 11:07:17 -04:00
Steven E. Harris 7162f5e6fa
Use Secret generator keys for SOPS format hint
Rather than inspecting the source file name supplied to kustomize's
Secret generator to determine the format of the SOPS-encrypted file
content, instead inspect the Secret key (when supplied separately from
the source file name) as a more reliable heuristic.

Doing so allows kustomization authors to name their SOPS-encrypted
output files with a ".json" extension accurately reflecting the format
in which SOPS writes its encrypted output, even if the encrypted
content itself is not in JSON format.

Signed-off-by: Steven E. Harris <seh@panix.com>
2022-04-26 11:07:17 -04:00
Hidde Beydals cda51febc7
Merge pull request #631 from cwyl02/ychen/exp-backoff-retry 2022-04-22 09:45:45 +02:00
Sanskar Jaiswal d499ed8063 Ensure generated temp dir is absolute on all OSes
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
2022-04-22 09:01:46 +02:00
York Chen 827f814f2f
feat: flags to configure exponential backoff retry
Signed-off-by: York Chen <ychen@d2iq.com>
2022-04-21 10:09:39 -04:00
Hidde Beydals 880587f20d generator: ensure remote build dirs can be reached
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-20 17:12:33 +02:00
Hidde Beydals f4528fb25d controllers: use own Kustomize FS implementation
For details, see: https://github.com/fluxcd/pkg/pull/262

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-15 15:48:23 +02:00
Hidde Beydals 105ebd9f47 controllers: improve decryptor and add tests
- Refactored recursion while iterating over Kustomization files.
  References of files that have been visited are cached, and not
  visited again. In addition, symlinks are confirmed to not traverse
  outside the working directory.
- Optimized various bits around (un)marshalling (encrypted) data, and
  YAML -> JSON -> YAML roundtrips are prevented where not required.
- Added support for decrypting INI Kustomize EnvSource references using
  the dedicated SOPS store for the format.
- Introduced support for decrypting Kustomize FileSources:
  https://pkg.go.dev/sigs.k8s.io/kustomize@v1.0.2/pkg/types#DataSources

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-14 09:17:43 +02:00
Hidde Beydals 62dafa1baf decryptor: switch Printf to Errorf
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-05 13:07:01 +02:00