Commit Graph

182 Commits

Author SHA1 Message Date
Yohan Belléguic 2741d0a150 fix typo in helmRepo secretRef spec CRD
When using a TLS authentication, user can provide a custom certificate
by setting the caFile key in the secret, not caCert.

Signed-off-by: Yohan Belléguic <yohan.belleguic@arkea.com>
2023-01-16 19:52:06 +05:30
Paulo Gomes cc75764412
api: Deprecate field gitImplementation
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-12-12 15:34:32 +00:00
Paulo Gomes 22e8b0ff7e
Update dependencies
Given that pkg/* now depends on fluxcd/go-git, this changes also apply
the same changes here.

New versions:
- github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0.
- github.com/distribution/distribution/v3 v3.0.0-20221111170714-3b8fbf975279.
- github.com/fluxcd/pkg/apis/event v0.2.0.
- github.com/fluxcd/pkg/helmtestserver v0.10.0.
- github.com/fluxcd/pkg/oci v0.15.0.
- github.com/fluxcd/pkg/runtime v0.24.0.
- github.com/fluxcd/pkg/sourceignore v0.3.0.
- github.com/google/go-containerregistry v0.12.1.
- github.com/google/go-containerregistry/pkg/authn/k8schain v0.0.0-20221114162634-781782aa2757.
- golang.org/x/crypto v0.3.0.
- helm.sh/helm/v3 v3.10.2.
- k8s.io/api v0.25.4.
- k8s.io/client-go v0.25.4.
- sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-17 11:31:45 +00:00
Stefan Prodan 65e1041492
Use Flux Event API v1beta1
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-11-09 11:06:23 +02:00
Soule BA 06a55590a5
Fix verification condition
Delete a failed verification condition at the beginning of the source
reconciliation and set `SourceVerifiedCondition` to false approprietly.

Set the `BuildOptions.Verify` to true as long as Verify is enabled in the
API fields.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 15:21:10 +02:00
Soule BA 0e97547eeb
implement Cosign verification for HelmCharts
If implemented, users will be able to enable chart verification for OCI
based helm charts.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:08 +02:00
Stefan Prodan 1931800661
Update dependencies
- `k8s.io/*` v0.25.3
- `helm.sh/helm/v3` v3.10.1
- `sigstore/cosign` v1.13.1
- `fluxcd/pkg/oci` v0.14.0
- `fluxcd/pkg/runtime` v0.22.0
- `golang.org/x/text` v0.4.0 (fix CVE-2022-32149)

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-10-20 18:11:36 +03:00
Sunny a6d7948667 Bucket: Add status.observedIgnore
Introduce status.observedIgnore in the Bucket API for consistency with
other sources with ignore.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny e996848555 GitRepo: Add observed content config in status
Replace content config checksum with explicit artifact content config
observations. It makes the observations of the controller more
transparent and easier to debug.

Introduces `observedIgnore`, `observedRecurseSubmodules` and
`observedInclude` status fields.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny 278a223bc6 OCIRepo: Add observed content config in status
Replace content config checksum with explicit artifact content config
observations. It makes the observations of the controller more
transparent and easier to debug.

Introduces `observedIgnore` and `observedLayerSelector` status fields.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny f4aed8baf8
OCIRepoReconciler: no-op reconcile improvements
Introduce contentConfigChecksum in the OCIRepository status to store a
checksum of the values that affect the source artifact. It is used to
detect when to rebuild an artifact when the spec changes.

The considerations for this are similar to the GitRepository
reconciler no-op clone implementation. Both reconcileSource and
reconcileArtifact need to consider the source configuration change
when deciding if the artifact in the storage is up-to-date.

Adds tests for reconcileSource and reconcileArtifact for the noop
cases.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-09-29 09:48:27 +03:00
Stefan Prodan 8614543e73
Update dependencies
- k8s.io/* v0.25.2
- helm.sh/helm/v3 v3.10.0
- sigs.k8s.io/controller-runtime v0.13.0
- cloud.google.com/go/storage v1.27.0
- fluxcd/pkg/runtime v0.19.0
- sigstore/sigstore v1.4.2
- github.com/fluxcd/git2go/v33 v33.0.9-flux (use Flux own fork)

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-28 14:45:19 +03:00
Stefan Prodan 4ec51ca306
Add option to copy the OCI layer to storage
Add on optional field to the `OCIRepository.spec.layerSelector` called `operation` that accepts one of the following values: `extract` or `copy`. When the operation is set to `copy`, instead of extracting the compressed layer, the controller copies the compressed blob as it is to storage, thus keeping the original content unaltered.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-23 19:00:30 +03:00
Stefan Prodan 07b532674c
Add omitempty to cosgin secretRef
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-20 14:07:11 +03:00
Furkan 697f260dba
Introduce Initial OCIRepository Source Verification
Fixes #863

Signed-off-by: Furkan <furkan.turkal@trendyol.com>
Co-authored-by: Batuhan <batuhan.apaydin@trendyol.com>
Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>
2022-09-20 14:07:10 +03:00
Hidde Beydals 27f4ed5a47 api: add custom validation for v1.Duration types
To solve discrepancies between parsing versus validation.

xref: https://github.com/kubernetes/apimachinery/issues/131

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-09-20 08:03:01 +00:00
Somtochi Onyekwere c38fafe128 Align controller logs to Kubernetes structured logging
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-08-31 14:24:40 +01:00
Stefan Prodan e1ad5a6fd3
Add `spec.insecure` to OCIRepository API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-31 11:10:25 +03:00
Stefan Prodan 6a367ec70b
Update Kubernetes packages to v1.25.0
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-26 12:26:38 +03:00
Soule BA ad3eb5ca47
Enable contextual login for helm OCI
If implemented, this pr will enable user to use the auto login feature
in order to automatically login to their provider of choice's container
registry (i.e. aws, gcr, acr).

Signed-off-by: Soule BA <soule@weave.works>
2022-08-25 22:27:35 +02:00
Stefan Prodan e5cb32b0f2
Add OCI layer selector to API docs
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-24 12:46:04 +03:00
Stefan Prodan 11dc0a3bc7
Select layer by OCI media type
Allow specifying the media type of the layer which should be extracted from the OCI artifact.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-23 12:25:18 +03:00
Stefan Prodan 1a59935858
Add OCI failure reasons to API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-05 13:24:06 +03:00
Stefan Prodan 8cc8798e6e
Add the provider field to the OCIRepository API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:28:50 +03:00
Stefan Prodan acc95d8c50
Add upstream source and revision to logs and events
Enrich the successful reconciliation event message with the upstream opencontainers annotations

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:11:15 +03:00
Stefan Prodan 05f9c0ee2b
Add the OCI metadata to the internal artifact
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:07 +03:00
Stefan Prodan 5072091eb5
Remove the default tag value from the CRD
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:07 +03:00
Stefan Prodan ada42eeaa7
Remove `spec.verify` from the API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:07 +03:00
Stefan Prodan ded0c2d78b
Add `oci://` prefix
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:06 +03:00
Stefan Prodan 07466730c0
Implement OCIRepository controller for public repos
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:06 +03:00
Stefan Prodan 46fe7a389c
Add OCIRepository kind to v1beta2 API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:05 +03:00
Philip Laine b47d468e8f
Upgrade to go 1.18
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2022-07-27 19:02:13 +03:00
Davin Kevin 889a505a4b fix(openapi): full regex for url to prevent error
In IDEA, the previous pattern led to an error because the regex wasn't complete.

Signed-off-by: Davin Kevin <davin.kevin@gmail.com>
2022-07-22 08:26:30 +00:00
Sunny f941f5ce24
Update go-yaml to v3.0.1
Fix CVE-2022-28948

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-06-30 00:21:59 +05:30
Paulo Gomes ea7027ac54
Update dependencies
- github.com/fluxcd/pkg/apis/meta to version 0.14.2.
- github.com/fluxcd/pkg/runtime to version 0.16.2.
- google.golang.org/api to version 0.83.0.
- k8s.io/api to version 0.24.1.
- github.com/fluxcd/pkg/apis/meta to version 0.14.2.
- k8s.io/apimachinery to version 0.24.1.
- github.com/fluxcd/pkg/helmtestserver to version 0.7.4.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-06-08 10:19:05 +01:00
Paulo Gomes b0c59d1fbb
Update dependencies
- fluxcd/pkg/apis/meta v0.14.1
- fluxcd/pkg/runtime v0.16.1
- gopkg.in/yaml.v3 v3.0.0

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-26 14:48:36 +01:00
Stefan Prodan f3a372da60
Update dependencies
- fluxcd/pkg/apis/meta v0.14.0
- fluxcd/pkg/runtime v0.16.0
- k8s.io/* v0.24.0
- helm.sh/helm/v3 v3.9.0-rc.1 (required by breaking changes in Kubernetes 1.24)

Note that fluxcd/pkg/runtime v0.16 comes with support for Kubernetes API Priority and Fairness feature.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-05-24 14:48:32 +03:00
Sunny 581695b4d6
gitrepo: Intro contentConfigChecksum & improvement
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.

Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.

Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.

The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Max Jonas Werner 841ed7ae66
[RFC 0002] Flux OCI support for Helm (#690)
* Add OCI Helm support

* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.

Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
2022-05-19 14:50:16 +02:00
Paulo Gomes 50cb97f331
Fix make verify failures
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-09 16:16:04 +01:00
Hidde Beydals a5e5fe13bd api: update dependencies
- github.com/fluxcd/pkg/apis/meta to v0.13.0
- k8s.io/apimachinery to v0.23.6

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-05-03 12:44:08 +02:00
Alexander Block 065a760752 docs: Remove all traces of "identity.pub" from docs
"identity.pub" is referenced multiple times in CRDs and docs. This secret
is however never used in any place. Instead, the public key is derived from
the "identity" private key.

This commit/PR removes all traces of "identity.pub" from v1beta2 CRDs and
docs.

Signed-off-by: Alexander Block <ablock84@gmail.com>
2022-04-29 21:17:53 +02:00
Paulo Gomes 009504b294 helm: optimise repository index loading
Avoid validating (and thus loading) indexes if the checksum already exists in storage.
In other words, if the YAML is identical to the Artifact in storage, the reconciliation should
be a no-op, and therefore can short-circuit long/heavy operations.

Co-authored-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-25 17:00:27 +02:00
Stefan Prodan 0f64fef2a1
Update dependencies
- helm.sh/helm/v3 v3.8.2
- k8s.io/api v0.23.5
- fluxcd/pkg/runtime v0.14.1

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-04-15 10:49:31 +03:00
Soule BA 0f9302827c
Add MIT Licence copyright notice
Signed-off-by: Soule BA <soule@weave.works>
2022-04-01 12:41:53 +02:00
Soule BA 366f5cfde8
Cache HelmRepository index files
If implemented, will provide users with a way to cache index files.

This addresses issues where the index file is loaded and unmarshalled in
concurrent reconciliation resulting in a heavy memory footprint.

The caching strategy used is cache aside, and the cache is a k/v store
with expiration.

The cache number of entries and ttl for entries are configurable.

The cache is optional and is disabled by default

Signed-off-by: Soule BA <soule@weave.works>
2022-04-01 12:41:52 +02:00
Sunny b41c717e16
controllers: emit event and log source up-to-date
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:06:16 +05:30
Sunny d939e98ec2
Introduce separate positive polarity conditions
Introduce separate positive polarity conditions which are used to set
Ready condition. Move the "artifact stored" ready condition into
ArtifactInStorage positive polarity condition. If ArtifactInStorage is
True and there's no negative polarity condition present, the Ready
condition is summarized with ArtifactInStorage condition value.

Also, update the priorities of the conditions. ArtifactInStorage has
higher priority than SourceVerfied condition. If both are present, the
Ready condition will have ArtifactInStorage.
The negative polarity conditions are reordered to have the most likely
actual cause of failure condition the highest priority, for example
StorageOperationFailed, followed by the conditions that are reconciled
first in the whole reconciliation so as to prioritize the first failure
which may be the cause of subsequent failures.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:02:59 +05:30
Hidde Beydals 6bf8dc5cca api/v1beta2: add note on Condition polarity
This was missing for `BuildFailedCondition` and
`StorageOperationFailedCondition`.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-17 13:13:45 +01:00
Hidde Beydals 93e15f25a2 api: update dependencies
- github.com/fluxcd/pkg/apis/meta to v0.12.1
- k8s.io/apimachinery to v0.23.4
- sigs.k8s.io/controller-runtime to v0.11.1

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-16 11:15:45 +01:00