Commit Graph

465 Commits

Author SHA1 Message Date
Sunny 7b44c9db0d Add progressive status in helmrepo-oci reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 55573f5eb6 Add progressive status in ocirepo reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 9c866ee49d Add progressive status in helmchart reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 3d6a5e1203 Add progressive status in helmrepo reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny ccf0b624a7 Add progressive status in bucket reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 887b5309bf Add progressive status in gitrepository reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny b115dda217 Use filepath instead of path in storage tests
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-21 05:25:49 +05:30
Sunny 6134dd97d5 test: Add tests for GC ignoring lock files
Add storage tests to ensure garbage collection ignores lock files for GC
count and deletes them eventually.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-21 05:25:45 +05:30
Sanskar Jaiswal bdd08bcb72 storage: take lock files into consideration while garbage collecting
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-12-20 16:28:34 +05:30
Sunny d551e59a06 Use Event v1 API metadata keys in notifications
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-19 20:21:49 +05:30
Soule BA 6b5d0b1335
Fix aliased chart dependencies resolution
If implemented, this fix the issue were aliased chart dependencies were
detected but not included in the final packaged chart.

Signed-off-by: Soule BA <soule@weave.works>
2022-12-16 23:12:27 +01:00
Sunny 8126c99a76 GitRepo: git impl. deprecation test cleanup
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-13 17:00:09 +05:30
Paulo Gomes 5ffa6a5a68
libgit2: Remove references to libgit2 from code
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-12-12 15:34:28 +00:00
Paulo Gomes 04dd4b54fb
build: Migrate to Go Native fuzz and improve reliability
Establish conventions which aligns with what is supported upstream
today, whilst expanding on documentation to ensure folks have
pointers on how to debug/check for issues going forwards.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-24 09:56:38 +00:00
Paulo Gomes 22e8b0ff7e
Update dependencies
Given that pkg/* now depends on fluxcd/go-git, this changes also apply
the same changes here.

New versions:
- github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0.
- github.com/distribution/distribution/v3 v3.0.0-20221111170714-3b8fbf975279.
- github.com/fluxcd/pkg/apis/event v0.2.0.
- github.com/fluxcd/pkg/helmtestserver v0.10.0.
- github.com/fluxcd/pkg/oci v0.15.0.
- github.com/fluxcd/pkg/runtime v0.24.0.
- github.com/fluxcd/pkg/sourceignore v0.3.0.
- github.com/google/go-containerregistry v0.12.1.
- github.com/google/go-containerregistry/pkg/authn/k8schain v0.0.0-20221114162634-781782aa2757.
- golang.org/x/crypto v0.3.0.
- helm.sh/helm/v3 v3.10.2.
- k8s.io/api v0.25.4.
- k8s.io/client-go v0.25.4.
- sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-17 11:31:45 +00:00
Sanskar Jaiswal b98b9a92b6 gitrepo: bump git packages
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-11-16 19:34:11 +05:30
Paulo Gomes bdcf708ef8
git: Replace Stalling error for git implementation
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-11 10:25:01 +00:00
Paulo Gomes 331fd64952
gogit: Add new ForceGoGitImplementation FeatureGate
ForceGoGitImplementation ignores the value set for gitImplementation
and ensures that go-git is used for all GitRepository objects.
This can be used to confirm that Flux instances won't break if/when
the libgit2 implementation was to be deprecated.

When enabled, libgit2 won't be initialized, nor will any git2go cgo
code be called.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-11 10:25:00 +00:00
Paulo Gomes 39e999d617
git: Load default feature gates
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-10 13:46:51 +00:00
Stefan Prodan 65e1041492
Use Flux Event API v1beta1
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-11-09 11:06:23 +02:00
Hidde Beydals 206fb838e0 Update dependencies
- cloud.google.com/go/storage to v1.28.0
- github.com/distribution/distribution/v3 to v3.0.0-20221108081720-e9a25da7a47e
- github.com/docker/cli to v20.10.21+incompatible
- github.com/google/go-containerregistry to v0.12.0
- github.com/google/go-containerregistry/pkg/authn/k8schain to v0.0.0-20221103173901-353a117661ae
- github.com/minio/minio-go/v7 to v7.0.43
- github.com/onsi/gomega to v1.24.0
- github.com/prometheus/client_golang to v1.14.0
- github.com/sigstore/sigstore to v1.4.5
- google.golang.org/api to v0.102.0
- k8s.io/utils to v0.0.0-20221107191617-1a15be271d1d
- sigs.k8s.io/cli-utils to v0.34.0
- sigs.k8s.io/controller-runtime to v0.13.1

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-11-08 14:16:00 +00:00
Paulo Gomes e87997c117
Use serror.NewStalling() and small PR fixes
This ensures that the event, notification and log
are configured correctly.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-10-31 10:16:00 +00:00
Sanskar Jaiswal 22e712b935
gitrepo: match against api types when creating git client
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-10-26 14:06:15 +01:00
Paulo Gomes d25ed0c8c4
build: Bump golang-with-libgit2 to v0.4.0
The new version uses libgit2 1.5.0 and requires git2go/v34.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-10-26 14:06:14 +01:00
Sanskar Jaiswal b6d6b593c8
gitrepo: refactor reconciler to use fluxcd/pkg/git
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-10-26 14:04:00 +01:00
Soule BA 06a55590a5
Fix verification condition
Delete a failed verification condition at the beginning of the source
reconciliation and set `SourceVerifiedCondition` to false approprietly.

Set the `BuildOptions.Verify` to true as long as Verify is enabled in the
API fields.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 15:21:10 +02:00
Soule BA 25673ac512
addressing review comments
Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:08 +02:00
Soule BA 0e97547eeb
implement Cosign verification for HelmCharts
If implemented, users will be able to enable chart verification for OCI
based helm charts.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:08 +02:00
Soule BA 55dd799dad
Remove test case on aws
This remove test case for contextual login on
oci://123456789000.dkr.ecr.us-east-2.amazonaws.com.

This is not longer a wrong url since f7c66eb06a
and we no longer error on nil auth.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:07 +02:00
Soule BA bb83270acc
Refactor to use authn for authentication as OCIrepository does
If implemented the oras registry loginOption will only be used internaly
with the specific ChartRepo struct.

This will permit reusing more easily feature developped with
googlecontainerregistry authn.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:07 +02:00
Sunny 15cdd85805 controllers: Allow deletion of suspended objects
Reorders the object suspended check in all the reconcilers to allow
deletion of objects when they are suspended. Objects used to get stuck
on delete because the finalizers were not getting removed due to the
suspended state.

Adds a generic test for all the reconcilers to check if a suspended
source object can be delete.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-20 17:22:11 +05:30
Sunny df1894b7ed
update dependencies
- replace github.com/darkowlzz/controller-check/status ->
  github.com/fluxcd/pkg/runtime/conditions/check
- github.com/fluxcd/pkg/runtime - v0.21.0
- github.com/fluxcd/pkg/oci - v0.13.0
- github.com/fluxcd/pkg/apis/meta - v0.17.0
- k8s.io/cli-runtime - v0.25.2

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-11 19:08:44 +05:30
Sunny a6d7948667 Bucket: Add status.observedIgnore
Introduce status.observedIgnore in the Bucket API for consistency with
other sources with ignore.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny e996848555 GitRepo: Add observed content config in status
Replace content config checksum with explicit artifact content config
observations. It makes the observations of the controller more
transparent and easier to debug.

Introduces `observedIgnore`, `observedRecurseSubmodules` and
`observedInclude` status fields.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny 278a223bc6 OCIRepo: Add observed content config in status
Replace content config checksum with explicit artifact content config
observations. It makes the observations of the controller more
transparent and easier to debug.

Introduces `observedIgnore` and `observedLayerSelector` status fields.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Soule BA f51c98ecad
Fail when verifying with insecure
If implemented we fails when trying to verify with insecure set. This
will likely change once cosign add support for insecure registries.

Signed-off-by: Soule BA <soule@weave.works>
2022-09-29 14:45:25 +02:00
Soule BA e2f4e5e7c5
Accept a slice of remote.Option for cosign verification
If implemented this enable passing a keychain, an authenticator and a
custom transport as remote.Option to the verifier. It enables contextual
login, self-signed certificates and insecure registries.

Signed-off-by: Soule BA <soule@weave.works>

refactor makeOptions

Reduce complexity by replacing the functional options with a flat out
conditional logic in makeOptions.

Signed-off-by: Soule BA <soule@weave.works>
2022-09-29 14:45:25 +02:00
Sunny dcd0db406e
OCIRepo: Implement source ignore
This implements source ignore in OCIRepositoryReconcilers'
reconcileArtifact so that the ignore rules are considered when building
the artifact.

Adds tests based on the artifact checksum change when ignore rules are
applied.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-09-29 09:48:32 +03:00
Sunny f4aed8baf8
OCIRepoReconciler: no-op reconcile improvements
Introduce contentConfigChecksum in the OCIRepository status to store a
checksum of the values that affect the source artifact. It is used to
detect when to rebuild an artifact when the spec changes.

The considerations for this are similar to the GitRepository
reconciler no-op clone implementation. Both reconcileSource and
reconcileArtifact need to consider the source configuration change
when deciding if the artifact in the storage is up-to-date.

Adds tests for reconcileSource and reconcileArtifact for the noop
cases.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-09-29 09:48:27 +03:00
Stefan Prodan 3f7d4630cc
Use the OCI artifact revision in status and events
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-26 13:30:28 +03:00
Stefan Prodan aae9d917fb
Optimise OCI artifacts reconciliation
- Fetch the upstream digest before validation and pulling
- Pull artifact only if the upstream digest is different from the one in storage
- Add the image tag to the revision string `<tag>/<digest-hex>` for a better UX
- Extract the layer processing to a dedicated function

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-24 10:30:46 +03:00
Stefan Prodan 4ec51ca306
Add option to copy the OCI layer to storage
Add on optional field to the `OCIRepository.spec.layerSelector` called `operation` that accepts one of the following values: `extract` or `copy`. When the operation is set to `copy`, instead of extracting the compressed layer, the controller copies the compressed blob as it is to storage, thus keeping the original content unaltered.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-23 19:00:30 +03:00
Max Jonas Werner 7a139647a2
add custom CA certificates to system certificates
When a custom CA certificate is provided in a Secret's `caCert` field
referenced in `HelmRelease.spec.secretRef` then that CA cert is now
added to the list of system certificates instead of it replacing the
system certificates. This makes HelmRepositories work in mixed
environments where charts are pulled from both, a public repository
and a private repository (e.g. through a chart dependency).

The test that is added as part of this change will fail without the
change and passes with it.

closes #866
closes fluxcd/helm-controller#519

Signed-off-by: Max Jonas Werner <max@e13.dev>
2022-09-23 10:57:00 +03:00
Sunny 90b7cec915 ocirepo: Fix event trace type value
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-09-22 15:42:53 +05:30
Sunny 64bd34f116 Use bipolarity option in gitrepo and ocirepo
Use the bipolarity condition options in OCIRepository and GitRepository
reconcilers.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-09-22 15:42:53 +05:30
Stefan Prodan 3b637a82fe
Add tests for keyless verification
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-22 12:25:40 +03:00
Stefan Prodan 7c92949d17
Retry failed verifications with exponential backoff
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-20 21:26:34 +03:00
Stefan Prodan 082028e115
Refactor internal OCI package
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-20 18:34:02 +03:00
Furkan 03af4f6418
fix: ocirepository_controller reviews
Signed-off-by: Furkan <furkan.turkal@trendyol.com>
2022-09-20 14:07:11 +03:00
Batuhan Apaydın 2db2715988
feat: add condition tests for verification logic
Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>
2022-09-20 14:07:11 +03:00