Commit Graph

486 Commits

Author SHA1 Message Date
Sanskar Jaiswal c3511cc834 add support for checking out to Git refs
Add a new field `.spec.ref.name` which points to a Git reference which
enables checking out to a particular commit pointed to by the specified
reference.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2023-02-16 17:33:11 +05:30
Somtochi Onyekwere 6f0384c50e Normalize path in url
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2023-02-15 03:54:18 +01:00
Max Jonas Werner bfa61d9ed0 Apply default permission mode to all files/dirs in an artifact archive
Files: 0644
Directories: 0755

closes #1019

Signed-off-by: Max Jonas Werner <max@e13.dev>
2023-02-14 14:44:25 +01:00
Hidde Beydals 9283894bbe Use MetaDigestKey from event API
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals f53bfd1dc1 Use Artifact.Path for HelmRepository index cache
Resolving it to a local path does not make it more unique, while
resulting in longer keys and a lot of safejoin calls.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals d62f4dc0c6 misc: order imports and align digest aliases
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals 0aaeeee5e9 controllers: RFC-0005 fmt for HelmRepository rev
This includes changes to the `ChartRepository`, to allow calculating
the revision and digest and tidy things.

In addition, the responsibility of caching the `IndexFile` has been
moved to the reconcilers. As this allowed to remove a lot of
complexities within the `ChartRepository`, and prevented passing on
the cache in general.

Change `HelmRepository`'s Revision to digest

Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals f00aeae09a controllers: use TransformLegacyRevision helper
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals 469c9387ee controllers: make OCIRepository compat with RFC-0005
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals b89b049bf1 controllers: make HelmChart compat with RFC-0005
This ensures the revision is correctly parsed for `Bucket` and
`GitRepository` sources from which a chart is built, either in the
legacy or new RFC-0005 format.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals fe8bc43f84 controllers: use RFC-0005 format for Git revision
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals 83b6fdcdd8 controllers: use digest for Bucket revision
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals a72badf16b reconcilers: include artifact digest in event meta
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Hidde Beydals 6e0a6f11d4 storage: calculate `Digest` for `Artifact`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-14 12:48:36 +01:00
Sunny 42bc3e8b0a helmrepo-oci: check before rec on type switching
When a HelmRepository with "default" spec.type is switched to "oci", the
existing HelmRepository is processed by HelmRepositoryReconciler by
running reconcileDelete() which removes all the previous status
information and allows the HelmRepositoryOCIReconciler to process the
object and add its own status data. But at times, when
HelmRepositoryOCIReconciler starts processing a HelmRepository with
stale status data from the client cache, it contains the stale
conditions that are owned only by HelmRepositoryReconciler and isn't
managed by HelmRepositoryOCIReconciler. This results in situations where
Ready is marked as True with the latest generation of the object and the
unmanaged stale conditions remain in the previous generation, resulting
in unexpected status conditions.

In the observed flaky tests,
`TestHelmRepositoryReconciler_ReconcileTypeUpdatePredicateFilter` would
fail because of stale ArtifactInStorage condition with previous
generation value.

This change adds a check in the HelmRepositoryOCIReconciler to start
processing the object only once the stale unmanaged conditions have been
removed.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-02-07 20:11:49 +05:30
Sunny 75cde08ff0 Use condition checker with gomega WithT
This allows using the condition checker as a test helper with proper
test like assertion failure and stacktrace.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-02-07 20:11:46 +05:30
Aurel Canciu 14a4a5eed6
Prevent panic when cloning empty git repository
This covers the edge case in which a user creates a GitRepository CR
referencing an empty Git repository. Currently, the controller will panic
in this situation since the returned commit pointer is nil.

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2023-02-07 13:53:31 +01:00
Stefan Prodan f89d07579f
Update dependencies
- k8s.io/* v0.26.1
- helm.sh/helm/v3 v3.11.0
- github.com/sigstore/sigstore v1.5.1
- github.com/google/go-containerregistry v0.13.0
- github.com/fluxcd/pkg/oci v0.18.0
- github.com/fluxcd/pkg/runtime v0.27.0
- cloud.google.com/go/storage v1.29.0
- github.com/Azure/azure-sdk-for-go/sdk/azcore v1.3.0
- sigs.k8s.io/controller-runtime v0.14.1

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2023-01-27 14:03:04 +02:00
Sunny f707193e90 Update git dependencies for bearer token support
Also update API spec to mention bearer token field in git secret.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-23 21:14:07 +05:30
Sunny e0cf840d02 Set rate limiter option in test reconcilers
Set the default rate limiter configuration used in main.go in the
test reconcilers as well.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-18 16:10:30 +05:30
Sunny baaa412779 Fix ocirepo testdata permissions
On Ubuntu, and maybe some others, the
`TestOCIRepository_reconcileArtifact` test fails due to difference in
file permission, which results in different artifact checksum. This is
due to the default umask on ubuntu. Reset the permission of the testdata
to fix the test on ubuntu.
There's a similar fix in `TestGitRepositoryReconciler_reconcileArtifact`
test.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-18 02:59:15 +05:30
Sunny 7b44c9db0d Add progressive status in helmrepo-oci reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 55573f5eb6 Add progressive status in ocirepo reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 9c866ee49d Add progressive status in helmchart reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 3d6a5e1203 Add progressive status in helmrepo reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny ccf0b624a7 Add progressive status in bucket reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny 887b5309bf Add progressive status in gitrepository reconciler
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-10 00:30:40 +05:30
Sunny b115dda217 Use filepath instead of path in storage tests
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-21 05:25:49 +05:30
Sunny 6134dd97d5 test: Add tests for GC ignoring lock files
Add storage tests to ensure garbage collection ignores lock files for GC
count and deletes them eventually.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-21 05:25:45 +05:30
Sanskar Jaiswal bdd08bcb72 storage: take lock files into consideration while garbage collecting
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-12-20 16:28:34 +05:30
Sunny d551e59a06 Use Event v1 API metadata keys in notifications
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-19 20:21:49 +05:30
Soule BA 6b5d0b1335
Fix aliased chart dependencies resolution
If implemented, this fix the issue were aliased chart dependencies were
detected but not included in the final packaged chart.

Signed-off-by: Soule BA <soule@weave.works>
2022-12-16 23:12:27 +01:00
Sunny 8126c99a76 GitRepo: git impl. deprecation test cleanup
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-12-13 17:00:09 +05:30
Paulo Gomes 5ffa6a5a68
libgit2: Remove references to libgit2 from code
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-12-12 15:34:28 +00:00
Paulo Gomes 04dd4b54fb
build: Migrate to Go Native fuzz and improve reliability
Establish conventions which aligns with what is supported upstream
today, whilst expanding on documentation to ensure folks have
pointers on how to debug/check for issues going forwards.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-24 09:56:38 +00:00
Paulo Gomes 22e8b0ff7e
Update dependencies
Given that pkg/* now depends on fluxcd/go-git, this changes also apply
the same changes here.

New versions:
- github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0.
- github.com/distribution/distribution/v3 v3.0.0-20221111170714-3b8fbf975279.
- github.com/fluxcd/pkg/apis/event v0.2.0.
- github.com/fluxcd/pkg/helmtestserver v0.10.0.
- github.com/fluxcd/pkg/oci v0.15.0.
- github.com/fluxcd/pkg/runtime v0.24.0.
- github.com/fluxcd/pkg/sourceignore v0.3.0.
- github.com/google/go-containerregistry v0.12.1.
- github.com/google/go-containerregistry/pkg/authn/k8schain v0.0.0-20221114162634-781782aa2757.
- golang.org/x/crypto v0.3.0.
- helm.sh/helm/v3 v3.10.2.
- k8s.io/api v0.25.4.
- k8s.io/client-go v0.25.4.
- sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-17 11:31:45 +00:00
Sanskar Jaiswal b98b9a92b6 gitrepo: bump git packages
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-11-16 19:34:11 +05:30
Paulo Gomes bdcf708ef8
git: Replace Stalling error for git implementation
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-11 10:25:01 +00:00
Paulo Gomes 331fd64952
gogit: Add new ForceGoGitImplementation FeatureGate
ForceGoGitImplementation ignores the value set for gitImplementation
and ensures that go-git is used for all GitRepository objects.
This can be used to confirm that Flux instances won't break if/when
the libgit2 implementation was to be deprecated.

When enabled, libgit2 won't be initialized, nor will any git2go cgo
code be called.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-11 10:25:00 +00:00
Paulo Gomes 39e999d617
git: Load default feature gates
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-11-10 13:46:51 +00:00
Stefan Prodan 65e1041492
Use Flux Event API v1beta1
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-11-09 11:06:23 +02:00
Hidde Beydals 206fb838e0 Update dependencies
- cloud.google.com/go/storage to v1.28.0
- github.com/distribution/distribution/v3 to v3.0.0-20221108081720-e9a25da7a47e
- github.com/docker/cli to v20.10.21+incompatible
- github.com/google/go-containerregistry to v0.12.0
- github.com/google/go-containerregistry/pkg/authn/k8schain to v0.0.0-20221103173901-353a117661ae
- github.com/minio/minio-go/v7 to v7.0.43
- github.com/onsi/gomega to v1.24.0
- github.com/prometheus/client_golang to v1.14.0
- github.com/sigstore/sigstore to v1.4.5
- google.golang.org/api to v0.102.0
- k8s.io/utils to v0.0.0-20221107191617-1a15be271d1d
- sigs.k8s.io/cli-utils to v0.34.0
- sigs.k8s.io/controller-runtime to v0.13.1

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-11-08 14:16:00 +00:00
Paulo Gomes e87997c117
Use serror.NewStalling() and small PR fixes
This ensures that the event, notification and log
are configured correctly.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-10-31 10:16:00 +00:00
Sanskar Jaiswal 22e712b935
gitrepo: match against api types when creating git client
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-10-26 14:06:15 +01:00
Paulo Gomes d25ed0c8c4
build: Bump golang-with-libgit2 to v0.4.0
The new version uses libgit2 1.5.0 and requires git2go/v34.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-10-26 14:06:14 +01:00
Sanskar Jaiswal b6d6b593c8
gitrepo: refactor reconciler to use fluxcd/pkg/git
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-10-26 14:04:00 +01:00
Soule BA 06a55590a5
Fix verification condition
Delete a failed verification condition at the beginning of the source
reconciliation and set `SourceVerifiedCondition` to false approprietly.

Set the `BuildOptions.Verify` to true as long as Verify is enabled in the
API fields.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 15:21:10 +02:00
Soule BA 25673ac512
addressing review comments
Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:08 +02:00
Soule BA 0e97547eeb
implement Cosign verification for HelmCharts
If implemented, users will be able to enable chart verification for OCI
based helm charts.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:08 +02:00
Soule BA 55dd799dad
Remove test case on aws
This remove test case for contextual login on
oci://123456789000.dkr.ecr.us-east-2.amazonaws.com.

This is not longer a wrong url since f7c66eb06a
and we no longer error on nil auth.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:07 +02:00