Commit Graph

68 Commits

Author SHA1 Message Date
Sanskar Jaiswal c3511cc834 add support for checking out to Git refs
Add a new field `.spec.ref.name` which points to a Git reference which
enables checking out to a particular commit pointed to by the specified
reference.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2023-02-16 17:33:11 +05:30
Hidde Beydals 964b2d3f00 api: introduce `Digest` field to `Artifact`
As discussed in RFC-0005, this introduces a `Digest` field to the
`Artifact` in favor of the now deprecated `Checksum`.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2023-02-10 20:30:59 +01:00
Sunny f707193e90 Update git dependencies for bearer token support
Also update API spec to mention bearer token field in git secret.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2023-01-23 21:14:07 +05:30
Yohan Belléguic 2741d0a150 fix typo in helmRepo secretRef spec CRD
When using a TLS authentication, user can provide a custom certificate
by setting the caFile key in the secret, not caCert.

Signed-off-by: Yohan Belléguic <yohan.belleguic@arkea.com>
2023-01-16 19:52:06 +05:30
Paulo Gomes 5ffa6a5a68
libgit2: Remove references to libgit2 from code
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-12-12 15:34:28 +00:00
Soule BA 06a55590a5
Fix verification condition
Delete a failed verification condition at the beginning of the source
reconciliation and set `SourceVerifiedCondition` to false approprietly.

Set the `BuildOptions.Verify` to true as long as Verify is enabled in the
API fields.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 15:21:10 +02:00
Soule BA 0e97547eeb
implement Cosign verification for HelmCharts
If implemented, users will be able to enable chart verification for OCI
based helm charts.

Signed-off-by: Soule BA <soule@weave.works>
2022-10-21 10:00:08 +02:00
Sunny a6d7948667 Bucket: Add status.observedIgnore
Introduce status.observedIgnore in the Bucket API for consistency with
other sources with ignore.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny e996848555 GitRepo: Add observed content config in status
Replace content config checksum with explicit artifact content config
observations. It makes the observations of the controller more
transparent and easier to debug.

Introduces `observedIgnore`, `observedRecurseSubmodules` and
`observedInclude` status fields.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny 278a223bc6 OCIRepo: Add observed content config in status
Replace content config checksum with explicit artifact content config
observations. It makes the observations of the controller more
transparent and easier to debug.

Introduces `observedIgnore` and `observedLayerSelector` status fields.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-10-10 23:06:02 +05:30
Sunny f4aed8baf8
OCIRepoReconciler: no-op reconcile improvements
Introduce contentConfigChecksum in the OCIRepository status to store a
checksum of the values that affect the source artifact. It is used to
detect when to rebuild an artifact when the spec changes.

The considerations for this are similar to the GitRepository
reconciler no-op clone implementation. Both reconcileSource and
reconcileArtifact need to consider the source configuration change
when deciding if the artifact in the storage is up-to-date.

Adds tests for reconcileSource and reconcileArtifact for the noop
cases.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-09-29 09:48:27 +03:00
Stefan Prodan 4ec51ca306
Add option to copy the OCI layer to storage
Add on optional field to the `OCIRepository.spec.layerSelector` called `operation` that accepts one of the following values: `extract` or `copy`. When the operation is set to `copy`, instead of extracting the compressed layer, the controller copies the compressed blob as it is to storage, thus keeping the original content unaltered.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-09-23 19:00:30 +03:00
Furkan 697f260dba
Introduce Initial OCIRepository Source Verification
Fixes #863

Signed-off-by: Furkan <furkan.turkal@trendyol.com>
Co-authored-by: Batuhan <batuhan.apaydin@trendyol.com>
Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>
2022-09-20 14:07:10 +03:00
Stefan Prodan e1ad5a6fd3
Add `spec.insecure` to OCIRepository API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-31 11:10:25 +03:00
Soule BA ad3eb5ca47
Enable contextual login for helm OCI
If implemented, this pr will enable user to use the auto login feature
in order to automatically login to their provider of choice's container
registry (i.e. aws, gcr, acr).

Signed-off-by: Soule BA <soule@weave.works>
2022-08-25 22:27:35 +02:00
Stefan Prodan e5cb32b0f2
Add OCI layer selector to API docs
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-24 12:46:04 +03:00
Stefan Prodan 11dc0a3bc7
Select layer by OCI media type
Allow specifying the media type of the layer which should be extracted from the OCI artifact.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-23 12:25:18 +03:00
Stefan Prodan 8cc8798e6e
Add the provider field to the OCIRepository API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:28:50 +03:00
Stefan Prodan 05f9c0ee2b
Add the OCI metadata to the internal artifact
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:07 +03:00
Stefan Prodan ada42eeaa7
Remove `spec.verify` from the API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:07 +03:00
Stefan Prodan 46fe7a389c
Add OCIRepository kind to v1beta2 API
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-08-02 13:07:05 +03:00
Sunny 581695b4d6
gitrepo: Intro contentConfigChecksum & improvement
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.

Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.

Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.

The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Max Jonas Werner 841ed7ae66
[RFC 0002] Flux OCI support for Helm (#690)
* Add OCI Helm support

* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.

Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
2022-05-19 14:50:16 +02:00
Alexander Block 065a760752 docs: Remove all traces of "identity.pub" from docs
"identity.pub" is referenced multiple times in CRDs and docs. This secret
is however never used in any place. Instead, the public key is derived from
the "identity" private key.

This commit/PR removes all traces of "identity.pub" from v1beta2 CRDs and
docs.

Signed-off-by: Alexander Block <ablock84@gmail.com>
2022-04-29 21:17:53 +02:00
Max Jonas Werner 1e83f0ecf5 fix API docs link
v0.3.0 of gen-crd-api-reference-docs has a bug that leads to it
rendering incorrect links for local types. This is fixed on the master
branch but not released, yet, so I'm pinning the version of the tool
to the latest current commit.

Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-04-29 19:27:53 +02:00
Sunny f59b48c339 Document HelmChart API v1beta2 spec
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-11 10:04:14 +01:00
Hidde Beydals cbffd82d8f Document GitRepository API v1beta2 spec
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Hidde Beydals 86d1d80bf2 Document HelmRepository API v1beta2 spec
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Hidde Beydals 5832296ef5 Document Bucket API v1beta2 spec
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Kevin McDermott f7105ea736 Implement Size field on archived artifacts
This adds a Size field to Artifacts, which reflects the number of bytes
written to the artifact when it's being archived.

Signed-off-by: Kevin McDermott <bigkevmcd@gmail.com>
2022-02-28 16:01:34 +01:00
Sunny 45df2d76c8 Update API descriptions and messages to be consistent
- Update v1beta2 API descriptions and reconciling messages to be
  consistent.
- Replace 'download' with 'fetch'. Since the status condition for
  download failure is called FetchFailed, using the term 'fetch' makes
  the messaging more consistent.
- Replace `BucketOperationSucceed` with `BucketOperationSucceeded` and
  generate api docs.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals 349739b7e4 Implement new runtime interfaces, prepare testenv
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.

Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny cc1e48243d Introduce v1beta2 API package
This commit introduces a v1beta2 API package for the staged breaking
changes around conditions and general usage of the API objects.

Signed-off-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Co-authored-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:34:35 +01:00
Paulo Gomes fcb62eedc7
Validate LIBGIT2 args are set correctly
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-09 15:52:46 +00:00
Paulo Gomes b63780505e
Add api-docs and manifests to make verify
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-01-13 10:16:13 +00:00
Stefan Prodan 058a016a60 Add ACL optional field to Source API
Cherry-picked from 525be388ec.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-11-22 11:41:55 +01:00
Hidde Beydals fb688ffe8a
storage: change Artifact checksum to SHA256
This changes the format of the Artifact checksum from SHA1 to SHA256 to
mitigate chosen-prefix and length extension attacks, and ensures it can
be used to secure content against malicious modifications.

Source consumers (including our own {kustomize,helm}-controllers)
should ensure the SHA256 of a downloaded artifact matches the
advertised checksum before making use of it.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-11 22:47:49 +01:00
Hidde Beydals c2495ae408 Fix generation of API documentation
The version was accidentally set to an invalid version, causing the
API documentation generation to fail.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-14 13:49:52 -05:00
Dylan Arbour 5e6abae9e8 Add ReconcileStrategy to HelmChart
This commit adds a `ReconcileStrategy` field to the `HelmChart` resource, which
allows defining when a new chart should be packaged and/or published if it
originates from a `Bucket` or `GitRepository` resource.

The two available strategies are:

- `ChartVersion`: creates a new artifact when the version of the Helm chart as
  defined in the `Chart.yaml` from the Source is different from the current
  version.
- `Revision`: creates a new artifact when the revision of the Source is
  different from the current revision.

For the `Revision` strategy, the (checksum part of the) revision of the
artifact the chart originatesfrom is added as SemVer metadata.

A chart from a `GitRepository` with Artifact revision
`main/f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer
`0.1.0` will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.

A chart from a `Bucket` with Artifact revision
`f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer `0.1.0`
will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.

Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
2021-10-14 13:49:52 -05:00
pa250194 0b971511ad Revert change to doc/api/source.md
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 12:30:58 -05:00
pa250194 a5588fb191 Added Comments for reconcileWithGCP and reconcileWithMinio
Signed-off-by: pa250194 <pa250194@ncr.com>
2021-09-16 09:49:56 -05:00
Hidde Beydals 1f27410b34 Update Helm to v3.6.1
v3.6.1 is a a security update from Helm, ensuring that credentials are
always only passed to the defined repository host.

Based on Helm user reports, disabling this behavior may be required for
some Helm repository solutions like Artifactory, and may be done by
setting `PassCredentials` in the `HelmRepositorySpec`.

For more information, see:
https://github.com/helm/helm/security/advisories/GHSA-56hp-xqp3-w2jf

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-06-18 12:31:23 +02:00
Philip Laine fcf7048992 Add include property to GitRepositories
Signed-off-by: Philip Laine <philip.laine@gmail.com>
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2021-05-11 09:46:50 +02:00
Dylan Arbour 4a834e1d2d Add `ValuesFiles` to HelmChart spec
Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
2021-04-19 09:16:53 -04:00
Stefan Prodan 664a568822
Add support for Git submodules with go-git
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-03-30 13:00:13 +03:00
Stefan Prodan 207ed99d72
Use LocalObjectReference from fluxcd/pkg/meta v0.7
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-01-21 14:18:42 +02:00
Philip Laine f044ebc253 Rename git2go option to libgit2
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-06 23:19:57 +01:00
Philip Laine 1b4eacc588 Refactor argument name to enable git2go
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-04 10:07:17 +01:00
Philip Laine 4af072aec0 Generate docs
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-02 20:18:29 +01:00
Philip Laine 2cb3754953 Add support for gitv2 servers
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-02 20:18:29 +01:00