Commit Graph

145 Commits

Author SHA1 Message Date
Max Jonas Werner ce072c7eda
better variable names; improved logging
When setup of one of the two controller reconciling HelmRepositories
fails, it's now possible to judge from the log which setup call failed
by regarding the "type" log field.

Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:31:37 +02:00
Max Jonas Werner bb4d886ba2
dockerconfigjson for OCI registry authentication
`loginOptionFromSecret` now derives username/password from a docker
config stored in Secrets of type "kubernetes.io/dockerconfigjson".

Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:31:37 +02:00
Max Jonas Werner 841ed7ae66
[RFC 0002] Flux OCI support for Helm (#690)
* Add OCI Helm support

* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.

Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
2022-05-19 14:50:16 +02:00
Soule BA 0df8dcccec
Enable dependency manager to use in memory cache
If implemented this will:
- enable the helmCharts dependency manager to use the helm in memry
  cache to retrieve reconciled HelmRepositories indexes.
- record cache events.

Signed-off-by: Soule BA <soule@weave.works>
2022-04-19 11:46:27 +02:00
Peter Gundel 0949d0adb9
Change all file permissions to octal format
This better represent permissions as Linux handles such information in
octal format, meaning that the left-most 0 has an important meaning
and is not to be ignored as normally integers would.

See https://github.com/fluxcd/source-controller/issues/603

Signed-off-by: Peter Gundel <mail@petergundel.de>
2022-04-12 21:24:34 +02:00
Paulo Gomes 4198191759
Add flags to configure exponential back-off retry
Add two new flags to enable users to configure exponential
back-off for Flux objects. The default values are now
set to 750ms for minimum retry time, and 15min for max.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-12 10:44:46 +01:00
Hidde Beydals e85ea781e2 helm: switch to our own chart loader package
This includes some rewiring of tests, and slight changes in how we work
with the local chart reference. `Path` is expected to be relative to
`WorkDir`, and both fields are now mandatory.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 11:47:28 +02:00
Sanskar Jaiswal 72a4982541 remove leftover timeout in reconcilers
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-04-07 22:33:07 +05:30
Sunny 5da74ca5a9
Add notify() in all the reconcilers
notify() is used to emit events for new artifact and failure recovery
scenarios. It's implemented in all the reconcilers.
Previously, when there used to be a failure due to any reason, on a
subsequent successful reconciliation, no notification was sent to
indicate that the failure has been resolved.
With notify(), the old version of the object is compared with the new
version of the object to determine if all, if any, of the failures have
been resolved and a notification is sent. The notification message is
the same that's sent in usual successful source reconciliation message
about stored artifact.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-04-07 21:39:16 +05:30
Sunny 44207f46d5
Avoid event logging GC failure
We try to avoid affecting the source reconciliation when there's a
garbage collection related failure.

The event logging was resulting in events and notifications related to
GC failure when the artifact directory isn't created in the first
reconciliation of an object.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-04-07 21:14:12 +05:30
Sanskar Jaiswal f8c27a85dd Garbage collect with provided retention options.
Introduce two new flags to configure the ttl of an artifact and the max
no. of files to retain for an artifact. Modify the gc process to
consider the options and use timeouts to prevent the controller from
hanging.
This helps in situations when the SC has already garbage collected the
current artifact but the advertised artifact url is still the same,
which leads to the server returning a 404.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-04-07 18:43:55 +05:30
Soule BA 7ff96a8b0c
Update the docs with a section regarding the cache usage
Signed-off-by: Soule BA <soule@weave.works>
2022-04-02 18:13:59 +02:00
Soule BA 366f5cfde8
Cache HelmRepository index files
If implemented, will provide users with a way to cache index files.

This addresses issues where the index file is loaded and unmarshalled in
concurrent reconciliation resulting in a heavy memory footprint.

The caching strategy used is cache aside, and the cache is a k/v store
with expiration.

The cache number of entries and ttl for entries are configurable.

The cache is optional and is disabled by default

Signed-off-by: Soule BA <soule@weave.works>
2022-04-01 12:41:52 +02:00
Sunny b41c717e16
controllers: emit event and log source up-to-date
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:06:16 +05:30
Sunny 86860ec913
Update all reconcilers with ArtifactInStorage cond
Update alll the other reconcilers similar to the GitRepository
reconcilers to introduce positive condition ArtifactInStorage and
reorder the status conditions.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:06:12 +05:30
Hidde Beydals 7ca393498c Prefix event annotations with API Group FQDN
This to facilitate improvements on the notification-controller side,
where annotations prefixed with the FQDN of the Group of the Involved
Object will be transformed into "fields".

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-23 19:28:05 +01:00
Sunny a102d95cf9
Prioritize StorageOperationFailedCondition
Prioritize StorageOperationFailedCondition over other artifact outdated
and unavailable conditions so that when artifact is failing due to
storage operation, it's visble in the ready status condition, making the
reason for not ready more accurate.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-16 21:41:39 +05:30
Sunny a2d6af126d
Add new condition StorageOperationFailedCondition
Introduce new condition StorageOperationFailedCondition for all the
failures related to the storage. It is a negative polarity condition and
is considered in computing summary of reconciliation.

Also, introduce more granular event reasons related to
StorageOperationFailedCondition for precise reasoning behind failures.
These replace the vague StorageOperationFailedReason.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-14 21:20:37 +05:30
Hidde Beydals 971caf92d5 controllers: finetune `eventLogf` (variant) docs
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Sunny 8d4fce1955 Add more details in helmchart spec doc
Details about the source reference, reconcile strategy and artifact
revision value based on the reconcile strategy.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-11 10:04:14 +01:00
Sunny f59b48c339 Document HelmChart API v1beta2 spec
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-11 10:04:14 +01:00
Paulo Gomes 3b4cc52419
Use uppercase TLS in error messages
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 13:02:11 +00:00
Paulo Gomes d9d789fdb1
Reuse transport for helm chart download
Reuses the same transport across different helm chart downloads,
whilst resetting the tlsconfig to avoid cross-contamination.

Crypto material is now only processed in-memory and does not
touch the disk.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 13:02:11 +00:00
Sunny f72a28a193 Use field owner in the patch helper
- Update summarize helper to have patch field owner.
- Updated the controllers to set the patch field owner.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 45df2d76c8 Update API descriptions and messages to be consistent
- Update v1beta2 API descriptions and reconciling messages to be
  consistent.
- Replace 'download' with 'fetch'. Since the status condition for
  download failure is called FetchFailed, using the term 'fetch' makes
  the messaging more consistent.
- Replace `BucketOperationSucceed` with `BucketOperationSucceeded` and
  generate api docs.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny d997876b07 Make generic SummarizeAndPatch()
summarizeAndPatch() was used by all the reconcilers with their own
object type. This creates a generic SummarizeAndPatch helper that takes
a conditions.Setter object and performs the same operations. All the
reconcilers are updated to use SummarizeAndPatch(). The process of
summarize and patch can be configured using the HelperOptions.

Introduce ResultProcessor to allow injecting middlewares in the
SummarizeAndPatch process.

Introduce RuntimeResultBuilder to allow defining how the reconciliation
result is computed for specific reconciler. This enabled different
reconcilers to have different meanings of the reconciliation results.

Introduce Conditions in summary package to store all the status
conditions related information of a reconciler. This is passed to
SummarizeAndPatch() to be used for summary and patch calculation.

Remove all the redundant summarizeAndPatch() tests per reconciler.

Add package internal/object containing helpers for interacting with
runtime.Object needed by the generic SummarizeAndPatch().

Add tests for ComputeReconcileResult().

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 9b5613732f storage: Return details about the deleted items
Update Storage.RemoveAll() and Storage.RemoveAllButCurrent() to return
the details about the deleted items. This helps emit useful information
about garbage collection in the controllers and ignore no-op garbage
collections.

RemoveAll() returns the path of the deleted directory if any.
RemoveAllButCurrent() returns a slice of path of all the deleted items
from a resource's artifact dir.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals fb45032eb6 controllers: only handle BuildError
All other errors returned by `build*` are already properly wrapped.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny 78882b3b36 Consolidate result conversion and computation
Consolidate BuildRuntimeResult() into summarizeAndPatch() to simplify
where the results are computed, summarized and patched.

Move the event recording and logging of context specific errors into
RecordContextualError() and call it in summarizeAndPatch().

Introduce Waiting error for wait and requeue scenarios. Update
ComputeReconcileResult() and RecordContextualError() to consider Waiting
error.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals 032ffb4d27 controllers: tweak events and logging
- Ensure all logged messages start with a lowercase.
- Make some pushed (and logged) events of type `EventTypeTrace` to
  prevent them from being sinked to the external event recorder, to
  prevent spam.
- Only log if artifact is up-to-date with upstream (instead of pushing
  an event).

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 527fce05df Rewrite HelmChartReconciler tests
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 8e107ea60e HelmChartReconciler refactor
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny e0e048ad6d helmchart: Replace GetInterval() with GetRequeueAfter()
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals 349739b7e4 Implement new runtime interfaces, prepare testenv
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.

Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 7c3c14997e Switch to v1beta2 API package
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:25 +01:00
Stefan Prodan 5be33770e8
Use patch instead of update when adding finalizers
This is needed to prevent source-controller from managing all the fields under `.spec`.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-01-13 18:18:32 +02:00
Stefan Prodan eacabe23a1
Log the error when tmp cleanup fails
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2022-01-07 10:37:06 +02:00
Aurel Canciu 22d0880e4d
Update flux pkg components
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2021-12-20 14:47:44 +01:00
Hidde Beydals 5ddeb0934f controllers: use short SHA in chart SemVer meta
As the full version can be used as a label value, the full SHA from the
reference takes up too much space from the 63 characters available in
total.

To mitigate against this, we now take a "short" version of the first 12
characters, which was still unique for the Linux kernel in 2019 with
875.000 commits:
http://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#Short-SHA-1

This should be sufficient to safely detect all changes within the
context of operations.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-12-02 22:28:18 +01:00
Hidde Beydals 905602bdfe controllers: return err on auth dir create failure
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals dbbef5add8 controllers: use `time.Since`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals 6a8b5889f1 controllers: absolute local path for cached chart
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-22 22:10:31 +01:00
Hidde Beydals 472eb12f43 controllers: set generation as version metadata
By providing the Generation of the object that is getting reconciled
as version metadata to the builder if any custom values files are
defined, the Artifact revision changes if the specification does,
ensuring consumers of the Artifact are able to react to changes in
values (and perform a release).

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals a1e9302b7d internal/helm: "value files" -> "values files"
Previous usage while consistent, was incorrect, and inconsitent with
the field in the API spec.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 2b8134ce20 internal/helm: introduce typed BuildError
This commit introduces a typed `BuildError` to be returned by
`Builder.Build` in case of a failure.

The `Reason` field in combination with `BuildErrorReason` can be used
to signal (or determine) the reason of a returned error within the
context of the build process.

At present this is used to determine the correct Condition Reason, but
in a future iteration this can be used to determine the negative
polarity condition that should be set to indicate a precise failure to
the user.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 32e19ebcd0 controllers: more tidying of wiring
Dealing with some loose ends around making observations, and code
style.

The loaded byes of a chart are used as a revision to ensure e.g.
periodic builds with unstable ordering of items do not trigger a false
positive.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 7d0f79f41b internal/helm: divide into subpackages
With all the logic that used to reside in the `controllers` package
factored into this package, it became cluttered. This commit tries to
bring a bit more structure in place.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 9abbdd80a6 controllers: rough wiring of Helm chart builder
This commit starts wiring the factored out Helm chart build logic into
the reconciler to ensure, validating the API capabilities.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 5a1fcc213b git: standardise commit and (PGP) verification
This commit refactors the previous `Commit` interface into a
standardised `Commit` struct. This object contains sufficient
information for referencing, observating and (PGP) verification.

- `libgit2` commit checkout does now return `HEAD/<SHA1>` as
  the branch is not taken into account.
- `git2go` objects are now properly `Free`d everywhere
- `Verify` logic is tested.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-10-27 00:43:31 +05:30
Dylan Arbour 27c385b957 Add ReconcileStrategy to HelmChart
This commit adds a `ReconcileStrategy` field to the `HelmChart` resource, which
allows defining when a new chart should be packaged and/or published if it
originates from a `Bucket` or `GitRepository` resource.

The two available strategies are:

- `ChartVersion`: creates a new artifact when the version of the Helm chart as
  defined in the `Chart.yaml` from the Source is different from the current
  version.
- `Revision`: creates a new artifact when the revision of the Source is
  different from the current revision.

For the `Revision` strategy, the (checksum part of the) revision of the
artifact the chart originatesfrom is added as SemVer metadata.

A chart from a `GitRepository` with Artifact revision
`main/f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer
`0.1.0` will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.

A chart from a `Bucket` with Artifact revision
`f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer `0.1.0`
will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.

Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
2021-10-08 11:07:11 +02:00