Commit Graph

1568 Commits

Author SHA1 Message Date
Hidde Beydals 3ce6e8d1b5
reconcile: improve wording `Stalled` condition
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-23 00:17:17 +01:00
Hidde Beydals 0535ae1114
predicates: notice source changing to `Ready=True`
This ensure that when a chart object has a temporary `Ready=False`
state, the predicate will notice the change to `Ready=True` and
cause an enqueue request.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-23 00:17:16 +01:00
Hidde Beydals 6ffdaddc06
action: omit logging on CRD apply no-op
As it does not offer real value, and creates noise when the logs are
included in an emitted event.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-23 00:17:15 +01:00
Hidde Beydals 7aad010664
controller: immediate requeue unfinished release
This improves continuity while the controller attempts to move the
release forward.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-23 00:17:14 +01:00
Hidde Beydals 5d1f34a029
controller: patch after setting `Reconciling=True`
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-23 00:17:13 +01:00
Hidde Beydals 20c00fd47a
action: provide a reason on release target changes
This to allow better feedback to the user on why the controller decided
to uninstall the release.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-23 00:17:09 +01:00
Hidde Beydals 580c72cd09
controller: adopt release based on v2beta1 state
This allows the controller to be updated from `v2beta1` to `v2beta2`
without triggering a release to settle state.

It does this by looking at the previous successful release as recorded
for the `v2beta1` object, and if found, recording a snapshot for it in
the new `History` field of the status.

This feature can be disabled by setting the `AdoptLegacyReleases`
feature flag to `false`.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-22 23:14:17 +01:00
Hidde Beydals eab8a50071
api: prepare `v2beta1` status for `v2beta2`
This allows users to delay updating their `apiVersion` declarations,
as the fields will be known in the previous version.

If we would not do this, the fields would get wiped when the `v2beta1`
resource is applied, potentially causing spurious upgrades.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-22 23:14:15 +01:00
Hidde Beydals 9bb8f02495
api: continue to record `LastAppliedRevision`
This ensures that certain UIs can continue to display information to
their users while they work on making better use of the new data
available in `v2beta2`.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-21 15:31:30 +01:00
Hidde Beydals 6f05228e85
reconcile: remove logs from test failure event
During tests, Helm logs a mixture of debug and timeout errors which
(unlike during install or upgrade) do not provide much useful
information about the reason of failure.

Given this, remove the logs from the output to reduce noise.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:56 +01:00
Hidde Beydals 28a7800528
reconcile: mark `Ready=Unknown` when awaiting test
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:55 +01:00
Hidde Beydals 70485017d2
controller: requeue on fixed interval on chart 404
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:54 +01:00
Hidde Beydals eacd97532c
reconcile: remove reconciler type from messages
Not of much relevance to the user, while it can be confusing for
unlock.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:54 +01:00
Hidde Beydals 533589ce39
api: change `MaxHistory` default to `5`
As anything beyond ~3 releases has no real value to the controller
itself, while the time it takes to perform a release increases with
the number of releases kept in history.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:53 +01:00
Hidde Beydals c5a017cb76
api: record observed releases in `Status.History`
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:53 +01:00
Hidde Beydals 517d42fe5d
build: incorporate condition changes in e2e
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:52 +01:00
Hidde Beydals 16da3ec66f
reconcile: allow unlock without current
As persisting to the API server before an unexpected termination is not
feasible, allow a release to be unlocked without it being observed.

This has the potential result of interfering with a user running `helm`
by hand, but the further process should be capable of dealing with any
unexpected failure resulting from this.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:51 +01:00
Hidde Beydals 2e0e22593f
reconcile: improve state determination
This decouples the state determination from deciding which action to
take, making it easier to reason about the different types of state
and what action should be taken to drive it forward.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:51 +01:00
Hidde Beydals 10277c7b26
api: add `LastAttemptedReleaseAction` to status
This can be used to reliable determine the active remediation strategy
which should be used. Including in edge cases where e.g. an uninstall
remediation is configured for upgrade failures.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:50 +01:00
Hidde Beydals 80d0878e96
controller: ignore `NotFound` API error on delete
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:50 +01:00
Hidde Beydals 2df90eb4cf
reconcile: improve observability between actions
- Change the log-level of "action determination" to "debug".
- Set `Ready=Unknown` while working on an install or upgrade.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:49 +01:00
Hidde Beydals 7c52fd255f
action: simplify chart diff logic
We actually only care about the chart name or version changing, as we
assume proper (immutable) versioning by the publisher of the chart
(either the user, or the source-controller).

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:49 +01:00
Hidde Beydals 096956fdfd
controller: properly record object metrics
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:48 +01:00
Hidde Beydals d0c4c14056
reconcile: improve uninstall w/o purging history
This improves the reconciliation of an uninstall when the release has
already been uninstalled while `KeepHistory` has been set, by detecting
the (sadly non-typed) error Helm produces as desired state.

Avoiding certain edge-cases where for example a deleted HelmRelease
would end up in an irrecoverable loop of uninstall attempts, after
being remediated (using an uninstall) before the deletion request.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:47 +01:00
Hidde Beydals 191bebfafd
reconcile: simplify `NextAction` logic
By looking at the type of the error, instead of doing a separate check
on `cur != nil`.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:47 +01:00
Hidde Beydals f156c3550e
reconcile: allow cfg of manager in atomic action
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:46 +01:00
Hidde Beydals ac9c2c3142
reconcile: ensure object patch on context cancel
As we are working with secondary state which we need to keep track of,
persisting the last state even when the context is canceled (due to
e.g. a controller shutdown) is important to improve the chances of
successfully being able to recover from any abrupt terminations.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:46 +01:00
Hidde Beydals 19be1b24ac
api: change format of `Snapshot#FullReleaseName`
From `<namespace>/<name>.<version>` to `<namespace>/<name>.v<version>`,
to better resemble the internal name format of e.g. Helm storage
Secrets.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:45 +01:00
Hidde Beydals 272329d86a
action: add `:` separator between ts and msg logs
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:45 +01:00
Hidde Beydals a6ae4c3fb9
reconcile: improve log levels of actions
This ensures the logs of the Kubernetes client used by Helm are
persisted to the log buffer, as they can contain important information
when an action times out.

In addition, move the logs from the Helm actions themselves to the
"debug" log level (while still including them in Kubernetes Events in
case of a failure), in favor of the logs produced by the `reconcile`
package itself. While moving the logs from the Helm storage to the
"trace" log level, as they only contain information about e.g. writes
to a Secret.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:44 +01:00
Hidde Beydals bc036c027f
reconcile: improve insights of progress in logs
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:43 +01:00
Hidde Beydals 5510175ccb
reconcile: tweak event messages
This in an attempt to maintain compatability with earlier documented
inclusion and exclusion lists for Alerts, like the following:

```
  eventSources:
    - kind: HelmRelease
      name: demo
  inclusionList:
    - ".*.upgrade.*succeeded.*"
```

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:43 +01:00
Hidde Beydals 94064da340
controller: add reconcile release tests
Plus some minor improvements to the logic, based on writing tests.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:42 +01:00
Hidde Beydals 882da27a5d
api: move `Current` and `Previous` into `History`
The primary reason for this is the alphabetical ordering of `kubectl
describe`, which caused the fields to be listed in separate places
instead of a bundle.

From a programmatic perspective, it is also great because it is now much
easier to reset any previous state when e.g. uninstalling a release. As
we can simply write an empty struct to erase any memory of a previous
release, instead of having to deal with multiple fields.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:42 +01:00
Hidde Beydals 7dfce0c738
api: introduce `APIVersion` in `Snapshot`
This will allow the controller to pick the right method for digest
calculations when we for example add new data into the calculation.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:41 +01:00
Hidde Beydals 9df9b176d9
api: various naming improvements
- Rename `HelmReleaseInfo` to `Snapshot`.
- Rename `HelmReleaseTestHook` to `TestHookStatus`.
- Rename `ObservedRelease` to `Observation`.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:41 +01:00
Hidde Beydals b2ba3d97ea
controller: improve deletion logic and add tests
This ensures certain edge-cases around the availability of the service
account and/or KubeConfig are handled.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:40 +01:00
Hidde Beydals fbd73ac399
controller: start w/ adding tests for HelmRelease
This adds base coverage for some of the simpler methods which do not
require extensive mocking.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:39 +01:00
Hidde Beydals 1dac82ad2c
reconcile: handle manually uninstalled release
This is a better way of dealing with this situation, as the previous
logic would result in an `ErrNoStorageUpdate`.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:39 +01:00
Hidde Beydals 5e3ad5d21a
reconcile: add `HelmChartTemplate` sub-reconciler
"With hope comes the potential for both triumph and tribulation."

Due to difficulties beyond the time I have at hands at present[1], the
separate reconciler which took care of ensuring the HelmChart of the
HelmRelease was kept up-to-date has been transformed into a
sub-reconciler.

The behavior of the sub-reconciler remains largely unchanged, except the
required changes to deal with the lack of possibilities to requeue.
Effectively, this means that instead of e.g. deleting the HelmChart
object, requeue, and create it again. This is now handled in a single
operation, unless the deletion fails.

[1]: The core of the issue is that deregistration of finalizers becomes
difficult due to the behavior of the patch helper, and unavailability of
list merges for patch operations on Custom Resources within Kubernetes.

This means that when two reconcilers simultaneously work on the
deregistration of the finalizers, and one succeeds before the other. The
last finishing reconciler will attempt to add the finalizer of the other
reconciler back, as it did exist at the start of their reconciliation
run.

Attempts to work around this (for example, by using an optimistic lock
on the patch operation of the finalizers field) would cause new issues.
As Kubernetes will then delete the object as soon as the patch has
succeeded, and before the reconciliation process actually ends.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:38 +01:00
Hidde Beydals dab2578c07
acl: introduce package to enable global config
This introduces an `acl` package in `internal` which globally configures
the allowance to namespaced references, instead of having to pass on a
variable everywhere.

For the sake of security, the default behavior of the package itself is
to _not_ allow cross namespace references. However, the behavior of the
controller remains unchanged, and the configuration flag still enables
the allowance by default.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:38 +01:00
Hidde Beydals e32c1a0f4a
reconcile: trim space from Helm error messages
Sadly, Helm more than often ends error messages with `\n\n`. Trim this
space to ensure we produce pretty messages.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:37 +01:00
Hidde Beydals 68c273b701
controller: handle delete before adding finalizer
When an object is marked as under deletion, the API server will reject
any attempt to register new finalizers. Given this, handling the
deletion timestamp always has to come before an attempt to register
the finalizer.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:37 +01:00
Hidde Beydals 866f076d1f
reconcile: share PatchHelper with controller
This ensures they both have the same observation on the last
modifications made to the object. Preventing possible scenarios where
a condition would not be removed because it wasn't set at the start of
the reconcile run, then added, and then removed. Causing it to go
unnoticed during the diff calculation.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:36 +01:00
Hidde Beydals bbefbc4ded
reconcile: use failure count in Stalled condition
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:35 +01:00
Hidde Beydals d802ba6cc1
controllers: roughly rewire HelmRelease reconciler
This adds the base wiring to get the controller to work with the
v2beta2 API and the newly introduced packages in `internal/`.

In essence, this means that from now on the controller will utilize all
new code for the reconciliation of the HelmRelease resource.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:35 +01:00
Hidde Beydals eee91b06fa
Introduce new `yaml` package with `Encode` func
Comparison versus `sigs.k8s.io/yaml#Marshal`:

```
BenchmarkEncode/EncodeWithSort-12         	    475	  2419063 ns/op	2235305 B/op	   5398 allocs/op
BenchmarkEncode/EncodeWithSort-12         	    498	  2406794 ns/op	2235300 B/op	   5398 allocs/op
BenchmarkEncode/EncodeWithSort-12         	    492	  2376460 ns/op	2235312 B/op	   5398 allocs/op
BenchmarkEncode/EncodeWithSort-12         	    496	  2406756 ns/op	2235323 B/op	   5398 allocs/op
BenchmarkEncode/EncodeWithSort-12         	    488	  2402969 ns/op	2235336 B/op	   5398 allocs/op
BenchmarkEncode/SigYAMLMarshal-12         	    202	  5791549 ns/op	3124841 B/op	  19324 allocs/op
BenchmarkEncode/SigYAMLMarshal-12         	    205	  5780248 ns/op	3123193 B/op	  19320 allocs/op
BenchmarkEncode/SigYAMLMarshal-12         	    207	  5762621 ns/op	3124537 B/op	  19324 allocs/op
BenchmarkEncode/SigYAMLMarshal-12         	    214	  5748899 ns/op	3121183 B/op	  19324 allocs/op
BenchmarkEncode/SigYAMLMarshal-12         	    211	  5682105 ns/op	3120592 B/op	  19325 allocs/op
```

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:34 +01:00
Hidde Beydals bb4e9b7cee
Update YAMLs to `helm.toolkit.fluxcd.io/v2beta2`
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:34 +01:00
Hidde Beydals deb0b14e43
api: make v2beta2 storage version
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:33 +01:00
Hidde Beydals 76f62ffc47
api: backport uninstall del propagation to v2beta2
Manual backport of the work done in #698, to keep things aligned.

Signed-off-by: Hidde Beydals <hidde@hhh.computer>
2023-11-20 12:06:10 +01:00