- Change the log-level of "action determination" to "debug".
- Set `Ready=Unknown` while working on an install or upgrade.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
We actually only care about the chart name or version changing, as we
assume proper (immutable) versioning by the publisher of the chart
(either the user, or the source-controller).
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This improves the reconciliation of an uninstall when the release has
already been uninstalled while `KeepHistory` has been set, by detecting
the (sadly non-typed) error Helm produces as desired state.
Avoiding certain edge-cases where for example a deleted HelmRelease
would end up in an irrecoverable loop of uninstall attempts, after
being remediated (using an uninstall) before the deletion request.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
As we are working with secondary state which we need to keep track of,
persisting the last state even when the context is canceled (due to
e.g. a controller shutdown) is important to improve the chances of
successfully being able to recover from any abrupt terminations.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
From `<namespace>/<name>.<version>` to `<namespace>/<name>.v<version>`,
to better resemble the internal name format of e.g. Helm storage
Secrets.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This ensures the logs of the Kubernetes client used by Helm are
persisted to the log buffer, as they can contain important information
when an action times out.
In addition, move the logs from the Helm actions themselves to the
"debug" log level (while still including them in Kubernetes Events in
case of a failure), in favor of the logs produced by the `reconcile`
package itself. While moving the logs from the Helm storage to the
"trace" log level, as they only contain information about e.g. writes
to a Secret.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This in an attempt to maintain compatability with earlier documented
inclusion and exclusion lists for Alerts, like the following:
```
eventSources:
- kind: HelmRelease
name: demo
inclusionList:
- ".*.upgrade.*succeeded.*"
```
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
The primary reason for this is the alphabetical ordering of `kubectl
describe`, which caused the fields to be listed in separate places
instead of a bundle.
From a programmatic perspective, it is also great because it is now much
easier to reset any previous state when e.g. uninstalling a release. As
we can simply write an empty struct to erase any memory of a previous
release, instead of having to deal with multiple fields.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This will allow the controller to pick the right method for digest
calculations when we for example add new data into the calculation.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This ensures certain edge-cases around the availability of the service
account and/or KubeConfig are handled.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This is a better way of dealing with this situation, as the previous
logic would result in an `ErrNoStorageUpdate`.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
"With hope comes the potential for both triumph and tribulation."
Due to difficulties beyond the time I have at hands at present[1], the
separate reconciler which took care of ensuring the HelmChart of the
HelmRelease was kept up-to-date has been transformed into a
sub-reconciler.
The behavior of the sub-reconciler remains largely unchanged, except the
required changes to deal with the lack of possibilities to requeue.
Effectively, this means that instead of e.g. deleting the HelmChart
object, requeue, and create it again. This is now handled in a single
operation, unless the deletion fails.
[1]: The core of the issue is that deregistration of finalizers becomes
difficult due to the behavior of the patch helper, and unavailability of
list merges for patch operations on Custom Resources within Kubernetes.
This means that when two reconcilers simultaneously work on the
deregistration of the finalizers, and one succeeds before the other. The
last finishing reconciler will attempt to add the finalizer of the other
reconciler back, as it did exist at the start of their reconciliation
run.
Attempts to work around this (for example, by using an optimistic lock
on the patch operation of the finalizers field) would cause new issues.
As Kubernetes will then delete the object as soon as the patch has
succeeded, and before the reconciliation process actually ends.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This introduces an `acl` package in `internal` which globally configures
the allowance to namespaced references, instead of having to pass on a
variable everywhere.
For the sake of security, the default behavior of the package itself is
to _not_ allow cross namespace references. However, the behavior of the
controller remains unchanged, and the configuration flag still enables
the allowance by default.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
Sadly, Helm more than often ends error messages with `\n\n`. Trim this
space to ensure we produce pretty messages.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
When an object is marked as under deletion, the API server will reject
any attempt to register new finalizers. Given this, handling the
deletion timestamp always has to come before an attempt to register
the finalizer.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This ensures they both have the same observation on the last
modifications made to the object. Preventing possible scenarios where
a condition would not be removed because it wasn't set at the start of
the reconcile run, then added, and then removed. Causing it to go
unnoticed during the diff calculation.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This adds the base wiring to get the controller to work with the
v2beta2 API and the newly introduced packages in `internal/`.
In essence, this means that from now on the controller will utilize all
new code for the reconciliation of the HelmRelease resource.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
- Use `Unknown` status for the `TestSuccess` condition when tests
have not been run yet.
- Update Ready summarization logic to incorportate conditions with an
Unknown status. Within the context of readiness, this always caises
Ready=False when the condition is included in the summarization.
- Variety of tiny fixes.
- Tiny nits in test mocks to prevent confusion.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This includes the "token" in the emitted events which is used to rate
limit events received by the notification-controller.
Either by using the already calculated config (values) digest, or by
calculating it for the current reconciliation request in scenarios
where it isn't available from made observations.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This provides more context to individual log entries (and the duration
between individual log lines) while e.g. printing them in an event.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit adds an atomic release reconciler, capable of stepping
through a series of Helm actions. In addition, it adds the last bits
around eventing and summarizing the end state of the Condition types
into e.g. a Ready condition.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This solves the issue where a release name composed out of e.g.
the target namespace and name of the HelmRelease itself would exceed
the >=53 character length. By calculating the SHA256 checksum of the
release name, taking the first 12 characters of this checksum and
appending it to the release named trimmed to 40 characters separated
by a hyphen (`<long-release-name>-abcdef12345678`).
Signed-off-by: Hidde Beydals <hello@hidde.co>
This gives more fine-grain control over what release must be targeted,
as we do not always want to rely on the current spec but rather on e.g.
a release we have made ourselves with a previous configuration for
garbage collection purposes.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This allows for requesting the count of non-empty values in the ring
buffer, and thus the number of log lines.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- Change the map with Helm release test hooks to a pointer map. This
allows (in combination with the constrains around JSON serialization)
to distinguish a release _without_ a test run from a release _with_
test run but no tests (an empty map).
- Add `GetTestHooks` and `SetTestHooks` methods to help circumvent some
of the common problems around working with a pointer map in Go (e.g.
not being capable of iterating over it using range).
- Add `HasBeenTested` and `HasTestInPhase` methods to help make
observations on captured release information.
- Add `StorageNamespace` to Status to allow for observations of
configuration changes which are mutating compared to the spec.
- Add `GetActiveRemediation` helper method to get the active
remediation strategy based on the presence of Current and/or Previous
release observations in the Status of the object.
- Add `ReleaseTargetChanged` helper method to determine if an immutable
release target changed has occurred, in which case e.g. garbage
collection needs to happen before performing any other action.
- Add `GetCurrent`, `HasCurrent`, `GetPrevious` and `HasPrevious`
helper methods to ease access to their values nested in the Status.
- Add `FullReleaseName` and `VersionedChartName` helper methods to e.g.
allow printing full name references in Condition and Event messages
which can be placed in a point in time based on metadata more
familiar to a user than for example the observed generation.
- Change `GetFailureCount` and `RetriesExhausted` signatures of
`Remediation` interface to take a pointer. This eases use of the API,
as generally speaking a (Kubernetes) API object is a pointer.
- Move methods from `HelmReleaseSpec` to `HelmRelease`, this is easier
to access and matches `GetConditions`, etc.
- Remove `DeploymentAction` interface and `GetDescription` from
`Remediation` interface as this is no longer of value.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This to allow the Flux CLI to e.g. enable the dry-run flag on an action
outside of the HelmRelease spec, and inject other (user input based)
modifications.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This provides a rough (but not flawless) outline for determining the
sub-reconciler which should run based on the state of the `HelmRelease`
API object, and the Helm storage.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This adds a `reconcile` package with the reconciliation and (status)
observation logic for individual Helm actions, but no glue to loop
through them till desired state.
All actions have individual `ActionReconciler` implementations which
construct their `action.Configuration` out of a factory, so the Helm
client can be shared between sub-reconcilers. They all present a
`ReconcilerType`, allowing an iterator to e.g. stop after running
every type just once.
The observation model can be explained as follows, but may lack some
minor details:
- The observed release has to match the release target of the
HelmRelease object
- ActionReconcilers of type "release" move Current to Previous
when they see a higher release revision. They then write the
new release to Current, and continue to observe writes to
revisions that match either version
- Remediation only updates Current
- Test updates Current and Current.TestHooks
- Unlock updates Current
After running the action, the reconcilers observe both the action
result and the state of the object. This allows them to distinguish
certain types of errors which are otherwise hard to detect.
For example, errors which do not cause drift to the Helm storage, or a
change of release version compared to Current for actions which do not
provide a version target flag.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This adds a `release` package which allows to create (minified)
`ObservedRelease` copy of a Helm release object. This
`ObservedRelease` contains sufficient data to detect changes
to the storage object made by Helm actions run manually, and a variety
of malicious changes (but not all, at present).
The data in an `ObservedRelease` can be filtered using a `DataFilter`,
this allows for example to filter out test hooks to prevent the
controller from taking action on a manually run `helm test`.
The consumer can combine the `ObservedRelease` with a Helm storage
observer to take snapshots of the release object as written to the
storage by a Helm action. To record this on a `HelmRelease` v2beta2 API
object, the `ObservedRelease` can be transformed into a
`HelmReleaseInfo` API object which can be recorded as either the
Current or Previous release in the status.
During the transformation, the digests of both the `ObservedRelease`
object and release config are calculated using the canonical algorithm.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit introduces an `action` package which allows the consumer to
run Helm actions using the instructions from a `HelmRelease` v2beta2
API object.
The actions do not determine if there is a desire be run, nor do they
record state on the object. This can however be injected by the caller
using the simplified observing Helm storage driver, which now iterates
over a list of callback functions after persisting an object instead
of keeping state.
This separation of concerns would allow e.g. the Flux CLI later on
to run actions (but with a dry-run flag or different storage
configuration) using the object in the same manner as the controller.
Some minor changes have been made to the `postrender` and `runner`
package to allow the code to co-exist while we are inbetween API
versions.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Hidde Beydals
Hidde Beydals
Jiri Tyr
Jiri Tyr