Likely after the upgrade to controller-runtime v0.15.0 a regression
surfaced for long-running reconciliations of HelmRelease resources (e.g.
for charts having pre-upgrade hooks taking a few minutes to complete).
This regression would cause the controller to immediately re-run the
upgrade after a successful upgrade, thus entering an almost-endless
loop.
Apparently, the only fix to this issue is to ensure
`.Status.LastReleaseRevision` is updated as soon as possible in the
reconiliation cycle rather than wait for the update at the end of the
cycle.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
This commit updates Kubernetes to v1.27, controller-runtime to
v0.15, and Helm to v3.12.
It deals with various breaking changes in controller-runtime, as
documented in the release notes:
https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.15.0
In short:
- `Watches` now use a `client.Object` instead of a `source.Kind`.
- `handler.MapFunc` signature accepts a Go context, which is used to
log any errors, instead of silently ignoring them and/or panicking.
- Max concurrent reconciles is configured on the manager, instead of
configuring them per reconciler instance.
- Various manager configuration options have been moved to new
structures and/or fields.
In addition to this, all other dependencies which had updates
available are updated to their latest versions as well.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This commit changes the way the checksum is calculated for the release
values, by stable sorting the keys. By doing this, an upgrade will not
be triggered when a key/value pair has just been moved, instead of
containing a real change of value.
To make it backwards compatible (and without triggering an upgrade due
to new ordering), the checksum without ordering is continued to be
calculated and compared against until removal in a future controller
release. However, only the checksum of the ordered values is taken note
of in the Status of the HelmRelease.
Co-authored-by: Hidde Beydals <hidde@hhh.computer>
Signed-off-by: longquan0104 <longquan0104@gmail.com>
Hotfix to deal with the issue, as we will soon be moving to the patch
logic used by already rewritten controllers. This is however a pain for
many users, so worth solving now.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
- Assing `ObjectMeta` field in Helm chart template.
- Ensure things are at least lightly mentioned in spec documentation.
- Add two simple test cases.
- Fix broken links to Kubernetes documentation.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
- github.com/fluxcd/pkg/apis/kustomize to v1.0.0
- github.com/fluxcd/pkg/apis/meta to v1.0.0
- k8s.io/apiextensions-apiserver to v0.26.3
- k8s.io/apimachinery to v0.26.3
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
There is no good reason for it to be exposed and available through a
public API, and this follows the new kubebuilder defaults.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This enhances the drift detection to allow detection without acting on
it by adding a `CorrectDrift` feature gate. When set to `false` (i.e.
`--feature-gates=DetectDrift=true,CorrectDrift=false`), the controller
will only log the difference it detected without acting on it.
This allows for a smoother transition to drift detection, as issues
with a number of existing releases can be solved before the feature
is enabled in full.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This commit adds support for recognizing cgroup v1 paths, and allows for
the configuration of alternative absolute path locations using
`--oom-watch-max-memory-path` and `--oom-watch-current-memory-path`.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This commit introduces an OOM watcher, which can be enabled using
`--feature-gates=OOMWatch=true`. The OOM watcher watches the current
memory usage as reported by cgroups via `memory.current` and cancels
the context when it reaches a certain threshold compared to
`memory.max` (default `95`%, configurable using
`--oom-watch-memory-threshold`).
This allows ongoing Helm processes to gracefully exit with a failure
before the controller is forcefully OOM killed, preventing a deadlock
of releases in a pending state.
The OOM watcher polls the `memory.current` file on an interval (default
`500ms`, configurable using `--oom-watch-interval`), as subscribing to
file updates using inotify is not possible for cgroups (v2) except for
`*.events` files. Which does provide signals using `memory.events`, but
these will generally be too late for our use case. As for example `high`
equals `max` in most containers, buying us little time to gracefully
stop our processes.
In addition, because we simply watch current usage compared to max
usage in bytes. This approach should work for cgroups v1 as well, given
this has (most of the time) files for these values available, albeit
at times at different locations. For which this commit does not
introduce a flag yet, but the library takes into account that it could
be configured at some point.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
As otherwise with a persistent discovery client and/or REST mapper
configuration, newly installed CRDs will not be recognized and cause a
`resource mapping not found for name` error.
In addition, remove the `ServerGroups` and `Invalidate` calls. As this
is later done (again) by Helm when gathering server capabilities.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This drops the twofold implementation in favor of a single
`MemoryRESTClientGetter` which can work with an arbitrary `rest.Config`.
The new `MemoryRESTClientGetter` lazy-loads and caches the objects it
initializes, thereby creating at most one instance of each object for
the duration of the reconcile of a single `HelmRelease` object.
Based on some initial tests, this seems to reduce the overal memory
footprint of the controller.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This reduces the amount of log lines pushed to `debug` by configuring the kube
client and storage loggers to only log to `trace`.
In addition, the log buffer used in events will now just contain the
most relevant information about a failure as reported by the Helm action
itself, and not the in-depth information from the underlying client
and/or storage.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This allows install and upgrade actions to use DNS lookups while
rendering Helm templates after it got disabled in Helm due to possible
security risks.
It is enabled (globally) on the controller by configuring
`--feature-gates=AllowDNSLookups=true`.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
As there are currently no other utilities to properly see what change
the controller detected, this allows people to have an insight into
the observed changes by configuring the controller with
`--log-level=debug`.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This allows a specific object from a release manifest to be excluded
from drift detection by labeling or annotating it with:
`helm.toolkit.fluxcd.io/diff: disabled`.
Using a Kustomize post renderer definition in a HelmRelease, this can
be used to ignore any object from an arbitrary chart.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This enables experimental drift detection of cluster state compared to
the current manifest data from the Helm storage's manifest blob.
Drift detection works based on the already proven approach of the
kustomize-controller's SSA package, and utilizes the managed field
configured by the controller since `v0.12.2`.
This feature is planned to go out of experimental once the further
controller rewrite has been finished, and the state of the Helm storage
itself is more fault tolerant.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This allows the applied CRDs to be traced using the same labels as
currently applied to resources using a Kustomize post-render.
Kustomize is not used here as the apply logic for CRDs is different
from the approach used during releasing, where we inject the labels
in such a way that they are written back to the Helm storage in the
rendered manifest. This to match Helm's logic from which our present
code is already derived (buth with support for policies).
This also moves the full responsibility of dealing with the install
of CRDs to ourselves, as we no longer fall back to Helm's logic when
`Create` is configured as a policy during a Helm install. As this
would not allow us to add the labels.
Signed-off-by: Hidde Beydals <hello@hidde.co>
You can re-enabled caching of secrets by starting the
controller with the argument '--feature-gates=CacheSecretsAndConfigMaps=true'
Signed-off-by: Mac Chaffee <machaffe@renci.org>
Fixing a regression introduced in #480 which would always pick the
namespace of the release. In addition, historically seen the
configuration of the impersonation username while making use of a
KubeConfig has never worked correctly, this has been adressed as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This is a partial cherry-pick of commit ae4f499e87, including
changes around `kube`. This to include some of the changes around the
construction of the ConfigFlags RESTClientGetter, as an attempt to
solve token refresh issues.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Two new flags were added to allow users to enable the
use of user.Exec and InsecureTLS in the kubeconfigs
provided remote apply reconciliations.
Breaking change: both functionalities are no longer
enabled by default.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This commit changes the default behavior of the Helm uninstall action
to wait for all resources to be deleted, and introduces a
`.spec.uninstall.disableWait` flag to disable this behavior.
Signed-off-by: Samuel Torres <samuelpirestorres@gmail.com>
Introduce the flag `--default-service-account` for allowing cluster admins to enforce impersonation for resources reconciliation.
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
Controller-runtime has been updated to `v0.9.0`, K8s dependencies to
`v0.21.1`, and all `fluxcd/pkg` and other dependencies to the versions
that have matching dependencies and/or build constraints.
This includes an update of Helm to `v3.6.0`, and an update of the
Kustomize API to match `v4.1.x`.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Aurel Canciu
Hidde Beydals
longquan0104
Stefan Prodan
Raffael Sahli
Hidde Beydals
stoetti
Mac Chaffee
Paulo Gomes
Tomek Rękawek
Samuel Torres
Sunny
Santosh Kewat