Remove the HelmRepositoryOCI reconciler and make HelmRepository of type
OCI static. The existing HelmRepository OCI objects are migrated to
static object by removing their finalizers and status. New
HelmRepository OCI objects go through one time migration to remove the
status. These are not reconciled again, unless the type is changed to
default. On type switching from HelmRepository default to OCI, the
finalizer, status and artifact are removed to make the object static. On
switching from OCI to default, a complete reconciliation of
HelmRepository takes place to build artifact and add status and
finalizer.
The HelmRepository .spec.url has a new validation to check the URL
scheme. This is to add some validation to HelmRepository OCI since it's
not backed by a reconciler for full validation.
Add HelmRepositoryOCIMigrationPredicate predicate to detect and allow
reconciliation of HelmRepository OCI objects that need migration. The
other predicates that filtered the HelmRepository events based on the
type have been removed as all the HelmRepositories will now be
reconciled by a single reconciler. HelmRepositoryOCIMigrationPredicate
readily allows non-OCI objects and only checks if a migration is needed
for OCI type object.
Add controller tests for different migration scenarios.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Move record suspend metrics next to readiness and duration metrics so
that it gets recorded along with others always at the end and the
metrics delete, which requires the knowledge of deleted finalizers,
applies to suspend too.
HelmRepository cache event metrics for a given helmrepo also continues
to be exported even after the object is deleted. This change deletes
the cache event metrics when the object is deleted.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This adds a `--interval-jitter-percentage` flag to the controller to
add a +/- percentage jitter to the interval defined in resources
(defaults to 10%).
Effectively, this results in a reconcilation every 4.5 - 5.5 minutes
for a resource with an interval of 5 minutes.
Main reason to add this change is to mitigate spikes in memory and
CPU usage caused by many resources being configured with the same
interval.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
If implemented user will be able to provide their own custom start and
bypass tls verification when interacting with OCI registries over https
to pull helmCharts.
Signed-off-by: Soule BA <soule@weave.works>
This deals with various breaking changes in controller-runtime, as
documented in the release notes:
https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.15.0
In short:
- `Watches` now use a `client.Object` instead of a `source.Kind`.
- `handler.MapFunc` signature accepts a Go context, which is used to
log any errors, instead of silently ignoring them and/or panicking.
- Fake clients used in tests are now configured using
`WithStatusSubresource` to enable the correct behavior for status
updates and patches.
- Max concurrent reconciles is configured on the manager, instead of
configuring them per reconciler instance.
- Various manager configuration options have been moved to new
structures and/or fields.
In addition to this, all other dependencies which had updates are
updated to their latest (compatible) versions as well.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This was an artifact of the desire to run `make run` without any custom
configuration, rather than it being a feature of the controller. The
setup of this has now been moved to the `Makefile` itself, including the
required configuration of the `--storage-adv-addr`.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
With this enhancement, the controller can be configured with
`--watch-label-selector`, after which only objects with this label will
be reconciled by the controller.
This allows for horizontal scaling of the source-controller, where each
controller can be deployed multiple times with a unique label selector
which is used as the sharding key.
Note that this also requires configuration of the `--storage-adv-addr`
to a unique address (in combination with a proper Service definition).
This to ensure the Artifacts handled by the sharding controller point
to a unique endpoint.
In addition, Source object kinds which have a dependency on another
kind (i.e. a HelmChart on a HelmRepository) need to have the same
labels applied to work as expected.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This uses the newly introduced helper from runtime, which also
configures the logger for `klog`.
Resulting in all logs now being properly formatted in, even when logged
by internal Kubernetes elements like the leader election or a dynamic
client.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
You can re-enable caching by starting the controller
with the argument '--feature-gates=CacheSecretsAndConfigMaps=true'
Signed-off-by: Mac Chaffee <machaffe@renci.org>
ForceGoGitImplementation ignores the value set for gitImplementation
and ensures that go-git is used for all GitRepository objects.
This can be used to confirm that Flux instances won't break if/when
the libgit2 implementation was to be deprecated.
When enabled, libgit2 won't be initialized, nor will any git2go cgo
code be called.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
1. moved chartRepo.Unload() from reconcileSource() to the defer func in reconcileArtifact to allow caching index in memory
2. added step to init memory cache in reconcileArtifact()
3. added step to save helmrepo index into memory cache in reconcileArtifact()
Signed-off-by: York Chen <ychen@d2iq.com>
Decommission libgit2 unmanaged transport and remove the related feature
gate, making managed transport the default.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Debugging connection issues can be extremely difficult, even more so at scale or when
concurrent connections are required to trigger specific issues.
Changes:
- Add a correlation identifier for each reconciliation, which allows for greater traceability when
going through all the reconciliation operations - including at transport level.
- Add transportType to segregate HTTP and SSH transport logging.
- SSH operations are now enriched with addr containing server address, and HTTP url.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This new package holds all Helm OCI registry-specific code now so we
have a single location to look for such code which makes it easier to
find yourself around.
Signed-off-by: Max Jonas Werner <mail@makk.es>
When setup of one of the two controller reconciling HelmRepositories
fails, it's now possible to judge from the log which setup call failed
by regarding the "type" log field.
Signed-off-by: Max Jonas Werner <mail@makk.es>
* Add OCI Helm support
* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.
Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
OptimizedGitClones decreases resource utilization for GitRepository
reconciliations. It supports both go-git and libgit2 implementations
when cloning repositories using branches or tags.
This is an opt-out feature, which can be disabled by starting the
controller with the argument '--feature-gates=OptimizedGitClones=false'.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The new flag allow users to set the list of hostkey algorithms
to use for ssh connections, enabling them to ensure specific
are/aren't used.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
If implemented this will:
- enable the helmCharts dependency manager to use the helm in memry
cache to retrieve reconciled HelmRepositories indexes.
- record cache events.
Signed-off-by: Soule BA <soule@weave.works>
As suggested by @pjbgf
Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Signed-off-by: Peter Gundel <mail@petergundel.de>
Matheus Pimenta
Stefan Prodan
Sunny
Hidde Beydals
Soule BA
Max Jonas Werner
Hidde Beydals
Mac Chaffee
Sanskar Jaiswal
Paulo Gomes
Somtochi Onyekwere
York Chen
Max Jonas Werner
edwinwalela
Peter Gundel