In TestHelmRepositoryReconciler_ReconcileTypeUpdatePredicateFilter, when
the type of HelmRepo is updated and immediately checked for the object
to be ready, if the check happens before the client cache is updated, it
results in observing the object to be ready in the previous generation.
This results in status check failure:
```
[Check-FAIL]: [Ready condition must be False when the ObservedGeneration is less than the object Generation, Ready condition must be False when any of the status condition's ObservedGeneration is less than the object Generation: [Ready ArtifactInStorage]]
```
Explicitly look for the object with the next generation to prevent such
failure.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- fluxcd/pkg/apis/meta v0.14.0
- fluxcd/pkg/runtime v0.16.0
- k8s.io/* v0.24.0
- helm.sh/helm/v3 v3.9.0-rc.1 (required by breaking changes in Kubernetes 1.24)
Note that fluxcd/pkg/runtime v0.16 comes with support for Kubernetes API Priority and Fairness feature.
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
This new package holds all Helm OCI registry-specific code now so we
have a single location to look for such code which makes it easier to
find yourself around.
Signed-off-by: Max Jonas Werner <mail@makk.es>
When setup of one of the two controller reconciling HelmRepositories
fails, it's now possible to judge from the log which setup call failed
by regarding the "type" log field.
Signed-off-by: Max Jonas Werner <mail@makk.es>
`loginOptionFromSecret` now derives username/password from a docker
config stored in Secrets of type "kubernetes.io/dockerconfigjson".
Signed-off-by: Max Jonas Werner <mail@makk.es>
When the Secret referenced in an OCI HelmRepository doesn't contain a
username and password, the controller doesn't panic, anymore.
Signed-off-by: Max Jonas Werner <mail@makk.es>
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.
Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.
Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.
The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce concrete and partial commits. Concrete commits have all the
information from remote including the hash and commit content. Partial
commits are based on locally available copy of a repo, they may only
contain the commit hash and reference.
IsConcreteCommit() can be used to find out if a given commit is based on
local information or full remote repo information.
Update go-git and libgit2 branch/tag clone optimization to return a
partial commit and no error.
Update and simplify the go-git and libgit2 tests for the same.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce a new field in the GitRepositoryReconciler to set the enabled
features. This makes it test friendly compared to using global flags for
setting and checking flags in the tests.
Enable default feature gates in all the GitRepo reconciler tests.
Add test cases for reconcileSource() to test the behavior of optimized
git clone when the Repo is ready and not ready. This ensures that the
full reconciliation is not skipped when GitRepo is not ready.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
For gradual migration to Generic error, update only the GitRepo
reconciler to use Generic error.
Replace the Waiting error for git no change scenario with a Generic
error with proper no-op, early return, error configurations. This
ensures that the no-op only results in log and K8s native events at
normal level.
Fixes a reconciliation issue when recovering from a failure state (with
previous success state and artifact in the storage) and optimized git
clone feature is on, which results in failure to persist as the git
optimization prevented full reconciliation due to already existing
artifact and removal of failure negative conditions on the object
status. In order to allow failure recovery, the git clone optimizations
are now only applied when the object is already in a ready state.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Add Generic error in RuntimeResultBuilder and ComputeReconcileResult
implementation with consideration to the error configurations.
Safeguards are added in the runtime result builder to ensure default
requeue after interval is set when is's set to zero or unset.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
ErrorActionHandler processes the reconciliation error results based on
their configurations. It performs actions like logging and event
recording based on the error configuration. More actions can be
accommodated in the future with more error configurations.
It can be a replacement for RecordContextualError() which does the same
operations but can't be configured much.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Generic error is an attempt to avoid creating new error type for every
new unique scenario. It can be used to configure and build custom error
handling behavior, logging and event recording at present.
Contextual errors, Stalling and Waiting error, have special meaning for
the reconciliation results. But the Event error type can be replaced
with Generic error with some specific configurations. The Event error
is kept for a gradual migation to Generic error. Similarly, the Generic
error can be used to easily create new error handling behaviors.
The error Config can be used to configure any of the errors, including
contextual errors, without altering their contextual meaning, to modify
how they are handled.
The error constructors configure the errors with common default
configurations. These configurations can be modified to alter the
behavior.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
* Add OCI Helm support
* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.
Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
Hashed known_hosts was previously only supported when using
go-git. Now both Git implementations benefit from this
features, and the code coverage across them can ensure no
future regression.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Both libgit2 and go-git now have tests that confirms
support for the supported authentication algorithms.
This test was created previously and is no longer
needed.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Connection caching was a feature created to resolve
upstream issues raised from concurrent ssh connections.
Some scenarios were based on multiple key exchange
operations happening at the same time.
This PR removes the connection caching, and instead:
- Services Session.StdoutPipe() as soon as possible,
as it is a known source of blocking SSH connections.
- Reuse SSH connection within the same subtransport,
eliminating the need for new handshakes when talking
with the same server.
- Simplifies the entire transport logic for better
maintainability.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
OptimizedGitClones decreases resource utilization for GitRepository
reconciliations. It supports both go-git and libgit2 implementations
when cloning repositories using branches or tags.
This is an opt-out feature, which can be disabled by starting the
controller with the argument '--feature-gates=OptimizedGitClones=false'.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
getBlankRepoAndRemote's callers are responsible for the disposal
of the returned objects. However, the caller does not expect to
need to dispose objects when err != nil, which may result to memory
leaks.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
No-op reconciliations are very inefficient, as they carry out
a full clone operation of the target repository even when
no changes have taken place.
This change will execute a remote-ls operation, and cancel
the clone operation if the remote tip commit is still the same
as the one observed on the last reconcilation. In such cases,
an git.NoChangesError is returned.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The new flag allow users to set the list of hostkey algorithms
to use for ssh connections, enabling them to ensure specific
are/aren't used.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The previous version would verify git status at time of `make verify` execution, but before all its dependencies executed.
If one of the dependencies resulted in a dirty git, the check would still pass.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Sunny