Add a new field `.spec.ref.name` which points to a Git reference which
enables checking out to a particular commit pointed to by the specified
reference.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
This covers the edge case in which a user creates a GitRepository CR
referencing an empty Git repository. Currently, the controller will panic
in this situation since the returned commit pointer is nil.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
ForceGoGitImplementation ignores the value set for gitImplementation
and ensures that go-git is used for all GitRepository objects.
This can be used to confirm that Flux instances won't break if/when
the libgit2 implementation was to be deprecated.
When enabled, libgit2 won't be initialized, nor will any git2go cgo
code be called.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Reorders the object suspended check in all the reconcilers to allow
deletion of objects when they are suspended. Objects used to get stuck
on delete because the finalizers were not getting removed due to the
suspended state.
Adds a generic test for all the reconcilers to check if a suspended
source object can be delete.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Replace content config checksum with explicit artifact content config
observations. It makes the observations of the controller more
transparent and easier to debug.
Introduces `observedIgnore`, `observedRecurseSubmodules` and
`observedInclude` status fields.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Context in the reconcilers were overwritten earlier after adding new
log field `reconcileID` in the logger. Since the `reconcileID` is now
set by controller-runtime, this is no longer needed. The logger in the
context already has the field set and when the context is passed to
other functions, they too have the logger with the reconcileID set.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Decommission libgit2 unmanaged transport and remove the related feature
gate, making managed transport the default.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
- Update the comments around artifact retention fields in Storage.
- Update the comments around reconcileStorage regarding artifact
retention and garbage collection.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
GitRepository introduced correlation ID to improve
transport level logging. This change aligns the other
reconcilers to the same approach.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Debugging connection issues can be extremely difficult, even more so at scale or when
concurrent connections are required to trigger specific issues.
Changes:
- Add a correlation identifier for each reconciliation, which allows for greater traceability when
going through all the reconciliation operations - including at transport level.
- Add transportType to segregate HTTP and SSH transport logging.
- SSH operations are now enriched with addr containing server address, and HTTP url.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Some scenarios could lead a goroutine to be running indefinetely within managed ssh.
Previously between the two git operations, the reconciliation
could take twice the timeout set for the Flux object.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Injects transport and auth options at the transport level directly to
bypass the inbuilt credentials callback because of it's several
shortcomings. Moves some of the pre-existing logic from the reconciler
to the checkout implementation.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
gitCheckout() should return typed errors only. This helps prevent
error type assertions by the caller to determine how to handle the
error. gitCheckout() also sets the appropriate conditions associated
with the error, if any.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.
Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.
Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.
The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce a new field in the GitRepositoryReconciler to set the enabled
features. This makes it test friendly compared to using global flags for
setting and checking flags in the tests.
Enable default feature gates in all the GitRepo reconciler tests.
Add test cases for reconcileSource() to test the behavior of optimized
git clone when the Repo is ready and not ready. This ensures that the
full reconciliation is not skipped when GitRepo is not ready.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
For gradual migration to Generic error, update only the GitRepo
reconciler to use Generic error.
Replace the Waiting error for git no change scenario with a Generic
error with proper no-op, early return, error configurations. This
ensures that the no-op only results in log and K8s native events at
normal level.
Fixes a reconciliation issue when recovering from a failure state (with
previous success state and artifact in the storage) and optimized git
clone feature is on, which results in failure to persist as the git
optimization prevented full reconciliation due to already existing
artifact and removal of failure negative conditions on the object
status. In order to allow failure recovery, the git clone optimizations
are now only applied when the object is already in a ready state.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
OptimizedGitClones decreases resource utilization for GitRepository
reconciliations. It supports both go-git and libgit2 implementations
when cloning repositories using branches or tags.
This is an opt-out feature, which can be disabled by starting the
controller with the argument '--feature-gates=OptimizedGitClones=false'.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
No-op reconciliations are very inefficient, as they carry out
a full clone operation of the target repository even when
no changes have taken place.
This change will execute a remote-ls operation, and cancel
the clone operation if the remote tip commit is still the same
as the one observed on the last reconcilation. In such cases,
an git.NoChangesError is returned.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Add two new flags to enable users to configure exponential
back-off for Flux objects. The default values are now
set to 750ms for minimum retry time, and 15min for max.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
notify() is used to emit events for new artifact and failure recovery
scenarios. It's implemented in all the reconcilers.
Previously, when there used to be a failure due to any reason, on a
subsequent successful reconciliation, no notification was sent to
indicate that the failure has been resolved.
With notify(), the old version of the object is compared with the new
version of the object to determine if all, if any, of the failures have
been resolved and a notification is sent. The notification message is
the same that's sent in usual successful source reconciliation message
about stored artifact.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
We try to avoid affecting the source reconciliation when there's a
garbage collection related failure.
The event logging was resulting in events and notifications related to
GC failure when the artifact directory isn't created in the first
reconciliation of an object.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce two new flags to configure the ttl of an artifact and the max
no. of files to retain for an artifact. Modify the gc process to
consider the options and use timeouts to prevent the controller from
hanging.
This helps in situations when the SC has already garbage collected the
current artifact but the advertised artifact url is still the same,
which leads to the server returning a 404.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Introduce separate positive polarity conditions which are used to set
Ready condition. Move the "artifact stored" ready condition into
ArtifactInStorage positive polarity condition. If ArtifactInStorage is
True and there's no negative polarity condition present, the Ready
condition is summarized with ArtifactInStorage condition value.
Also, update the priorities of the conditions. ArtifactInStorage has
higher priority than SourceVerfied condition. If both are present, the
Ready condition will have ArtifactInStorage.
The negative polarity conditions are reordered to have the most likely
actual cause of failure condition the highest priority, for example
StorageOperationFailed, followed by the conditions that are reconciled
first in the whole reconciliation so as to prioritize the first failure
which may be the cause of subsequent failures.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
The GitRepository object with included artifact should not stall when
the included artifact is not available since there's no way to signal a
reconciliation when the included artifact becomes available. The
reconciliation should fail and retry until the included artifact becomes
available.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Sanskar Jaiswal
Hidde Beydals
Aurel Canciu
Stefan Prodan
Sunny
Paulo Gomes
Somtochi Onyekwere
Santosh Kaluskar