Injects transport and auth options at the transport level directly to
bypass the inbuilt credentials callback because of it's several
shortcomings. Moves some of the pre-existing logic from the reconciler
to the checkout implementation.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
- cloud.google.com/go/storage to v1.22.1
- github.com/ProtonMail/go-crypto to v0.0.0-20220517143526-88bb52951d5b
- github.com/distribution/distribution/v3 to v3.0.0-20220516112011-c202b9b0d7b7
- github.com/docker/cli to v20.10.16+incompatible
- github.com/fluxcd/gitkit to v0.5.1
- github.com/fluxcd/pkg/helmtestserver to v0.7.2
- github.com/fluxcd/pkg/ssh to v0.4.0
- github.com/phayes/freeport to v0.0.0-20220201140144-74d24b5ae9f5
- github.com/prometheus/client_golang to v1.12.2
- golang.org/x/crypto to v0.0.0-20220518034528-6f7dac969898
- golang.org/x/net to v0.0.0-20220524220425-1d687d428aca
- golang.org/x/sync to v0.0.0-20220513210516-0976fa681c29
- google.golang.org/api to v0.81.0
- sigs.k8s.io/cli-utils to v0.31.1
Signed-off-by: Hidde Beydals <hello@hidde.co>
gitCheckout() should return typed errors only. This helps prevent
error type assertions by the caller to determine how to handle the
error. gitCheckout() also sets the appropriate conditions associated
with the error, if any.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Users environmental proxy settings should not impact
the execution of the tests. The changes override both
HTTP_PROXY and HTTPS_PROXY to ensure that is the case.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Some test cases rely on checksum to match in order to pass.
Those checksums were calculated based on file headers which
contain their file modes.
In Ubuntu, the umask is set to 002 by default, resulting in
the tests files having different permissions then when the
same files are cloned on another Linux machine with umask
set to 022.
This change ensures that the files are always set (to 0644 and
the directories to 0755) before running the aforementioned tests.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
The observed generation must be set only when actual observation is
made. When an actual observation is made, some conditions are set on the
object. Introduce a helper function
addPatchOptionWithStatusObservedGeneration() to set the patcher option
WithStatusObservedGeneration only when there's any condition in the
status.
Updates the existing tests that depended on this behavior.
This fixes the issue where the observed generation is set by the patcher
when a reconciler does an early return for setting the finalizers only.
With this, the observed generation will be updated only when some
observations are made on the object based on the usual rules of success
result, no error, ignore error and stalled condition.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
In TestHelmRepositoryReconciler_ReconcileTypeUpdatePredicateFilter, when
the type of HelmRepo is updated and immediately checked for the object
to be ready, if the check happens before the client cache is updated, it
results in observing the object to be ready in the previous generation.
This results in status check failure:
```
[Check-FAIL]: [Ready condition must be False when the ObservedGeneration is less than the object Generation, Ready condition must be False when any of the status condition's ObservedGeneration is less than the object Generation: [Ready ArtifactInStorage]]
```
Explicitly look for the object with the next generation to prevent such
failure.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- fluxcd/pkg/apis/meta v0.14.0
- fluxcd/pkg/runtime v0.16.0
- k8s.io/* v0.24.0
- helm.sh/helm/v3 v3.9.0-rc.1 (required by breaking changes in Kubernetes 1.24)
Note that fluxcd/pkg/runtime v0.16 comes with support for Kubernetes API Priority and Fairness feature.
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
This new package holds all Helm OCI registry-specific code now so we
have a single location to look for such code which makes it easier to
find yourself around.
Signed-off-by: Max Jonas Werner <mail@makk.es>
When setup of one of the two controller reconciling HelmRepositories
fails, it's now possible to judge from the log which setup call failed
by regarding the "type" log field.
Signed-off-by: Max Jonas Werner <mail@makk.es>
`loginOptionFromSecret` now derives username/password from a docker
config stored in Secrets of type "kubernetes.io/dockerconfigjson".
Signed-off-by: Max Jonas Werner <mail@makk.es>
When the Secret referenced in an OCI HelmRepository doesn't contain a
username and password, the controller doesn't panic, anymore.
Signed-off-by: Max Jonas Werner <mail@makk.es>
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.
Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.
Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.
The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce concrete and partial commits. Concrete commits have all the
information from remote including the hash and commit content. Partial
commits are based on locally available copy of a repo, they may only
contain the commit hash and reference.
IsConcreteCommit() can be used to find out if a given commit is based on
local information or full remote repo information.
Update go-git and libgit2 branch/tag clone optimization to return a
partial commit and no error.
Update and simplify the go-git and libgit2 tests for the same.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce a new field in the GitRepositoryReconciler to set the enabled
features. This makes it test friendly compared to using global flags for
setting and checking flags in the tests.
Enable default feature gates in all the GitRepo reconciler tests.
Add test cases for reconcileSource() to test the behavior of optimized
git clone when the Repo is ready and not ready. This ensures that the
full reconciliation is not skipped when GitRepo is not ready.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
For gradual migration to Generic error, update only the GitRepo
reconciler to use Generic error.
Replace the Waiting error for git no change scenario with a Generic
error with proper no-op, early return, error configurations. This
ensures that the no-op only results in log and K8s native events at
normal level.
Fixes a reconciliation issue when recovering from a failure state (with
previous success state and artifact in the storage) and optimized git
clone feature is on, which results in failure to persist as the git
optimization prevented full reconciliation due to already existing
artifact and removal of failure negative conditions on the object
status. In order to allow failure recovery, the git clone optimizations
are now only applied when the object is already in a ready state.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Add Generic error in RuntimeResultBuilder and ComputeReconcileResult
implementation with consideration to the error configurations.
Safeguards are added in the runtime result builder to ensure default
requeue after interval is set when is's set to zero or unset.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
ErrorActionHandler processes the reconciliation error results based on
their configurations. It performs actions like logging and event
recording based on the error configuration. More actions can be
accommodated in the future with more error configurations.
It can be a replacement for RecordContextualError() which does the same
operations but can't be configured much.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Generic error is an attempt to avoid creating new error type for every
new unique scenario. It can be used to configure and build custom error
handling behavior, logging and event recording at present.
Contextual errors, Stalling and Waiting error, have special meaning for
the reconciliation results. But the Event error type can be replaced
with Generic error with some specific configurations. The Event error
is kept for a gradual migation to Generic error. Similarly, the Generic
error can be used to easily create new error handling behaviors.
The error Config can be used to configure any of the errors, including
contextual errors, without altering their contextual meaning, to modify
how they are handled.
The error constructors configure the errors with common default
configurations. These configurations can be modified to alter the
behavior.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
* Add OCI Helm support
* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.
Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
Sanskar Jaiswal
Stefan Prodan
Hidde Beydals
Paulo Gomes
Sunny
Max Jonas Werner
Max Jonas Werner