This allows using the condition checker as a test helper with proper
test like assertion failure and stacktrace.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Reorders the object suspended check in all the reconcilers to allow
deletion of objects when they are suspended. Objects used to get stuck
on delete because the finalizers were not getting removed due to the
suspended state.
Adds a generic test for all the reconcilers to check if a suspended
source object can be delete.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
When a custom CA certificate is provided in a Secret's `caCert` field
referenced in `HelmRelease.spec.secretRef` then that CA cert is now
added to the list of system certificates instead of it replacing the
system certificates. This makes HelmRepositories work in mixed
environments where charts are pulled from both, a public repository
and a private repository (e.g. through a chart dependency).
The test that is added as part of this change will fail without the
change and passes with it.
closes#866closesfluxcd/helm-controller#519
Signed-off-by: Max Jonas Werner <max@e13.dev>
1. moved chartRepo.Unload() from reconcileSource() to the defer func in reconcileArtifact to allow caching index in memory
2. added step to init memory cache in reconcileArtifact()
3. added step to save helmrepo index into memory cache in reconcileArtifact()
Signed-off-by: York Chen <ychen@d2iq.com>
Refactor the OCI HelmRepo reconciler to remove extra custom status
conditions and manage Ready, Reconciling and Stalled conditions within
the reconciler, in-line.
The internal/reconcile/summarize package uses the patch helper
conditions summary before patching which results in overwriting the
Ready condition with Reconciling condition as it's a negative polarity
condition.
For OCI HelmRepository, since it's a simple reconciler, it'd be better
to not introduce extra conditions and utilize the three base conditions
to represent the status. In order to have the same consistent status
conditions, a new summarization is written within the reconciler based
on the context. It takes into consideration a lot of the details from
the internal/reconcile/summarize package and handles certain scenarios
in context specific ways. All the result and error abstractions are
removed since they are only needed when using internal/reconcile
package.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
In TestHelmRepositoryReconciler_ReconcileTypeUpdatePredicateFilter, when
the type of HelmRepo is updated and immediately checked for the object
to be ready, if the check happens before the client cache is updated, it
results in observing the object to be ready in the previous generation.
This results in status check failure:
```
[Check-FAIL]: [Ready condition must be False when the ObservedGeneration is less than the object Generation, Ready condition must be False when any of the status condition's ObservedGeneration is less than the object Generation: [Ready ArtifactInStorage]]
```
Explicitly look for the object with the next generation to prevent such
failure.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
* Add OCI Helm support
* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.
Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This change prevents Reconciling and ArtifactOutdated conditions to be
set on HelmRepo when the checksum of a cached repo index changes.
Adds some tests to ensure that when the repo index is cached, the
revision and checksum of the returned artifact are the same as on the
existing object status.
Also adds checks for the returned artifact and chartRepo from
reconcileSource, to ensure that chartRepo is populated and the checksum
of a new potential artifact is always empty, as it's populated when the
artifact is written in the storage.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
I assume using "interval" for timeouts was an accident and "timeout" was
actually meant to be used. This also fixes flakiness of tests.
Signed-off-by: Alexander Block <ablock84@gmail.com>
This fixes the immediate issue of the nil pointer dereference but we
still haven't isolated the actual cause of the size being nil to begin
with. This is ongoing work and as soon as we have boiled that down to
the simplest case we will provide a regression test for that case.
closes#680
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Hidde Beydals <hiddeco@users.noreply.github.com>
Azure SDK dependencies cannot be updated, as this requires us to move to
Go 1.18.
- cloud.google.com/go/storage to v1.22.0
- github.com/ProtonMail/go-crypto to v0.0.0-20220407094043-a94812496cf5
- github.com/darkowlzz/controller-check to v0.0.0-20220325122359-11f5827b7981
- github.com/elazarl/goproxy to v0.0.0-20220403042543-a53172b9392e
- github.com/fluxcd/pkg/gittestserver to v0.5.2
- github.com/go-logr/logr to v1.2.3
- github.com/minio/minio-go/v7 to v7.0.24
- github.com/onsi/gomega to v1.19.0
- golang.org/x/crypto to v0.0.0-20220411220226-7b82a4e95df4
- google.golang.org/api to v0.74.0
Signed-off-by: Hidde Beydals <hello@hidde.co>
As suggested by @pjbgf
Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Signed-off-by: Peter Gundel <mail@petergundel.de>
notify() is used to emit events for new artifact and failure recovery
scenarios. It's implemented in all the reconcilers.
Previously, when there used to be a failure due to any reason, on a
subsequent successful reconciliation, no notification was sent to
indicate that the failure has been resolved.
With notify(), the old version of the object is compared with the new
version of the object to determine if all, if any, of the failures have
been resolved and a notification is sent. The notification message is
the same that's sent in usual successful source reconciliation message
about stored artifact.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce two new flags to configure the ttl of an artifact and the max
no. of files to retain for an artifact. Modify the gc process to
consider the options and use timeouts to prevent the controller from
hanging.
This helps in situations when the SC has already garbage collected the
current artifact but the advertised artifact url is still the same,
which leads to the server returning a 404.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Update alll the other reconcilers similar to the GitRepository
reconcilers to introduce positive condition ArtifactInStorage and
reorder the status conditions.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Reuses the same transport across different helm chart downloads,
whilst resetting the tlsconfig to avoid cross-contamination.
Crypto material is now only processed in-memory and does not
touch the disk.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This adds a Size field to Artifacts, which reflects the number of bytes
written to the artifact when it's being archived.
Signed-off-by: Kevin McDermott <bigkevmcd@gmail.com>
- Update v1beta2 API descriptions and reconciling messages to be
consistent.
- Replace 'download' with 'fetch'. Since the status condition for
download failure is called FetchFailed, using the term 'fetch' makes
the messaging more consistent.
- Replace `BucketOperationSucceed` with `BucketOperationSucceeded` and
generate api docs.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
summarizeAndPatch() was used by all the reconcilers with their own
object type. This creates a generic SummarizeAndPatch helper that takes
a conditions.Setter object and performs the same operations. All the
reconcilers are updated to use SummarizeAndPatch(). The process of
summarize and patch can be configured using the HelperOptions.
Introduce ResultProcessor to allow injecting middlewares in the
SummarizeAndPatch process.
Introduce RuntimeResultBuilder to allow defining how the reconciliation
result is computed for specific reconciler. This enabled different
reconcilers to have different meanings of the reconciliation results.
Introduce Conditions in summary package to store all the status
conditions related information of a reconciler. This is passed to
SummarizeAndPatch() to be used for summary and patch calculation.
Remove all the redundant summarizeAndPatch() tests per reconciler.
Add package internal/object containing helpers for interacting with
runtime.Object needed by the generic SummarizeAndPatch().
Add tests for ComputeReconcileResult().
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Consolidate BuildRuntimeResult() into summarizeAndPatch() to simplify
where the results are computed, summarized and patched.
Move the event recording and logging of context specific errors into
RecordContextualError() and call it in summarizeAndPatch().
Introduce Waiting error for wait and requeue scenarios. Update
ComputeReconcileResult() and RecordContextualError() to consider Waiting
error.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Remove ArtifactUnavailable condition and use Reconciling condition to
convey the same.
- Make Reconciling condition affect the ready condition.
- Introduce summarizeAndPatch() to calculate the final status conditions
and patch them.
- Introduce reconcile() to iterate through the sub-reconcilers and
execute them.
- Simplify logging and event recording
- Introduce controller-check/status checks to assert that the status
conditions are valid at the end of the tests.
- Create variables for various condition groups: owned conditions, ready
dependencies and ready dependencies negative.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit rewrites the `HelmRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying to
adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- Refactoring of some Helm elements to make them easier to use within
the new reconciler logic.
- Integration tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.
Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- Ensure the proper path is garbage collected for libgit2 repositories,
as the `Path` method on the repository object returns the `.git`
directory, and not the root path.
- Ensure the Helm test server does not get swapped during tests,
with as side-effect that no obsolete temporary directories remain.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit upgrades the `controller-runtime` dependency to `v0.7.0`,
including all changes required to make all wiring work again.
- Upgrade `runtime` to v0.6.0 to include `controller-runtime` changes.
- Loggers have been removed from the reconcilers and are now retrieved
from the `context.Context` passed to the `Reconcile` method and
downwards functions.
- Logger configuration flags are now bound to the flag set using
`BindFlags` from `runtime/logger`, ensuring the same contract across
GitOps Toolkit controllers, and the `--log-json` flag has been
deprecated in favour of the `--log-encoding=json` default.
- The `ChangePredicate` from `runtime` has changed to a
`ReconcilateAtChangedPredicate`, and is now chained with the
`GenerationChangedPredicate` from `controller-runtime` using
`predicate.Or`.
- Signatures that made use of `runtime.Object` have changed to
`client.Object`, removing the requirement to e.g. call
`runtime.Object#Object`.
- The `client.MatchingField` function was deprecated, and has been
replaced with `client.MatchingFields{}`.
- The `leader-election-role` was changed, as leader election now works
via the `coordination/v1` API.
Other notable changes:
- `util.ObjectKey` was added to easily construct a `client.ObjectKey` /
`types.NamespacedName` from a `metav1.Object`.
- The `SourceIndexKey` constant has been split out into
`{GitRepository,HelmRepository,Bucket}IndexKey` constants.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit ensures that resources will only return early if they are
already in a `Ready==True` state. If not, but the status object somehow
still reports that it has an artifact, the reconciliation will continue
to ensure and/or guarantee state, and to prevent a deadlock from
happening.
Sunny
Max Jonas Werner
York Chen
Stefan Prodan
Soule BA
Max Jonas Werner
Eng Zer Jun
Alexander Block
Max Jonas Werner
Hidde Beydals
Peter Gundel
Sanskar Jaiswal
Paulo Gomes
Kevin McDermott
Aurel Canciu