This wraps the errors which are returned instead of logging them, as
the returned error is logged at the end of the reconcile run.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit consolidates the `DownloadFailed` and `CheckoutFailed`
Condition types into a new more generic `FetchFailed` type to simplify
the API and observations by consumers.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit rewrites the `BucketReconciler` to new standards, while
implementing the newly introduced Condition types, and trying to
adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- Refactor of reconciler logic, including more efficient detection of
changes to bucket objects by making use of the etag data available,
and downloading of object files in parallel with a limited number of
workers (4).
- Integration tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
The problem with `GetInterval()` was that the returned type was of
`metav1.Duration`, while almost anywhere it was used, a type of
`time.Duration` was requested. The result of this was that we had to
call `GetInterval().Duration` all the time, which would become a bit
cumbersome after awhile.
To prevent this, we introduce a new `GetRequeueAfter() time.Duration`
method, which both results the right type, and bears a name that is
easier to remember where the value is used most; while setting the
`Result.RequeueAfter` during reconcile operations.
The introduced of this method deprecates `GetInterval()`, which should
be removed in a future MINOR release.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit changes the checksum method which is used to calculate the
revision of a Bucket source, so that the file paths are taken into
account and directory structure changes can be observed.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit makes the filtering applied during the archiving
configurable by introducing an optional `ArchiveFileFilter`
callback argument and a `SourceIgnoreFilter` implementation.
`SourceIgnoreFilter` filters out files matching
sourceignore.VCSPatterns and any of the provided patterns.
If an empty gitignore.Pattern slice is given, the matcher is set to
sourceignore.NewDefaultMatcher.
The `GitRepository` now loads the ignore patterns before archiving
the repository contents by calling `sourceignore.LoadIgnorePatterns`
and other helpers. The loading behavior is **breaking** as
`.sourceignore` files in the (subdirectories of the) repository are
now still taken into account if `spec.ignore` for a resource is
defined, overwriting is still possible by creating an overwriting
rule in the `spec.ignore` of the resource.
This change also makes it possible for the `BucketReconciler` to not
configure a callback at all and prevent looking for ignore
matches twice. To finalize the bucket refactor, a change to the
reconciler has been made to look for a `.sourceignore` file in
the root of the bucket to provide an additional way of configuring
(global) exclusions.
Signed-off-by: Hidde Beydals <hello@hidde.co>
To include a bug fix to the `ReconcilateAtChangedPredicate`
and renaming to `ReconcileRequestedPredicate`.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit upgrades the `controller-runtime` dependency to `v0.7.0`,
including all changes required to make all wiring work again.
- Upgrade `runtime` to v0.6.0 to include `controller-runtime` changes.
- Loggers have been removed from the reconcilers and are now retrieved
from the `context.Context` passed to the `Reconcile` method and
downwards functions.
- Logger configuration flags are now bound to the flag set using
`BindFlags` from `runtime/logger`, ensuring the same contract across
GitOps Toolkit controllers, and the `--log-json` flag has been
deprecated in favour of the `--log-encoding=json` default.
- The `ChangePredicate` from `runtime` has changed to a
`ReconcilateAtChangedPredicate`, and is now chained with the
`GenerationChangedPredicate` from `controller-runtime` using
`predicate.Or`.
- Signatures that made use of `runtime.Object` have changed to
`client.Object`, removing the requirement to e.g. call
`runtime.Object#Object`.
- The `client.MatchingField` function was deprecated, and has been
replaced with `client.MatchingFields{}`.
- The `leader-election-role` was changed, as leader election now works
via the `coordination/v1` API.
Other notable changes:
- `util.ObjectKey` was added to easily construct a `client.ObjectKey` /
`types.NamespacedName` from a `metav1.Object`.
- The `SourceIndexKey` constant has been split out into
`{GitRepository,HelmRepository,Bucket}IndexKey` constants.
Signed-off-by: Hidde Beydals <hello@hidde.co>
As GCP's S3 interoperability does not implement the ListObjectsV2
implementation.
Ref: https://cloud.google.com/storage/docs/interoperability
> Note: While most actions are interoperable with the Amazon S3 V2 SDK,
> listing objects can only be performed using the Amazon S3 V1 list
> objects method.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Use SetResourceCondition as a generic method to set conditions for CRs,
implmeneting the ObjectWithStatusConditions interface used as input
type.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
Updates to use metav1.Condition type and removes references for
deprecated corev1.Condition* constants and uses the new k8s api/meta
helpers in place of the old pkg/apis/meta types.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
This commit ensures that resources will only return early if they are
already in a `Ready==True` state. If not, but the status object somehow
still reports that it has an artifact, the reconciliation will continue
to ensure and/or guarantee state, and to prevent a deadlock from
happening.
Hidde Beydals
Somtochi Onyekwere
Stefan Prodan
Aurel Canciu
Philip Laine