- Attempt short-circuiting clone only when the artifact is already in the
storage.
- A successful no-op clone need not return an error, but a partial
commit which contains only a hash + reference.
- On no-op clone, reconcileSource() populates the source build dir by
copying the existing artifact and lets the reconciliation continue.
- Reconciliation is not skipped to allow other subreconcilers to operate
on other parts of GitRepo object like include, ignore, etc, when
attributes associated with them change but the remote repo has not
changed.
- Add a function IsConcreteCommit() to differentiate between partial and
concrete commit.
- Update and simplify go-git and libgit2 no-op clone tests.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
"identity.pub" is referenced multiple times in CRDs and docs. This secret
is however never used in any place. Instead, the public key is derived from
the "identity" private key.
This commit/PR removes all traces of "identity.pub" from v1beta2 CRDs and
docs.
Signed-off-by: Alexander Block <ablock84@gmail.com>
Avoid validating (and thus loading) indexes if the checksum already exists in storage.
In other words, if the YAML is identical to the Artifact in storage, the reconciliation should
be a no-op, and therefore can short-circuit long/heavy operations.
Co-authored-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
If implemented, will provide users with a way to cache index files.
This addresses issues where the index file is loaded and unmarshalled in
concurrent reconciliation resulting in a heavy memory footprint.
The caching strategy used is cache aside, and the cache is a k/v store
with expiration.
The cache number of entries and ttl for entries are configurable.
The cache is optional and is disabled by default
Signed-off-by: Soule BA <soule@weave.works>
Introduce separate positive polarity conditions which are used to set
Ready condition. Move the "artifact stored" ready condition into
ArtifactInStorage positive polarity condition. If ArtifactInStorage is
True and there's no negative polarity condition present, the Ready
condition is summarized with ArtifactInStorage condition value.
Also, update the priorities of the conditions. ArtifactInStorage has
higher priority than SourceVerfied condition. If both are present, the
Ready condition will have ArtifactInStorage.
The negative polarity conditions are reordered to have the most likely
actual cause of failure condition the highest priority, for example
StorageOperationFailed, followed by the conditions that are reconciled
first in the whole reconciliation so as to prioritize the first failure
which may be the cause of subsequent failures.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Introduce new condition StorageOperationFailedCondition for all the
failures related to the storage. It is a negative polarity condition and
is considered in computing summary of reconciliation.
Also, introduce more granular event reasons related to
StorageOperationFailedCondition for precise reasoning behind failures.
These replace the vague StorageOperationFailedReason.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This adds a Size field to Artifacts, which reflects the number of bytes
written to the artifact when it's being archived.
Signed-off-by: Kevin McDermott <bigkevmcd@gmail.com>
Status content could be very long compare to other fields. Moving it to
the end helps improve the visibility of other fields.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Update v1beta2 API descriptions and reconciling messages to be
consistent.
- Replace 'download' with 'fetch'. Since the status condition for
download failure is called FetchFailed, using the term 'fetch' makes
the messaging more consistent.
- Replace `BucketOperationSucceed` with `BucketOperationSucceeded` and
generate api docs.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit rewrites the `HelmRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying to
adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- Refactoring of some Helm elements to make them easier to use within
the new reconciler logic.
- Integration tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit consolidates the `DownloadFailed` and `CheckoutFailed`
Condition types into a new more generic `FetchFailed` type to simplify
the API and observations by consumers.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit rewrites the `BucketReconciler` to new standards, while
implementing the newly introduced Condition types, and trying to
adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- Refactor of reconciler logic, including more efficient detection of
changes to bucket objects by making use of the etag data available,
and downloading of object files in parallel with a limited number of
workers (4).
- Integration tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit consolidates the `DownloadFailed` and `CheckoutFailed`
Condition types into a new more generic `FetchFailed` type to simplify
the API and observations by consumers.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit rewrites the `GitRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying
to adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- First (integration) tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Embedding runtime.Object in Source interface makes the Source type more
useful to interact with k8s API machinery.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
The problem with `GetInterval()` was that the returned type was of
`metav1.Duration`, while almost anywhere it was used, a type of
`time.Duration` was requested. The result of this was that we had to
call `GetInterval().Duration` all the time, which would become a bit
cumbersome after awhile.
To prevent this, we introduce a new `GetRequeueAfter() time.Duration`
method, which both results the right type, and bears a name that is
easier to remember where the value is used most; while setting the
`Result.RequeueAfter` during reconcile operations.
The introduction of this method deprecates `GetInterval()`, which
should be removed in a future MINOR release.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Co-authored-by: Hidde Beydals <hello@hidde.co>
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.
Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit introduces new Condition types to the v1beta1 API,
facilitating easier observation of (potentially) problematic state for
end-users.
- `ArtifactUnavailableCondition`: indicates there is no artifact
available for the resource. This Condition should be set by the
reconciler as soon as it observes the absence of an artifact for a
source.
- `CheckoutFailedCondition`: indicates a transient or persistent
checkout failure. This Condition should be set by the reconciler as
soon as it observes a Git checkout failure, including any
prerequisites like the unavailability of the referenced Secret used
for authentication. It should be deleted as soon as a successful
checkout has been observed again.
- `SourceVerifiedCondition`: indicates the integrity of the source has
been verified. The Condition should be set to True or False by the
reconciler based on the result of the integrity check.
If there is no verification mode and/or secret configured, the
Condition should be removed.
- `IncludeUnavailableCondition`: indicates one of the referenced
includes is not available. This Condition should for example be set
by the reconciler when the include does not exist, or does not have
an artifact. If the includes become available, it should be deleted.
- `ArtifactOutdatedCondition`: indicates the current artifact of the
source is outdated. This Condition should for example be set by the
reconciler when it notices there is a newer revision for an artifact,
or the previously included artifacts differ from the current available
ones. The Condition should be removed after writing a new artifact
to the storage.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit introduces a v1beta2 API package for the staged breaking
changes around conditions and general usage of the API objects.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Co-authored-by: Hidde Beydals <hello@hidde.co>
Sunny
Alexander Block
Paulo Gomes
Soule BA
Hidde Beydals
Kevin McDermott