- Update summarize helper to have patch field owner.
- Updated the controllers to set the patch field owner.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Use the etagIndex to provide more information about the artifact in
NewArtifact events and remove the revision from the event message. The
revision is still kept in the event annotations.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Update v1beta2 API descriptions and reconciling messages to be
consistent.
- Replace 'download' with 'fetch'. Since the status condition for
download failure is called FetchFailed, using the term 'fetch' makes
the messaging more consistent.
- Replace `BucketOperationSucceed` with `BucketOperationSucceeded` and
generate api docs.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
summarizeAndPatch() was used by all the reconcilers with their own
object type. This creates a generic SummarizeAndPatch helper that takes
a conditions.Setter object and performs the same operations. All the
reconcilers are updated to use SummarizeAndPatch(). The process of
summarize and patch can be configured using the HelperOptions.
Introduce ResultProcessor to allow injecting middlewares in the
SummarizeAndPatch process.
Introduce RuntimeResultBuilder to allow defining how the reconciliation
result is computed for specific reconciler. This enabled different
reconcilers to have different meanings of the reconciliation results.
Introduce Conditions in summary package to store all the status
conditions related information of a reconciler. This is passed to
SummarizeAndPatch() to be used for summary and patch calculation.
Remove all the redundant summarizeAndPatch() tests per reconciler.
Add package internal/object containing helpers for interacting with
runtime.Object needed by the generic SummarizeAndPatch().
Add tests for ComputeReconcileResult().
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Update Storage.RemoveAll() and Storage.RemoveAllButCurrent() to return
the details about the deleted items. This helps emit useful information
about garbage collection in the controllers and ignore no-op garbage
collections.
RemoveAll() returns the path of the deleted directory if any.
RemoveAllButCurrent() returns a slice of path of all the deleted items
from a resource's artifact dir.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Consolidate BuildRuntimeResult() into summarizeAndPatch() to simplify
where the results are computed, summarized and patched.
Move the event recording and logging of context specific errors into
RecordContextualError() and call it in summarizeAndPatch().
Introduce Waiting error for wait and requeue scenarios. Update
ComputeReconcileResult() and RecordContextualError() to consider Waiting
error.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Ensure all logged messages start with a lowercase.
- Make some pushed (and logged) events of type `EventTypeTrace` to
prevent them from being sinked to the external event recorder, to
prevent spam.
- Only log if artifact is up-to-date with upstream (instead of pushing
an event).
Signed-off-by: Hidde Beydals <hello@hidde.co>
- Remove ArtifactUnavailable condition and use Reconciling condition to
convey the same.
- Make Reconciling condition affect the ready condition.
- Introduce summarizeAndPatch() to calculate the final status conditions
and patch them.
- Introduce reconcile() to iterate through the sub-reconcilers and
execute them.
- Simplify logging and event recording
- Introduce controller-check/status checks to assert that the status
conditions are valid at the end of the tests.
- Create variables for various condition groups: owned conditions, ready
dependencies and ready dependencies negative.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit rewrites the `HelmRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying to
adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- Refactoring of some Helm elements to make them easier to use within
the new reconciler logic.
- Integration tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- Remove ArtifactUnavailable condition and use Reconciling condition to
convey the same.
- Make Reconciling condition affect the ready condition.
- Introduce summarizeAndPatch() to calculate the final status conditions
and patch them.
- Introduce reconcile() to iterate through the sub-reconcilers and
execute them.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Introduce mock GCP Server to test the gcp bucket client against mocked
gcp server results.
- Add tests for reconcileGCPSource().
- Patch GCPClient.BucketExists() to return no error when the bucket
doesn't exists. This keeps the GCP client compatible with the minio
client.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Add `BucketReconciler.reconcileArtifact` tests based on
`GitRepositoryReconciler.reconcileArtifact` test cases.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit consolidates the `DownloadFailed` and `CheckoutFailed`
Condition types into a new more generic `FetchFailed` type to simplify
the API and observations by consumers.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit rewrites the `BucketReconciler` to new standards, while
implementing the newly introduced Condition types, and trying to
adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- Refactor of reconciler logic, including more efficient detection of
changes to bucket objects by making use of the etag data available,
and downloading of object files in parallel with a limited number of
workers (4).
- Integration tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
The artifacts built in reconcileInclude() should be persisted by writing
it to includes. reconcileArtifact() later adds includes to the git repo
object status and persists it.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Remove ArtifactUnavailable condition and use Reconciling condition to
convey the same.
- Make Reconciling condition affect the ready condition.
- Introduce summarizeAndPatch() to calculate the final status conditions
and patch them.
- Introduce reconcile() to iterate through the sub-reconcilers and
execute them.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
While deleting, patching an object with new status results in "not
found" error because the object is already deleted. The patching
operation first patches the status conditions, the rest of the object
and, at the very end, the rest of the status. When an object is
deleted, the garbage collection results in the artifact in the status
to be updated, resulting in a diff that is attempted to be patched
when the deferred patch runs.
Since the status patching runs at the very end, the object gets deleted
before it can be patched.
Ignore "not found" error while patching when the delete timestamp is
set.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
SourceVerifiedCondition is a normal condition, remove it from negative
polarity conditions. Add SourceVerifiedCondition in patch option
WithOwnedConditions.
Also, Update the signature of reconcileInclude() to remove include
being passed and overwritten in the first line. Include is available
as part of the passed source object.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This wraps the errors which are returned instead of logging them, as
the returned error is logged at the end of the reconcile run.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit consolidates the `DownloadFailed` and `CheckoutFailed`
Condition types into a new more generic `FetchFailed` type to simplify
the API and observations by consumers.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Adds test cases for reconcileArtifact to check if old status
conditions are removed after new artifact is created.
Adds a test case to verify that the latest artifact symlink points to
the created artifact.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This tests the status conditions update in the gitrepository reconciler.
Given a mix of old status conditions, on a successful reconciliation,
the status condition is set to Ready=True.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Use the created artifact server test storage in reconcileInclude
test's GitRepositoryReconciler and cleanup the created storage.
Fix the test assertions to check the copied artifact directories in
the correct path. Also, update the tests to expect artifacts in the
include `toPath` to exist.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Mention the current revision in the up-to-date log message.
- Ensure any error that is "swallowed" (not returned) is logged to
ensure they are visible within the logs, and not just by inspecting
the object.
Signed-off-by: Hidde Beydals <hello@hidde.co>
The problem with `GetInterval()` was that the returned type was of
`metav1.Duration`, while almost anywhere it was used, a type of
`time.Duration` was requested. The result of this was that we had to
call `GetInterval().Duration` all the time, which would become a bit
cumbersome after awhile.
To prevent this, we introduce a new `GetRequeueAfter() time.Duration`
method, which both results the right type, and bears a name that is
easier to remember where the value is used most; while setting the
`Result.RequeueAfter` during reconcile operations.
The introduced of this method deprecates `GetInterval()`, which should
be removed in a future MINOR release.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Fixes error returned from target path validation check and adds more
test cases for TestGitRepositoryReconciler_reconcileArtifact.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
NOTE: This should be amended with the previous commit which has
commented out tests.
Update reconcileSource() to work with the test case where no secret is
set. A minimal auth options is created and used for git checkout.
Update TestGitRepositoryReconciler_verifyCommitSignature() to use the
new git.Commit type.
Update TestGitRepositoryReconciler_reconcileSource_checkoutStrategy to
add skipForImplementation for branch commit test case.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit rewrites the `GitRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying
to adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- First (integration) tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.
Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.
Signed-off-by: Hidde Beydals <hello@hidde.co>
go-git: Include the commit message in the returned commit object.
libgit2: Set the URL in the checkout error.
Add new method Commit.ShortMessage() for returning short commit
message.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This avoids skipping the directories when generating the archive
tarball.
This makes it easier to scan directory trees.
Signed-off-by: Kevin McDermott <kevin@weave.works>
Ensure that directories are included.
This avoids skipping the directories when generating the archive
tarball.
This makes it easier to scan directory trees.
Signed-off-by: Kevin McDermott <kevin@weave.works>
Refactor logic to install helper tools into one function in the
Makefile. Add support for envtest to help install tools like kubectl,
etcd which helps users run tests more conveniently.
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
As the full version can be used as a label value, the full SHA from the
reference takes up too much space from the 63 characters available in
total.
To mitigate against this, we now take a "short" version of the first 12
characters, which was still unique for the Linux kernel in 2019 with
875.000 commits:
http://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#Short-SHA-1
This should be sufficient to safely detect all changes within the
context of operations.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Test case "Setting valid valuesFile attribute" and the tests around it
aren't isolated and most of the time pass because of the results from
the previous tests being re-read as the test expectation match the
previous test results. Failures are very rare to reproduce, even in
the CI they aren't seen but it failed very frequently on my computer,
especially this specific case because unlike the other cases, there is
just one file to be merged, which invalidates the chart result from
the previous cases.
In order to ensure the test wait for the chart to be updated by its
action and not by any other previous updates, status condition message
seems to be the most reliable way, as it also contains the paths of the
files that were merged.
With this change, I could no longer reproduce the failure on my
computer.
Reordering the tests makes this issue more clear.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
By providing the Generation of the object that is getting reconciled
as version metadata to the builder if any custom values files are
defined, the Artifact revision changes if the specification does,
ensuring consumers of the Artifact are able to react to changes in
values (and perform a release).
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit introduces a typed `BuildError` to be returned by
`Builder.Build` in case of a failure.
The `Reason` field in combination with `BuildErrorReason` can be used
to signal (or determine) the reason of a returned error within the
context of the build process.
At present this is used to determine the correct Condition Reason, but
in a future iteration this can be used to determine the negative
polarity condition that should be set to indicate a precise failure to
the user.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Dealing with some loose ends around making observations, and code
style.
The loaded byes of a chart are used as a revision to ensure e.g.
periodic builds with unstable ordering of items do not trigger a false
positive.
Signed-off-by: Hidde Beydals <hello@hidde.co>
With all the logic that used to reside in the `controllers` package
factored into this package, it became cluttered. This commit tries to
bring a bit more structure in place.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit starts wiring the factored out Helm chart build logic into
the reconciler to ensure, validating the API capabilities.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- Ensure the proper path is garbage collected for libgit2 repositories,
as the `Path` method on the repository object returns the `.git`
directory, and not the root path.
- Ensure the Helm test server does not get swapped during tests,
with as side-effect that no obsolete temporary directories remain.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This changes the format of the Artifact checksum from SHA1 to SHA256 to
mitigate chosen-prefix and length extension attacks, and ensures it can
be used to secure content against malicious modifications.
Source consumers (including our own {kustomize,helm}-controllers)
should ensure the SHA256 of a downloaded artifact matches the
advertised checksum before making use of it.
Signed-off-by: Hidde Beydals <hello@hidde.co>
For signed commit verification, this code errors out on line 303:
✗ GitRepository reconciliation failed: ''PGP public keys secret error: expected pointer, but got nil
Pointer was not initialized with a concrete instance of the Secret struct
Signed-off-by: Kingdon Barrett <yebyen@gmail.com>
This commit changes the `gogit` behavior for commit checkouts,
now allowing one to reference to just a commit while omitting any
branch reference. Doing this creates an Artifact with a
`HEAD/<commit>` revision.
If both a `branch` and `commit` are defined, the commit is expected
to exist within the branch. This results in a more efficient clone
of just the target branch, and also makes this change backwards
compatible.
Fixes#407Fixes#315
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit refactors the previous `Commit` interface into a
standardised `Commit` struct. This object contains sufficient
information for referencing, observating and (PGP) verification.
- `libgit2` commit checkout does now return `HEAD/<SHA1>` as
the branch is not taken into account.
- `git2go` objects are now properly `Free`d everywhere
- `Verify` logic is tested.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit moves the previous `AuthStrategy` wiring to a more generic
`AuthOptions`, breaking free from implementation specific details in
the `git` package.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit adds a `ReconcileStrategy` field to the `HelmChart` resource, which
allows defining when a new chart should be packaged and/or published if it
originates from a `Bucket` or `GitRepository` resource.
The two available strategies are:
- `ChartVersion`: creates a new artifact when the version of the Helm chart as
defined in the `Chart.yaml` from the Source is different from the current
version.
- `Revision`: creates a new artifact when the revision of the Source is
different from the current revision.
For the `Revision` strategy, the (checksum part of the) revision of the
artifact the chart originatesfrom is added as SemVer metadata.
A chart from a `GitRepository` with Artifact revision
`main/f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer
`0.1.0` will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.
A chart from a `Bucket` with Artifact revision
`f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer `0.1.0`
will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.
Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
Added Support for Google Cloud Storage with Workload Identity as Source Provider. This enables the use of GCP without enabling S3 compatible access.
Signed-off-by: pa250194 <pa250194@ncr.com>
This ensures the checksum is predictable, and not influenced by e.g.
different runtime configuration settings, or FS specific data.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit changes the checksum method which is used to calculate the
revision of a Bucket source, so that the file paths are taken into
account and directory structure changes can be observed.
Signed-off-by: Hidde Beydals <hello@hidde.co>
controller-runtime's client package provides ObjectKeyFromObject() to
extract NamespacedName from a given object. ObjectKey() in
internal/util package is a helper for the same. Replace the internal
helper with controller-runtime's helper for the same.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
v3.6.1 is a a security update from Helm, ensuring that credentials are
always only passed to the defined repository host.
Based on Helm user reports, disabling this behavior may be required for
some Helm repository solutions like Artifactory, and may be done by
setting `PassCredentials` in the `HelmRepositorySpec`.
For more information, see:
https://github.com/helm/helm/security/advisories/GHSA-56hp-xqp3-w2jf
Signed-off-by: Hidde Beydals <hello@hidde.co>
Sunny
Hidde Beydals
Paulo Gomes
Kevin McDermott
Stefan Prodan
Sanskar Jaiswal
Tom Huang
Aurel Canciu
Kingdon Barrett
pa250194
Dylan Arbour
Philip Laine