The problem with `GetInterval()` was that the returned type was of
`metav1.Duration`, while almost anywhere it was used, a type of
`time.Duration` was requested. The result of this was that we had to
call `GetInterval().Duration` all the time, which would become a bit
cumbersome after awhile.
To prevent this, we introduce a new `GetRequeueAfter() time.Duration`
method, which both results the right type, and bears a name that is
easier to remember where the value is used most; while setting the
`Result.RequeueAfter` during reconcile operations.
The introduced of this method deprecates `GetInterval()`, which should
be removed in a future MINOR release.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Fixes error returned from target path validation check and adds more
test cases for TestGitRepositoryReconciler_reconcileArtifact.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
NOTE: This should be amended with the previous commit which has
commented out tests.
Update reconcileSource() to work with the test case where no secret is
set. A minimal auth options is created and used for git checkout.
Update TestGitRepositoryReconciler_verifyCommitSignature() to use the
new git.Commit type.
Update TestGitRepositoryReconciler_reconcileSource_checkoutStrategy to
add skipForImplementation for branch commit test case.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit rewrites the `GitRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying
to adhere better to Kubernetes API conventions.
More specifically it introduces:
- Implementation of more explicit Condition types to highlight
abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
conditions.
- First (integration) tests that solely rely on `testenv` and do not
use Ginkgo.
There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.
Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.
Signed-off-by: Hidde Beydals <hello@hidde.co>
go-git: Include the commit message in the returned commit object.
libgit2: Set the URL in the checkout error.
Add new method Commit.ShortMessage() for returning short commit
message.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This avoids skipping the directories when generating the archive
tarball.
This makes it easier to scan directory trees.
Signed-off-by: Kevin McDermott <kevin@weave.works>
Ensure that directories are included.
This avoids skipping the directories when generating the archive
tarball.
This makes it easier to scan directory trees.
Signed-off-by: Kevin McDermott <kevin@weave.works>
Refactor logic to install helper tools into one function in the
Makefile. Add support for envtest to help install tools like kubectl,
etcd which helps users run tests more conveniently.
Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
As the full version can be used as a label value, the full SHA from the
reference takes up too much space from the 63 characters available in
total.
To mitigate against this, we now take a "short" version of the first 12
characters, which was still unique for the Linux kernel in 2019 with
875.000 commits:
http://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#Short-SHA-1
This should be sufficient to safely detect all changes within the
context of operations.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Test case "Setting valid valuesFile attribute" and the tests around it
aren't isolated and most of the time pass because of the results from
the previous tests being re-read as the test expectation match the
previous test results. Failures are very rare to reproduce, even in
the CI they aren't seen but it failed very frequently on my computer,
especially this specific case because unlike the other cases, there is
just one file to be merged, which invalidates the chart result from
the previous cases.
In order to ensure the test wait for the chart to be updated by its
action and not by any other previous updates, status condition message
seems to be the most reliable way, as it also contains the paths of the
files that were merged.
With this change, I could no longer reproduce the failure on my
computer.
Reordering the tests makes this issue more clear.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
By providing the Generation of the object that is getting reconciled
as version metadata to the builder if any custom values files are
defined, the Artifact revision changes if the specification does,
ensuring consumers of the Artifact are able to react to changes in
values (and perform a release).
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit introduces a typed `BuildError` to be returned by
`Builder.Build` in case of a failure.
The `Reason` field in combination with `BuildErrorReason` can be used
to signal (or determine) the reason of a returned error within the
context of the build process.
At present this is used to determine the correct Condition Reason, but
in a future iteration this can be used to determine the negative
polarity condition that should be set to indicate a precise failure to
the user.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Dealing with some loose ends around making observations, and code
style.
The loaded byes of a chart are used as a revision to ensure e.g.
periodic builds with unstable ordering of items do not trigger a false
positive.
Signed-off-by: Hidde Beydals <hello@hidde.co>
With all the logic that used to reside in the `controllers` package
factored into this package, it became cluttered. This commit tries to
bring a bit more structure in place.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit starts wiring the factored out Helm chart build logic into
the reconciler to ensure, validating the API capabilities.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- Ensure the proper path is garbage collected for libgit2 repositories,
as the `Path` method on the repository object returns the `.git`
directory, and not the root path.
- Ensure the Helm test server does not get swapped during tests,
with as side-effect that no obsolete temporary directories remain.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This changes the format of the Artifact checksum from SHA1 to SHA256 to
mitigate chosen-prefix and length extension attacks, and ensures it can
be used to secure content against malicious modifications.
Source consumers (including our own {kustomize,helm}-controllers)
should ensure the SHA256 of a downloaded artifact matches the
advertised checksum before making use of it.
Signed-off-by: Hidde Beydals <hello@hidde.co>
For signed commit verification, this code errors out on line 303:
✗ GitRepository reconciliation failed: ''PGP public keys secret error: expected pointer, but got nil
Pointer was not initialized with a concrete instance of the Secret struct
Signed-off-by: Kingdon Barrett <yebyen@gmail.com>
This commit changes the `gogit` behavior for commit checkouts,
now allowing one to reference to just a commit while omitting any
branch reference. Doing this creates an Artifact with a
`HEAD/<commit>` revision.
If both a `branch` and `commit` are defined, the commit is expected
to exist within the branch. This results in a more efficient clone
of just the target branch, and also makes this change backwards
compatible.
Fixes#407Fixes#315
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit refactors the previous `Commit` interface into a
standardised `Commit` struct. This object contains sufficient
information for referencing, observating and (PGP) verification.
- `libgit2` commit checkout does now return `HEAD/<SHA1>` as
the branch is not taken into account.
- `git2go` objects are now properly `Free`d everywhere
- `Verify` logic is tested.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit moves the previous `AuthStrategy` wiring to a more generic
`AuthOptions`, breaking free from implementation specific details in
the `git` package.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit adds a `ReconcileStrategy` field to the `HelmChart` resource, which
allows defining when a new chart should be packaged and/or published if it
originates from a `Bucket` or `GitRepository` resource.
The two available strategies are:
- `ChartVersion`: creates a new artifact when the version of the Helm chart as
defined in the `Chart.yaml` from the Source is different from the current
version.
- `Revision`: creates a new artifact when the revision of the Source is
different from the current revision.
For the `Revision` strategy, the (checksum part of the) revision of the
artifact the chart originatesfrom is added as SemVer metadata.
A chart from a `GitRepository` with Artifact revision
`main/f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer
`0.1.0` will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.
A chart from a `Bucket` with Artifact revision
`f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer `0.1.0`
will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`.
Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
Added Support for Google Cloud Storage with Workload Identity as Source Provider. This enables the use of GCP without enabling S3 compatible access.
Signed-off-by: pa250194 <pa250194@ncr.com>
This ensures the checksum is predictable, and not influenced by e.g.
different runtime configuration settings, or FS specific data.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit changes the checksum method which is used to calculate the
revision of a Bucket source, so that the file paths are taken into
account and directory structure changes can be observed.
Signed-off-by: Hidde Beydals <hello@hidde.co>
controller-runtime's client package provides ObjectKeyFromObject() to
extract NamespacedName from a given object. ObjectKey() in
internal/util package is a helper for the same. Replace the internal
helper with controller-runtime's helper for the same.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
v3.6.1 is a a security update from Helm, ensuring that credentials are
always only passed to the defined repository host.
Based on Helm user reports, disabling this behavior may be required for
some Helm repository solutions like Artifactory, and may be done by
setting `PassCredentials` in the `HelmRepositorySpec`.
For more information, see:
https://github.com/helm/helm/security/advisories/GHSA-56hp-xqp3-w2jf
Signed-off-by: Hidde Beydals <hello@hidde.co>
Unlike Bucket resources which are matched by key as presented by S3,
ignore rules for GitRepository objects do have a domain: the temporary
directory of the Git repository.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This likely happened because the byte buffer response was already
being read by the chart loader, making it empty by the time the
artifact was written to storage.
As an alternative, and because it makes the code a tiny bit less
obnoxious: write the data to a temp file first, and later decide
what file to copy over and use as an stored artifact.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit makes the filtering applied during the archiving
configurable by introducing an optional `ArchiveFileFilter`
callback argument and a `SourceIgnoreFilter` implementation.
`SourceIgnoreFilter` filters out files matching
sourceignore.VCSPatterns and any of the provided patterns.
If an empty gitignore.Pattern slice is given, the matcher is set to
sourceignore.NewDefaultMatcher.
The `GitRepository` now loads the ignore patterns before archiving
the repository contents by calling `sourceignore.LoadIgnorePatterns`
and other helpers. The loading behavior is **breaking** as
`.sourceignore` files in the (subdirectories of the) repository are
now still taken into account if `spec.ignore` for a resource is
defined, overwriting is still possible by creating an overwriting
rule in the `spec.ignore` of the resource.
This change also makes it possible for the `BucketReconciler` to not
configure a callback at all and prevent looking for ignore
matches twice. To finalize the bucket refactor, a change to the
reconciler has been made to look for a `.sourceignore` file in
the root of the bucket to provide an additional way of configuring
(global) exclusions.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Adds a test that loads the helmChart from the updated resource and
verifies that `testOverride` (the value overrode in the test fixtures)
changes from `false` to `true`.
Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
This commit adds a test specifically for RecurseSubmodules. It takes a
bit more preparation, since it needs a repo using submodules to start
with. go-git doesn't appear to support adding submodules
programmatically, so the preparation is done in part by execing `git`.
Signed-off-by: Michael Bridgen <michael@weave.works>
This commit fixes a bug where local chart dependencies would not be
detected correctly due to the absence of a repository URL.
Signed-off-by: Hidde Beydals <hello@hidde.co>
`io.Read` was used incorrectly to read from the override file provided
by the user.
This is now replaced with `ioutil.ReadFile` for better handling and
error reporting.
Fixes#263
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
To include a bug fix to the `ReconcilateAtChangedPredicate`
and renaming to `ReconcileRequestedPredicate`.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit upgrades the `controller-runtime` dependency to `v0.7.0`,
including all changes required to make all wiring work again.
- Upgrade `runtime` to v0.6.0 to include `controller-runtime` changes.
- Loggers have been removed from the reconcilers and are now retrieved
from the `context.Context` passed to the `Reconcile` method and
downwards functions.
- Logger configuration flags are now bound to the flag set using
`BindFlags` from `runtime/logger`, ensuring the same contract across
GitOps Toolkit controllers, and the `--log-json` flag has been
deprecated in favour of the `--log-encoding=json` default.
- The `ChangePredicate` from `runtime` has changed to a
`ReconcilateAtChangedPredicate`, and is now chained with the
`GenerationChangedPredicate` from `controller-runtime` using
`predicate.Or`.
- Signatures that made use of `runtime.Object` have changed to
`client.Object`, removing the requirement to e.g. call
`runtime.Object#Object`.
- The `client.MatchingField` function was deprecated, and has been
replaced with `client.MatchingFields{}`.
- The `leader-election-role` was changed, as leader election now works
via the `coordination/v1` API.
Other notable changes:
- `util.ObjectKey` was added to easily construct a `client.ObjectKey` /
`types.NamespacedName` from a `metav1.Object`.
- The `SourceIndexKey` constant has been split out into
`{GitRepository,HelmRepository,Bucket}IndexKey` constants.
Signed-off-by: Hidde Beydals <hello@hidde.co>
As part of the feature implementation to support helm chart
dependencies, the functionality for allowing values files overwriting
from any location scoped to the same source was altered. This should fix
the problem by allowing users to load files from any arbitrary location
as long as it's in the context of the same source from where the helm
chart itself is loaded.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
The controller logic is a serious candidate to be refactored so that
errors can be taken into account better, and do not always result in a
requeue. But this will do for the time being.
Signed-off-by: Hidde Beydals <hello@hidde.co>
As GCP's S3 interoperability does not implement the ListObjectsV2
implementation.
Ref: https://cloud.google.com/storage/docs/interoperability
> Note: While most actions are interoperable with the Amazon S3 V2 SDK,
> listing objects can only be performed using the Amazon S3 V1 list
> objects method.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Following the rules described in
https://helm.sh/docs/chart_best_practices/conventions/#chart-names.
This guards against people following the wrong guidance of Artifactory,
that supports and promotes repository indexes with e.g. '/' in the
chart names.
In a future version this should be moved to a validation webhook, but
there are still discussions ongoing around the TLS certificates for
this.
Signed-off-by: Hidde Beydals <hello@hidde.co>
We had a hardcoded assumption that the SSH user for a Git repository is
always "git". This is however not true in all scenarios, for example
when one is making use of Gerrit for team code collaboration, as users
there have their own username for (SSH) Git operations.
This commit changes the logic of the auth strategy helpers to:
1. Select the auth strategy based on the protocol of the parsed URL,
instead of a simple rely on a correct prefix.
2. Use the user information from the parsed URL to configure the user
for the public key authentication strategy, with a fallback to `git`
if none is defined.
Signed-off-by: Hidde Beydals <hello@hidde.co>
To enqueue a new reconciliation for the HelmChart sources as soon as
the revision of their upstream source changes.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Use SetResourceCondition as a generic method to set conditions for CRs,
implmeneting the ObjectWithStatusConditions interface used as input
type.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
Updates to use metav1.Condition type and removes references for
deprecated corev1.Condition* constants and uses the new k8s api/meta
helpers in place of the old pkg/apis/meta types.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
It looks like the use of chartutil.ProcessDependencies in the HelmChart
Controller was not correct, this method seems to be used in Helm only
during install/upgrade. The intention was to load the dependencies but
this seems to not be needed as it's already done through the loaders
(loader.Load).
The use of this method caused a regression where Chart.yaml files would
be overwritten and registered subcharts that had aliases would be
renamed using the alias name. While this is an expected behaviour of
chartutil.ProcessDependencies it is not what the controller should do
to the chart during (re)packaging.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
Non-packaged charts that don't have their dependencies present in
charts/ will now have these dependencies built using the
DependencyManager. The idea behind it is to replicate the logic
implemeneted in Helm's downloader.Manager with the support for already
existing HelmRepository resources and their chart retrieval capabilities.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
To facilitate an inexpensive lookup when collecting credentials and
index artifacts while working with chart dependencies.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
The feature allows the source-controller to load packaged helm charts
for HelmChart resource artifacts from GitRepository and Bucket sources
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
This commit ensures that resources will only return early if they are
already in a `Ready==True` state. If not, but the status object somehow
still reports that it has an artifact, the reconciliation will continue
to ensure and/or guarantee state, and to prevent a deadlock from
happening.
During high custom resource count / low interval tests, I was greated
with a `cannot patch resource "events"` message. This happened due to
event compaction, where it will perform a patch instead of a create.
By giving the role the permission to do so this should no longer pose
a problem.
When a delete of a resource is requested a `deletionTimestamp` is set
on the resource by the requester, this also results in a generation
change of the resource.
If the resource is under reconciliation while this timestamp is set, and
had not produced an artifact earlier on, this becomes a problem as the
artifact metadata is used to determine what should be garbage collected
on a deletion, resulting in stray files for resources that are no longer
present.
To resolve this for now, we always create a new artifact object for the
resource when `all==true` on the GC method call, and no longer rely on
the presence of the artifact object on the resource itself.
This includes a change to how the revision for HelmRepository sources is
recorded, as this will now equal to the generated timestamp from the index
in RFC3339Nano format.
As the storage base directory is determined during runtime, and
artifacts may live longer than that if they are e.g. stored in a
persistent volume but the mount path configuration changes.
Given that:
* The produced artifact as advertisted in the path should always
be a regular file (including the exclusion of symlinks).
* The produced artifact should be readable, so any type of error
should count as "does not exist".
We should use `os.Lstat` to not follow symlinks; return `false`
on any error we run in to, or return if the file mode information
reports a regular file.
Due to required domain changes for the helm-controller so that it
can co-exist in a cluster with the Helm Operator, other Toolkit
components are moving to a *.toolklit.fluxcd.io domain too.
filepath.Walk can return a `nil` for the stat value, when it does, the
directory is invalid and the error will be set. This causes a
panic+crash if the directory does not currently exist when
RemoveAllButCurrent is called.
The following patch makes the behavior an error instead.
Signed-off-by: Erik Hollensbe <github@hollensbe.org>
- return reconciliation error so that controller runtime metrics record failures
- change structure logging labels to match the controller runtime format
- log the reconciliation duration for all kinds
- normalise log messages and labels across all controllers
This commit changes the file excludes for tarballs generated for
Git repository artifacts from a fixed set of strings to include
exclusion files files. It currently takes `.sourceignore` and
in the root of the given directory into account.
In addition to this the Git VCS related files that are ignored have
been extended to not only include the .git/ directory, but also the
.gitignore, .gitmodules and .gitattributes files. Mimicking part of
the --exclude-vcs flag not available on all tar versions.
This commit adds a timeout field to the GitRepositorySpec to be used
during the git clone operation when reconciling the resource.
When no interval is defined the default timeout returned by the getter
is 20 seconds.
The timeout can not be added yet to the Helm related sources as it
is currently not possible to inject anything custom into the HTTP
client from the Helm HTTP getter except for the authentication
options built in. A submit has been submitted to make this possible
and is waiting for review.
This commit includes some context changes to the other reconcilers
to tidy them up and make them depend on a single background context.
It also includes some added docblocks that crossed my path.
Introduces new helpers and config structs to all reconcilers to
set the max concurrent reconciles number.
Introduces a new flag `--concurrent` to configure the number of
concurrent reconciles per reconciler, defaults to `2`.
Hidde Beydals
Sunny
Paulo Gomes
Kevin McDermott
Stefan Prodan
Sanskar Jaiswal
Tom Huang
Aurel Canciu
Kingdon Barrett
pa250194
Dylan Arbour
Philip Laine
Michael Bridgen
Somtochi Onyekwere
Raffael Sahli
Philip Laine
Thomas Runyon
Léo Martinez
Michael Bridgen
Erik Hollensbe
Hidde Beydals