If implemented user will be able to provide their own custom start and
bypass tls verification when interacting with OCI registries over https
to pull helmCharts.
Signed-off-by: Soule BA <soule@weave.works>
In Reconcile() methods, move the object deletion above add finalizer.
Finalizers can't be set when an object is being deleted.
Introduce a cacheless client in suite_test to use for testing this
change. It ensures that the Reconcile() call always operates on the
latest version of the object which has the deletion timestamp and
existing finalizer.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Add tests to test Cosign support for insecure registries. Furthermore,
refactor OCI test utils to be more user friendly and enable accurate
testing of HTTPS and HTTP OCI registries by circumnavigating Docker's
automatic connection downgrade for registries hosted on localhost.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Add support for verifying insecure HTTP OCI repositories with cosign. If
`.spec.insecure` set to true, then cosign uses plain HTTP connections to
communicate with the registry.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Add support for specifying TLS auth data via `.spec.certSecretRef` in
HelmRepository and log a deprecation warning if TLS is configured via
`.spec.secretRef`. Introduce (and refactor) Helm client builder and
auth helpers to reduce duplicated code and increase uniformity and
testability.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
Add `.spec.proxySecretRef.name` to the `GitRepository` API to allow
referencing a secret containing the proxy settings to be used for all
remote Git operations for the particular `GitRepository` object.
It takes precedence over any proxy configured through enviornment
variables.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
This is required because the test fails with Git >=v2.41.0 due to
changes to commands used by the Git test server. Causing the server to
return an error when cloning an empty repository, instead of yielding
an empty object.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This adds a bit back which got removed in
69f567bdc7, as there are reasons for the
controller to perform a non-optimized clone.
However, we always want to attempt the optimized version first without
it being put behind a feature gate. Which was the original intent of
the referenced commit.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
Remove the `OptimizedGitClones` feature gate, making optimized Git
clones when using a branch or tag to checkout, the default behavior.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
This deals with various breaking changes in controller-runtime, as
documented in the release notes:
https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.15.0
In short:
- `Watches` now use a `client.Object` instead of a `source.Kind`.
- `handler.MapFunc` signature accepts a Go context, which is used to
log any errors, instead of silently ignoring them and/or panicking.
- Fake clients used in tests are now configured using
`WithStatusSubresource` to enable the correct behavior for status
updates and patches.
- Max concurrent reconciles is configured on the manager, instead of
configuring them per reconciler instance.
- Various manager configuration options have been moved to new
structures and/or fields.
In addition to this, all other dependencies which had updates are
updated to their latest (compatible) versions as well.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This commit properly sets `IgnoreTlog` to `true` when a public key is
provided to check the signature against, which matches the (silent)
default behavior from cosign v1.
However, during this exercise it has become apparant that this
assumption isn't necessarily true. As you can theoretically have a
custom key and a tlog entry.
Given this, we should inventarise the possible configuration options
and the potential value they have to users (e.g. defining a custom
Rekor URL seems to be valuable as well), and extend our API to
facilitate these needs.
In addition to the above, the CTLog public keys are now properly
retrieved to avoid a `none of the CTFE keys have been found` error.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This commit ensures that files with exec permissions set continue to be
executable by the user extracting the archive.
This is not of use to any of Flux itself, but does help downstream
dependents making use of the controller to facilitate artifact
acquisitions for their (CI/CD) software suite.
Co-authored-by: Hidde Beydals <hidde@hhh.computer>
Signed-off-by: Rashed Kamal <krashed@vmware.com>
Given:
- None of the methods of the `Storage` are mutating the storage
itself.
- It must be instantiated to be usable, as there is a strict
reliance on values.
- The struct itself is light.
This seems to be more fitting.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This commits adds verification of the digest of the artifact in storage
to all reconcilers which manage artifacts.
When the artifact does not have a digest or if it mismatches with the
file in storage, the file is removed from the storage and status of the
object.
This hardens the storage against potential tampering, in addition to
resolving an issue where users upgrading from a (much) older version of
the controller would run into an error after the checksum field was
removed from the API.
This would cause the controller to not advertise any checksum at all,
while not producing a new one until a new revision was detected.
Resulting in fetch failures for consumers while they would try to
verify the digest of the advertised artifact.
While not strictly part of this exercise, some of the tests were
altered to prepare the storage used in test cases to become isolated
by strictly using the `storage` provided via the callback. Actually
isolating this has however been left as a task at a later moment.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
In #1001 bits around the Helm repository reconciliation logic were
rewritten, mostly based on the documented behavior instead of the
actual code. This resulted in the reintroduction of a YAML marshal of
the (sorted) index YAML instead of reliance of just the checksum of the
file.
This to take situations into account in which a repository would e.g.
provide a new random order on every generation. However, this approach
is (extremely) expensive as the marshal goes through a JSON -> YAML
loop, eating lots of RAM in the process.
As the further (silently) introduced behavior has not resulted in any
reported issues, I deem this approach safe and better than e.g.
encoding to just JSON which would still require a substantial amount of
memory.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This includes changes to the `ChartRepository`, to allow calculating
the revision and digest and tidy things.
In addition, the responsibility of caching the `IndexFile` has been
moved to the reconcilers. As this allowed to remove a lot of
complexities within the `ChartRepository`, and prevented passing on
the cache in general.
Change `HelmRepository`'s Revision to digest
Signed-off-by: Hidde Beydals <hello@hidde.co>
This allows using the condition checker as a test helper with proper
test like assertion failure and stacktrace.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Add chart address in the OCI chart download failure error message to make
it clear about the chart URL that was attempted to download.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
You can re-enable caching by starting the controller
with the argument '--feature-gates=CacheSecretsAndConfigMaps=true'
Signed-off-by: Mac Chaffee <machaffe@renci.org>
Replace the patch Helper with SerialPatcher which is used for
progressive status patching.
Update the tests to use progressive status reasons in tests.
Add ProgressingWithRetry Reconciling reason for failed
reconciliation result to indicate a finished failure operation.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Update the implementations of the helm repository downloaders to return
implementation specific typed error from GetChartVersion(). This is
needed to distinguish between persistent build error and transient build
error.
In the case of OCI charts, a transient network failure shouldn't be
considered a persistent build failure of the chart and should be
retried.
Two repository errors, ErrReference and ErrExternal are introduced for
the repository downloader implementations to provide enough context
about the failure which can be used by the caller to add appropriate
context as per the needs. In case of chart builder, it adds the build
error context based on the repository error value.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
If implemented, this fix the issue were aliased chart dependencies were
detected but not included in the final packaged chart.
Signed-off-by: Soule BA <soule@weave.works>
ForceGoGitImplementation ignores the value set for gitImplementation
and ensures that go-git is used for all GitRepository objects.
This can be used to confirm that Flux instances won't break if/when
the libgit2 implementation was to be deprecated.
When enabled, libgit2 won't be initialized, nor will any git2go cgo
code be called.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Delete a failed verification condition at the beginning of the source
reconciliation and set `SourceVerifiedCondition` to false approprietly.
Set the `BuildOptions.Verify` to true as long as Verify is enabled in the
API fields.
Signed-off-by: Soule BA <soule@weave.works>
If implemented the oras registry loginOption will only be used internaly
with the specific ChartRepo struct.
This will permit reusing more easily feature developped with
googlecontainerregistry authn.
Signed-off-by: Soule BA <soule@weave.works>
Add setters and getters for spec.suspend and status.artifact.
This is needed for writing generic tests for any source kind.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
If implemented this enable passing a keychain, an authenticator and a
custom transport as remote.Option to the verifier. It enables contextual
login, self-signed certificates and insecure registries.
Signed-off-by: Soule BA <soule@weave.works>
refactor makeOptions
Reduce complexity by replacing the functional options with a flat out
conditional logic in makeOptions.
Signed-off-by: Soule BA <soule@weave.works>
- Update Go to 1.19 in CI
- Use Go 1.19 in base image
- Update controller-gen v0.8.0 and regenerate manifests
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
When a custom CA certificate is provided in a Secret's `caCert` field
referenced in `HelmRelease.spec.secretRef` then that CA cert is now
added to the list of system certificates instead of it replacing the
system certificates. This makes HelmRepositories work in mixed
environments where charts are pulled from both, a public repository
and a private repository (e.g. through a chart dependency).
The test that is added as part of this change will fail without the
change and passes with it.
closes#866closesfluxcd/helm-controller#519
Signed-off-by: Max Jonas Werner <max@e13.dev>
This introduces the consideration of bipolarity conditions in the status
condition summary for Ready condition. The summarize.HelperOptions can
now be configured with a list of bipolarity conditions which are used in
SummarizeAndPatch() to set the Ready condition to failing bipolarity
condition with the highest priority.
Bipolarity condition is not a typical status property. It is a mix of
positive and negative polarities. It's "normal-true" and
"abnormal-false". Failing bipolarity conditions are prioritized over
other conditions to show the actual reason of failure on the Ready
status.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
If implemented, this pr will enable user to use the auto login feature
in order to automatically login to their provider of choice's container
registry (i.e. aws, gcr, acr).
Signed-off-by: Soule BA <soule@weave.works>
This change allows for a HelmRepository to point to e.g. "ghcr.io" and
then a HelmRelease pointing to the chart "stefanprodan/charts/podinfo"
in its `.spec.chart.spec.chart` field.
Related discussion: https://github.com/fluxcd/flux2/discussions/2959
Signed-off-by: Max Jonas Werner <max@e13.dev>
Taking this shortcut has two benefits:
1. It allows charts to be fetched from AWS's public container registry
at public.ecr.aws
2. It makes reconciling a HelmChart faster by skipping one or more
potentially expensive API calls to the registry.
I adapted the unit tests to the new behavior that the
OCIChartRepository doesn't fail anymore for the case where a specific
chart version has been requested that doesn't actually exist in the
registry.
refs #845
Signed-off-by: Max Jonas Werner <max@e13.dev>
1. moved chartRepo.Unload() from reconcileSource() to the defer func in reconcileArtifact to allow caching index in memory
2. added step to init memory cache in reconcileArtifact()
3. added step to save helmrepo index into memory cache in reconcileArtifact()
Signed-off-by: York Chen <ychen@d2iq.com>
Decommission libgit2 unmanaged transport and remove the related feature
gate, making managed transport the default.
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
If implemented, this make sure than we clear only referenced
downloaders.
It is also checked if the repository url is supported.
Signed-off-by: Soule BA <soule@weave.works>
If implemented, the source controller will be able to resolve charts
dependencies from OCI repositories.
The remote builder has been refactored as part of this work.
Signed-off-by: Soule BA <soule@weave.works>
Since the transport reuse is dependent on the garbage collection, the
result is inconsistent. It fails frequently when running the tests with
the go race detector. Remove the test.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
If implemented this fix the issue where we previously did a string
ordering of matching semver versions when retrieving a list of tags from
an OCI registry.
Signed-off-by: Soule BA <soule@weave.works>
Registry login option should verify that the obtained dockerconfig
credentials are for the same host. When the helmrepo URL and the URL in
docker auth config don't match, the docker config store returns an
empty auth config, instead of failing. This results in accepting empty
username and password. The HelmRepo would appear to be ready in such
situation because the creds are empty, no login is attempted. But when
a HelmChart tries to use the login options, it'd fail.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
The observed generation must be set only when actual observation is
made. When an actual observation is made, some conditions are set on the
object. Introduce a helper function
addPatchOptionWithStatusObservedGeneration() to set the patcher option
WithStatusObservedGeneration only when there's any condition in the
status.
Updates the existing tests that depended on this behavior.
This fixes the issue where the observed generation is set by the patcher
when a reconciler does an early return for setting the finalizers only.
With this, the observed generation will be updated only when some
observations are made on the object based on the usual rules of success
result, no error, ignore error and stalled condition.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This new package holds all Helm OCI registry-specific code now so we
have a single location to look for such code which makes it easier to
find yourself around.
Signed-off-by: Max Jonas Werner <mail@makk.es>
Add Generic error in RuntimeResultBuilder and ComputeReconcileResult
implementation with consideration to the error configurations.
Safeguards are added in the runtime result builder to ensure default
requeue after interval is set when is's set to zero or unset.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
ErrorActionHandler processes the reconciliation error results based on
their configurations. It performs actions like logging and event
recording based on the error configuration. More actions can be
accommodated in the future with more error configurations.
It can be a replacement for RecordContextualError() which does the same
operations but can't be configured much.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Generic error is an attempt to avoid creating new error type for every
new unique scenario. It can be used to configure and build custom error
handling behavior, logging and event recording at present.
Contextual errors, Stalling and Waiting error, have special meaning for
the reconciliation results. But the Event error type can be replaced
with Generic error with some specific configurations. The Event error
is kept for a gradual migation to Generic error. Similarly, the Generic
error can be used to easily create new error handling behaviors.
The error Config can be used to configure any of the errors, including
contextual errors, without altering their contextual meaning, to modify
how they are handled.
The error constructors configure the errors with common default
configurations. These configurations can be modified to alter the
behavior.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
* Add OCI Helm support
* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.
Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
OptimizedGitClones decreases resource utilization for GitRepository
reconciliations. It supports both go-git and libgit2 implementations
when cloning repositories using branches or tags.
This is an opt-out feature, which can be disabled by starting the
controller with the argument '--feature-gates=OptimizedGitClones=false'.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
SummarizeAndPatch() should also consider the object's status conditions
when computing and returning the runtime results to avoid any
inconsistency in the runtime result and status condition of the object.
When an object's Ready condition is False, the reconciler should retry
unless it's in stalled condition.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Avoid validating (and thus loading) indexes if the checksum already exists in storage.
In other words, if the YAML is identical to the Artifact in storage, the reconciliation should
be a no-op, and therefore can short-circuit long/heavy operations.
Co-authored-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
If implemented this will:
- enable the helmCharts dependency manager to use the helm in memry
cache to retrieve reconciled HelmRepositories indexes.
- record cache events.
Signed-off-by: Soule BA <soule@weave.works>
Azure SDK dependencies cannot be updated, as this requires us to move to
Go 1.18.
- cloud.google.com/go/storage to v1.22.0
- github.com/ProtonMail/go-crypto to v0.0.0-20220407094043-a94812496cf5
- github.com/darkowlzz/controller-check to v0.0.0-20220325122359-11f5827b7981
- github.com/elazarl/goproxy to v0.0.0-20220403042543-a53172b9392e
- github.com/fluxcd/pkg/gittestserver to v0.5.2
- github.com/go-logr/logr to v1.2.3
- github.com/minio/minio-go/v7 to v7.0.24
- github.com/onsi/gomega to v1.19.0
- golang.org/x/crypto to v0.0.0-20220411220226-7b82a4e95df4
- google.golang.org/api to v0.74.0
Signed-off-by: Hidde Beydals <hello@hidde.co>
As suggested by @pjbgf
Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Signed-off-by: Peter Gundel <mail@petergundel.de>
This better represent permissions as Linux handles such information in
octal format, meaning that the left-most 0 has an important meaning
and is not to be ignored as normally integers would.
See https://github.com/fluxcd/source-controller/issues/603
Signed-off-by: Peter Gundel <mail@petergundel.de>
This includes some rewiring of tests, and slight changes in how we work
with the local chart reference. `Path` is expected to be relative to
`WorkDir`, and both fields are now mandatory.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This introduces our own `secureloader` package, with a directory
loader that's capable of following symlinks while validating they stay
within a certain root boundary.
Signed-off-by: Hidde Beydals <hello@hidde.co>
We require these to be able to mimic Helm's own directory loader, and
surprisingly (for `ignore` at least), these are not public.
Signed-off-by: Hidde Beydals <hello@hidde.co>
notify() is used to emit events for new artifact and failure recovery
scenarios. It's implemented in all the reconcilers.
Previously, when there used to be a failure due to any reason, on a
subsequent successful reconciliation, no notification was sent to
indicate that the failure has been resolved.
With notify(), the old version of the object is compared with the new
version of the object to determine if all, if any, of the failures have
been resolved and a notification is sent. The notification message is
the same that's sent in usual successful source reconciliation message
about stored artifact.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
If implemented, will provide users with a way to cache index files.
This addresses issues where the index file is loaded and unmarshalled in
concurrent reconciliation resulting in a heavy memory footprint.
The caching strategy used is cache aside, and the cache is a k/v store
with expiration.
The cache number of entries and ttl for entries are configurable.
The cache is optional and is disabled by default
Signed-off-by: Soule BA <soule@weave.works>
The experimental managed transport can also leverage TransportPool,
moving it to its own package to accommodate that use case.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This is required after the update of `cloud.google.com/go/storage` from
`v1.16.0` to `v1.21.0`, due to changed `STORAGE_EMULATOR_HOST`
behavior.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Reuses the same transport across different helm chart downloads,
whilst resetting the tlsconfig to avoid cross-contamination.
Crypto material is now only processed in-memory and does not
touch the disk.
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This commit introduces a BucketProvider interface for fetch operations
against object storage provider buckets. Allowing for easier
introduction of new provider implementations.
The algorithm for conditionally downloading object files is the same,
whether you are using GCP storage or an S3/Minio-compatible
bucket. The only thing that differs is how the respective clients
handle enumerating through the objects in the bucket; by implementing
just that in each provider, I can have the select-and-fetch code in
once place.
The client implementations do now include safe-guards to ensure the
fetched object is the same as metadata has been collected for. In
addition, minor changes have been made to the object fetch operation
to take into account that:
- Etags can change between composition of index and actual fetch, in
which case the etag is now updated.
- Objects can disappear between composition of index and actual fetch,
in which case the item is removed from the index.
Lastly, the requirement for authentication has been removed (and not
referring to a Secret at all is thus allowed), to provide support
for e.g. public buckets.
Co-authored-by: Hidde Beydals <hello@hidde.co>
Co-authored by: Michael Bridgen <michael@weave.works>
Signed-off-by: pa250194 <pa250194@ncr.com>
- Update summarize helper to have patch field owner.
- Updated the controllers to set the patch field owner.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
summarizeAndPatch() was used by all the reconcilers with their own
object type. This creates a generic SummarizeAndPatch helper that takes
a conditions.Setter object and performs the same operations. All the
reconcilers are updated to use SummarizeAndPatch(). The process of
summarize and patch can be configured using the HelperOptions.
Introduce ResultProcessor to allow injecting middlewares in the
SummarizeAndPatch process.
Introduce RuntimeResultBuilder to allow defining how the reconciliation
result is computed for specific reconciler. This enabled different
reconcilers to have different meanings of the reconciliation results.
Introduce Conditions in summary package to store all the status
conditions related information of a reconciler. This is passed to
SummarizeAndPatch() to be used for summary and patch calculation.
Remove all the redundant summarizeAndPatch() tests per reconciler.
Add package internal/object containing helpers for interacting with
runtime.Object needed by the generic SummarizeAndPatch().
Add tests for ComputeReconcileResult().
Signed-off-by: Sunny <darkowlzz@protonmail.com>
If a local reference does not contain a path to a valid file, returning
`ErrChartReference` is more correct to signal the reference is invalid.
This also indirectly causes the reconciler to signal a Suspend, as the
source or resource requires a change before a reattempt might be
successful.
Signed-off-by: Hidde Beydals <hello@hidde.co>
Consolidate BuildRuntimeResult() into summarizeAndPatch() to simplify
where the results are computed, summarized and patched.
Move the event recording and logging of context specific errors into
RecordContextualError() and call it in summarizeAndPatch().
Introduce Waiting error for wait and requeue scenarios. Update
ComputeReconcileResult() and RecordContextualError() to consider Waiting
error.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
- Ensure all logged messages start with a lowercase.
- Make some pushed (and logged) events of type `EventTypeTrace` to
prevent them from being sinked to the external event recorder, to
prevent spam.
- Only log if artifact is up-to-date with upstream (instead of pushing
an event).
Signed-off-by: Hidde Beydals <hello@hidde.co>
In most of the reconcilers we have a repetative pattern of using part of
the object metadata to construct a temporary file path.
This commit introduces helpers as an abstraction, for both the creation
of a temporary directory based on `client.Object` type and object
metadata, and the generation of an arbitrary random temporary path
string.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- internal/error - Contains internal error type used across the
source-controller reconcilers.
- internal/reconcile - Contains helper abstractions for the
controller-runtime reconcile Result type and functions to
interact with the abstractions.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Due to the temporary file having a random suffix, it would result in
the notification-controller not rate limiting the messages as they
are "unique".
For the close error (which also makes use of the name), we keep the
information as it might be of importance to figure out why the close
failed.
Signed-off-by: Hidde Beydals <hello@hidde.co>
There was an unfinished code path that should have continued validating
the paths within the package. This commit completes it.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This makes it possible to signal reference (validation) errors
happening before the build process actually starts dealing with
the chart.
At present, this does not have a more specific counterpart in the API,
but this is expected to change when the conditions logic is revised.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- Add some more documentation around chart builders
- Ensure correct indentation in some doc comments
- Provide example of using `errors.Is` for typed `BuildError`
- Mention "bytes" in file size limit errors
- Add missing copyright header
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit introduces a typed `BuildError` to be returned by
`Builder.Build` in case of a failure.
The `Reason` field in combination with `BuildErrorReason` can be used
to signal (or determine) the reason of a returned error within the
context of the build process.
At present this is used to determine the correct Condition Reason, but
in a future iteration this can be used to determine the negative
polarity condition that should be set to indicate a precise failure to
the user.
Signed-off-by: Hidde Beydals <hello@hidde.co>
- For remote builds, if the build option has a version metadata, the
chart should be repackaged with the provided version.
- Update internal/helm/testdata/charts/helmchart-0.1.0.tgz to include
value files for testing merge chart values.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
With all the logic that used to reside in the `controllers` package
factored into this package, it became cluttered. This commit tries to
bring a bit more structure in place.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit refactors the `ChartBuilder` that used to be a do-it-all
struct into an interace with two implementations:
- `LocalChartBuilder`: to build charts from a source on the local
filesystem, either from a directory or from a packaged chart.
- `RemoteChartBuilder`: to build charts from a remote Helm repository
index.
The new logic within the builders validates the size of the Helm size
it works with based on the `Max*Size` global variables in the internal
`helm` package, to address the recommendation from the security audit.
In addition, changes `ClientOptionsFromSecret` takes now a directory
argument which temporary files are placed in, making it easier to
perform a garbage collection of the whole directory at the end of a
reconcile run.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit starts with the creation of a `ChartBuilder` to facilitate
the (conditional) build of a chart outside of the reconciler logic.
The builder can be configured with a set of (modifying) options, which
define together with the type of chart source what steps are taken
during the build.
To better facilitate the builder's needs and attempt to be more
efficient, changes have been made to the `DependencyBuilder` and
`ChartRepository` around (order of) operations and/or lazy-load
capabilities.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit starts with the optimization of the `DepenendencyManager`,
ensuring the chart indexes are lazy loaded, and replacing the
(limitless) concurrency with a configurable number of workers with a
default of 1.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commits adds simple caching capabilities to the
`ChartRepository`, which makes it possible to load the `Index` from a
defined `CachePath` using `LoadFromCache()`, and to download the index
to a new `CachePath` using `CacheIndex()`.
In addition, the repository tests have been updated to make use of
Gomega, and some missing ones have been added.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commits adds `LoadChartMetadataFromArchive` and
`LoadChartMetadataFromDir` helpers to the internal `helm` package
to be able to make observations to the Helm metadata file without
loading the chart in full.
The helpers are compatible with charts of the v1 format (with a
separate `requirements.yaml` file), and an additional
`LoadChartMetadata` helper is available to automatically call the
right `LoadChartMetadataFrom*` version by looking at the file
description of the given path.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit updates `github.com/libgit2/git2go` to `v31.6.1` (with
`libgit2` `1.1.1`), and changes the container image build process so
that it makes use of `ghcr.io/hiddeco/golang-with-libgit2`.
This image provides a pre-build dynamic `libgit2` dependency linked
against OpenSSL and LibSSH2 (without gcrypt), and a set of cross-compile
build tools (see
[rationale](https://github.com/hiddeco/golang-with-libgit2#rationale) and
[usage](https://github.co/hiddeco/golang-with-libgit2#usage) for more
detailed information).
The linked set of dependency should solve most known issues around
unsupport private key types, but does not resolve the issues with ECDSA*
and ED25519 hostkeys yet. Solving this requires a newer version of
`libgit2` (`>=1.2.0`), which currently does not seem to work properly
with `git2go/v32`.
Some small changes have been made to the `libgit2` package to address
(future) deprecations.
Signed-off-by: Hidde Beydals <hello@hidde.co>
controller-runtime's client package provides ObjectKeyFromObject() to
extract NamespacedName from a given object. ObjectKey() in
internal/util package is a helper for the same. Replace the internal
helper with controller-runtime's helper for the same.
Signed-off-by: Sunny <darkowlzz@protonmail.com>
This commit fixes a bug where local chart dependencies would not be
detected correctly due to the absence of a repository URL.
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit upgrades the `controller-runtime` dependency to `v0.7.0`,
including all changes required to make all wiring work again.
- Upgrade `runtime` to v0.6.0 to include `controller-runtime` changes.
- Loggers have been removed from the reconcilers and are now retrieved
from the `context.Context` passed to the `Reconcile` method and
downwards functions.
- Logger configuration flags are now bound to the flag set using
`BindFlags` from `runtime/logger`, ensuring the same contract across
GitOps Toolkit controllers, and the `--log-json` flag has been
deprecated in favour of the `--log-encoding=json` default.
- The `ChangePredicate` from `runtime` has changed to a
`ReconcilateAtChangedPredicate`, and is now chained with the
`GenerationChangedPredicate` from `controller-runtime` using
`predicate.Or`.
- Signatures that made use of `runtime.Object` have changed to
`client.Object`, removing the requirement to e.g. call
`runtime.Object#Object`.
- The `client.MatchingField` function was deprecated, and has been
replaced with `client.MatchingFields{}`.
- The `leader-election-role` was changed, as leader election now works
via the `coordination/v1` API.
Other notable changes:
- `util.ObjectKey` was added to easily construct a `client.ObjectKey` /
`types.NamespacedName` from a `metav1.Object`.
- The `SourceIndexKey` constant has been split out into
`{GitRepository,HelmRepository,Bucket}IndexKey` constants.
Signed-off-by: Hidde Beydals <hello@hidde.co>
As part of the feature implementation to support helm chart
dependencies, the functionality for allowing values files overwriting
from any location scoped to the same source was altered. This should fix
the problem by allowing users to load files from any arbitrary location
as long as it's in the context of the same source from where the helm
chart itself is loaded.
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
Soule BA
Hidde Beydals
Sunny
Sanskar Jaiswal
Kevin McDermott
Stefan Prodan
Rashed Kamal
Hidde Beydals
Somtochi Onyekwere
Mac Chaffee
Paulo Gomes
Max Jonas Werner
Batuhan Apaydın
Furkan
York Chen
Max Jonas Werner
Max Jonas Werner
Eng Zer Jun
Peter Gundel
Peter Gundel
pa250194
Aurel Canciu