Commit Graph

133 Commits

Author SHA1 Message Date
Soule BA 869c73d0ad
secretRef take precedence over provider
if secretRef is provided, we do not attempt to resolve oidc

Signed-off-by: Soule BA <soule@weave.works>
2022-09-09 14:43:35 +02:00
Soule BA ad3eb5ca47
Enable contextual login for helm OCI
If implemented, this pr will enable user to use the auto login feature
in order to automatically login to their provider of choice's container
registry (i.e. aws, gcr, acr).

Signed-off-by: Soule BA <soule@weave.works>
2022-08-25 22:27:35 +02:00
Max Jonas Werner 64c1b065a8
allow for charts from OCI registries to specify a chart path
This change allows for a HelmRepository to point to e.g. "ghcr.io" and
then a HelmRelease pointing to the chart "stefanprodan/charts/podinfo"
in its `.spec.chart.spec.chart` field.

Related discussion: https://github.com/fluxcd/flux2/discussions/2959

Signed-off-by: Max Jonas Werner <max@e13.dev>
2022-08-08 15:44:43 +02:00
Max Jonas Werner b86572b98e
don't fetch tags when exact version is used in HelmRepository
Taking this shortcut has two benefits:

1. It allows charts to be fetched from AWS's public container registry
   at public.ecr.aws
2. It makes reconciling a HelmChart faster by skipping one or more
   potentially expensive API calls to the registry.

I adapted the unit tests to the new behavior that the
OCIChartRepository doesn't fail anymore for the case where a specific
chart version has been requested that doesn't actually exist in the
registry.

refs #845

Signed-off-by: Max Jonas Werner <max@e13.dev>
2022-07-27 13:25:43 +02:00
York Chen d5a75f6b2f feat: cache helmrepo early after reconcile
1. moved chartRepo.Unload() from reconcileSource() to the defer func in reconcileArtifact to allow caching index in memory
2. added step to init memory cache in reconcileArtifact()
3. added step to save helmrepo index into memory cache in reconcileArtifact()

Signed-off-by: York Chen <ychen@d2iq.com>
2022-07-21 18:17:26 +01:00
Sanskar Jaiswal f5ada743d5 libgit2: decommission unmanaged transport
Decommission libgit2 unmanaged transport and remove the related feature
gate, making managed transport the default.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-07-20 18:56:37 +05:30
Soule BA e4209dba8a
Fix SIGSEGV when resolving charts dependencies
If implemented, this make sure than we clear only referenced
downloaders.

It is also checked if the repository url is supported.

Signed-off-by: Soule BA <soule@weave.works>
2022-07-13 14:59:38 +02:00
Soule BA 361b975bf4
Enable remote dependencies from OCI repositories
If implemented, the source controller will be able to resolve charts
dependencies from OCI repositories.

The remote builder has been refactored as part of this work.

Signed-off-by: Soule BA <soule@weave.works>
2022-07-06 19:11:01 +02:00
Soule BA b402e546bc
Refactor repository logic
Signed-off-by: Soule BA <soule@weave.works>
2022-07-06 19:11:00 +02:00
Somtochi Onyekwere 35a7ea1efa Remove trailing slash
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2022-06-29 11:05:08 +01:00
Sunny d72a189e88
internal/helm/getter: remove transport reuse test
Since the transport reuse is dependent on the garbage collection, the
result is inconsistent. It fails frequently when running the tests with
the go race detector. Remove the test.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-06-14 09:25:52 +01:00
Soule BA a163ea1dff
Fix sorting semver from OCI repository tags
If implemented this fix the issue where we previously did a string
ordering of matching semver versions when retrieving a list of tags from
an OCI registry.

Signed-off-by: Soule BA <soule@weave.works>
2022-06-07 22:20:36 +02:00
Soule BA 0d4d78fc6b
Fix repository cache regression
If implemented this make sure we don't unload an index before caching it
during a chart build phase.

Signed-off-by: Soule BA <soule@weave.works>
2022-06-06 13:15:10 +02:00
Sunny f79fd034fd
registry: repo URL and dockerconfig URL mismatch
Registry login option should verify that the obtained dockerconfig
credentials are for the same host. When the helmrepo URL and the URL in
docker auth config don't match, the docker config store returns an
empty auth config, instead of failing. This results in accepting empty
username and password. The HelmRepo would appear to be ready in such
situation because the creds are empty, no login is attempted. But when
a HelmChart tries to use the login options, it'd fail.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-06-01 00:09:36 +05:30
Sanskar Jaiswal 972d1cac2a fix docs, error handling and managed proxy auth
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Sanskar Jaiswal 7501e8622c add method to disable features internally
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-27 14:01:23 +05:30
Sunny 321317971f
reconcile: Set observed gen when conditions exist
The observed generation must be set only when actual observation is
made. When an actual observation is made, some conditions are set on the
object. Introduce a helper function
addPatchOptionWithStatusObservedGeneration() to set the patcher option
WithStatusObservedGeneration only when there's any condition in the
status.

Updates the existing tests that depended on this behavior.

This fixes the issue where the observed generation is set by the patcher
when a reconciler does an early return for setting the finalizers only.
With this, the observed generation will be updated only when some
observations are made on the object based on the usual rules of success
result, no error, ignore error and stalled condition.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-25 14:28:18 +05:30
Paulo Gomes a0d0a6312d
libgit2: enable managed transport by default
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-24 10:46:32 +01:00
Max Jonas Werner bb569bec1f
include Secret name in returned errors
Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:32:42 +02:00
Max Jonas Werner 7cfd94effb
fix func doc
Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:32:42 +02:00
Max Jonas Werner a3be7e5d3d
document generateBuildResult
Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:32:42 +02:00
Max Jonas Werner c795da2280
introduce `internal/helm/registry` package
This new package holds all Helm OCI registry-specific code now so we
have a single location to look for such code which makes it easier to
find yourself around.

Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:32:38 +02:00
Sunny 5d154a83dc
Introduce Generic error in reconcile
Add Generic error in RuntimeResultBuilder and ComputeReconcileResult
implementation with consideration to the error configurations.

Safeguards are added in the runtime result builder to ensure default
requeue after interval is set when is's set to zero or unset.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Sunny 30fe0dc6aa
Introduce ErrorActionHandler ResultProcessor
ErrorActionHandler processes the reconciliation error results based on
their configurations. It performs actions like logging and event
recording based on the error configuration. More actions can be
accommodated in the future with more error configurations.

It can be a replacement for RecordContextualError() which does the same
operations but can't be configured much.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Sunny ddd26f4d52
Introduce Generic error and error Config
Generic error is an attempt to avoid creating new error type for every
new unique scenario. It can be used to configure and build custom error
handling behavior, logging and event recording at present.
Contextual errors, Stalling and Waiting error, have special meaning for
the reconciliation results. But the Event error type can be replaced
with Generic error with some specific configurations. The Event error
is kept for a gradual migation to Generic error. Similarly, the Generic
error can be used to easily create new error handling behaviors.

The error Config can be used to configure any of the errors, including
contextual errors, without altering their contextual meaning, to modify
how they are handled.

The error constructors configure the errors with common default
configurations. These configurations can be modified to alter the
behavior.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:03 +05:30
Max Jonas Werner 841ed7ae66
[RFC 0002] Flux OCI support for Helm (#690)
* Add OCI Helm support

* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.

Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
2022-05-19 14:50:16 +02:00
Paulo Gomes 262efc08bc
Add git-optimized-clones feature gate
OptimizedGitClones decreases resource utilization for GitRepository
reconciliations. It supports both go-git and libgit2 implementations
when cloning repositories using branches or tags.

This is an opt-out feature, which can be disabled by starting the
controller with the argument '--feature-gates=OptimizedGitClones=false'.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 14:39:21 +01:00
Sunny 2240106c8e
summarize: Consider obj status condition in result
SummarizeAndPatch() should also consider the object's status conditions
when computing and returning the runtime results to avoid any
inconsistency in the runtime result and status condition of the object.
When an object's Ready condition is False, the reconciler should retry
unless it's in stalled condition.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-04-30 10:12:21 +05:30
Eng Zer Jun 10b92da433
test: use `T.TempDir` to create temporary test directory
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-29 23:01:39 +08:00
Paulo Gomes 009504b294 helm: optimise repository index loading
Avoid validating (and thus loading) indexes if the checksum already exists in storage.
In other words, if the YAML is identical to the Artifact in storage, the reconciliation should
be a no-op, and therefore can short-circuit long/heavy operations.

Co-authored-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-25 17:00:27 +02:00
Soule BA 0df8dcccec
Enable dependency manager to use in memory cache
If implemented this will:
- enable the helmCharts dependency manager to use the helm in memry
  cache to retrieve reconciled HelmRepositories indexes.
- record cache events.

Signed-off-by: Soule BA <soule@weave.works>
2022-04-19 11:46:27 +02:00
Hidde Beydals 18b59351c1 Update dependencies
Azure SDK dependencies cannot be updated, as this requires us to move to
Go 1.18.

- cloud.google.com/go/storage to v1.22.0
- github.com/ProtonMail/go-crypto to v0.0.0-20220407094043-a94812496cf5
- github.com/darkowlzz/controller-check to v0.0.0-20220325122359-11f5827b7981
- github.com/elazarl/goproxy to v0.0.0-20220403042543-a53172b9392e
- github.com/fluxcd/pkg/gittestserver to v0.5.2
- github.com/go-logr/logr to v1.2.3
- github.com/minio/minio-go/v7 to v7.0.24
- github.com/onsi/gomega to v1.19.0
- golang.org/x/crypto to v0.0.0-20220411220226-7b82a4e95df4
- google.golang.org/api to v0.74.0

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-15 16:41:42 +02:00
Peter Gundel 37551f1964
Reduce file permission
As suggested by @pjbgf

Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>

Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Signed-off-by: Peter Gundel <mail@petergundel.de>
2022-04-13 17:14:45 +02:00
Peter Gundel 0949d0adb9
Change all file permissions to octal format
This better represent permissions as Linux handles such information in
octal format, meaning that the left-most 0 has an important meaning
and is not to be ignored as normally integers would.

See https://github.com/fluxcd/source-controller/issues/603

Signed-off-by: Peter Gundel <mail@petergundel.de>
2022-04-12 21:24:34 +02:00
Hidde Beydals 9a17fd53e7 helm: attach loader to helm.MaxChartFileSize
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 11:47:28 +02:00
Hidde Beydals e85ea781e2 helm: switch to our own chart loader package
This includes some rewiring of tests, and slight changes in how we work
with the local chart reference. `Path` is expected to be relative to
`WorkDir`, and both fields are now mandatory.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 11:47:28 +02:00
Hidde Beydals b9063d7362 helm: add more test coverage for secureloader
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 11:47:28 +02:00
Hidde Beydals 6fc066b1b6 helm: introduce customized chart loaders
This introduces our own `secureloader` package, with a directory
loader that's capable of following symlinks while validating they stay
within a certain root boundary.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 09:36:14 +02:00
Hidde Beydals 5ae30cb4aa helm: drop github.com/pkg/errors
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 09:36:14 +02:00
Hidde Beydals 25f54ee80e sympath: provide abs path after eval symlink
This can be used to detect traversion outside of a certain path scope
while walking.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 09:36:14 +02:00
Hidde Beydals ad597b352c helm: copy internal ignore and sympath modules
We require these to be able to mimic Helm's own directory loader, and
surprisingly (for `ignore` at least), these are not public.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 09:36:14 +02:00
Sunny 5da74ca5a9
Add notify() in all the reconcilers
notify() is used to emit events for new artifact and failure recovery
scenarios. It's implemented in all the reconcilers.
Previously, when there used to be a failure due to any reason, on a
subsequent successful reconciliation, no notification was sent to
indicate that the failure has been resolved.
With notify(), the old version of the object is compared with the new
version of the object to determine if all, if any, of the failures have
been resolved and a notification is sent. The notification message is
the same that's sent in usual successful source reconciliation message
about stored artifact.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-04-07 21:39:16 +05:30
Soule BA 7ff96a8b0c
Update the docs with a section regarding the cache usage
Signed-off-by: Soule BA <soule@weave.works>
2022-04-02 18:13:59 +02:00
Soule BA 0f9302827c
Add MIT Licence copyright notice
Signed-off-by: Soule BA <soule@weave.works>
2022-04-01 12:41:53 +02:00
Soule BA 366f5cfde8
Cache HelmRepository index files
If implemented, will provide users with a way to cache index files.

This addresses issues where the index file is loaded and unmarshalled in
concurrent reconciliation resulting in a heavy memory footprint.

The caching strategy used is cache aside, and the cache is a k/v store
with expiration.

The cache number of entries and ttl for entries are configurable.

The cache is optional and is disabled by default

Signed-off-by: Soule BA <soule@weave.works>
2022-04-01 12:41:52 +02:00
Paulo Gomes 3819ac37bc
Move TransportPool to its own package
The experimental managed transport can also leverage TransportPool,
moving it to its own package to accommodate that use case.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-25 19:08:50 +00:00
Hidde Beydals aa40a3a313 mock/gcs: strip /storage/v1 URI element
This is required after the update of `cloud.google.com/go/storage` from
`v1.16.0` to `v1.21.0`, due to changed `STORAGE_EMULATOR_HOST`
behavior.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-16 11:16:47 +01:00
Paulo Gomes 7d61553463
Remove dependency to helm v2
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 17:58:15 +00:00
Paulo Gomes f63681f372
Improve TransportPool documentation
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 13:53:31 +00:00
Paulo Gomes d9d789fdb1
Reuse transport for helm chart download
Reuses the same transport across different helm chart downloads,
whilst resetting the tlsconfig to avoid cross-contamination.

Crypto material is now only processed in-memory and does not
touch the disk.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 13:02:11 +00:00