Commit Graph

346 Commits

Author SHA1 Message Date
Max Jonas Werner ce072c7eda
better variable names; improved logging
When setup of one of the two controller reconciling HelmRepositories
fails, it's now possible to judge from the log which setup call failed
by regarding the "type" log field.

Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:31:37 +02:00
Max Jonas Werner bb4d886ba2
dockerconfigjson for OCI registry authentication
`loginOptionFromSecret` now derives username/password from a docker
config stored in Secrets of type "kubernetes.io/dockerconfigjson".

Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:31:37 +02:00
Max Jonas Werner 1070d1287a
fix nil pointer dereference
When the Secret referenced in an OCI HelmRepository doesn't contain a
username and password, the controller doesn't panic, anymore.

Signed-off-by: Max Jonas Werner <mail@makk.es>
2022-05-24 10:31:36 +02:00
Sunny 581695b4d6
gitrepo: Intro contentConfigChecksum & improvement
Introduce contentConfigChecksum in the GitRepository.Status to track the
configurations that affect the content of the artifact. It is used to
detect a change in the configuration that requires rebuilding the whole
artifact. This helps skip the reconciliation early when we find out that
the remote repository has not changed.

Moves fetching the included repositories in reconcileSource() to collect
enough information in reconcileSource() to be able to decide if the full
reconciliation can be skipped. This results in reconcileInclude() to
just copy artifact to the source build directory.

Introduce a gitCheckout() method to perform construction of all the git
checkout options and perform the checkout operation. This helps to
easily perform checkout multiple times when we need it in
reconcileSource(). When we check with the remote repository if there's
an update, and find out that there's no update, we check if any other
configurations that affect the source content has changed, like
includes, ignore rules, etc. If there's a change, we need to perform a
full checkout of the remote repository in order to fetch the complete
source. The git checkout no-op optimization is enabled in this method
based on the presence of an artifact in the storage.

The failure notification handler is modifed to handle the recovery of a
no-op reconcile failure and create a notification message accordingly
with the partial commit.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Sunny 5b77f65f46
gitrepo: Enable default feature gates in tests
Introduce a new field in the GitRepositoryReconciler to set the enabled
features. This makes it test friendly compared to using global flags for
setting and checking flags in the tests.

Enable default feature gates in all the GitRepo reconciler tests.

Add test cases for reconcileSource() to test the behavior of optimized
git clone when the Repo is ready and not ready. This ensures that the
full reconciliation is not skipped when GitRepo is not ready.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Sunny 4882cea274
Replace Event error with Generic error in GitRepo
For gradual migration to Generic error, update only the GitRepo
reconciler to use Generic error.

Replace the Waiting error for git no change scenario with a Generic
error with proper no-op, early return, error configurations. This
ensures that the no-op only results in log and K8s native events at
normal level.

Fixes a reconciliation issue when recovering from a failure state (with
previous success state and artifact in the storage) and optimized git
clone feature is on, which results in failure to persist as the git
optimization prevented full reconciliation due to already existing
artifact and removal of failure negative conditions on the object
status. In order to allow failure recovery, the git clone optimizations
are now only applied when the object is already in a ready state.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-05-20 19:52:18 +05:30
Max Jonas Werner 841ed7ae66
[RFC 0002] Flux OCI support for Helm (#690)
* Add OCI Helm support

* users will be able to declare OCI HelmRepository by using the `.spec.type` field of the HelmRepository API. Contrary to the HTTP/S HelmRepository no index.yaml is reconciled from source, instead a simple url and credentials validation is performed.
* For backwards-compatibility, an empty `.spec.type` field leads to the HelmRepository being treated as a plain old HTTP Helm repository.
* users will be able to declare the new OCI HelmRepository type as source using the .Spec.SourceRef field of the HelmChart API. This will result in reconciling a chart from an OCI repository.
* Add registryTestServer in the test suite and OCI HelmRepository test case
* Add a new OCI chart repository type that manage tags and charts from an OCI registry.
* Adapat RemoteBuilder to accept both repository types
* discard output from OCI registry client; The client has no way to set a verbosity level and spamming the controller logs with "Login succeeded" every time the object is reconciled doesn't help much.

Signed-off-by: Soule BA <soule@weave.works>
Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Soule BA <soule@weave.works>
2022-05-19 14:50:16 +02:00
Paulo Gomes 8b50367849
libgit2: Add support for hashed known_hosts
Hashed known_hosts was previously only supported when using
go-git. Now both Git implementations benefit from this
features, and the code coverage across them can ensure no
future regression.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-16 16:57:22 +01:00
Paulo Gomes 262efc08bc
Add git-optimized-clones feature gate
OptimizedGitClones decreases resource utilization for GitRepository
reconciliations. It supports both go-git and libgit2 implementations
when cloning repositories using branches or tags.

This is an opt-out feature, which can be disabled by starting the
controller with the argument '--feature-gates=OptimizedGitClones=false'.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 14:39:21 +01:00
Paulo Gomes 87e03431b0
Emit waiting error when no changes occurred since last reconciliation
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 11:40:10 +01:00
Sanskar Jaiswal 6a793a55f6
Optimize libgit2 checkout tag strategy through condition no-ops
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-05-11 11:40:05 +01:00
Paulo Gomes 860d7051da
libgit2: optimise checkout branch clones
No-op reconciliations are very inefficient, as they carry out
a full clone operation of the target repository even when
no changes have taken place.

This change will execute a remote-ls operation, and cancel
the clone operation if the remote tip commit is still the same
as the one observed on the last reconcilation. In such cases,
an git.NoChangesError is returned.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-11 11:40:04 +01:00
Paulo Gomes 2b59150fbe
tests: algorithms test coverage for go-git and libgit2
Assures support for:
- Authentication Key Types
  - rsa
  - ecdsa P256
  - ecdsa P384
  - ecdsa P521
  - ed25519
- Key Exchange Algoritms:
  - diffie-hellman-group14-sha1
  - diffie-hellman-group14-sha256
  - curve25519-sha256
  - ecdh-sha2-nistp256
  - ecdh-sha2-nistp384
  - ecdh-sha2-nistp521
  - curve25519-sha256@libssh.org
- HostKey Algoritms:
  - ssh-rsa
  - rsa-sha2-256
  - rsa-sha2-512
  - ecdsa-sha2-nistp256
  - ecdsa-sha2-nistp384
  - ecdsa-sha2-nistp521
  - ssh-ed25519

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-05-09 13:31:54 +01:00
Eng Zer Jun 10b92da433
test: use `T.TempDir` to create temporary test directory
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-29 23:01:39 +08:00
Sunny eeaa958866
helmrepo: same revision different checksum condn
This change prevents Reconciling and ArtifactOutdated conditions to be
set on HelmRepo when the checksum of a cached repo index changes.

Adds some tests to ensure that when the repo index is cached, the
revision and checksum of the returned artifact are the same as on the
existing object status.
Also adds checks for the returned artifact and chartRepo from
reconcileSource, to ensure that chartRepo is populated and the checksum
of a new potential artifact is always empty, as it's populated when the
artifact is written in the storage.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-04-27 13:03:51 +05:30
Paulo Gomes 009504b294 helm: optimise repository index loading
Avoid validating (and thus loading) indexes if the checksum already exists in storage.
In other words, if the YAML is identical to the Artifact in storage, the reconciliation should
be a no-op, and therefore can short-circuit long/heavy operations.

Co-authored-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-25 17:00:27 +02:00
Alexander Block fa09aa2a39 tests: Use proper timeouts in gitrepository and helmrepository tests
I assume using "interval" for timeouts was an accident and "timeout" was
actually meant to be used. This also fixes flakiness of tests.

Signed-off-by: Alexander Block <ablock84@gmail.com>
2022-04-25 14:30:25 +02:00
Max Jonas Werner c3d5dac4a8
fix panic when HelmRepository's artifact size is nil
This fixes the immediate issue of the nil pointer dereference but we
still haven't isolated the actual cause of the size being nil to begin
with. This is ongoing work and as soon as we have boiled that down to
the simplest case we will provide a regression test for that case.

closes #680

Signed-off-by: Max Jonas Werner <mail@makk.es>
Co-authored-by: Hidde Beydals <hiddeco@users.noreply.github.com>
2022-04-22 10:29:56 +02:00
Soule BA 0df8dcccec
Enable dependency manager to use in memory cache
If implemented this will:
- enable the helmCharts dependency manager to use the helm in memry
  cache to retrieve reconciled HelmRepositories indexes.
- record cache events.

Signed-off-by: Soule BA <soule@weave.works>
2022-04-19 11:46:27 +02:00
Hidde Beydals 18b59351c1 Update dependencies
Azure SDK dependencies cannot be updated, as this requires us to move to
Go 1.18.

- cloud.google.com/go/storage to v1.22.0
- github.com/ProtonMail/go-crypto to v0.0.0-20220407094043-a94812496cf5
- github.com/darkowlzz/controller-check to v0.0.0-20220325122359-11f5827b7981
- github.com/elazarl/goproxy to v0.0.0-20220403042543-a53172b9392e
- github.com/fluxcd/pkg/gittestserver to v0.5.2
- github.com/go-logr/logr to v1.2.3
- github.com/minio/minio-go/v7 to v7.0.24
- github.com/onsi/gomega to v1.19.0
- golang.org/x/crypto to v0.0.0-20220411220226-7b82a4e95df4
- google.golang.org/api to v0.74.0

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-15 16:41:42 +02:00
Peter Gundel 37551f1964
Reduce file permission
As suggested by @pjbgf

Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>

Co-authored-by: Paulo Gomes <paulo.gomes.uk@gmail.com>
Signed-off-by: Peter Gundel <mail@petergundel.de>
2022-04-13 17:14:45 +02:00
Peter Gundel 0949d0adb9
Change all file permissions to octal format
This better represent permissions as Linux handles such information in
octal format, meaning that the left-most 0 has an important meaning
and is not to be ignored as normally integers would.

See https://github.com/fluxcd/source-controller/issues/603

Signed-off-by: Peter Gundel <mail@petergundel.de>
2022-04-12 21:24:34 +02:00
Paulo Gomes 4198191759
Add flags to configure exponential back-off retry
Add two new flags to enable users to configure exponential
back-off for Flux objects. The default values are now
set to 750ms for minimum retry time, and 15min for max.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-04-12 10:44:46 +01:00
Hidde Beydals e85ea781e2 helm: switch to our own chart loader package
This includes some rewiring of tests, and slight changes in how we work
with the local chart reference. `Path` is expected to be relative to
`WorkDir`, and both fields are now mandatory.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-04-11 11:47:28 +02:00
Sanskar Jaiswal 72a4982541 remove leftover timeout in reconcilers
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-04-07 22:33:07 +05:30
Sunny 5da74ca5a9
Add notify() in all the reconcilers
notify() is used to emit events for new artifact and failure recovery
scenarios. It's implemented in all the reconcilers.
Previously, when there used to be a failure due to any reason, on a
subsequent successful reconciliation, no notification was sent to
indicate that the failure has been resolved.
With notify(), the old version of the object is compared with the new
version of the object to determine if all, if any, of the failures have
been resolved and a notification is sent. The notification message is
the same that's sent in usual successful source reconciliation message
about stored artifact.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-04-07 21:39:16 +05:30
Sunny 44207f46d5
Avoid event logging GC failure
We try to avoid affecting the source reconciliation when there's a
garbage collection related failure.

The event logging was resulting in events and notifications related to
GC failure when the artifact directory isn't created in the first
reconciliation of an object.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-04-07 21:14:12 +05:30
Sanskar Jaiswal f8c27a85dd Garbage collect with provided retention options.
Introduce two new flags to configure the ttl of an artifact and the max
no. of files to retain for an artifact. Modify the gc process to
consider the options and use timeouts to prevent the controller from
hanging.
This helps in situations when the SC has already garbage collected the
current artifact but the advertised artifact url is still the same,
which leads to the server returning a 404.

Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
2022-04-07 18:43:55 +05:30
Soule BA 7ff96a8b0c
Update the docs with a section regarding the cache usage
Signed-off-by: Soule BA <soule@weave.works>
2022-04-02 18:13:59 +02:00
Soule BA 366f5cfde8
Cache HelmRepository index files
If implemented, will provide users with a way to cache index files.

This addresses issues where the index file is loaded and unmarshalled in
concurrent reconciliation resulting in a heavy memory footprint.

The caching strategy used is cache aside, and the cache is a k/v store
with expiration.

The cache number of entries and ttl for entries are configurable.

The cache is optional and is disabled by default

Signed-off-by: Soule BA <soule@weave.works>
2022-04-01 12:41:52 +02:00
Sunny b41c717e16
controllers: emit event and log source up-to-date
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:06:16 +05:30
Sunny 86860ec913
Update all reconcilers with ArtifactInStorage cond
Update alll the other reconcilers similar to the GitRepository
reconcilers to introduce positive condition ArtifactInStorage and
reorder the status conditions.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:06:12 +05:30
Sunny d939e98ec2
Introduce separate positive polarity conditions
Introduce separate positive polarity conditions which are used to set
Ready condition. Move the "artifact stored" ready condition into
ArtifactInStorage positive polarity condition. If ArtifactInStorage is
True and there's no negative polarity condition present, the Ready
condition is summarized with ArtifactInStorage condition value.

Also, update the priorities of the conditions. ArtifactInStorage has
higher priority than SourceVerfied condition. If both are present, the
Ready condition will have ArtifactInStorage.
The negative polarity conditions are reordered to have the most likely
actual cause of failure condition the highest priority, for example
StorageOperationFailed, followed by the conditions that are reconciled
first in the whole reconciliation so as to prioritize the first failure
which may be the cause of subsequent failures.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-30 17:02:59 +05:30
Sunny a62fee5ac0
gitrepo: Do not stall when no included artifact
The GitRepository object with included artifact should not stall when
the included artifact is not available since there's no way to signal a
reconciliation when the included artifact becomes available. The
reconciliation should fail and retry until the included artifact becomes
available.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-25 16:28:37 +05:30
Hidde Beydals 7ca393498c Prefix event annotations with API Group FQDN
This to facilitate improvements on the notification-controller side,
where annotations prefixed with the FQDN of the Group of the Involved
Object will be transformed into "fields".

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-23 19:28:05 +01:00
Sunny eed118b8c5
test: source ignore for subdirectories in a repo
Add gitrepository controller test for source ignore in a repository with
subdirectories where the subdirectories are part of the ignore patterns.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-23 17:17:06 +05:30
Hidde Beydals 501c8be467 fix: configure domain for .sourceignore rules
This ensures the scope while looking for matches is properly set, and
rules are thus correctly matched.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-23 10:29:08 +01:00
Sunny a102d95cf9
Prioritize StorageOperationFailedCondition
Prioritize StorageOperationFailedCondition over other artifact outdated
and unavailable conditions so that when artifact is failing due to
storage operation, it's visble in the ready status condition, making the
reason for not ready more accurate.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-16 21:41:39 +05:30
Paulo Gomes 822788b79e
Fix word casing
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:21 +05:30
Paulo Gomes aa3288112e
Implement Managed Transport for libgit2
libgit2 network operations are blocking and do not provide timeout nor context capabilities,
leading for several reports by users of the controllers hanging indefinitely.

By using managed transport, golang primitives such as http.Transport and net.Dial can be used
to ensure timeouts are enforced.

Co-Authored-by: Sunny <darkowlzz@protonmail.com>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-16 16:22:20 +05:30
Sunny a2d6af126d
Add new condition StorageOperationFailedCondition
Introduce new condition StorageOperationFailedCondition for all the
failures related to the storage. It is a negative polarity condition and
is considered in computing summary of reconciliation.

Also, introduce more granular event reasons related to
StorageOperationFailedCondition for precise reasoning behind failures.
These replace the vague StorageOperationFailedReason.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-14 21:20:37 +05:30
Hidde Beydals 971caf92d5 controllers: finetune `eventLogf` (variant) docs
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Sunny 8d4fce1955 Add more details in helmchart spec doc
Details about the source reference, reconcile strategy and artifact
revision value based on the reconcile strategy.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-11 10:04:14 +01:00
Sunny f59b48c339 Document HelmChart API v1beta2 spec
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-03-11 10:04:14 +01:00
Hidde Beydals cbffd82d8f Document GitRepository API v1beta2 spec
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Hidde Beydals 86d1d80bf2 Document HelmRepository API v1beta2 spec
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Hidde Beydals 5832296ef5 Document Bucket API v1beta2 spec
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-11 10:04:14 +01:00
Hidde Beydals 94c8185d87 controllers: wire Azure Blob Bucket Provider
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-03-03 13:20:48 +01:00
Paulo Gomes 3b4cc52419
Use uppercase TLS in error messages
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 13:02:11 +00:00
Paulo Gomes d9d789fdb1
Reuse transport for helm chart download
Reuses the same transport across different helm chart downloads,
whilst resetting the tlsconfig to avoid cross-contamination.

Crypto material is now only processed in-memory and does not
touch the disk.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-03-02 13:02:11 +00:00