Commit Graph

1133 Commits

Author SHA1 Message Date
Sunny ba7cbd31f1 bucket: Add more reconciler design improvements
- Remove ArtifactUnavailable condition and use Reconciling condition to
  convey the same.
- Make Reconciling condition affect the ready condition.
- Introduce summarizeAndPatch() to calculate the final status conditions
  and patch them.
- Introduce reconcile() to iterate through the sub-reconcilers and
  execute them.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 848534a8f1 bucket: Ignore patch error not found on delete
Ignore "not found" error while patching when the delete timestamp is
set.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny f472cadab4 Add bucket controller tests for reconcileGCPSource
- Introduce mock GCP Server to test the gcp bucket client against mocked
  gcp server results.
- Add tests for reconcileGCPSource().
- Patch GCPClient.BucketExists() to return no error when the bucket
  doesn't exists. This keeps the GCP client compatible with the minio
  client.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 25779241ec Add more reconcileMinioSource test cases
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny c79a55baa8 BucketReconciler: Add reconcileArtifact tests
Add `BucketReconciler.reconcileArtifact` tests based on
`GitRepositoryReconciler.reconcileArtifact` test cases.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals 84301e6370 Consolidate condition types into `FetchFailed`
This commit consolidates the `DownloadFailed` and `CheckoutFailed`
Condition types into a new more generic `FetchFailed` type to simplify
the API and observations by consumers.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 89ba8374b6 Rewrite `BucketReconciler` to new standards
This commit rewrites the `BucketReconciler` to new standards, while
implementing the newly introduced Condition types, and trying to
adhere better to Kubernetes API conventions.

More specifically it introduces:

- Implementation of more explicit Condition types to highlight
  abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
  objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
  conditions.
- Refactor of reconciler logic, including more efficient detection of
  changes to bucket objects by making use of the etag data available,
  and downloading of object files in parallel with a limited number of
  workers (4).
- Integration tests that solely rely on `testenv` and do not
  use Ginkgo.

There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny 52f4a2a800 bucket: Replace GetInterval() with GetRequeueAfter()
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny c4fa79c85e gitrepo: Fix reconcileInclude() includes
The artifacts built in reconcileInclude() should be persisted by writing
it to includes. reconcileArtifact() later adds includes to the git repo
object status and persists it.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 2acb721c35 gitrepo: Use internal/util for creating temp dir
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 5767291b58 gitrepo: Add more reconciler design improvements
- Remove ArtifactUnavailable condition and use Reconciling condition to
  convey the same.
- Make Reconciling condition affect the ready condition.
- Introduce summarizeAndPatch() to calculate the final status conditions
  and patch them.
- Introduce reconcile() to iterate through the sub-reconcilers and
  execute them.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny d9a947c909 gitrepo: Ignore patch error not found on delete
While deleting, patching an object with new status results in "not
found" error because the object is already deleted. The patching
operation first patches the status conditions, the rest of the object
and, at the very end, the rest of the status. When an object is
deleted, the garbage collection results in the artifact in the status
to be updated, resulting in a diff that is attempted to be patched
when the deferred patch runs.
Since the status patching runs at the very end, the object gets deleted
before it can be patched.
Ignore "not found" error while patching when the delete timestamp is
set.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 25ae83ad7d gitrepo: Fix SourceVerifiedCondition condition type
SourceVerifiedCondition is a normal condition, remove it from negative
polarity conditions. Add SourceVerifiedCondition in patch option
WithOwnedConditions.

Also, Update the signature of reconcileInclude() to remove include
being passed and overwritten in the first line. Include is available
as part of the passed source object.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals 88dc2e6ed6 chore: ensure Git server dir is removed after test
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals dbaf21b974 Wrap err with context instead of logging twice
This wraps the errors which are returned instead of logging them, as
the returned error is logged at the end of the reconcile run.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals a437ed69bd Consolidate condition types into `FetchFailed`
This commit consolidates the `DownloadFailed` and `CheckoutFailed`
Condition types into a new more generic `FetchFailed` type to simplify
the API and observations by consumers.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny 3d5698856a gitrepo: test reconcileArtifact condtns & symlink
Adds test cases for reconcileArtifact to check if old status
conditions are removed after new artifact is created.
Adds a test case to verify that the latest artifact symlink points to
the created artifact.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 59b3e5da5d gitrepo: Add tests for old conditions update
This tests the status conditions update in the gitrepository reconciler.
Given a mix of old status conditions, on a successful reconciliation,
the status condition is set to Ready=True.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 7e71185594 gitrepo: reconcileInclude test assertion fixes
Use the created artifact server test storage in reconcileInclude
test's GitRepositoryReconciler and cleanup the created storage.

Fix the test assertions to check the copied artifact directories in
the correct path. Also, update the tests to expect artifacts in the
include `toPath` to exist.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals c2e6875284 Tweak logged messages
- Mention the current revision in the up-to-date log message.
- Ensure any error that is "swallowed" (not returned) is logged to
  ensure they are visible within the logs, and not just by inspecting
  the object.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 1e326e8f1c source: `GetRequeueAfter` in place of `GetInterval`
The problem with `GetInterval()` was that the returned type was of
`metav1.Duration`, while almost anywhere it was used, a type of
`time.Duration` was requested. The result of this was that we had to
call `GetInterval().Duration` all the time, which would become a bit
cumbersome after awhile.

To prevent this, we introduce a new `GetRequeueAfter() time.Duration`
method, which both results the right type, and bears a name that is
easier to remember where the value is used most; while setting the
`Result.RequeueAfter` during reconcile operations.

The introduced of this method deprecates `GetInterval()`, which should
be removed in a future MINOR release.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 67bd2cb81c Replace %q in messages with '%s'
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny 18180776e7 controllers: Add more tests for reconcileArtifact
Fixes error returned from target path validation check and adds more
test cases for TestGitRepositoryReconciler_reconcileArtifact.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny b814070bc2 Fixes to gitrepo reconciler tests
NOTE: This should be amended with the previous commit which has
commented out tests.

Update reconcileSource() to work with the test case where no secret is
set. A minimal auth options is created and used for git checkout.

Update TestGitRepositoryReconciler_verifyCommitSignature() to use the
new git.Commit type.

Update TestGitRepositoryReconciler_reconcileSource_checkoutStrategy to
add skipForImplementation for branch commit test case.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Hidde Beydals 31d2e6d65c Rewrite `GitRepositoryReconciler` to new standards
This commit rewrites the `GitRepositoryReconciler` to new standards,
while implementing the newly introduced Condition types, and trying
to adhere better to Kubernetes API conventions.

More specifically it introduces:

- Implementation of more explicit Condition types to highlight
  abnormalities.
- Extensive usage of the `conditions` subpackage from `runtime`.
- Better and more conflict-resilient (status)patching of reconciled
  objects using the `patch` subpackage from runtime.
- Proper implementation of kstatus' `Reconciling` and `Stalled`
  conditions.
- First (integration) tests that solely rely on `testenv` and do not
  use Ginkgo.

There are a couple of TODOs marked in-code, these are suggestions for
the future and should be non-blocking.
In addition to the TODOs, more complex and/or edge-case test scenarios
may be added as well.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 5ab2f6219b internal/util: introduce temp dir/path helpers
In most of the reconcilers we have a repetative pattern of using part of
the object metadata to construct a temporary file path.

This commit introduces helpers as an abstraction, for both the creation
of a temporary directory based on `client.Object` type and object
metadata, and the generation of an arbitrary random temporary path
string.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny 5df4acb710 Add internal packages error and reconcile
- internal/error - Contains internal error type used across the
  source-controller reconcilers.
- internal/reconcile - Contains helper abstractions for the
  controller-runtime reconcile Result type and functions to
  interact with the abstractions.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 6789aaf147 api: Embed runtime.Object in Source interface
Embedding runtime.Object in Source interface makes the Source type more
useful to interact with k8s API machinery.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny dc9d8b7f66 Add gomega matcher for artifact
Signed-off-by: Sunny <darkowlzz@protonmail.com>

Co-authored-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny b1233dc24f Move Artifact conditions to conditions
Also, introduce FetchFailedCondition for generic fetch failures.

Signed-off-by: Sunny <darkowlzz@protonmail.com>

Co-authored-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Sunny a1efbad15a Use new events and metrics helpers in main.go
Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-23 12:35:30 +01:00
Sunny 9eb6833d4d source: Add `GetRequeueAfter`
The problem with `GetInterval()` was that the returned type was of
`metav1.Duration`, while almost anywhere it was used, a type of
`time.Duration` was requested. The result of this was that we had to
call `GetInterval().Duration` all the time, which would become a bit
cumbersome after awhile.

To prevent this, we introduce a new `GetRequeueAfter() time.Duration`
method, which both results the right type, and bears a name that is
easier to remember where the value is used most; while setting the
`Result.RequeueAfter` during reconcile operations.

The introduction of this method deprecates `GetInterval()`, which
should be removed in a future MINOR release.

Signed-off-by: Sunny <darkowlzz@protonmail.com>

Co-authored-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals e190059cc7 Introduce `artifactSet` to replace `hasArtifactUpdated`
NOTE: Remove `hasArtifactUpdated` in the future once it's no longer
used.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 349739b7e4 Implement new runtime interfaces, prepare testenv
This commit ensures all API objects implement the interfaces used by
the runtime package to work with conditions, etc., and prepares the
test suite to work with the `pkg/runtime/testenv` wrapper.

Changes are made in a backwards compatible way (that being: the
existing code can still be build and works as expected), but without
proper dependency boundaries. The result of this is that the API
package temporary depends on the runtime package, which is resolved
when all reconcilers have been refactored and the API package does
no longer contain condition modifying functions.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals e42eedd392 Introduce more explicit Condition types
This commit introduces new Condition types to the v1beta1 API,
facilitating easier observation of (potentially) problematic state for
end-users.

- `ArtifactUnavailableCondition`: indicates there is no artifact
  available for the resource. This Condition should be set by the
  reconciler as soon as it observes the absence of an artifact for a
  source.
- `CheckoutFailedCondition`: indicates a transient or persistent
  checkout failure. This Condition should be set by the reconciler as
  soon as it observes a Git checkout failure, including any
  prerequisites like the unavailability of the referenced Secret used
  for authentication. It should be deleted as soon as a successful
  checkout has been observed again.
- `SourceVerifiedCondition`: indicates the integrity of the source has
  been verified. The Condition should be set to True or False by the
  reconciler based on the result of the integrity check.
  If there is no verification mode and/or secret configured, the
  Condition should be removed.
- `IncludeUnavailableCondition`: indicates one of the referenced
  includes is not available. This Condition should for example be set
  by the reconciler when the include does not exist, or does not have
  an artifact. If the includes become available, it should be deleted.
- `ArtifactOutdatedCondition`: indicates the current artifact of the
  source is outdated. This Condition should for example be set by the
  reconciler when it notices there is a newer revision for an artifact,
  or the previously included artifacts differ from the current available
  ones. The Condition should be removed after writing a new artifact
  to the storage.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:30 +01:00
Hidde Beydals 7c3c14997e Switch to v1beta2 API package
Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:35:25 +01:00
Sunny cc1e48243d Introduce v1beta2 API package
This commit introduces a v1beta2 API package for the staged breaking
changes around conditions and general usage of the API objects.

Signed-off-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Sunny <darkowlzz@protonmail.com>
Co-authored-by: Hidde Beydals <hello@hidde.co>
2022-02-23 12:34:35 +01:00
Hidde Beydals bd3d7817d0
Merge pull request #589 from fluxcd/libgit2-branch-checkout 2022-02-22 17:15:37 +01:00
Hidde Beydals eff40e22e9 git/libgit2: assert proper test of default branch
If there is no configuration set for `init.defaultBranch`, it does not
return an error but an empty string. We now take this into account so
we do not overwrite the default, and make the default `master` to match
with libgit2 defaults.

In addition, some comments have been added to not get confused about
what commits we are checking against.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-22 16:31:56 +01:00
Hidde Beydals 15c064abdf git/libgit2: set CheckoutForce on branch strategy
In the recent update from libgit2 1.1.x to 1.3.x, something seems to
have changed upstream. Resulting in the clone of a branch ending up
with a semi-bare file system state (in other words: without any files
present in the directory).

This commit patches the clone behavior to set the `CheckoutForce`
strategy as `CheckoutOption`, which mitigates the issue.

In addition, test cases have been added to ensure we do not run into
this again by asserting the state of the branch after cloning.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-22 16:31:56 +01:00
Paulo Gomes b65cf9d535
Merge pull request #583 from pjbgf/controller-runtime
Upgrade controller-runtime to v0.11.1 and docker/distribution to v2.8.0
2022-02-18 09:12:27 +00:00
Paulo Gomes 6e46d7fe55
Upgrade docker/distribution to v2.8.0
Fixes https://github.com/advisories/GHSA-qq97-vm5h-rrhg

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-17 20:51:48 +00:00
Paulo Gomes 00ff9fb2fa
Upgrade controller-runtime to v0.11.1
Fix for CVE-2022-21698 by upgrading the trasient dependency github.com/prometheus/client_golang.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-17 20:51:47 +00:00
Stefan Prodan 35d785ea62
Merge pull request #584 from pjbgf/maintainers
Add pjbgf to Maintainers
2022-02-17 13:42:01 +02:00
Paulo Gomes ec89a2d067
Add pjbgf to Maintainers
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-17 11:24:02 +00:00
Hidde Beydals 2c4818fda5
Merge pull request #573 from pjbgf/bump-libgit2 2022-02-16 13:55:42 +01:00
Paulo Gomes 8429708997
Upgrade libgit2 to libgit2-1.3.0-2
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 11:39:11 +00:00
Paulo Gomes f0d7a6bb48
Update libgit2 attributions
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 10:30:27 +00:00
Paulo Gomes 514126c4b8
Fix make test on arm64 runners
The environment variables set at the Makefile were causing go install to
yield a corrupted file for setup-envtest. To fix the issue, such operation
is now always executed in a clean bash.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 10:17:33 +00:00
Paulo Gomes e5d032fe9c
Add libgit2 checkout test with ED25519 key
This adds a test to detect any regression in libgit2's ED25519 key
support. go-git supports ED25519 but not the current version of
libgit2 used in flux. The updates to libgit2 in v1.2.0 adds support
for ED25519. This test would help ensure the right version of libgit2
is used.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
2022-02-16 10:17:32 +00:00