Commit Graph

191 Commits

Author SHA1 Message Date
Dylan Arbour 4a834e1d2d Add `ValuesFiles` to HelmChart spec
Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
2021-04-19 09:16:53 -04:00
Hidde Beydals b5004a93bc Make Storage#Archive file filtering configurable
This commit makes the filtering applied during the archiving
configurable by introducing an optional `ArchiveFileFilter`
callback argument and a `SourceIgnoreFilter` implementation.

`SourceIgnoreFilter` filters out files matching
sourceignore.VCSPatterns and any of the provided patterns.
If an empty gitignore.Pattern slice is given, the matcher is set to
sourceignore.NewDefaultMatcher.

The `GitRepository` now loads the ignore patterns before archiving
the repository contents by calling `sourceignore.LoadIgnorePatterns`
and other helpers. The loading behavior is **breaking** as
`.sourceignore` files in the (subdirectories of the) repository are
now still taken into account if `spec.ignore` for a resource is
defined, overwriting is still possible by creating an overwriting
rule in the `spec.ignore` of the resource.

This change also makes it possible for the `BucketReconciler` to not
configure a callback at all and prevent looking for ignore
matches twice. To finalize the bucket refactor, a change to the
reconciler has been made to look for a `.sourceignore` file in
the root of the bucket to provide an additional way of configuring
(global) exclusions.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-04-13 15:34:20 +02:00
Hidde Beydals cca2c4a362 Check ignore matches before Bucket item downloads
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-04-13 15:34:20 +02:00
Dylan Arbour c0bd4ab7d0 Test values overrides
Adds a test that loads the helmChart from the updated resource and
verifies that `testOverride` (the value overrode in the test fixtures)
changes from `false` to `true`.

Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>
2021-04-08 18:57:05 -04:00
Stefan Prodan 9a08c0cc52
Add well-known CI configs to exclusion list
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-03-31 14:39:12 +03:00
Michael Bridgen 681ddd5db0 Test RecurseSubmodules
This commit adds a test specifically for RecurseSubmodules. It takes a
bit more preparation, since it needs a repo using submodules to start
with. go-git doesn't appear to support adding submodules
programmatically, so the preparation is done in part by execing `git`.

Signed-off-by: Michael Bridgen <michael@weave.works>
2021-03-31 12:22:10 +01:00
Stefan Prodan 664a568822
Add support for Git submodules with go-git
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-03-30 13:00:13 +03:00
Stefan Prodan f0016cfad1
Enable self-signed certs for go-git
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-03-29 13:23:32 +03:00
Somtochi Onyekwere 2624ba93a3 Record suspension metric
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
2021-03-17 14:04:21 +01:00
Raffael Sahli 1e19503359
break before default
Signed-off-by: Raffael Sahli <raffael.sahli@doodle.com>
2021-02-09 17:35:02 +01:00
Raffael Sahli bc3c4e2a36
fixes writing chart twice which results in a 0bytes tgz
Signed-off-by: Raffael Sahli <raffael.sahli@doodle.com>
2021-02-09 10:17:07 +01:00
Hidde Beydals fac1afa2a8 Move `git/common` to `git`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-02-08 14:05:05 +01:00
Philip Laine c063484761 Add custom certificate validation
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2021-02-08 12:19:22 +01:00
Thomas Runyon 8428054575 Properly escape outer loop for present dependency
To prevent dependencies that are already present locally to be included.

Signed-off-by: Thomas Runyon <runyontr@gmail.com>
2021-02-02 16:16:40 +01:00
Hidde Beydals fcc5fc8d32 typo: 'seperated' -> 'separated'
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-02-01 15:14:13 +01:00
Hidde Beydals 2c09df6570 Be more verbose about invalid chart name
Includes a change to _not_ requeue after validation failure, as
there is no chance on recovery.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-01-29 15:26:54 +01:00
Hidde Beydals 3cdc897236 Assume local dependency without Helm repository
This commit fixes a bug where local chart dependencies would not be
detected correctly due to the absence of a repository URL.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-01-28 12:28:19 +01:00
Léo Martinez c1512d34a0
rely on index checksum for HelmRepository reconcile operation
Fixes #256

Signed-off-by: Léo Martinez <leo84.martinez@gmail.com>
2021-01-25 19:11:57 +01:00
Stefan Prodan 207ed99d72
Use LocalObjectReference from fluxcd/pkg/meta v0.7
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-01-21 14:18:42 +02:00
Aurel Canciu f1b5768200
Fix values file override
`io.Read` was used incorrectly to read from the override file provided
by the user.
This is now replaced with `ioutil.ReadFile` for better handling and
error reporting.

Fixes #263

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2021-01-21 13:40:12 +02:00
Aurel Canciu 778b07cdd4
Fix unreachable code issue in helmchart reconciler
Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2021-01-18 23:06:30 +02:00
Hidde Beydals 34c4290cda Alin HelmChart source indexer and watches
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-01-18 17:08:25 +01:00
Hidde Beydals 566ce8aa15 Upgrade runtime package to v0.6.2
To include a bug fix to the `ReconcilateAtChangedPredicate`
and renaming to `ReconcileRequestedPredicate`.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-01-14 11:39:47 +01:00
Hidde Beydals fda481efe9 Upgrade controller-runtime to v0.7.0
This commit upgrades the `controller-runtime` dependency to `v0.7.0`,
including all changes required to make all wiring work again.

- Upgrade `runtime` to v0.6.0 to include `controller-runtime` changes.
- Loggers have been removed from the reconcilers and are now retrieved
  from the `context.Context` passed to the `Reconcile` method and
  downwards functions.
- Logger configuration flags are now bound to the flag set using
  `BindFlags` from `runtime/logger`, ensuring the same contract across
  GitOps Toolkit controllers, and the `--log-json` flag has been
  deprecated in favour of the `--log-encoding=json` default.
- The `ChangePredicate` from `runtime` has changed to a
  `ReconcilateAtChangedPredicate`, and is now chained with the
  `GenerationChangedPredicate` from `controller-runtime` using
  `predicate.Or`.
- Signatures that made use of `runtime.Object` have changed to
  `client.Object`, removing the requirement to e.g. call
  `runtime.Object#Object`.
- The `client.MatchingField` function was deprecated, and has been
  replaced with `client.MatchingFields{}`.
- The `leader-election-role` was changed, as leader election now works
  via the `coordination/v1` API.

Other notable changes:

- `util.ObjectKey` was added to easily construct a `client.ObjectKey` /
  `types.NamespacedName` from a `metav1.Object`.
- The `SourceIndexKey` constant has been split out into
  `{GitRepository,HelmRepository,Bucket}IndexKey` constants.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-18 19:17:10 +01:00
Aurel Canciu a55c502bb4
Fix HelmChart valuesFile chart path restriction
As part of the feature implementation to support helm chart
dependencies, the functionality for allowing values files overwriting
from any location scoped to the same source was altered. This should fix
the problem by allowing users to load files from any arbitrary location
as long as it's in the context of the same source from where the helm
chart itself is loaded.

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-12-17 12:57:41 +02:00
Hidde Beydals 7cefdee6d8 Prevent HelmChart controller panic on init failure
The controller logic is a serious candidate to be refactored so that
errors can be taken into account better, and do not always result in a
requeue. But this will do for the time being.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-16 15:25:17 +01:00
Hidde Beydals 29a051c5f4 Refactor and document DependencyManager
Mostly to re-use the fields of the structure instead of copying things
around.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-15 15:37:42 +01:00
Hidde Beydals 8d0b54e431 Make proper use of errgroup context
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-15 09:56:40 +01:00
Hidde Beydals bc890874e1 Add safe guards for relative paths
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-15 09:56:40 +01:00
Hidde Beydals 9881bc2845 Use ListObjects V1 fallback for GCS S3 endpoints
As GCP's S3 interoperability does not implement the ListObjectsV2
implementation.

Ref: https://cloud.google.com/storage/docs/interoperability

> Note: While most actions are interoperable with the Amazon S3 V2 SDK,
> listing objects can only be performed using the Amazon S3 V1 list
> objects method.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-15 09:41:45 +01:00
Hidde Beydals 5f3c014966 Validate provided name for charts from HelmRepos
Following the rules described in
https://helm.sh/docs/chart_best_practices/conventions/#chart-names.

This guards against people following the wrong guidance of Artifactory,
that supports and promotes repository indexes with e.g. '/' in the
chart names.

In a future version this should be moved to a validation webhook, but
there are still discussions ongoing around the TLS certificates for
this.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-09 15:37:10 +01:00
Philip Laine f044ebc253 Rename git2go option to libgit2
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-06 23:19:57 +01:00
Philip Laine 1b4eacc588 Refactor argument name to enable git2go
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-04 10:07:17 +01:00
Philip Laine cb12736a84 Update pkg git server
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-03 12:56:38 +01:00
Philip Laine 47e4e54550 Add test to show that libgit2 fails with self signed test
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2020-12-02 23:28:32 +01:00
Philip Laine 5ff65d7ae7 Add user support to git2go implementation
Signed-off-by: Philip Laine <philip.laine@gmail.com>
2020-12-02 20:18:29 +01:00
Philip Laine 2cb3754953 Add support for gitv2 servers
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-12-02 20:18:29 +01:00
Hidde Beydals 67b10aad22 Respect configured user in SSH Git repository URL
We had a hardcoded assumption that the SSH user for a Git repository is
always "git". This is however not true in all scenarios, for example
when one is making use of Gerrit for team code collaboration, as users
there have their own username for (SSH) Git operations.

This commit changes the logic of the auth strategy helpers to:

1. Select the auth strategy based on the protocol of the parsed URL,
   instead of a simple rely on a correct prefix.
2. Use the user information from the parsed URL to configure the user
   for the public key authentication strategy, with a fallback to `git`
   if none is defined.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-30 16:26:23 +01:00
Hidde Beydals 254de30f6a List HelmChart for GitRepository change
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-26 18:03:04 +01:00
Hidde Beydals 0c904a1879 Watch chart sources for revision changes
To enqueue a new reconciliation for the HelmChart sources as soon as
the revision of their upstream source changes.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-26 13:02:55 +01:00
Stefan Prodan 5579dc8c96
Implement source suspension
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2020-11-26 12:58:00 +02:00
Hidde Beydals 0b4586253c Add LastHandledReconcileAt to source statuses
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-19 13:21:59 +01:00
Aurel Canciu b80f450b68
Switch to new pkg/apis/meta SetResourceCondition
Use SetResourceCondition as a generic method to set conditions for CRs,
implmeneting the ObjectWithStatusConditions interface used as input
type.

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-11-19 13:08:33 +02:00
Aurel Canciu 00bb853d0e
Refactor to adopt k8s standardized Condition type
Updates to use metav1.Condition type and removes references for
deprecated corev1.Condition* constants and uses the new k8s api/meta
helpers in place of the old pkg/apis/meta types.

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-11-19 13:08:33 +02:00
Hidde Beydals a3f177328a Move Helm dependency manager to `helm` package
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-19 09:50:50 +01:00
Hidde Beydals 2f50e3ff59 Make GC behavior depend on deletion timestamp
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-18 23:03:44 +01:00
Hidde Beydals 93d1559b50 Prevent GC from wiping artifacts in the same NS
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-18 22:56:08 +01:00
Stefan Prodan 81d3ecf612
Patch status sub-resource
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2020-11-12 17:02:39 +02:00
Philip Laine 01d0053871 Fix duration default values
Signed-off-by: Philip Laine <philip.laine@xenit.se>
2020-11-02 09:55:39 +01:00
Aurel Canciu 960ad882c7
Refactor dependency_manager_tests.go
Cleanup some bits especially in the local chart suite.

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-30 19:16:28 +02:00