Commit Graph

74 Commits

Author SHA1 Message Date
Hidde Beydals 3bda2c931d internal/util: introduce temp dir/path helpers
In most of the reconcilers we have a repetative pattern of using part of
the object metadata to construct a temporary file path.

This commit introduces helpers as an abstraction, for both the creation
of a temporary directory based on `client.Object` type and object
metadata, and the generation of an arbitrary random temporary path
string.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-02-15 15:40:58 +05:30
Sunny 729941ab2d Add internal packages error and reconcile
- internal/error - Contains internal error type used across the
  source-controller reconcilers.
- internal/reconcile - Contains helper abstractions for the
  controller-runtime reconcile Result type and functions to
  interact with the abstractions.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2022-02-15 15:40:58 +05:30
Hidde Beydals 8aad5520b6 helm: remove temp file name from index cache err
Due to the temporary file having a random suffix, it would result in
the notification-controller not rate limiting the messages as they
are "unique".

For the close error (which also makes use of the name), we keep the
information as it might be of importance to figure out why the close
failed.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2022-01-13 10:54:37 +01:00
Hidde Beydals fb0d7f24c8 internal/helm: validate loaded chart metadata obj
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 10:14:00 +01:00
Hidde Beydals 16d4a4899b internal/helm: ensure err is checked before defer
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals ee1cb49b0c internal/helm: check size of meta files in package
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:35:27 +01:00
Hidde Beydals 87c7c80e0a internal/helm: validate package while loading meta
There was an unfinished code path that should have continued validating
the paths within the package. This commit completes it.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:35:27 +01:00
Hidde Beydals 2392326ba9 internal/helm: doc block nitpicks
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:16:20 +01:00
Hidde Beydals 88ff049ab0 internal/helm: ensure cached chart name matches
This helps detect e.g. path or chart name reference changes.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals c202ad59aa helm/internal: add `ErrChartReference`
This makes it possible to signal reference (validation) errors
happening before the build process actually starts dealing with
the chart.

At present, this does not have a more specific counterpart in the API,
but this is expected to change when the conditions logic is revised.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals dcd5dd3db1 internal/helm: various nitpicks
- Add some more documentation around chart builders
- Ensure correct indentation in some doc comments
- Provide example of using `errors.Is` for typed `BuildError`
- Mention "bytes" in file size limit errors
- Add missing copyright header

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals a1e9302b7d internal/helm: "value files" -> "values files"
Previous usage while consistent, was incorrect, and inconsitent with
the field in the API spec.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Sunny 37ac5a9679 internal/helm: test load funcs for max size cases
This includes a change of the defaults to more acceptible (higher)
values.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Hidde Beydals 2b8134ce20 internal/helm: introduce typed BuildError
This commit introduces a typed `BuildError` to be returned by
`Builder.Build` in case of a failure.

The `Reason` field in combination with `BuildErrorReason` can be used
to signal (or determine) the reason of a returned error within the
context of the build process.

At present this is used to determine the correct Condition Reason, but
in a future iteration this can be used to determine the negative
polarity condition that should be set to indicate a precise failure to
the user.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 4fd6e6ef60 internal/helm: add more tests
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals ef0517372b internal/helm: tweak and test chart build summary
This makes the string less verbose and deals with the safe handling
of some edge-case build states.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Sunny dd3afce3be internal/helm: add cached chart build tests
Cached chart build tests for both local and remote builder.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Sunny 753abed30c internal/helm: add remote builder tests
- For remote builds, if the build option has a version metadata, the
  chart should be repackaged with the provided version.
- Update internal/helm/testdata/charts/helmchart-0.1.0.tgz to include
  value files for testing merge chart values.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Sunny 7c910e37a2 internal/helm: local builder & dep manager test
Add more chart local builder and dependency manager tests.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Hidde Beydals 7d0f79f41b internal/helm: divide into subpackages
With all the logic that used to reside in the `controllers` package
factored into this package, it became cluttered. This commit tries to
bring a bit more structure in place.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 52459c899d internal/helm: make ChartBuilder an interface
This commit refactors the `ChartBuilder` that used to be a do-it-all
struct into an interace with two implementations:

- `LocalChartBuilder`: to build charts from a source on the local
  filesystem, either from a directory or from a packaged chart.
- `RemoteChartBuilder`: to build charts from a remote Helm repository
  index.

The new logic within the builders validates the size of the Helm size
it works with based on the `Max*Size` global variables in the internal
`helm` package, to address the recommendation from the security audit.

In addition, changes `ClientOptionsFromSecret` takes now a directory
argument which temporary files are placed in, making it easier to
perform a garbage collection of the whole directory at the end of a
reconcile run.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals f5f212ff43 internal/helm: introduce ChartBuilder
This commit starts with the creation of a `ChartBuilder` to facilitate
the (conditional) build of a chart outside of the reconciler logic.

The builder can be configured with a set of (modifying) options, which
define together with the type of chart source what steps are taken
during the build.

To better facilitate the builder's needs and attempt to be more
efficient, changes have been made to the `DependencyBuilder` and
`ChartRepository` around (order of) operations and/or lazy-load
capabilities.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals d60131d16b internal/helm: optimize dependency manager
This commit starts with the optimization of the `DepenendencyManager`,
ensuring the chart indexes are lazy loaded, and replacing the
(limitless) concurrency with a configurable number of workers with a
default of 1.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 44c1863334 internal/helm: add repository cache helpers
This commits adds simple caching capabilities to the
`ChartRepository`, which makes it possible to load the `Index` from a
defined `CachePath` using `LoadFromCache()`, and to download the index
to a new `CachePath` using `CacheIndex()`.

In addition, the repository tests have been updated to make use of
Gomega, and some missing ones have been added.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 8537a0f8fa internal/helm: add helpers to load chart metadata
This commits adds `LoadChartMetadataFromArchive` and
`LoadChartMetadataFromDir` helpers to the internal `helm` package
to be able to make observations to the Helm metadata file without
loading the chart in full.

The helpers are compatible with charts of the v1 format (with a
separate `requirements.yaml` file), and an additional
`LoadChartMetadata` helper is available to automatically call the
right `LoadChartMetadataFrom*` version by looking at the file
description of the given path.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals d3c27c37f8 Update github.com/libgit2/git2go to v31.6.1
This commit updates `github.com/libgit2/git2go` to `v31.6.1` (with
`libgit2` `1.1.1`), and changes the container image build process so
that it makes use of `ghcr.io/hiddeco/golang-with-libgit2`.

This image provides a pre-build dynamic `libgit2` dependency linked
against OpenSSL and LibSSH2 (without gcrypt), and a set of cross-compile
build tools (see
[rationale](https://github.com/hiddeco/golang-with-libgit2#rationale) and
[usage](https://github.co/hiddeco/golang-with-libgit2#usage) for more
detailed information).

The linked set of dependency should solve most known issues around
unsupport private key types, but does not resolve the issues with ECDSA*
and ED25519 hostkeys yet. Solving this requires a newer version of
`libgit2` (`>=1.2.0`), which currently does not seem to work properly
with `git2go/v32`.

Some small changes have been made to the `libgit2` package to address
(future) deprecations.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-09-30 16:36:20 +02:00
Hidde Beydals c4d7e46b90 Drop deprecated `io/ioutil`
The package has been deprecated since Go 1.16, see:
https://golang.org/doc/go1.16#ioutil

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-07-29 09:58:00 +02:00
Sunny 9825a60b74
Use ObjectKeyFromObject instead of ObjectKey
controller-runtime's client package provides ObjectKeyFromObject() to
extract NamespacedName from a given object. ObjectKey() in
internal/util package is a helper for the same. Replace the internal
helper with controller-runtime's helper for the same.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-07-26 04:35:08 +05:30
Stefan Prodan fd88f934ee
Update chart tests
helm.sh/helm/v3 v3.5.2 no longer accepts chart versions that are not strict semver

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-02-23 14:19:56 +02:00
Hidde Beydals 5f4f91ed78 Copy loop iterator var for use by goroutine
This fixes a bug where only the last dependency would be downloaded.

https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-02-18 01:39:07 +01:00
Hidde Beydals 3cdc897236 Assume local dependency without Helm repository
This commit fixes a bug where local chart dependencies would not be
detected correctly due to the absence of a repository URL.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-01-28 12:28:19 +01:00
Hidde Beydals fda481efe9 Upgrade controller-runtime to v0.7.0
This commit upgrades the `controller-runtime` dependency to `v0.7.0`,
including all changes required to make all wiring work again.

- Upgrade `runtime` to v0.6.0 to include `controller-runtime` changes.
- Loggers have been removed from the reconcilers and are now retrieved
  from the `context.Context` passed to the `Reconcile` method and
  downwards functions.
- Logger configuration flags are now bound to the flag set using
  `BindFlags` from `runtime/logger`, ensuring the same contract across
  GitOps Toolkit controllers, and the `--log-json` flag has been
  deprecated in favour of the `--log-encoding=json` default.
- The `ChangePredicate` from `runtime` has changed to a
  `ReconcilateAtChangedPredicate`, and is now chained with the
  `GenerationChangedPredicate` from `controller-runtime` using
  `predicate.Or`.
- Signatures that made use of `runtime.Object` have changed to
  `client.Object`, removing the requirement to e.g. call
  `runtime.Object#Object`.
- The `client.MatchingField` function was deprecated, and has been
  replaced with `client.MatchingFields{}`.
- The `leader-election-role` was changed, as leader election now works
  via the `coordination/v1` API.

Other notable changes:

- `util.ObjectKey` was added to easily construct a `client.ObjectKey` /
  `types.NamespacedName` from a `metav1.Object`.
- The `SourceIndexKey` constant has been split out into
  `{GitRepository,HelmRepository,Bucket}IndexKey` constants.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-18 19:17:10 +01:00
Aurel Canciu a55c502bb4
Fix HelmChart valuesFile chart path restriction
As part of the feature implementation to support helm chart
dependencies, the functionality for allowing values files overwriting
from any location scoped to the same source was altered. This should fix
the problem by allowing users to load files from any arbitrary location
as long as it's in the context of the same source from where the helm
chart itself is loaded.

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-12-17 12:57:41 +02:00
Hidde Beydals 29a051c5f4 Refactor and document DependencyManager
Mostly to re-use the fields of the structure instead of copying things
around.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-15 15:37:42 +01:00
Hidde Beydals 8d0b54e431 Make proper use of errgroup context
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-15 09:56:40 +01:00
Hidde Beydals bc890874e1 Add safe guards for relative paths
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-12-15 09:56:40 +01:00
Hidde Beydals a3f177328a Move Helm dependency manager to `helm` package
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-11-19 09:50:50 +01:00
Aurel Canciu bfd8d4bc5a
Change copyright wording to accomodate convention
Re: fluxcd/toolkit#380

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-28 18:08:44 +02:00
Aurel Canciu f3370d8004
Extract chart repo url normalization logic
Move the logic to helm/utils exported as func
NormalizeChartRepositoryURL

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-28 14:57:12 +02:00
Hidde Beydals c9eb8f03c0 Switch to Masterminds/semver and pkg/version libs
Co-authored-by: Illia Ovchynnikov <illia.ovchynnikov@gmail.com>
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-10-28 12:21:08 +01:00
Illia Ovchynnikov 394b5c3bd0
Pick the most recent chart/tag for ambiguous semver matches
Signed-off-by: Illia Ovchynnikov <illia.ovchynnikov@gmail.com>
2020-10-27 20:26:28 +01:00
Hidde Beydals 582c2092b6 Change copyright to Flux authors
Signed-off-by: Hidde Beydals <hello@hidde.co>
2020-10-27 17:54:22 +01:00
Aurel Canciu 684624b1a0
Add support for loading packaged helm charts
The feature allows the source-controller to load packaged helm charts
for HelmChart resource artifacts from GitRepository and Bucket sources

Signed-off-by: Aurel Canciu <aurelcanciu@gmail.com>
2020-10-27 11:15:17 +02:00
Hidde Beydals 8bf7d8f440 Factor out Helm repo index and chart download 2020-09-24 12:40:04 +02:00
Hidde Beydals 969a46f4d6 internal/helm: return callback on empty TLS config
...and no-op on empty valuesFile string.
2020-09-21 23:27:38 +02:00
Hidde Beydals 7268c8b61d Refactor and factor out chart values replacement 2020-09-11 12:16:41 +02:00
Hidde Beydals 1ab12869ac Make storage file writes atomic 2020-09-10 12:02:32 +02:00
Hidde Beydals f35dcc3656 Upgrade blang/semver from v3.5.0 to v4.0.0
v4.0.0 is fully compatible with Go Modules.
2020-08-31 10:03:47 +02:00
Hidde Beydals d38b8fe193 Support proper semver ranges for Helm charts
This commit changes the semver range parser to `blang/semver`, which
is also used to parse semver tags for GitRepository sources.
2020-08-31 10:03:47 +02:00
Hidde Beydals a0357172bc Helm: allow configuration of TLS certs without CA 2020-07-20 21:33:55 +02:00