Commit Graph

962 Commits

Author SHA1 Message Date
Hidde Beydals 6cadb04cbd main: remove redundant else
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals ab089ddb07 libgit2: use `bytes.Equal` instead of `.Compare`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals 9c1c8149a5 sourceignore: return err of nested load
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals 16d4a4899b internal/helm: ensure err is checked before defer
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals c793cd59da controllers: record suspension for HelmRepository
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals 905602bdfe controllers: return err on auth dir create failure
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals dbbef5add8 controllers: use `time.Since`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:51:44 +01:00
Hidde Beydals baae990aea
Merge pull request #502 from fluxcd/metadata-load-validation 2021-11-23 09:51:36 +01:00
Hidde Beydals ee1cb49b0c internal/helm: check size of meta files in package
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:35:27 +01:00
Hidde Beydals 87c7c80e0a internal/helm: validate package while loading meta
There was an unfinished code path that should have continued validating
the paths within the package. This commit completes it.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-23 09:35:27 +01:00
Hidde Beydals 131c074a21
Merge pull request #500 from fluxcd/helmchart-cache-path 2021-11-23 06:59:27 +01:00
Hidde Beydals 6a8b5889f1 controllers: absolute local path for cached chart
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-22 22:10:31 +01:00
Stefan Prodan 5701b7945f
Merge pull request #498 from fluxcd/update-deps
Update github.com/minio/minio-go to `v7.0.15`
2021-11-22 17:48:55 +02:00
Hidde Beydals cd7da9df8c Drop deprecated `io/ioutil`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-22 16:31:25 +01:00
Hidde Beydals 78f4bdc8f6 Update github.com/minio/minio-go to `v7.0.15`
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-22 16:27:10 +01:00
Stefan Prodan de09b6ee9b Update controller-runtime to v0.10.2
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-11-22 14:36:43 +01:00
Hidde Beydals c31bb0f069
Merge pull request #495 from fluxcd/acl-api 2021-11-22 12:02:24 +01:00
Stefan Prodan 058a016a60 Add ACL optional field to Source API
Cherry-picked from 525be388ec.

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
2021-11-22 11:41:55 +01:00
Hidde Beydals e15e0ac6b8
Merge pull request #496 from fluxcd/readme-patch 2021-11-22 11:41:44 +01:00
Hidde Beydals 852a2ae5e8 Update test shield link
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-22 11:40:18 +01:00
Hidde Beydals d776221681
Merge pull request #494 from darkowlzz/helmchart-e2e-test-val-merge 2021-11-22 11:40:07 +01:00
Sunny 144766d03c
controllers: Fix helmchart values file merge test
Test case "Setting valid valuesFile attribute" and the tests around it
aren't isolated and most of the time pass because of the results from
the previous tests being re-read as the test expectation match the
previous test results. Failures are very rare to reproduce, even in
the CI they aren't seen but it failed very frequently on my computer,
especially this specific case because unlike the other cases, there is
just one file to be merged, which invalidates the chart result from
the previous cases.
In order to ensure the test wait for the chart to be updated by its
action and not by any other previous updates, status condition message
seems to be the most reliable way, as it also contains the paths of the
files that were merged.
With this change, I could no longer reproduce the failure on my
computer.
Reordering the tests makes this issue more clear.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-22 15:46:02 +05:30
Hidde Beydals d5e05983f8
Merge pull request #485 from fluxcd/helmchart-reconciler-dev 2021-11-22 10:30:33 +01:00
Hidde Beydals 2392326ba9 internal/helm: doc block nitpicks
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:16:20 +01:00
Hidde Beydals 88ff049ab0 internal/helm: ensure cached chart name matches
This helps detect e.g. path or chart name reference changes.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 472eb12f43 controllers: set generation as version metadata
By providing the Generation of the object that is getting reconciled
as version metadata to the builder if any custom values files are
defined, the Artifact revision changes if the specification does,
ensuring consumers of the Artifact are able to react to changes in
values (and perform a release).

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals c202ad59aa helm/internal: add `ErrChartReference`
This makes it possible to signal reference (validation) errors
happening before the build process actually starts dealing with
the chart.

At present, this does not have a more specific counterpart in the API,
but this is expected to change when the conditions logic is revised.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals dcd5dd3db1 internal/helm: various nitpicks
- Add some more documentation around chart builders
- Ensure correct indentation in some doc comments
- Provide example of using `errors.Is` for typed `BuildError`
- Mention "bytes" in file size limit errors
- Add missing copyright header

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 4de8f1f862 Allow configuration of Helm file limits
This allows custom configuration of the Helm file read limits, allowing
a user to overwrite them to their likenings if the defaults are too
restrictive for their specific setup using arguments:

`--helm-{index,chart,chart-file}-max-size`

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals a1e9302b7d internal/helm: "value files" -> "values files"
Previous usage while consistent, was incorrect, and inconsitent with
the field in the API spec.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Sunny 37ac5a9679 internal/helm: test load funcs for max size cases
This includes a change of the defaults to more acceptible (higher)
values.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Hidde Beydals 2b8134ce20 internal/helm: introduce typed BuildError
This commit introduces a typed `BuildError` to be returned by
`Builder.Build` in case of a failure.

The `Reason` field in combination with `BuildErrorReason` can be used
to signal (or determine) the reason of a returned error within the
context of the build process.

At present this is used to determine the correct Condition Reason, but
in a future iteration this can be used to determine the negative
polarity condition that should be set to indicate a precise failure to
the user.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 4fd6e6ef60 internal/helm: add more tests
Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals ef0517372b internal/helm: tweak and test chart build summary
This makes the string less verbose and deals with the safe handling
of some edge-case build states.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Sunny dd3afce3be internal/helm: add cached chart build tests
Cached chart build tests for both local and remote builder.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Sunny 753abed30c internal/helm: add remote builder tests
- For remote builds, if the build option has a version metadata, the
  chart should be repackaged with the provided version.
- Update internal/helm/testdata/charts/helmchart-0.1.0.tgz to include
  value files for testing merge chart values.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Sunny 7c910e37a2 internal/helm: local builder & dep manager test
Add more chart local builder and dependency manager tests.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
2021-11-19 17:04:00 +01:00
Hidde Beydals 32e19ebcd0 controllers: more tidying of wiring
Dealing with some loose ends around making observations, and code
style.

The loaded byes of a chart are used as a revision to ensure e.g.
periodic builds with unstable ordering of items do not trigger a false
positive.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 7d0f79f41b internal/helm: divide into subpackages
With all the logic that used to reside in the `controllers` package
factored into this package, it became cluttered. This commit tries to
bring a bit more structure in place.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 9abbdd80a6 controllers: rough wiring of Helm chart builder
This commit starts wiring the factored out Helm chart build logic into
the reconciler to ensure, validating the API capabilities.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 52459c899d internal/helm: make ChartBuilder an interface
This commit refactors the `ChartBuilder` that used to be a do-it-all
struct into an interace with two implementations:

- `LocalChartBuilder`: to build charts from a source on the local
  filesystem, either from a directory or from a packaged chart.
- `RemoteChartBuilder`: to build charts from a remote Helm repository
  index.

The new logic within the builders validates the size of the Helm size
it works with based on the `Max*Size` global variables in the internal
`helm` package, to address the recommendation from the security audit.

In addition, changes `ClientOptionsFromSecret` takes now a directory
argument which temporary files are placed in, making it easier to
perform a garbage collection of the whole directory at the end of a
reconcile run.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals d23bcbb5db controllers: wire ChartRepository in reconciler
This wires the `ChartRepository` changes into the reconciler to ensure
it works.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals f5f212ff43 internal/helm: introduce ChartBuilder
This commit starts with the creation of a `ChartBuilder` to facilitate
the (conditional) build of a chart outside of the reconciler logic.

The builder can be configured with a set of (modifying) options, which
define together with the type of chart source what steps are taken
during the build.

To better facilitate the builder's needs and attempt to be more
efficient, changes have been made to the `DependencyBuilder` and
`ChartRepository` around (order of) operations and/or lazy-load
capabilities.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals d60131d16b internal/helm: optimize dependency manager
This commit starts with the optimization of the `DepenendencyManager`,
ensuring the chart indexes are lazy loaded, and replacing the
(limitless) concurrency with a configurable number of workers with a
default of 1.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 44c1863334 internal/helm: add repository cache helpers
This commits adds simple caching capabilities to the
`ChartRepository`, which makes it possible to load the `Index` from a
defined `CachePath` using `LoadFromCache()`, and to download the index
to a new `CachePath` using `CacheIndex()`.

In addition, the repository tests have been updated to make use of
Gomega, and some missing ones have been added.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals 8537a0f8fa internal/helm: add helpers to load chart metadata
This commits adds `LoadChartMetadataFromArchive` and
`LoadChartMetadataFromDir` helpers to the internal `helm` package
to be able to make observations to the Helm metadata file without
loading the chart in full.

The helpers are compatible with charts of the v1 format (with a
separate `requirements.yaml` file), and an additional
`LoadChartMetadata` helper is available to automatically call the
right `LoadChartMetadataFrom*` version by looking at the file
description of the given path.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 17:04:00 +01:00
Hidde Beydals cc2bc5691a
Merge pull request #493 from fluxcd/update-deps 2021-11-19 14:31:52 +01:00
Hidde Beydals 4ce894a62c Update opencontainers/image-spec to v1.0.2
Another patch for CVE-2021-41190.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 13:58:30 +01:00
Hidde Beydals f933cb5923 Update docker/cli to v20.10.9
To mitigate warnings for CVE-2021-41092. Because even if there is no
impact whatsoever, we are nice people.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 13:58:30 +01:00
Hidde Beydals 19eb3d5ce7 Update docker/distribution to v2.7.0-rc.0
This mitigates another warning for CVE-2017-11468, which is mostly
triggered because a part of Helm depends on it that our code paths
never reach.

Signed-off-by: Hidde Beydals <hello@hidde.co>
2021-11-19 13:58:30 +01:00