While Fleet config files are typically named `fleet.yaml`, the recent
introduction of user-driven bundle scanning enables a config file to be
named arbitrarily, in which case it must still be excluded from the
corresponding bundle. Fleet now takes care of this without any action
needed from the user.
Integration tests also demonstrate that `.fleetignore` files can be
leveraged to exclude config files living in the same directory as
the config file referenced explicitly through user-driven bundle
scanning. Fleet would otherwise not exclude those files from the bundle.
While we had recently added support for globbing for path-based secrets
to classic bundle scanning, that had been omitted for user-driven bundle
scanning.
This commit remedies this, ensuring that globbing support works
regardless of the chosen bundle scanning mode.
The Helm deployer now honours Helm's `atomic` flag for installs, not just
updates.
It also supports overwriting releases in `pending-install` state, which can
unblock deployments.
A new integration test case simulates an existing release with
`pending-install` status, by creating the corresponding Helm secret, as
Fleet uses Helm's secret storage back-end, then validates that Fleet can
install a release with the same name on top of it.
---------
Co-authored-by: Mario Manno <mario.manno@suse.com>
Co-authored-by: Corentin Néau <tan.neau@suse.com>
Values files may be referenced by file name alone, or with a directory
prefix. Both variants now lead to values files being excluded from
bundles resources, regardless of where a `fleet.yaml` file referencing
them lives (inside or outside of a chart directory).
A bundle's name can be computed from a GitRepo's name and the path
leading to the bundle. However, overriding that name is also possible,
by populating the `name` field in a `fleet.yaml` file.
A new integration test case covers and demonstrates this.
* Support globbing in helmSecretNameForPaths
Helm access secrets referenced in a GitRepo's `helmSecretNameForPaths`
field now support globbing, such as `my-path/*`, or `foo?bar`.
When more than one path listed in the `helmSecretNameForPaths` secret
matches a given GitRepo path, then the first match listed in the
`helmSecretNameForPaths` secret will be used.
* Ensure deterministic order of iteration over auth map
When more than one patterns match a given bundle path, the first
matching pattern, in lexical order, will be used.
* Enables OCI storage by default
Changes the env variable from `EXPERIMENTAL_OCI_STORAGE` to `OCI_STORAGE`
The feature is enabled by default and in order to disable it users should pass `OCI_STORAGE=false` in `extraEnv`
Refers to: https://github.com/rancher/fleet/issues/3818
---------
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
* Adds a label to internal/cloned secrets.
It also adds an event when the oci artifact is purged and could not be deleted for whatever reason.
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
* Detect bundle naming collision in HelmOps reconciler
When building a HelmOps bundle, the reconciler now aborts with an error
if a non-HelmOps bundle with the same name exists in the same namespace.
* Detect bundle naming collision in `fleet apply`
When creating or updating a bundle, `fleet apply` checks the spec of any
existing bundle with the same name in the same namespace. If one bundle
is a HelmOps bundle while the other is not, `fleet apply` will abort the
update and output an error message.
* Simplify collision detection in `fleet apply`
Relying on the fact that `fleet apply` does not support creating HelmOps
bundles at this point, there is no need to generalise collision
detection to any mismatch in HelmOps state between a possibly existing
bundle and the one to be created or updated.
Instead, checking whether a HelmOps bundle already exists is enough to
establish a collision.
* Use controller-runtime client in fleet apply
This change is made to eliminate dependencies on Wrangler in the Fleet CLI and also to support future proposed changes related to reading secrets for OCI registries.
It also facilitates the creation of new integration tests, as the client interface is simplified and a single mock is used.
* Delete cluster instantiation and cache start in the cli
* Avoid adding new verbs that are not strictly required
Progress is not working. This also wraps more errors and skips .git
folders when scanning cloned git repos.
Co-authored-by: Mario Manno <mario.manno@suse.com>
* Adds driven bundle scan
This adds a new feature to let the user define the bundles to be considered by Fleet.
It adds a new field `bundles` that is a list of path and config files.
The path is the base path for the resources contained in the Bundle and the config file is optional
and it's the Bundle's fleet.yaml file (that may have any custom name).
This is an example showing the new fields:
```yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: driven
namespace: fleet-local
spec:
repo: https://github.com/0xavi0/fleet-test-data
branch: driven-scan-example
bundles:
- path: driven/helm
- path: driven/simple
- path: driven/kustomize
options: dev.yaml
- path: driven/kustomize
options: test.yaml
```
Given the root driven folder with the contents:
```
driven
|___helm
| |__ fleet.yaml
|
|___simple
| |__ configmap.yaml
| |__ service.yaml
|
|___kustomize
|__ base
| |__ kustomization.yaml
| |__ secret.yaml
|
|__ overlays
| |__ dev
| | |__ kustomization.yaml
| | |__ secret.yaml
| |__ prod
| | |__ kustomization.yaml
| | |__ secret.yaml
| |__ test
| |__ kustomization.yaml
| |__ secret.yaml
|__ dev.yaml
|__ prod.yaml
|__ test.yaml
```
The GitRepo above would describe the following bundles:
**Helm based bundle (path: `driven/helm`, no config file)**
This one has a `fleet.yaml` file in the root folder and we're not specifying any config file.
In this case Fleet will read the `fleet.yaml` file in the root folder and will load the helm chart resources.
**Raw yaml folder based bundle (path: `driven/simple`, no config file)**
In this case the folder contains 2 yamls and no `fleet.yaml` in its root. We are not specifying any config file,
so the folder is considered as a bundle made of raw yaml files.
It will contain:
- configmap.yaml
- service.yaml
**Kustomize bundle (path: `driven/kustomize`, config file: `dev.yaml`)**
In this case we are specifying a config file (with the format of a `fleet.yaml` file).
Fleet will load all the resources found in the `drive/kustomize` folder and will load the config found
in the `dev.yaml` file.
The `dev.yaml` has the following content:
```yaml
namespace: kustomize-dev
kustomize:
dir: "overlays/dev"
```
Which means it points to the `overlays/dev` kustomization.
This Bundle will load all the resources, but will only apply the `overlays/dev` kustomization
**Kustomize bundle (path: `driven/kustomize`, config file: `test.yaml`)**
This Bundle will be equivalent to the `dev` one described above, but pointint to the `overlays/test` kustomization.
Note that the `prod` kustomization is not used in this case, which shows that not all the options defined in the kustomization must be deployed.
Possible improvements:
As we can be loading the same folder multiple times it would be better to just load the resources once and cache them for the next repeated folder.
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
Each Bundle and BundleDeployment can have a secret to store options in. The secrets have the same name and namespace as the resource. For now they only store the helm values.
* Remove unused function
This function became obsolete when using a delaying enqueue handler for
drift detection; see fleet#3401 [1].
[1]: https://github.com/rancher/fleet/pull/3401
* Modernise output setting in deployer integration tests
This makes use of `SetOut` and `SetErr` instead of `SetOutput`, which is
deprecated.
* Remove unused parameter from test helper methods
That parameter is no longer necessary now that our test infrastructure
uses `default` as its namespace.
* Remove unecessary files from bundle resources
This will make the bundle and content resources smaller. However, it
should not change the helm chart that is deployed. The files are parsed
into deployment options and have no use after the fleet apply stage.
Minor changes:
* Extract loaddirectory args to struct
* Remove unused key
* Clean up creation of agent manifests
* Separate bundledeployment construction from creation
* Remove slow unit test
This test relied on a network timeout and took up to 3 minutes.
Adds message context to fleet apply errors
Adds context to message in fleet apply. Specially to those running inside loops.
For example, when loading directories, remote charts, etc...
Contexts are wrapped in top of the previous one.
It also fixes the `gitjob` init container to use `log.sh` and redirect the errors to `/dev/termination-log`
Related to: https://github.com/rancher/fleet/issues/3234
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
* Changes after review
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
* More changes after review
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
---------
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
This introduces new environment variables for integrationtests:
* CI_SILENCE_CTRL - no logging output, even when test fails
* CI_KUBECONFIG - write kubeconfig for testenv to this path
* CI_USE_EXISTING_CLUSTER - integrationtests don't setup testenv, but
use existing cluster from current KUBECONFIG
The job also deletes all completed gitjobs, not leaving the last one active, as
https://github.com/rancher/fleet/pull/2903 was merged.
Also fixes the ServiceAccount to be able to list and delete jobs.
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
This ensures that, when updating bundles and imagescans, `fleet apply` only uses labels and annotations from fleet.yaml. It prevents pre-existing labels and annotations from being kept on those resources even if they are no longer reflected in fleet.yaml.
Co-authored-by: Corentin Néau <tan.neau@suse.com>
Eventually is not needed when testing for resources in bundles as we get
the bundle from `cli.GetBundleListFromOutput` and the buffer read will be
in its final state once `cli.GetBundleListFromOutput` returns.
Also addind custom Gomega matchers to avoid using helper functions and
to get the exact line when any test assertion fails.
Already existing tests have been changed to use the custom matchers and
don't use Eventually, also.
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
A new flag `disableDependencyUpdate` is added to the `fleet.yaml`
file in order to disable the feature, which is active by default.
A new package `helmupdater` is added to implement the dependencies
update.
The implementation is based on helm's when ussing the
`--dependency-update` flag in the install command.
Dependencies are applied to the bundle (upstream) to they're resolved
already when applying downstream.
Refers to: https://github.com/rancher/fleet/issues/1672
Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
* Rename CLI apply.Apply to apply.CreateBundles
* Refactor target package
* new UpdatePartitions
* new file rollout.go
* Extract BundleDeployment to func
* Remove global fleet args, remove unused args
These arguments were not shared between all commands.
There is one set of commands (apply, cleanup) which use the fleet/client
pkg and another set which uses a controller-runtime client. Same is true
for the debug flag.
* Move fleet apply helm test into apply folder
Both apply/ and helm/ are testing the fleet apply sub command
* Fix typos
---------
Co-authored-by: Corentin Néau <tan.neau@suse.com>
even when fleet.yaml does not reference a helm chart.
Also make sure that the values are actually evaluated and that no
potential default value is interfering with the test.
* Move codegen into cmd folder
* Split pkgs by command/binary into internal folder
* Update go imports after moving
* Remove cmd/agent/agent, cmd/controller/fleetcontroller packages
* Remove package comments, which indicated the binary the package belongs to
* Move fleetcli's main.go into cmd/
DoNotDeploy: BundleDeployments will not be created for targets that matches a targetCustomization that has DoNotDeploy set to true, which means no resources will be deployed.
OverrideTargets: Targets defined in a GitRepo will be overridden if a Bundle defines overrideTargets in the fleet.yaml
Signed-off-by: raul <raul.cabello@suse.com>
Add HelmSecretNameForPaths in GitRepo
This allows to specify different credentials for each Bundle that will be generated for each path.
HelmSecretName will be ignored if HelmSecretNameForPaths is present.
Tests returned an "address already in use" error on very rare occasions. This was due to the previous helm repository was not fully closed before creating the new one, as it might take an extra milliseconds to fully close after calling closed. Retry if "address already in use" error is returned after 100 milliseconds for a maximum of 10 attempts
keepResources can be specified in the GitRepo or in a fleet.yaml. If true it will not delete resources after the GitRepo or Bundle is deleted. It will delete helm release secrets instead.
Signed-off-by: raul <raul.cabello@suse.com>
Run the helm integration tests in different port, so they can be run in parallel.
Use Serial instead of Ordered, as the order doesn't matter.
Signed-off-by: raul <raul.cabello@suse.com>
Corentin Néau