chore: Set latest docusaurus version (#300)

Made with ❤️️ by updatecli

Co-authored-by: updatecli-bot <bot@rancher.io>
This commit is contained in:
github-actions[bot] 2025-07-24 13:37:56 +02:00 committed by GitHub
parent 9d32f4f7a8
commit e0c757b395
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
56 changed files with 7944 additions and 0 deletions

View File

@ -0,0 +1,44 @@
# Architecture
Fleet has two primary components. The Fleet controller and the cluster agents. These
components work in a two-stage pull model. The Fleet controller will pull from git and the
cluster agents will pull from the Fleet controller.
## Fleet Controller
The Fleet controller is a set of Kubernetes controllers running in any standard Kubernetes
cluster. The only API exposed by the Fleet controller is the Kubernetes API, there is no
custom API for the fleet controller.
## Cluster Agents
One cluster agent runs in each cluster and is responsible for talking to the Fleet controller.
The only communication from cluster to Fleet controller is by this agent and all communication
goes from the managed cluster to the Fleet controller. The fleet manager does not initiate
connections to downstream clusters. This means managed clusters can run in private networks and behind
NATs. The only requirement is the cluster agent needs to be able to communicate with the
Kubernetes API of the cluster running the Fleet controller. The one exception to this is if you use
the [manager initiated](./cluster-registration.md#manager-initiated) cluster registration flow. This is not required, but
an optional pattern.
The cluster agents are not assumed to have an "always on" connection. They will resume operation as
soon as they can connect. Future enhancements will probably add the ability to schedule times of when
the agent checks in, as it stands right now they will always attempt to connect.
## Security
The Fleet controller dynamically creates service accounts, manages their RBAC and then gives the
tokens to the downstream clusters. Clusters are registered by optionally expiring cluster registration tokens.
The cluster registration token is used only during the registration process to generate a credential specific
to that cluster. After the cluster credential is established the cluster "forgets" the cluster registration
token.
The service accounts given to the clusters only have privileges to list `BundleDeployment` in the namespace created
specifically for that cluster. It can also update the `status` subresource of `BundleDeployment` and the `status`
subresource of it's `Cluster` resource.
## Component Overview
An overview of the components and how they interact on a high level.
![Components](/img/FleetComponents.svg)

View File

@ -0,0 +1,117 @@
# Create a Bundle Resource
Bundles are automatically created by Fleet when a `GitRepo` is created. In most cases `Bundles` should not be created
manually by the user. If you want to deploy resources from a git repository use a
[GitRepo](https://fleet.rancher.io/gitrepo-add) instead.
If you want to deploy resources without a git repository follow this guide to create a `Bundle`.
:::note
If you want to deploy resources without running a Fleet controller, also take a look at the [Fleet CLI](ref-bundle-stages#examining-the-bundle-lifecycle-with-the-cli).
:::
When creating a `GitRepo` Fleet will fetch the resources from a git repository, and add them to a Bundle.
When creating a `Bundle` resources need to be explicitly specified in the `Bundle` Spec.
Resources can be compressed with gz. See [here](https://github.com/rancher/rancher/blob/main/pkg/controllers/provisioningv2/managedchart/managedchart.go#L149-L153)
an example of how Rancher uses compression in go code.
If you would like to deploy in downstream clusters, you need to define targets. Targets work similarly to targets in `GitRepo`.
See [Mapping to Downstream Clusters](https://fleet.rancher.io/gitrepo-targets#defining-targets).
The following example creates a nginx `Deployment` in the local cluster:
```yaml
kind: Bundle
apiVersion: fleet.cattle.io/v1alpha1
metadata:
# Any name can be used here
name: my-bundle
# For single cluster use fleet-local, otherwise use the namespace of
# your choosing
namespace: fleet-local
spec:
resources:
# List of all resources that will be deployed
- content: |
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
name: nginx.yaml
targets:
- clusterName: local
```
## Targets
The bundle can target multiple clusters. It uses the same [targeting as the GitRepo](gitrepo-targets#target-matching).
Additional [customization options](gitrepo-targets#supported-customizations) can be specified per target:
```yaml
targets:
- clusterSelector:
matchLabels:
env: dev
defaultNamespace: lab-1
helm:
values:
replicas: 1
```
## Limitations
Helm options related to downloading the helm chart will be ignored. The helm chart is downloaded by the fleet-cli, which creates the bundles. The bundle has to contain all the resources from the chart. Therefore the bundle will ignore:
* `spec.helm.repo`
* `spec.helm.charts`
You can't use a `fleet.yaml` in resources, it is only used by the fleet-cli to create bundles.
The `spec.targetRestrictions` field is not useful, as it is an allow list for targets specified in `spec.targets`. It is not needed, since `targets` are explicitly given in a bundle and an empty `targetRestrictions` defaults to allow.
## Convert a Helm Chart into a Bundle
You can use the Fleet CLI to convert a Helm chart into a bundle.
For example, you can download and convert the "external secrets" operator chart like this:
```
cat > targets.yaml <<EOF
targets:
- clusterSelector: {}
EOF
mkdir app
cat > app/fleet.yaml <<EOF
defaultNamespace: external-secrets
helm:
repo: https://charts.external-secrets.io
chart: external-secrets
EOF
fleet apply --compress --targets-file=targets.yaml -n fleet-default -o - external-secrets app > eso-bundle.yaml
kubectl apply -f eso-bundle.yaml
```
Make sure you use a cluster selector in `targets.yaml`, that matches all clusters you want to deploy to.
The blog post on [Fleet: Multi-Cluster Deployment with the Help of External Secrets](https://www.suse.com/c/rancher_blog/fleet-multi-cluster-deployment-with-the-help-of-external-secrets/) has more information.

View File

@ -0,0 +1,318 @@
# Generating Diffs to Ignore Modified GitRepos
Continuous Delivery in Rancher is powered by Fleet. When a user adds a GitRepo CR, then Continuous Delivery creates the associated fleet bundles.
You can access these bundles by navigating to the Cluster Explorer (Dashboard UI), and selecting the `Bundles` section.
The bundled charts may have some objects that are amended at runtime, for example:
* in a ValidatingWebhookConfiguration, the `caBundle` is empty and the CA cert is injected by the cluster.
* an installed chart may create a job, which is then deleted once completed
This leads the status of the bundle and associated GitRepo to be reported as "Modified"
![](/img/ModifiedGitRepo.png)
Associated Bundle
![](/img/ModifiedBundle.png)
Fleet bundles support the ability to specify a custom [jsonPointer patch](http://jsonpatch.com/).
With the patch, users can instruct fleet to ignore:
* object modifications
* entire objects
## Ignoring object modifications
### Simple Example
In this simple example, we create a Service and ConfigMap that we apply a bundle diff onto.
https://github.com/rancher/fleet-test-data/tree/master/bundle-diffs
### Gatekeeper Example
In this example, we are trying to deploy opa-gatekeeper using Continuous Delivery to our clusters.
The opa-gatekeeper bundle associated with the opa GitRepo is in modified state.
Each path in the GitRepo CR, has an associated Bundle CR. The user can view the Bundles, and the associated diff needed in the Bundle status.
In our case the differences detected are as follows:
```yaml
summary:
desiredReady: 1
modified: 1
nonReadyResources:
- bundleState: Modified
modifiedStatus:
- apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
name: gatekeeper-validating-webhook-configuration
patch: '{"$setElementOrder/webhooks":[{"name":"validation.gatekeeper.sh"},{"name":"check-ignore-label.gatekeeper.sh"}],"webhooks":[{"clientConfig":{"caBundle":"Cg=="},"name":"validation.gatekeeper.sh","rules":[{"apiGroups":["*"],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["*"]}]},{"clientConfig":{"caBundle":"Cg=="},"name":"check-ignore-label.gatekeeper.sh","rules":[{"apiGroups":[""],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["namespaces"]}]}]}'
- apiVersion: apps/v1
kind: Deployment
name: gatekeeper-audit
namespace: cattle-gatekeeper-system
patch: '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"manager"}],"containers":[{"name":"manager","resources":{"limits":{"cpu":"1000m"}}}],"tolerations":[]}}}}'
- apiVersion: apps/v1
kind: Deployment
name: gatekeeper-controller-manager
namespace: cattle-gatekeeper-system
patch: '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"manager"}],"containers":[{"name":"manager","resources":{"limits":{"cpu":"1000m"}}}],"tolerations":[]}}}}'
```
Based on this summary, there are three objects which need to be patched.
We will look at these one at a time.
#### 1. ValidatingWebhookConfiguration:
The gatekeeper-validating-webhook-configuration validating webhook has two ValidatingWebhooks in its spec.
In cases where more than one element in the field requires a patch, that patch will refer these to as `$setElementOrder/ELEMENTNAME`
From this information, we can see the two ValidatingWebhooks in question are:
```
"$setElementOrder/webhooks": [
{
"name": "validation.gatekeeper.sh"
},
{
"name": "check-ignore-label.gatekeeper.sh"
}
],
```
Within each ValidatingWebhook, the fields that need to be ignore are as follows:
```
{
"clientConfig": {
"caBundle": "Cg=="
},
"name": "validation.gatekeeper.sh",
"rules": [
{
"apiGroups": [
"*"
],
"apiVersions": [
"*"
],
"operations": [
"CREATE",
"UPDATE"
],
"resources": [
"*"
]
}
]
},
```
and
```
{
"clientConfig": {
"caBundle": "Cg=="
},
"name": "check-ignore-label.gatekeeper.sh",
"rules": [
{
"apiGroups": [
""
],
"apiVersions": [
"*"
],
"operations": [
"CREATE",
"UPDATE"
],
"resources": [
"namespaces"
]
}
]
}
```
In summary, we need to ignore the fields `rules` and `clientConfig.caBundle` in our patch specification.
The field webhook in the ValidatingWebhookConfiguration spec is an array, so we need to address the elements by their index values.
![](/img/WebhookConfigurationSpec.png)
Based on this information, our diff patch would look as follows:
```yaml
- apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
name: gatekeeper-validating-webhook-configuration
operations:
- {"op": "remove", "path":"/webhooks/0/clientConfig/caBundle"}
- {"op": "remove", "path":"/webhooks/0/rules"}
- {"op": "remove", "path":"/webhooks/1/clientConfig/caBundle"}
- {"op": "remove", "path":"/webhooks/1/rules"}
```
#### 2. Deployment gatekeeper-controller-manager:
The gatekeeper-controller-manager deployment is modified since there are cpu limits and tolerations applied (which are not in the actual bundle).
```
{
"spec": {
"template": {
"spec": {
"$setElementOrder/containers": [
{
"name": "manager"
}
],
"containers": [
{
"name": "manager",
"resources": {
"limits": {
"cpu": "1000m"
}
}
}
],
"tolerations": []
}
}
}
}
```
In this case, there is only 1 container in the deployment container spec, and that container has cpu limits and tolerations added.
Based on this information, our diff patch would look as follows:
```yaml
- apiVersion: apps/v1
kind: Deployment
name: gatekeeper-controller-manager
namespace: cattle-gatekeeper-system
operations:
- {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
- {"op": "remove", "path": "/spec/template/spec/tolerations"}
```
#### 3. Deployment gatekeeper-audit:
The gatekeeper-audit deployment is modified in a similarly, to the gatekeeper-controller-manager, with additional cpu limits and tolerations applied.
```
{
"spec": {
"template": {
"spec": {
"$setElementOrder/containers": [
{
"name": "manager"
}
],
"containers": [
{
"name": "manager",
"resources": {
"limits": {
"cpu": "1000m"
}
}
}
],
"tolerations": []
}
}
}
}
```
Similar to gatekeeper-controller-manager, there is only 1 container in the deployments container spec, and that has cpu limits and tolerations added.
Based on this information, our diff patch would look as follows:
```yaml
- apiVersion: apps/v1
kind: Deployment
name: gatekeeper-audit
namespace: cattle-gatekeeper-system
operations:
- {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
- {"op": "remove", "path": "/spec/template/spec/tolerations"}
```
#### Combining It All Together
We can now combine all these patches as follows:
```yaml
diff:
comparePatches:
- apiVersion: apps/v1
kind: Deployment
name: gatekeeper-audit
namespace: cattle-gatekeeper-system
operations:
- {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
- {"op": "remove", "path": "/spec/template/spec/tolerations"}
- apiVersion: apps/v1
kind: Deployment
name: gatekeeper-controller-manager
namespace: cattle-gatekeeper-system
operations:
- {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits/cpu"}
- {"op": "remove", "path": "/spec/template/spec/tolerations"}
- apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
name: gatekeeper-validating-webhook-configuration
operations:
- {"op": "remove", "path":"/webhooks/0/clientConfig/caBundle"}
- {"op": "remove", "path":"/webhooks/0/rules"}
- {"op": "remove", "path":"/webhooks/1/clientConfig/caBundle"}
- {"op": "remove", "path":"/webhooks/1/rules"}
```
We can add these now to the bundle directly to test and also commit the same to the `fleet.yaml` in your GitRepo.
Once these are added, the GitRepo should deploy and be in "Active" status.
## Ignoring entire objects
When installing a chart such as [Consul](https://developer.hashicorp.com/consul/docs/k8s/helm), a job named
`consul-server-acl-init` is created, then deleted once it has successfully completed.
That chart can be installed by creating a `GitRepo` pointing to a git repository using a `fleet.yaml` such as:
```yaml
defaultNamespace: consul
helm:
releaseName: test-consul
chart: "consul"
repo: "https://helm.releases.hashicorp.com"
values:
global:
name: consul
acls:
manageSystemACLs: true
```
Installing this chart will result in the `GitRepo` reporting a `Modified` status, with job `consul-server-acl-init`
missing, once that job has completed.
This can be remedied with the following bundle diff in our `fleet.yaml`:
```yaml
diff:
comparePatches:
- apiVersion: batch/v1
kind: Job
namespace: consul
name: consul-server-acl-init
operations:
- {"op":"ignore"}
```

View File

@ -0,0 +1,5 @@
---
title: "Next 🚧"
---
We are still working on the next release.

View File

@ -0,0 +1,5 @@
---
title: "Next 🚧"
---
We are still working on the next release.

View File

@ -0,0 +1,33 @@
---
title: ""
sidebar_label: "fleet-agent"
---
## fleet-agent
```
fleet-agent [flags]
```
### Options
```
--agent-scope string An identifier used to scope the agent bundleID names, typically the same as namespace
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
-h, --help help for fleet-agent
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
--namespace string system namespace is the namespace, the agent runs in, e.g. cattle-fleet-system
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) (default true)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
```
### SEE ALSO
* [fleet-agent clusterstatus](fleet-agent/fleet-agent_clusterstatus) - Continuously report resource status to the upstream cluster
* [fleet-agent register](fleet-agent/fleet-agent_register) - Register agent with an upstream cluster

View File

@ -0,0 +1,26 @@
---
title: ""
sidebar_label: "fleet-agent clusterstatus"
---
## fleet-agent clusterstatus
Continuously report resource status to the upstream cluster
```
fleet-agent clusterstatus [flags]
```
### Options
```
--checkin-interval string How often to post cluster status
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
-h, --help help for clusterstatus
--kubeconfig string kubeconfig file for agent's cluster
--namespace string system namespace is the namespace, the agent runs in, e.g. cattle-fleet-system
```
### SEE ALSO
* [fleet-agent](./)

View File

@ -0,0 +1,25 @@
---
title: ""
sidebar_label: "fleet-agent register"
---
## fleet-agent register
Register agent with an upstream cluster
```
fleet-agent register [flags]
```
### Options
```
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
-h, --help help for register
--kubeconfig string kubeconfig file for agent's cluster
--namespace string system namespace is the namespace, the agent runs in, e.g. cattle-fleet-system
```
### SEE ALSO
* [fleet-agent](./)

View File

@ -0,0 +1,24 @@
---
title: ""
sidebar_label: "fleet cleanup"
---
## fleet cleanup
Clean up outdated resources
```
fleet cleanup [flags]
```
### Options
```
-h, --help help for cleanup
```
### SEE ALSO
* [fleet](../fleet) -
* [fleet cleanup clusterregistration](./fleet_cleanup_clusterregistration) - Clean up outdated cluster registrations
* [fleet cleanup gitjob](./fleet_cleanup_gitjob) - Clean up outdated git jobs

View File

@ -0,0 +1,30 @@
---
title: ""
sidebar_label: "fleet cleanup clusterregistration"
---
## fleet cleanup clusterregistration
Clean up outdated cluster registrations
```
fleet cleanup clusterregistration [flags]
```
### Options
```
--context string kubeconfig context for authentication
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
--factor string Factor to increase delay between deletes (default: 1.1)
-h, --help help for clusterregistration
-k, --kubeconfig string kubeconfig for authentication
--max string Maximum delay between deletes (default: 5s)
--min string Minimum delay between deletes (default: 10ms)
-n, --namespace string namespace (default "fleet-local")
```
### SEE ALSO
* [fleet cleanup](./fleet_cleanup) - Clean up outdated resources

View File

@ -0,0 +1,28 @@
---
title: ""
sidebar_label: "fleet cleanup gitjob"
---
## fleet cleanup gitjob
Clean up outdated git jobs
```
fleet cleanup gitjob [flags]
```
### Options
```
--batch-size int Number of git jobs to retrieve at once (default 5000)
--context string kubeconfig context for authentication
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
-h, --help help for gitjob
-k, --kubeconfig string kubeconfig for authentication
-n, --namespace string namespace (default "fleet-local")
```
### SEE ALSO
* [fleet cleanup](./fleet_cleanup) - Clean up outdated resources

View File

@ -0,0 +1,26 @@
---
title: ""
sidebar_label: "fleet"
---
## fleet
```
fleet [flags]
```
### Options
```
-h, --help help for fleet
```
### SEE ALSO
* [fleet apply](./fleet_apply) - Create bundles from directories, and output them or apply them on a cluster
* [fleet cleanup](./cleanup/fleet_cleanup) - Clean up outdated resources
* [fleet deploy](./fleet_deploy) - Deploy a bundledeployment/content resource to a cluster, by creating a Helm release. This will not deploy the bundledeployment/content resources directly to the cluster.
* [fleet gitcloner](./fleet_gitcloner) - Clones a git repository
* [fleet target](./fleet_target) - Print available targets for a bundle

View File

@ -0,0 +1,67 @@
---
title: ""
sidebar_label: "fleet apply"
---
## fleet apply
Create bundles from directories, and output them or apply them on a cluster
```
fleet apply [flags] BUNDLE_NAME PATH...
```
### Options
```
-b, --bundle-file string Location of the raw Bundle resource yaml
--cacerts-file string Path of custom cacerts for helm repo
--commit string Commit to assign to the bundle
-c, --compress Force all resources to be compress
--context string kubeconfig context for authentication
--correct-drift Rollback any change made from outside of Fleet
--correct-drift-force Use --force when correcting drift. Resources can be deleted and recreated
--correct-drift-keep-fail-history Keep helm history for failed rollbacks
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
--delete-namespace Delete GitRepo target namespace after the GitRepo or Bundle is deleted
-f, --file string Location of the fleet.yaml
--helm-credentials-by-path-file string Path of file containing helm credentials for paths
--helm-repo-url-regex string Helm credentials will be used if the helm repo matches this regex. Credentials will always be used if this is empty or not provided
-h, --help help for apply
--keep-resources Keep resources created after the GitRepo or Bundle is deleted
-k, --kubeconfig string kubeconfig for authentication
-l, --label strings Labels to apply to created bundles
-n, --namespace string namespace (default "fleet-local")
--oci-basic-http Use HTTP to access the OCI regustry
--oci-insecure Allow connections to OCI registry without certs
--oci-password-file string Path of file containing basic auth password for OCI registry
--oci-reference string OCI registry reference
--ociusername string Basic auth username for OCI registry
-o, --output string Output contents to file or - for stdout
--password-file string Path of file containing basic auth password for helm repo
--paused Create bundles in a paused state
-a, --service-account string Service account to assign to bundle created
--ssh-privatekey-file string Path of ssh-private-key for helm repo
--sync-generation int Generation number used to force sync the deployment
--target-namespace string Ensure this bundle goes to this target namespace
--targets-file string Addition source of targets and restrictions to be append
--username string Basic auth username for helm repo
--driven-scan Use driven scan. Bundles are defined by the user.
Paths will be defined by the tuple <base, options> separated by a separator.
The default separator is the character ':'. If you need to use a different one,
use the --driven-scan-sep flag.
--driven-scan-sep Separator used for user-driven bundle generation paths. (default: ':')
```
### Good to know
`fleet apply` now reads known_hosts data from an environment variable (`FLEET_KNOWN_HOSTS`), similarly to what the git
cloner does.
This means that, should you want to run `fleet apply` directly on your host machine, providing it a custom
value of `KNOWN_HOSTS`, you will need to export `FLEET_KNOWN_HOSTS` beforehand or run it as
`FLEET_KNOWN_HOSTS=<your_entries> fleet apply <args>`.
### SEE ALSO
* [fleet](./fleet)

View File

@ -0,0 +1,33 @@
---
title: ""
sidebar_label: "fleet deploy"
---
## fleet deploy
Deploy a bundledeployment/content resource to a cluster, by creating a Helm release. This will not deploy the bundledeployment/content resources directly to the cluster.
```
fleet deploy [flags]
```
### Options
```
-a, --agent-namespace string Set the agent namespace, normally cattle-fleet-system. If set, fleet agent will garbage collect the helm release, i.e. delete it if the bundledeployment is missing.
-d, --dry-run Print the resources that would be deployed, but do not actually deploy them
-h, --help help for deploy
-i, --input-file string Location of the YAML file containing the content and the bundledeployment resource
--kube-version string For dry runs, sets the Kubernetes version to assume when validating Chart Kubernetes version constraints.
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
-n, --namespace string Set the default namespace. Deploy helm chart into this namespace.
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) (default true)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
```
### SEE ALSO
* [fleet](./fleet) -

View File

@ -0,0 +1,30 @@
---
title: ""
sidebar_label: "fleet gitcloner"
---
## fleet gitcloner
Clones a git repository
```
fleet gitcloner [REPO] [PATH] [flags]
```
### Options
```
-b, --branch string git branch
--ca-bundle-file string CA bundle file
-h, --help help for gitcloner
--insecure-skip-tls do not verify tls certificates
--known-hosts-file string known hosts file
--password-file string password file for basic auth
--revision string git revision
--ssh-private-key-file string ssh private key file path
-u, --username string user name for basic auth
```
### SEE ALSO
* [fleet](./fleet) -

View File

@ -0,0 +1,31 @@
---
title: ""
sidebar_label: "fleet target"
---
## fleet target
Print available targets for a bundle
```
fleet target [flags]
```
### Options
```
-b, --bundle-file string Location of the Bundle resource yaml
-l, --dump-input-list Dump the live resources, which impact targeting, like clusters, as YAML
-h, --help help for target
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
-n, --namespace string Override the namespace of the bundle. Targeting searches this namespace for clusters.
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) (default true)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
```
### SEE ALSO
* [fleet](./fleet) -

View File

@ -0,0 +1,30 @@
---
title: ""
sidebar_label: "fleet test"
---
## fleet test
Match a bundle to a target and render the output (deprecated)
```
fleet test [flags]
```
### Options
```
-b, --bundle-file string Location of the raw Bundle resource yaml
-f, --file string Location of the fleet.yaml
-g, --group string Cluster group to match against
-L, --group-label strings Cluster group labels to match against
-h, --help help for test
-l, --label strings Cluster labels to match against
-N, --name string Cluster name to match against
-q, --quiet Just print the match and don't print the resources
-t, --target string Explicit target to match
```
### SEE ALSO
* [fleet](./fleet) -

View File

@ -0,0 +1,34 @@
---
title: ""
sidebar_label: "fleet-controller"
---
## fleet-controller
```
fleet-controller [flags]
```
### Options
```
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
--disable-metrics disable metrics
-h, --help help for fleet-controller
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
--namespace string namespace to watch (default "cattle-fleet-system")
--shard-id string only manage resources labeled with a specific shard ID
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) (default true)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
```
### SEE ALSO
* [fleet-controller agentmanagement](fleet-controller_agentmanagement) -
* [fleet-controller cleanup](fleet-controller_cleanup) -
* [fleet-controller gitjob](fleet-controller_gitjob) -

View File

@ -0,0 +1,34 @@
---
title: ""
sidebar_label: "fleet-controller agentmanagement"
---
## fleet-controller agentmanagement
```
fleet-controller agentmanagement [flags]
```
### Options
```
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
--disable-bootstrap disable local cluster components
-h, --help help for agentmanagement
--kubeconfig string kubeconfig file
--namespace string namespace to watch
```
### Options inherited from parent commands
```
--disable-metrics disable metrics
--shard-id string only manage resources labeled with a specific shard ID
```
### SEE ALSO
* [fleet-controller](./) -

View File

@ -0,0 +1,30 @@
---
title: ""
sidebar_label: "fleet-controller cleanup"
---
## fleet-controller cleanup
```
fleet-controller cleanup [flags]
```
### Options
```
-h, --help help for cleanup
--kubeconfig string kubeconfig file
--namespace string namespace to watch
```
### Options inherited from parent commands
```
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
```
### SEE ALSO
* [fleet-controller](./)

View File

@ -0,0 +1,32 @@
---
title: ""
sidebar_label: "fleet-controller gitjob"
---
## fleet-controller gitjob
```
fleet-controller gitjob [flags]
```
### Options
```
--debug Turn on debug logging
--debug-level int If debugging is enabled, set klog -v=X
--disable-metrics Disable the metrics server.
--gitjob-image string The gitjob image that will be used in the generated job. (default "rancher/fleet:dev")
-h, --help help for gitjob
--kubeconfig string Kubeconfig file
--leader-elect Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager. (default true)
--listen string The port the webhook listens. (default ":8080")
--metrics-bind-address string The address the metric endpoint binds to. (default ":8081")
--namespace string namespace to watch (default "cattle-fleet-system")
--shard-id string only manage resources labeled with a specific shard ID
--shard-node-selector string node selector to apply to jobs based on the shard ID, if any
```
### SEE ALSO
* [fleet-controller](./)

View File

@ -0,0 +1,22 @@
# Create Cluster Groups
Clusters in a namespace can be put into a cluster group. A cluster group is essentially a named selector.
The only parameter for a cluster group is essentially the selector.
When you get to a certain scale cluster groups become a more reasonable way to manage your clusters.
Cluster groups serve the purpose of giving aggregated
status of the deployments and then also a simpler way to manage targets.
A cluster group is created by creating a `ClusterGroup` resource like below
```yaml
kind: ClusterGroup
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: production-group
namespace: clusters
spec:
# This is the standard metav1.LabelSelector format to match clusters by labels
selector:
matchLabels:
env: prod
```

View File

@ -0,0 +1,423 @@
import {versions} from '@site/src/fleetVersions';
import CodeBlock from '@theme/CodeBlock';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Register Downstream Clusters
## Overview
There are two specific styles to registering clusters. These styles will be referred
to as **agent-initiated** and **manager-initiated** registration. Typically one would
go with the agent-initiated registration but there are specific use cases in which
manager-initiated is a better workflow.
### Agent-Initiated Registration
Agent-initiated refers to a pattern in which the downstream cluster installs an agent with a
[cluster registration token](#create-cluster-registration-tokens) and optionally a client ID. The cluster
agent will then make a API request to the Fleet manager and initiate the registration process. Using
this process the Manager will never make an outbound API request to the downstream clusters and will thus
never need to have direct network access. The downstream cluster only needs to make outbound HTTPS
calls to the manager.
It is not commonly used in Rancher. Rancher does not need to make an outbound
API request to the downstream cluster, as it uses a tunnel to provide that
connectivity.
### Manager-Initiated Registration
Manager-initiated registration is a process in which you register an existing Kubernetes cluster
with the Fleet manager and the Fleet manager will make an API call to the downstream cluster to
deploy the agent. This style can place additional network access requirements because the Fleet
manager must be able to communicate with the downstream cluster API server for the registration process.
After the cluster is registered there is no further need for the manager to contact the downstream
cluster API. This style is more compatible if you wish to manage the creation of all your Kubernetes
clusters through GitOps using something like [cluster-api](https://github.com/kubernetes-sigs/cluster-api)
or [Rancher](https://github.com/rancher/rancher).
```mermaid
graph TD
subgraph "Upstream (Management Cluster)"
direction LR
subgraph "Flow 1: Agent-Initiated"
direction TB
A0(Optional: Admin Creates Cluster with clientID) --> A1
A1(Admin Creates<br>ClusterRegistrationToken) --> A2{Fleet Controller Creates Secret<br>for a temporary 'import' ServiceAccount}
end
subgraph "Flow 2: Manager-Initiated (for existing cluster)"
direction TB
B1(Admin Creates Kubeconfig Secret<br>for an existing cluster) --> B2(Admin Creates Cluster Resource<br>referencing the Kubeconfig Secret.<br>Can define a clientID here)
B2 --> B3{Fleet Controller uses admin-provided<br>kubeconfig to deploy agent}
end
end
subgraph "Downstream (Managed Cluster)"
direction LR
subgraph "Agent Install (Flow 1)"
direction TB
A3(Admin installs Fleet Agent via Helm<br>using the 'import' token secret.<br>Can provide clientID)
end
subgraph "Agent Deployed (Flow 2)"
direction TB
B4(Agent & bootstrap secret are deployed.<br>Bootstrap contains an 'import' kubeconfig.)
end
end
subgraph "Common Registration Stages (Identity Handshake)"
direction TB
C1(Agent pod starts, using its local 'agent' SA.<br>Finds & uses the 'import' kubeconfig<br>from the bootstrap secret to talk to Upstream.)
C1 --> C2(Using its 'import' identity, Agent creates<br>a ClusterRegistration resource on Upstream)
C2 --> C3{Upstream Controller creates a permanent<br>'request' ServiceAccount & a new,<br>long-term kubeconfig/secret for it.}
C3 --> C4(Agent receives and persists the<br>'request' SA credentials.<br>The temporary bootstrap secret is deleted.)
C4 --> C5{Upstream Controller creates a dedicated<br>Cluster Namespace for this agent.}
C5 --> C6(✅ Agent Fully Registered.<br>Uses its 'request' identity to watch<br>for workloads in its namespace.)
end
%% Styling
style A0 fill:#e0f2fe,stroke:#0ea5e9,stroke-width:2px
style A1 fill:#e0f2fe,stroke:#0ea5e9,stroke-width:2px
style B1 fill:#e0f2fe,stroke:#0ea5e9,stroke-width:2px
style A3 fill:#d1fae5,stroke:#10b981,stroke-width:2px
style B2 fill:#e0f2fe,stroke:#0ea5e9,stroke-width:2px
style A2 fill:#fef3c7,stroke:#f59e0b,stroke-width:2px
style B3 fill:#fef3c7,stroke:#f59e0b,stroke-width:2px
style B4 fill:#fef3c7,stroke:#f59e0b,stroke-width:2px
style C1 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px
style C2 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px
style C3 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px
style C4 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px
style C5 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px
style C6 fill:#dcfce7,stroke:#22c55e,stroke-width:2px,font-weight:bold
%% Connections
A2 --> A3
B3 --> B4
A3 --> C1
B4 --> C1
```
## Agent Initiated
A downstream cluster is registered by installing an agent via helm and using the **cluster registration token** and optionally a **client ID** or **cluster labels**.
The cluster resource on upstream does not have a kubecConfigSecret field, as
the Fleet manager does not need to communicate with the downstream cluster API
server.
However, a bundle for the agent is created and the agent will update itself from the bundle.
The bundle uses a different naming scheme, so the downstream cluster will end up with two helm charts.
:::info
It's not necessary to configure the Fleet manager for [multi cluster](./installation.md#configuration-for-multi-cluster), as the downstream agent we install via Helm will connect to the Kubernetes API of the upstream cluster directly.
Agent-initiated registration is normally not used with Rancher.
:::
### Cluster Registration Token and Client ID
The **cluster registration token** is a credential that will authorize the downstream cluster agent to be
able to initiate the registration process. This is required.
The [cluster registration token](./architecture.md#security) is manifested as a `values.yaml` file that will be passed to the `helm install` process.
Alternatively one can pass the token directly to the helm install command via `--set token="$token"`.
There are two styles of registering an agent. You can have the cluster for this agent dynamically created, in which
case you will probably want to specify **cluster labels** upon registration. Or you can have the agent register to a predefined
cluster in the Fleet manager, in which case you will need a **client ID**. The former approach is typically the easiest.
### Install Agent For a New Cluster
The Fleet agent is installed as a Helm chart. Following are explanations how to determine and set its parameters.
First, follow the [cluster registration token instructions](#create-cluster-registration-tokens) to obtain the `values.yaml` which contains
the registration token to authenticate against the Fleet cluster.
Second, optionally you can define labels that will assigned to the newly created cluster upon registration. After
registration is completed an agent cannot change the labels of the cluster. To add cluster labels add
`--set-string labels.KEY=VALUE` to the below Helm command. To add the labels `foo=bar` and `bar=baz` then you would
add `--set-string labels.foo=bar --set-string labels.bar=baz` to the command line.
```shell
# Leave blank if you do not want any labels
CLUSTER_LABELS="--set-string labels.example=true --set-string labels.env=dev"
```
Third, set variables with the Fleet cluster's API Server URL and CA, for the downstream cluster to use for connecting.
```shell
API_SERVER_URL=https://<API_URL>:6443
API_SERVER_CA_DATA=...
```
If the API server is not listening on the https port (443), the `API_SERVER_URL` should include the port, e.g. `https://<API_URL>:6443`. The URL can be found in the `.kube/config` file.
Value in `API_SERVER_CA_DATA` can be obtained from a `.kube/config` file with valid data to connect to the upstream cluster
(under the `certificate-authority-data` key). Alternatively it can be obtained from within the upstream cluster itself,
by looking up the default ServiceAccount secret name (typically prefixed with `default-token-`, in the default namespace),
under the `ca.crt` key.
:::warning Kubectl Context
__Ensure you are installing to the right cluster__:
Helm will use the default context in `${HOME}/.kube/config` to deploy the agent. Use `--kubeconfig` and `--kube-context`
to change which cluster Helm is installing to.
:::
:::caution Fleet in Rancher
Rancher has separate helm charts for Fleet and uses a different repository.
:::
Add Fleet's Helm repo.
<CodeBlock language="bash">
{`helm repo add fleet https://rancher.github.io/fleet-helm-charts/`}
</CodeBlock>
:::caution
__Use proper namespace and release name__:
For the agent chart the namespace must be `cattle-fleet-system` and the release name `fleet-agent`
:::
Finally, install the agent using Helm.
<Tabs>
<TabItem value="helm" label="Install" default>
<CodeBlock language="bash">
{`helm -n cattle-fleet-system install --create-namespace --wait \\
$CLUSTER_LABELS \\
--values values.yaml \\
--set apiServerCA="$API_SERVER_CA_DATA" \\
--set apiServerURL="$API_SERVER_URL" \\
fleet-agent fleet/fleet-agent`}
</CodeBlock>
</TabItem>
<TabItem value="validate" label="Validate">
You can check that status of the fleet pods by running the below commands.
```shell
# Ensure kubectl is pointing to the right cluster
kubectl -n cattle-fleet-system logs -l app=fleet-agent
kubectl -n cattle-fleet-system get pods -l app=fleet-agent
```
</TabItem>
</Tabs>
The agent should now be deployed.
Additionally you should see a new cluster registered in the Fleet manager. Below is an example of checking that a new cluster
was registered in the `clusters` [namespace](./namespaces.md). Please ensure your `${HOME}/.kube/config` is pointed to the Fleet
manager to run this command.
```shell
kubectl -n clusters get clusters.fleet.cattle.io
```
```
NAME BUNDLES-READY NODES-READY SAMPLE-NODE LAST-SEEN STATUS
cluster-ab13e54400f1 1/1 1/1 k3d-cluster2-server-0 2020-08-31T19:23:10Z
```
### Install Agent For a Predefined Cluster
Client IDs are for the purpose of predefining clusters in the Fleet manager with existing labels and repos targeted to them.
A client ID is not required and is just one approach to managing clusters.
The **client ID** is a unique string that will identify the cluster.
This string is user generated and opaque to the Fleet manager and agent. It is assumed to be sufficiently unique. For security reasons one should not be able to easily guess this value
as then one cluster could impersonate another. The client ID is optional and if not specified the UID field of the `kube-system` namespace
resource will be used as the client ID. Upon registration if the client ID is found on a `Cluster` resource in the Fleet manager it will associate
the agent with that `Cluster`. If no `Cluster` resource is found with that client ID a new `Cluster` resource will be created with the specific
client ID.
The Fleet agent is installed as a Helm chart. The only parameters to the helm chart installation should be the cluster registration token, which
is represented by the `values.yaml` file and the client ID. The client ID is optional.
First, create a `Cluster` in the Fleet Manager with the random client ID you have chosen.
```yaml
kind: Cluster
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: my-cluster
namespace: clusters
spec:
clientID: "really-random"
```
Second, follow the [cluster registration token instructions](#create-cluster-registration-tokens) to obtain the `values.yaml` file to be used.
Third, setup your environment to use the client ID.
```shell
CLUSTER_CLIENT_ID="really-random"
```
:::note
__Use proper namespace and release name__:
For the agent chart the namespace must be `cattle-fleet-system` and the release name `fleet-agent`
:::
:::note
__Ensure you are installing to the right cluster__:
Helm will use the default context in `${HOME}/.kube/config` to deploy the agent. Use `--kubeconfig` and `--kube-context`
to change which cluster Helm is installing to.
:::
Add Fleet's Helm repo.
<CodeBlock language="bash">
{`helm repo add fleet https://rancher.github.io/fleet-helm-charts/`}
</CodeBlock>
Finally, install the agent using Helm.
<Tabs>
<TabItem value="helm2" label="Install" default>
<CodeBlock language="bash">
{`helm -n cattle-fleet-system install --create-namespace --wait \\
--set clientID="$CLUSTER_CLIENT_ID" \\
--values values.yaml \\
fleet-agent fleet/fleet-agent`}
</CodeBlock>
</TabItem>
<TabItem value="validate2" label="Validate">
You can check that status of the fleet pods by running the below commands.
```shell
# Ensure kubectl is pointing to the right cluster
kubectl -n cattle-fleet-system logs -l app=fleet-agent
kubectl -n cattle-fleet-system get pods -l app=fleet-agent
```
</TabItem>
</Tabs>
The agent should now be deployed.
Additionally you should see a new cluster registered in the Fleet manager. Below is an example of checking that a new cluster
was registered in the `clusters` [namespace](./namespaces.md). Please ensure your `${HOME}/.kube/config` is pointed to the Fleet
manager to run this command.
```shell
kubectl -n clusters get clusters.fleet.cattle.io
```
```
NAME BUNDLES-READY NODES-READY SAMPLE-NODE LAST-SEEN STATUS
my-cluster 1/1 1/1 k3d-cluster2-server-0 2020-08-31T19:23:10Z
```
### Create Cluster Registration Tokens
:::info
__Not needed for Manager-initiated registration__:
For manager-initiated registrations the token is managed by the Fleet manager and does
not need to be manually created and obtained.
:::
For an agent-initiated registration the downstream cluster must have a [cluster registration token](./architecture.md#security).
Cluster registration tokens are used to establish a new identity for a cluster. Internally
cluster registration tokens are managed by creating Kubernetes service accounts that have the
permissions to create `ClusterRegistrationRequests` within a specific namespace. Once the
cluster is registered a new `ServiceAccount` is created for that cluster that is used as
the unique identity of the cluster. The agent is designed to forget the cluster registration
token after registration. While the agent will not maintain a reference to the cluster registration
token after a successful registration please note that usually other system bootstrap scripts do.
Since the cluster registration token is forgotten, if you need to re-register a cluster you must
give the cluster a new registration token.
#### Token TTL
Cluster registration tokens can be reused by any cluster in a namespace. The tokens can be given a TTL
such that it will expire after a specific time.
#### Create a new Token
The `ClusterRegistationToken` is a namespaced type and should be created in the same namespace
in which you will create `GitRepo` and `ClusterGroup` resources. For in depth details on how namespaces
are used in Fleet refer to the documentation on [namespaces](./namespaces.md). Create a new
token with the below YAML.
```yaml
kind: ClusterRegistrationToken
apiVersion: "fleet.cattle.io/v1alpha1"
metadata:
name: new-token
namespace: clusters
spec:
# A duration string for how long this token is valid for. A value <= 0 or null means infinite time.
ttl: 240h
```
After the `ClusterRegistrationToken` is created, Fleet will create a corresponding `Secret` with the same name.
As the `Secret` creation is performed asynchronously, you will need to wait until it's available before using it.
One way to do so is via the following one-liner:
```shell
while ! kubectl --namespace=clusters get secret new-token; do sleep 5; done
```
#### Obtaining Token Value (Agent values.yaml)
The token value contains YAML content for a `values.yaml` file that is expected to be passed to `helm install`
to install the Fleet agent on a downstream cluster.
Such value is contained in the `values` field of the `Secret` mentioned above. To obtain the YAML content for the
above example one can run the following one-liner:
```shell
kubectl --namespace clusters get secret new-token -o 'jsonpath={.data.values}' | base64 --decode > values.yaml
```
Once the `values.yaml` is ready it can be used repeatedly by clusters to register until the TTL expires.
## Manager Initiated
The manager-initiated registration flow is accomplished by creating a
`Cluster` resource in the Fleet Manager that refers to a Kubernetes
`Secret` containing a valid kubeconfig file in the data field called `value`.
:::info
If you are using Fleet standalone *without Rancher*, it must be installed as described in [installation details](./installation.md#configuration-for-multi-cluster).
The manager-initiated registration is used when you add a cluster from the Rancher dashboard.
:::
### Create Kubeconfig Secret
The format of this secret is intended to match the [format](https://cluster-api.sigs.k8s.io/developer/architecture/controllers/cluster.html#secrets) of the kubeconfig
secret used in [cluster-api](https://github.com/kubernetes-sigs/cluster-api).
This means you can use `cluster-api` to create a cluster that is dynamically registered with Fleet.
```yaml title="Kubeconfig Secret Example"
kind: Secret
apiVersion: v1
metadata:
name: my-cluster-kubeconfig
namespace: clusters
data:
value: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIHNlcnZlcjogaHR0cHM6Ly9leGFtcGxlLmNvbTo2NDQzCiAgbmFtZTogY2x1c3Rlcgpjb250ZXh0czoKLSBjb250ZXh0OgogICAgY2x1c3RlcjogY2x1c3RlcgogICAgdXNlcjogdXNlcgogIG5hbWU6IGRlZmF1bHQKY3VycmVudC1jb250ZXh0OiBkZWZhdWx0CmtpbmQ6IENvbmZpZwpwcmVmZXJlbmNlczoge30KdXNlcnM6Ci0gbmFtZTogdXNlcgogIHVzZXI6CiAgICB0b2tlbjogc29tZXRoaW5nCg==
```
### Create Cluster Resource
The cluster resource needs to reference the kubeconfig secret.
```yaml title="Cluster Resource Example"
apiVersion: fleet.cattle.io/v1alpha1
kind: Cluster
metadata:
name: my-cluster
namespace: clusters
labels:
demo: "true"
env: dev
spec:
kubeConfigSecret: my-cluster-kubeconfig
```

View File

@ -0,0 +1,51 @@
# Core Concepts
Fleet is fundamentally a set of Kubernetes custom resource definitions (CRDs) and controllers
to manage GitOps for a single Kubernetes cluster or a large-scale deployment of Kubernetes clusters.
:::info
For more on the naming conventions of CRDs, click [here](./troubleshooting.md#naming-conventions-for-crds).
:::
Below are some of the concepts of Fleet that will be useful throughout this documentation:
* **Fleet Manager**: The centralized component that orchestrates the deployments of Kubernetes assets
from git. In a multi-cluster setup, this will typically be a dedicated Kubernetes cluster. In a
single cluster setup, the Fleet manager will be running on the same cluster you are managing with GitOps.
* **Fleet controller**: The controller(s) running on the Fleet manager orchestrating GitOps. In practice,
the Fleet manager and Fleet controllers are used fairly interchangeably.
* **Single Cluster Style**: This is a style of installing Fleet in which the manager and downstream cluster are the
same cluster. This is a very simple pattern to quickly get up and running with GitOps.
* **Multi Cluster Style**: This is a style of running Fleet in which you have a central manager that manages a large
number of downstream clusters.
* **Fleet agent**: Every managed downstream cluster will run an agent that communicates back to the Fleet manager.
This agent is just another set of Kubernetes controllers running in the downstream cluster.
* **GitRepo**: Git repositories that are watched by Fleet are represented by the type `GitRepo`.
>**Example installation order via `GitRepo` custom resources when using Fleet for the configuration management of downstream clusters:**
>
> 1. Install [Calico](https://github.com/projectcalico/calico) CRDs and controllers.
> 2. Set one or multiple cluster-level global network policies.
> 3. Install [GateKeeper](https://github.com/open-policy-agent/gatekeeper). Note that **cluster labels** and **overlays** are critical features in Fleet as they determine which clusters will get each part of the bundle.
> 4. Set up and configure ingress and system daemons.
* **Bundle**: An internal unit used for the orchestration of resources from git.
When a `GitRepo` is scanned it will produce one or more bundles. Bundles are a collection of
resources that get deployed to a cluster. `Bundle` is the fundamental deployment unit used in Fleet. The
contents of a `Bundle` may be Kubernetes manifests, Kustomize configuration, or Helm charts.
Regardless of the source the contents are dynamically rendered into a Helm chart by the agent
and installed into the downstream cluster as a helm release.
- To see the **life cycle of a bundle**, click [here](./ref-bundle-stages.md).
* **BundleDeployment**: When a `Bundle` is deployed to a cluster an instance of a `Bundle` is called a `BundleDeployment`.
A `BundleDeployment` represents the state of that `Bundle` on a specific cluster with its cluster specific
customizations. The Fleet agent is only aware of `BundleDeployment` resources that are created for
the cluster the agent is managing.
- For an example of how to deploy Kubernetes manifests across clusters using Fleet customization, click [here](./gitrepo-targets.md#customization-per-cluster).
* **Downstream Cluster**: Clusters to which Fleet deploys manifests are referred to as downstream clusters. In the single cluster use case, the Fleet manager Kubernetes cluster is both the manager and downstream cluster at the same time.
* **Cluster Registration Token**: Tokens used by agents to register a new cluster.

View File

@ -0,0 +1,310 @@
# Create a GitRepo Resource
## Create GitRepo Instance
Git repositories are registered by creating a `GitRepo` resource in Kubernetes. Refer
to the [creating a deployment tutorial](./tut-deployment.md) for examples.
[Git Repository Contents](./gitrepo-content.md) has detail about the content of the Git repository.
The available fields of the GitRepo custom resource are documented in the [GitRepo resource reference](./ref-gitrepo.md)
### Proper Namespace
Git repos are added to the Fleet manager using the `GitRepo` custom resource type. The `GitRepo` type is namespaced. By default, Rancher will create two Fleet workspaces: **fleet-default** and **fleet-local**.
- `fleet-default` will contain all the downstream clusters that are already registered through Rancher.
- `fleet-local` will contain the local cluster by default.
If you are using Fleet in a [single cluster](./concepts.md) style, the namespace will always be **fleet-local**. Check [here](https://fleet.rancher.io/namespaces#fleet-local) for more on the `fleet-local` namespace.
For a [multi-cluster](./concepts.md) style, please ensure you use the correct repo that will map to the right target clusters.
## Override Workload's Namespace
The `targetNamespace` field will override any namespace in the bundle. If the deployment contains cluster scoped resources, it will fail.
It takes precendence over all other namespace definitions:
`gitRepo.targetNamespace > fleet.yaml namespace > namespace in workload's manifest > fleet.yaml defaultNamespace`
Workload namespace definitions can be restricted with `allowedTargetNamespaces` in the `GitRepoRestriction` resource.
## Adding A Private Git Repository
Fleet supports both HTTP and SSH auth keys for private repositories. To use this, you have to create a secret in the
`GitRepo`'s namespace.
For example, to generate a private SSH key:
```text
ssh-keygen -t rsa -b 4096 -m pem -C "user@email.com"
```
Note: The private key format has to be in `EC PRIVATE KEY`, `RSA PRIVATE KEY` or `PRIVATE KEY` and should not contain a passphase.
Put your private key into secret, use the namespace the GitRepo is in:
```text
kubectl create secret generic ssh-key -n fleet-default --from-file=ssh-privatekey=/file/to/private/key --type=kubernetes.io/ssh-auth
```
Now the `clientSecretName` must be specified in the repo definition:
```text
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: sample-ssh
# This namespace is special and auto-wired to deploy to the local cluster
namespace: fleet-local
spec:
# Everything from this repo will be run in this cluster. You trust me right?
repo: "git@github.com:rancher/fleet-examples"
# or
# repo: "ssh://git@github.com/rancher/fleet-examples"
clientSecretName: ssh-key
paths:
- simple
```
:::caution
Private key with passphrase is not supported.
:::
:::caution
The key has to be in PEM format.
:::
### Known hosts
:::warning
If you don't add one or more public keys into the secret, any server's public key will be trusted and added. (`ssh -o
stricthostkeychecking=yes` will be used), unless you install Fleet with chart value `insecureSkipHostKeyChecks` set to
`false`.
:::
Fleet supports injecting `known_hosts` into an SSH secret. Here is an example of how to add it:
Fetch the public key hash (taking Github as an example)
```text
ssh-keyscan -H github.com
```
And add it into the secret:
```text
apiVersion: v1
kind: Secret
metadata:
name: ssh-key
type: kubernetes.io/ssh-auth
stringData:
ssh-privatekey: <private-key>
known_hosts: |-
|1|YJr1VZoi6dM0oE+zkM0do3Z04TQ=|7MclCn1fLROZG+BgR4m1r8TLwWc= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
```
#### Strict host key checks
Chart value `insecureSkipHostKeyChecks` defines how Fleet behaves with regards to `known_hosts` when establishing SSH
connections.
When that value is set to `false`, Fleet will enforce strict host key checks, meaning that it will fail to establish any
SSH connections to hosts for which no matching `known_hosts` entry can be found.
This is the default behaviour from Fleet v0.13 onwards.
`known_hosts` entries are sourced in priority from secrets referenced in `GitRepo`s, e.g. `helmSecretName` for accessing
Helm charts or `clientSecretName` for cloning git repositories.
Note that this is compatible with Fleet looking for a `gitcredential` secret if no secret is referenced in the
`GitRepo`.
If no such secret exists, or no `known_hosts` entries are available in that secret, then Fleet uses its own
`known-hosts` config map, newly created at installation time with static entries for the most widely used git providers.
Host key fingerprints added to the config map are sourced, respectively:
* from [here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints) for
Github
* from [here](https://docs.gitlab.com/ee/user/gitlab_com/index.html#ssh-known_hosts-entries) for Gitlab
* from [here](https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/) for
Bitbucket, which also provides a `curl` command to fetch them in `known_hosts`-friendly format: `curl
https://bitbucket.org/site/ssh`
* from [here](https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops)
for Azure DevOps
The absence of the config map, should no secret be available, is considered a symptom of an incomplete Fleet deployment,
and reported as such.
Fleet only uses a _single_ source of `known_hosts` entries at a time. This means that, even if a secret contains invalid
(or insufficient) entries, then Fleet will not look for valid entries in the config map. This applies to a secret
referenced in a `GitRepo` as well as to a possible `gitcredential` secret, if no secret is referenced in the `GitRepo`.
### Using HTTP Auth
Create a secret containing username and password. You can replace the password with a personal access token if necessary. Also see [HTTP secrets in Github](./troubleshooting#http-secrets-in-github).
kubectl create secret generic basic-auth-secret -n fleet-default --type=kubernetes.io/basic-auth --from-literal=username=$user --from-literal=password=$pat
Just like with SSH, reference the secret in your GitRepo resource via `clientSecretName`.
spec:
repo: https://github.com/fleetrepoci/gitjob-private.git
branch: main
clientSecretName: basic-auth-secret
### Using Custom CA Bundles
Validating a repository using a certificate signed by a custom Certificate Authority can be done by specifying a
`cabundle` field in a `GitRepo`.
:::info
Note that if secrets specifying CA bundles exist, for instance if Fleet is installed with Rancher (see
[this](https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/add-tls-secrets#using-a-private-ca-signed-certificate)
and
[that](https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-references/helm-chart-options#additional-trusted-cas)),
Fleet will use those CA bundles if no CA bundle is specified in the `GitRepo`.
:::
## Using Private Helm Repositories
:::warning
The credentials will be used unconditionally for all Helm repositories referenced by the gitrepo resource.
Make sure you don't leak credentials by mixing public and private repositories. Use [different helm credentials for each path](#use-different-helm-credentials-for-each-path),
or split them into different gitrepos, or use `helmRepoURLRegex` to limit the scope of credentials to certain servers.
:::
For a private Helm repo, users can reference a secret with the following keys:
1. `username` and `password` for basic http auth if the Helm HTTP repo is behind basic auth.
2. `cacerts` for custom CA bundle if the Helm repo is using a custom CA.
:::info
Note that if secrets specifying CA bundles exist, for instance if Fleet is installed with Rancher (see
[this](https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/add-tls-secrets#using-a-private-ca-signed-certificate)
and
[that](https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-references/helm-chart-options#additional-trusted-cas)),
Fleet will use those CA bundles if no CA bundle is specified in the Helm secret.
:::
3. `ssh-privatekey` for ssh private key if repo is using ssh protocol. Private key with passphase is not supported currently.
For example, to add a secret in kubectl, run
`kubectl create secret -n $namespace generic helm --from-literal=username=foo --from-literal=password=bar --from-file=cacerts=/path/to/cacerts --from-file=ssh-privatekey=/path/to/privatekey.pem`
After secret is created, specify the secret to `gitRepo.spec.helmSecretName`. Make sure secret is created under the same namespace with gitrepo.
### Use different helm credentials for each path
Fleet allows you to define unique credentials for each Helm chart path in a Git repository using the `helmSecretNameForPaths` field.
:::info
If `gitRepo.spec.helmSecretNameForPaths` is defined, `gitRepo.spec.helmSecretName` is ignored.
:::
Create a file named `secrets-path.yaml` that specifies credentials for each path in your `GitRepo`. The keys must match the full path to a bundle directory (a folder containing a `fleet.yaml file`), which may have more segments than the entry under `paths:`. If a path listed in the GitRepo is not included in this file, Fleet does not use credentials for it.
:::note
The file should be named `secrets-path.yaml`, otherwise Fleet will not be able to use it.
:::
Example `GitRepo` resource:
```yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: gitrepo
namespace: fleet-local
spec:
helmSecretNameForPaths: test-multipasswd
repo: https://github.com/0xavi0/fleet-examples
branch: helm-multi-passwd
paths:
- single-cluster/test-multipasswd
```
Example `secrets-path.yaml`:
```yaml
single-cluster/test-multipasswd/passwd:
username: fleet-ci
password: foo
insecureSkipVerify: true
```
Another example with two distinct paths:
```yaml
path-one: # path path-one must exist in the repository
username: user
password: pass
path-two: # path path-one must exist in the repository
username: user2
password: pass2
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCiAgICBNSUlEblRDQ0FvV2dBd0lCQWdJVUNwMHB2SVJTb2c0eHJKN2Q1SUI2ME1ka0k1WXdEUVlKS29aSWh2Y05BUUVMCiAgICBCUUF3WGpFTE1Ba0dBMVVFQmhNQ1FWVXhFekFSQmdOVkJBZ01DbE52YldVdFUzUmhkR1V4SVRBZkJnTlZCQW9NCiAgICBHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpERVhNQlVHQTFVRUF3d09jbUZ1WTJobGNpNXRlUzV2CiAgICBjbWN3SGhjTk1qTXdOREkzTVRVd056VXpXaGNOTWpnd05ESTFNVFV3TnpVeldqQmVNUXN3Q1FZRFZRUUdFd0pCCiAgICBWVEVUTUJFR0ExVUVDQXdLVTI5dFpTMVRkR0YwWlRFaE1COEdBMVVFQ2d3WVNXNTBaWEp1WlhRZ1YybGtaMmwwCiAgICBjeUJRZEhrZ1RIUmtNUmN3RlFZRFZRUUREQTV5WVc1amFHVnlMbTE1TG05eVp6Q0NBU0l3RFFZSktvWklodmNOCiAgICBBUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTXBvZE5TMDB6NDc1dnVSc2ZZcTFRYTFHQVl3QU92anV4MERKTHY5CiAgICBrZFhwT091dGdjMU8yWUdqNUlCVGQzVmpISmFJYUg3SDR2Rm84RlBaMG9zcU9YaFg3eUM4STdBS3ZhOEE5VmVmCiAgICBJVXp6Vlo1cCs1elNxRjdtZTlOaUNiL0pVSkZLT0ZsTkF4cjZCcXhoMEIyN1VZTlpjaUIvL1V0L0I2eHJuVE55CiAgICBoRzJiNzk4bjg4bFZqY3EzbEE0djFyM3VzWGYxVG5aS2t2UEN4ZnFHYk5OdTlpTjdFZnZHOWoyekdHcWJvcDRYCiAgICBXY3VSa3N3QkgxZlRNS0ZrbGcrR1VsZkZPMGFzL3phalVOdmdweTlpdVBMZUtqZTVWcDBiMlBLd09qUENpV2d4CiAgICBabDJlVDlNRnJjV0F3NTg3emE5NDBlT1Era2pkdmVvUE5sU2k3eVJMMW96YlRka0NBd0VBQWFOVE1GRXdIUVlECiAgICBWUjBPQkJZRUZEQkNkYjE4M1hsU0tWYzBxNmJSTCt0dVNTV3lNQjhHQTFVZEl3UVlNQmFBRkRCQ2RiMTgzWGxTCiAgICBLVmMwcTZiUkwrdHVTU1d5TUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCiAgICBBQ1BCVERkZ0dCVDVDRVoxd1pnQmhKdm9GZTk2MUJqVCtMU2RxSlpsSmNRZnlnS0hyNks5ZmZaY1ZlWlBoMVU0CiAgICB3czBuWGNOZiszZGJlTjl4dVBiY0VqUWlQaFJCcnRzalE1T1JiVHdYWEdBdzlYbDZYTkl6YjN4ZDF6RWFzQXZPCiAgICBJMjM2ZHZXQ1A0dWoycWZqR0FkQjJnaXU2b2xHK01CWHlneUZKMElzRENraldLZysyWEdmU3lyci9KZU1vZlFBCiAgICB1VU9wcFVGdERYd0lrUW1VTGNVVUxWcTdtUVNQb0lzVkNNM2hKNVQzczdUSWtHUDZVcGVSSjgzdU9LbURYMkRHCiAgICBwVWVQVHBuVWVLOVMzUEVKTi9XcmJSSVd3WU1OR29qdDRKWitaK1N6VE1aVkh0SlBzaGpjL1hYOWZNU1ZXQmlzCiAgICBQRW5MU256MDQ4OGFUQm5SUFlnVXFsdz0KICAgIC0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0=
sshPrivateKey: ICAgIC0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQogICAgTUlJRFF6Q0NBaXNDRkgxTm5YUWI5SlV6anNBR3FSc3RCYncwRlFpak1BMEdDU3FHU0liM0RRRUJDd1VBTUY0eAogICAgQ3pBSkJnTlZCQVlUQWtGVk1STXdFUVlEVlFRSURBcFRiMjFsTFZOMFlYUmxNU0V3SHdZRFZRUUtEQmhKYm5SbAogICAgY201bGRDQlhhV1JuYVhSeklGQjBlU0JNZEdReEZ6QVZCZ05WQkFNTURuSmhibU5vWlhJdWJYa3ViM0puTUI0WAogICAgRFRJek1EUXlOekUxTVRBMU5Gb1hEVEkwTURReU5qRTFNVEExTkZvd1hqRUxNQWtHQTFVRUJoTUNRVlV4RXpBUgogICAgQmdOVkJBZ01DbE52YldVdFUzUmhkR1V4SVRBZkJnTlZCQW9NR0VsdWRHVnlibVYwSUZkcFpHZHBkSE1nVUhSNQogICAgSUV4MFpERVhNQlVHQTFVRUF3d09jbUZ1WTJobGNpNXRlUzV2Y21jd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQogICAgQTRJQkR3QXdnZ0VLQW9JQkFRRGd6UUJJTW8xQVFHNnFtYmozbFlYUTFnZjhYcURTbjdyM2lGcVZZZldDVWZOSwogICAgaGZwampTRGpOMmRWWEV2UXA3R0t3akFHUElFbXR5RmxyUW5rUGtnTGFSaU9jSDdNN0p2c3ZIa0Ewd0g0dzJ2QgogICAgUEp6aVlINWh2MUE2WS9NcFM5bVkvQUVxVm80TUJkdnNZQzc3MFpCbzVBMitIUEtMd1YzMVZyYlhhTytWeUJtNAogICAgSmJhZHlNUk40N3BKRWdPMjJaYVRXL3Y3S1dKdjNydGJTMlZVSkNlU0piWlpsN09ocHhLRTVocStmK0RWaU1mcQogICAgTWx4ODNEV2pVSlVkV3lqVUZYVlk0bEdVaUtrRWVtSlVuSlVyY1ErOXE1SzVaWmhyRjhoRXhKRjhiZTZjemVzeAogICAga1VWN3dKb1RjWkd2bUhYSk1FNmtrQXh4Mmh3bU8wSFcyQWdDdTJZekFnTUJBQUV3RFFZSktvWklodmNOQVFFTAogICAgQlFBRGdnRUJBS1BpTWdXc1dCTnJvRkY2aWpYL2xMM3FxaWc4TjlkR1VPWDIyRVJDU1RTekNONjM0ZTFkZUhsdQogICAgbTc5OU11Q3hvWSsyZWluNlV1cFMvTEV6cnpvU2dDVWllQzQrT3ZralF5eGJpTFR6bW1OWEFnd09TM3RvTHRGWAogICAgbytmWWpSMU9xcHVPS29kMkhiYjliczRWcXdaNHEvMlVKbXE2Q01pYjZKZUE2VFJvK2Rkc0pUM2dDOFhWL1Z1MAogICAgNnkwdjJxdTM0bm1MYjFxOHFTS1RwZXYyQmwzQUJGY3NyS0JvNHFieUM2bnBTbnpZenNYcS90SlFLclplNE4vMgogICAgUXIzd1dxQ0pDVWUrMWVsT3A2b0JVcXNWSnc3aHk3YzRLc1Fna09ERDJkc2NuNEF1NGJhWlY2QmpySm1USVY0aQogICAgeXJ1dk9oZ2lINklGUVdDWmVQM2s0MU5obWRzRTNHQT0KICAgIC0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
```
Supported fields per path:
| Field | Description |
| -- | --- |
| `username` | Registry or repository username |
| `password` | Registry or repository password |
| `caBundle` | Base64-encoded CA certificate bundle |
| `sshPrivateKey` | Base64-encoded SSH private key |
| `insecureSkipVerify` | Boolean value to skip TLS verification |
To create the secret, run the following command.
```bash
kubectl create secret generic test-multipasswd -n fleet-local --from-file=secrets-path.yaml
```
:::note
The secret must be created in the same namespace as the `GitRepo` resource.
:::
If you use [rancher-backups](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher) and want to include this secret in your backups, label it with `resources.cattle.io/backup: true`:
```bash
kubectl label secret path-auth-secret -n fleet-local resources.cattle.io/backup=true
```
:::note
Ensure the backup is encrypted to protect sensitive credentials.
:::
## Storing Credentials in Git
It's recommended not to store credentials in Git. Even if the repository is properly protected, the secrets are at risk when cloning, etc.
As a workaround tools like SOPS can be used to encrypt the credentials.
Instead it is recommended to reference secrets in the downstream cluster. For manifest-style and kustomize-style bundles this must be done in the manifests, e.g. by [mounting the secrets](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#create-a-pod-that-has-access-to-the-secret-data-through-a-volume) or [referencing them as environment variables](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables).
Helm-style bundles can use [valuesFrom](gitrepo-content#using-valuesfrom) to read values from a secret in the downstream cluster.
When using Kubernetes [encryption at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) and storing credentials in Git, it is recommended to configure the upstream cluster to include several Fleet CRDs in encryption resource list:
```
- secrets
- bundles.fleet.cattle.io
- bundledeployments.fleet.cattle.io
- contents.fleet.cattle.io
```
# Troubleshooting
See Fleet Troubleshooting section [here](./troubleshooting.md).

View File

@ -0,0 +1,355 @@
# Git Repository Contents
Fleet will create bundles from a git repository. This happens either explicitly by specifying paths, or when a `fleet.yaml` is found.
Each bundle is created from paths in a GitRepo and modified further by reading the discovered `fleet.yaml` file.
Bundle lifecycles are tracked between releases by the helm releaseName field added to each bundle. If the releaseName is not
specified within fleet.yaml it is generated from `GitRepo.name + path`. Long names are truncated and a `-<hash>` prefix is added.
**The git repository has no explicitly required structure.** It is important
to realize the scanned resources will be saved as a resource in Kubernetes so
you want to make sure the directories you are scanning in git do not contain
arbitrarily large resources. Right now there is a limitation that the resources
deployed must **gzip to less than 1MB**.
## How repos are scanned
Multiple paths can be defined for a `GitRepo` and each path is scanned independently.
Internally each scanned path will become a [bundle](./concepts.md) that Fleet will manage,
deploy, and monitor independently.
The following files are looked for to determine the how the resources will be deployed.
| File | Location | Meaning |
|------|----------|---------|
| **Chart.yaml**:| / relative to `path` or custom path from `fleet.yaml` | The resources will be deployed as a Helm chart. Refer to the `fleet.yaml` for more options. |
| **kustomization.yaml**:| / relative to `path` or custom path from `fleet.yaml` | The resources will be deployed using Kustomize. Refer to the `fleet.yaml` for more options. |
| **fleet.yaml** | Any subpath | If any fleet.yaml is found a new [bundle](./concepts.md) will be defined. This allows mixing charts, kustomize, and raw YAML in the same repo |
| ** *.yaml ** | Any subpath | If a `Chart.yaml` or `kustomization.yaml` is not found then any `.yaml` or `.yml` file will be assumed to be a Kubernetes resource and will be deployed. |
| **overlays/`{name}`** | / relative to `path` | When deploying using raw YAML (not Kustomize or Helm) `overlays` is a special directory for customizations. |
### Alternative scan, explicitly defined by the user
In addition to the previously described method, Fleet also supports a more direct, user-driven approach for defining Bundles.
In this mode, Fleet will load all resources found within the specified base directory. It will only attempt to locate a `fleet.yaml` file at the root of that directory if an options file is not explicitly provided.
Unlike the traditional scanning method, this one is not recursive and does not attempt to find Bundle definitions other than those explicitly specified by the user.
#### Example File Structure
```
driven
|___helm
| |__ fleet.yaml
|
|___simple
| |__ configmap.yaml
| |__ service.yaml
|
|___kustomize
|__ base
| |__ kustomization.yaml
| |__ secret.yaml
|
|__ overlays
| |__ dev
| | |__ kustomization.yaml
| | |__ secret.yaml
| |__ prod
| | |__ kustomization.yaml
| | |__ secret.yaml
| |__ test
| |__ kustomization.yaml
| |__ secret.yaml
|__ dev.yaml
|__ prod.yaml
|__ test.yaml
```
#### Corresponding GitRepo Definition
```
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: driven
namespace: fleet-local
spec:
repo: https://github.com/0xavi0/fleet-test-data
branch: driven-scan-example
bundles:
- base: driven/helm
- base: driven/simple
- base: driven/kustomize
options: dev.yaml
- base: driven/kustomize
options: test.yaml
```
In the example above, the user explicitly defines four Bundles to be generated.
* In the first case, the base directory is specified as `driven/helm`. As shown in the directory structure, this path contains a `fleet.yaml` file, which will be used to configure the Bundle.
* In the second case, the base directory is `driven/simple`, which contains only Kubernetes resource manifests (`configmap.yaml` and `service.yaml`). Since no `fleet.yaml` or options file is specified, Fleet will generate a Bundle using the default behavior—simply packaging all resources found within the directory.
* The third and fourth cases both reference the same base directory: `driven/kustomize`. However, each specifies a different options file (`dev.yaml` and `test.yaml`, respectively). These options files define overlay-specific configuration for each environment (e.g., dev, test) by selecting the appropriate kustomize overlay subdirectories and applying them on top of the shared base.
Fleet will process these as distinct Bundles, even though they originate from the same base path, because the provided options files point to different configurations.
An example of the files used in the third and fourth Bundles would be the following: (These files follow the exact same format as `fleet.yaml`, but since we can now reference them by name, we can use one that best suits our needs)
```yaml
namespace: kustomize-dev
kustomize:
dir: "overlays/dev"
```
It is important to note that any path defined in these files must be relative to the base directory used when the Bundle was described.
For example, with the previously mentioned structure, we are defining the base directory as `driven/kustomize`. That is the directory we need to use as the root for the paths used in Kustomize files.
We could decide to place the `dev.yaml` file at the path `driven/kustomize/overlays/dev` (this is supported), and then define the Bundle as:
```yaml
bundles:
- base: driven/kustomize
options: overlays/dev/dev.yaml
```
However, the path defined within `dev.yaml` should still be relative to `driven/kustomize`.
This is because when Fleet reads the options files, it always uses the base directory as the root.
In other words, with the previous example... this would be incorrect:
```yaml
namespace: kustomize-dev
kustomize:
dir: "."
```
And the correct definition should still be:
```yaml
namespace: kustomize-dev
kustomize:
dir: "overlays/dev"
```
With this new way of defining Bundles, Fleet becomes much more direct and also simplifies the adoption of deployments using kustomize.
In the example, we can see a complete kustomize use case where for each Bundle, we can specify which version we want.
With the previous scanning option, Fleet cannot determine which YAML we want to use to configure the Bundle, so it attempts to find it on its own (Which, at times, does not provide enough flexibility.)
### Excluding files and directories from bundles
Fleet supports file and directory exclusion by means of `.fleetignore` files, in a similar fashion to how `.gitignore`
files behave in git repositories:
* Glob syntax is used to match files or directories, using Golang's
[`filepath.Match`](https://pkg.go.dev/path/filepath#Match)
* Empty lines are skipped, and can therefore be used to improve readability
* Characters like white spaces and `#` can be escaped with a backslash
* Trailing spaces are ignored, unless escaped
* Comments, ie lines starting with unescaped `#`, are skipped
* A given line can match a file or a directory, even if no separator is provided: eg. `subdir/*` and `subdir` are both
valid `.fleetignore` lines, and `subdir` matches both files and directories called `subdir`
* A match may be found for a file or directory at any level below the directory where a `.fleetignore` lives, ie
`foo.yaml` will match `./foo.yaml` as well as `./path/to/foo.yaml`
* Multiple `.fleetignore` files are supported. For instance, in the following directory structure, only
`root/something.yaml`, `bar/something2.yaml` and `foo/something.yaml` will end up in a bundle:
```
root/
├── .fleetignore # contains `ignore-always.yaml'
├── something.yaml
├── bar
│   ├── .fleetignore # contains `something.yaml`
│   ├── ignore-always.yaml
│   ├── something2.yaml
│   └── something.yaml
└── foo
   ├── ignore-always.yaml
   └── something.yaml
```
This currently comes with a few limitations, the following not being supported:
* Double asterisks (`**`)
* Explicit inclusions with `!`
## `fleet.yaml`
The `fleet.yaml` is an optional file that can be included in the git repository to change the behavior of how
the resources are deployed and customized. The `fleet.yaml` is always at the root relative to the `path` of the `GitRepo`
and if a subdirectory is found with a `fleet.yaml` a new [bundle](./concepts.md) is defined that will then be
configured differently from the parent bundle.
:::caution
__Helm chart dependencies__:
Fleet automatically handles updating Helm chart dependencies, unless flag `disableDependencyUpdate` (`false` by
default) is set to `true`.
If automatic dependencies updates are disabled, it is up to the user to fulfill the dependency list for the Helm charts.
As such, you must manually run `helm dependencies update $chart` OR run `helm dependencies build $chart` prior to
install. See the [Fleet
docs](https://ranchermanager.docs.rancher.com/integrations-in-rancher/fleet/overview#helm-chart-dependencies) in Rancher for
more information.
:::
The available fields are documented in the [fleet.yaml reference](./ref-fleet-yaml.md)
For a private Helm repo, users can reference a secret from the git repo resource.
See [Using Private Helm Repositories](./gitrepo-add.md#using-private-helm-repositories) for more information.
## Using Helm Values
__How changes are applied to `values.yaml`__:
- Note that the most recently applied changes to the `values.yaml` will override any previously existing values.
- When changes are applied to the `values.yaml` from multiple sources at the same time, the values will update in the following order: `helm.values` -> `helm.valuesFiles` -> `helm.valuesFrom`. That means `valuesFrom` will take precedence over both, `valuesFiles` and `values`.
![](/img/FleetValuesStages.svg)
The targeting step can treat the values as a template and fill in information from the `clusters.fleet.cattle.io` resource. More information can be found in [Helm values templating](./ref-fleet-yaml#templating).
This can be turned off in `fleet.yaml`, by setting `disablePreProcess`, e.g. to avoid conflicts with other templating languages.
It is not necessary to reference a chart's own `values.yaml` via `valuesFiles:`. The `values.yaml` file contained in the
chart will always be used as a default when the agent installs the chart.
:::note Credentials in Values
If the chart generates certificates or passwords in its templates, these values must be overridden. Otherwise the chart could be continuously deployed as these values change.
Credentials loaded from the downstream cluster with `valuesFrom` are by default encrypted at rest, when [data encryption](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) is enabled in Kubernetes. Credentials contained in the default `values.yaml` file, or defined via `values:` or `valuesFiles` are not, as they are loaded from the repository when the bundle is created.
Hardened clusters should add the Fleet CRDs to the [list of resources encrypted at rest](gitrepo-add#storing-credentials-in-git), on the upstream cluster, when storing credentials in the bundles.
:::
### Using ValuesFrom
These examples showcase the style and format for using `valuesFrom`. ConfigMaps and Secrets should be created in *downstream clusters*.
Example [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/):
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-values
namespace: default
data:
values.yaml: |-
replication: true
replicas: 2
serviceType: NodePort
```
Example [Secret](https://kubernetes.io/docs/concepts/configuration/secret/):
```yaml
apiVersion: v1
kind: Secret
metadata:
name: secret-values
namespace: default
stringData:
values.yaml: |-
replication: true
replicas: 3
serviceType: NodePort
```
A secret like that, can be created from a YAML file `secretdata.yaml` by running the following kubectl command: `kubectl create secret generic secret-values --from-file=values.yaml=secretdata.yaml`
The resources can then be referenced from a `fleet.yaml`:
```yaml
helm:
chart: simple-chart
valuesFrom:
- secretKeyRef:
name: secret-values
namespace: default
key: values.yaml
- configMapKeyRef:
name: configmap-values
namespace: default
key: values.yaml
values:
replicas: "4"
```
## Per Cluster Customization
The `GitRepo` defines which clusters a git repository should be deployed to and the `fleet.yaml` in the repository
determines how the resources are customized per target.
All clusters and cluster groups in the same namespace as the `GitRepo` will be evaluated against all targets of that
`GitRepo`. The targets list is evaluated one by one and if there is a match the resource will be deployed to the cluster.
If no match is made against the target list on the `GitRepo` then the resources will not be deployed to that cluster.
Once a target cluster is matched the `fleet.yaml` from the git repository is then consulted for customizations. The
`targetCustomizations` in the `fleet.yaml` will be evaluated one by one and the first match will define how the
resource is to be configured. If no match is made the resources will be deployed with no additional customizations.
There are three approaches to matching clusters for both `GitRepo` `targets` and `fleet.yaml` `targetCustomizations`.
One can use cluster selectors, cluster group selectors, or an explicit cluster group name. All criteria is additive so
the final match is evaluated as "clusterSelector && clusterGroupSelector && clusterGroup". If any of the three have the
default value it is dropped from the criteria. The default value is either null or "". It is important to realize
that the value `{}` for a selector means "match everything."
```yaml
targetCustomizations:
- name: all
# Match everything
clusterSelector: {}
- name: none
# Selector ignored
clusterSelector: null
```
When matching a cluster by name, make sure to use the name of the
`clusters.fleet.cattle.io` resource. The Rancher UI also has a provisioning and
a management cluster resource. Since the management cluster resource is not
namespaced, its name is different and contains a random suffix.
```yaml
targetCustomizations:
- name: prod
clusterName: fleetname
```
See [Mapping to Downstream Clusters](gitrepo-targets#customization-per-cluster) for more information and a list of supported customizations.
## Raw YAML Resource Customization
When using Kustomize or Helm the `kustomization.yaml` or the `helm.values` will control how the resource are
customized per target cluster. If you are using raw YAML then the following simple mechanism is built-in and can
be used. The `overlays/` folder in the git repo is treated specially as folder containing folders that
can be selected to overlay on top per target cluster. The resource overlay content
uses a file name based approach. This is different from kustomize which uses a resource based approach. In kustomize
the resource Group, Kind, Version, Name, and Namespace identify resources and are then merged or patched. For Fleet
the overlay resources will override or patch content with a matching file name.
```shell
# Base files
deployment.yaml
svc.yaml
# Overlay files
# The following file will be added
overlays/custom/configmap.yaml
# The following file will replace svc.yaml
overlays/custom/svc.yaml
# The following file will patch deployment.yaml
overlays/custom/deployment_patch.yaml
```
A file named `foo` will replace a file called `foo` from the base resources or a previous overlay. In order to patch
the contents of a file the convention of adding `_patch.` (notice the trailing period) to the filename is used. The string `_patch.`
will be replaced with `.` from the file name and that will be used as the target. For example `deployment_patch.yaml`
will target `deployment.yaml`. The patch will be applied using JSON Merge, Strategic Merge Patch, or JSON Patch.
Which strategy is used is based on the file content. Even though JSON strategies are used, the files can be written
using YAML syntax.
## Cluster and Bundle State
See [Cluster and Bundle state](./ref-status-fields.md).
## Nested GitRepo CRs
Nested `GitRepo CRs` (defining a `GitRepo` that points to a repository containing one or more `GitRepo` resources) is supported.
You can use this feature to take advantage of `GitOps` in your `GitRepo` resources or, for example, to split complex scenarios into more than one `GitRepo` resource.
When finding a `GitRepo` in a `Bundle` Fleet will simply deploy it as any other resource.
See [this example](https://github.com/rancher/fleet-examples/tree/master/single-cluster/multi-gitrepo).

View File

@ -0,0 +1,193 @@
# Mapping to Downstream Clusters
[Fleet in Rancher](https://ranchermanager.docs.rancher.com/integrations-in-rancher/fleet) allows users to manage clusters easily as if they were one cluster.
Users can deploy bundles, which can be comprised of deployment manifests or any other Kubernetes resource, across clusters using grouping configuration.
:::info
__Multi-cluster Only__:
This approach only applies if you are running Fleet in a multi-cluster style
If no targets are specified, i.e. when using a single-cluster, the bundles target the default cluster group.
:::
When deploying `GitRepos` to downstream clusters the clusters must be mapped to a target.
## Defining Targets
The deployment targets of `GitRepo` is done using the `spec.targets` field to
match clusters or cluster groups. The YAML specification is as below.
```yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: myrepo
namespace: clusters
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- simple
# Targets are evaluated in order and the first one to match is used. If
# no targets match then the evaluated cluster will not be deployed to.
targets:
# The name of target. This value is largely for display and logging.
# If not specified a default name of the format "target000" will be used
- name: prod
# A selector used to match clusters. The structure is the standard
# metav1.LabelSelector format. If clusterGroupSelector or clusterGroup is specified,
# clusterSelector will be used only to further refine the selection after
# clusterGroupSelector and clusterGroup is evaluated.
clusterSelector:
matchLabels:
env: prod
# A selector used to match cluster groups.
clusterGroupSelector:
matchLabels:
region: us-east
# A specific clusterGroup by name that will be selected
clusterGroup: group1
# A specific cluster by name that will be selected
clusterName: cluster1
```
## Target Matching
All clusters and cluster groups in the same namespace as the `GitRepo` will be evaluated against all targets.
If any of the targets match the cluster then the `GitRepo` will be deployed to the downstream cluster. If
no match is made, then the `GitRepo` will not be deployed to that cluster.
There are three approaches to matching clusters.
One can use cluster selectors, cluster group selectors, or an explicit cluster group name. All criteria is additive so
the final match is evaluated as "clusterSelector && clusterGroupSelector && clusterGroup". If any of the three have the
default value it is dropped from the criteria. The default value is either null or "". It is important to realize
that the value `{}` for a selector means "match everything."
```yaml
targets:
# Match everything
- clusterSelector: {}
# Selector ignored
- clusterSelector: null
```
You can also match clusters by name:
```yaml
targets:
- clusterName: fleetname
```
When using Fleet in Rancher, make sure to put the name of the `clusters.fleet.cattle.io` resource.
## Default Target
If no target is set for the `GitRepo` then the default targets value is applied. The default targets value is as below.
```yaml
targets:
- name: default
clusterGroup: default
```
This means if you wish to setup a default location non-configured GitRepos will go to, then just create a cluster group called default
and add clusters to it.
## Customization per Cluster
:::info
The `targets:` in the `GitRepo` resource select clusters to deploy on. The `targetCustomizations:` in `fleet.yaml` override Helm values only and do not change targeting.
:::
To demonstrate how to deploy Kubernetes manifests across different clusters with customization using Fleet, we will use [multi-cluster/helm/fleet.yaml](https://github.com/rancher/fleet-examples/blob/master/multi-cluster/helm/fleet.yaml).
**Situation:** User has three clusters with three different labels: `env=dev`, `env=test`, and `env=prod`. User wants to deploy a frontend application with a backend database across these clusters.
**Expected behavior:**
- After deploying to the `dev` cluster, database replication is not enabled.
- After deploying to the `test` cluster, database replication is enabled.
- After deploying to the `prod` cluster, database replication is enabled and Load balancer services are exposed.
**Advantage of Fleet:**
Instead of deploying the app on each cluster, Fleet allows you to deploy across all clusters following these steps:
1. Deploy gitRepo `https://github.com/rancher/fleet-examples.git` and specify the path `multi-cluster/helm`.
2. Under `multi-cluster/helm`, a Helm chart will deploy the frontend app service and backend database service.
3. The following rule will be defined in `fleet.yaml`:
```
targetCustomizations:
- name: dev
helm:
values:
replication: false
clusterSelector:
matchLabels:
env: dev
- name: test
helm:
values:
replicas: 3
clusterSelector:
matchLabels:
env: test
- name: prod
helm:
values:
serviceType: LoadBalancer
replicas: 3
clusterSelector:
matchLabels:
env: prod
```
**Result:**
Fleet will deploy the Helm chart with your customized `values.yaml` to the different clusters.
>**Note:** Configuration management is not limited to deployments but can be expanded to general configuration management. Fleet is able to apply configuration management through customization among any set of clusters automatically.
### Supported Customizations
* [DefaultNamespace](/ref-crds#bundledeploymentoptions)
* [ForceSyncGeneration](/ref-crds#bundledeploymentoptions)
* [KeepResources](/ref-crds#bundledeploymentoptions)
* [ServiceAccount](/ref-crds#bundledeploymentoptions)
* [TargetNamespace](/ref-crds#bundledeploymentoptions)
* [Helm.Atomic](/ref-crds#helmoptions)
* [Helm.Chart](/ref-crds#helmoptions)
* [Helm.DisablePreProcess](/ref-crds#helmoptions)
* [Helm.Force](/ref-crds#helmoptions)
* [Helm.ReleaseName](/ref-crds#helmoptions)
* [Helm.Repo](/ref-crds#helmoptions)
* [Helm.TakeOwnership](/ref-crds#helmoptions)
* [Helm.TimeoutSeconds](/ref-crds#helmoptions)
* [Helm.ValuesFrom](/ref-crds#helmoptions)
* [Helm.Values](/ref-crds#helmoptions)
* [Helm.Version](/ref-crds#helmoptions)
:::warning important information
Overriding the version of a Helm chart via target customizations will lead to bundles containing _all_ versions, ie the
default one and the custom one(s), of the chart, to accommodate all clusters. This in turn means that Fleet will
deploy larger bundles.
As Fleet stores bundles via etcd, this may cause issues on some clusters where resultant bundle sizes may exceed
etcd's configured maximum blob size. See [this issue](https://github.com/rancher/fleet/issues/1650) for more details.
:::
* [Helm.WaitForJobs](/ref-crds#helmoptions)
* [Kustomize.Dir](/ref-crds#kustomizeoptions)
* [YAML.Overlays](/ref-crds#yamloptions)
* [Diff.ComparePatches](/ref-crds#diffoptions)
## Additional Examples
Examples using raw Kubernetes YAML, Helm charts, Kustomize, and combinations
of the three are in the [Fleet Examples repo](https://github.com/rancher/fleet-examples/).

View File

@ -0,0 +1,120 @@
# Glossary
## Agent
In Fleet's context, an agent is a Kubernetes deployment responsible for deploying workloads to its cluster.
This entails monitoring a specific namespace on the upstream cluster, and deploying any bundle deployments, living in that namespace, to the downstream cluster where the agent lives.
## Bundle
A bundle is a Fleet-specific resource (also known as a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) in Kubernetes) representing a workload, or set of user resources, to be deployed. It is typically generated by Fleet from a path of a git repository.
## Chart
See [this definition](https://helm.sh/docs/topics/charts/) of a Helm chart.
## Cluster
A cluster refers to:
* a [Kubernetes cluster](https://kubernetes.io/docs/concepts/architecture/) managed by Fleet
* a `Cluster` [resource](https://github.com/rancher/fleet/blob/main/pkg/apis/fleet.cattle.io/v1alpha1/cluster_types.go#L59) in Fleet's API, which Fleet uses to manage that Kubernetes cluster
## Continuous Delivery/Deployment
Definitions and distinctions between Continuous _Delivery_ and Continuous _Deployment_ greatly vary, for instance
depending on:
* whether the deployment step is included in the process, and to which environment: production or other?
* what triggers a deployment: is it a manual or automated step?
This much is clear, though: Fleet's goal is to make it easier to automate deployments.
## Custom Resource
See [this official definition](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/).
In short, a custom resource is a resource defined for the purposes of an application (in our case Fleet), to extend the set of resources supported by the Kubernetes API (pods, deployments, services, etc).
## Custom Resource Definition
See [this explanation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions) from the Kubernetes docs.
## Deployment
A deployment may refer to:
* a [Kubernetes deployment](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/deployment-v1/),
whether part of a user workload or part of Fleet itself, such as an agent deployment, controller deployments.
* the action of deploying a user workload, which means Fleet reading configuration (`GitRepo`, `fleet.yaml`, etc) and,
as a result, creating resources on target clusters.
## Downstream Cluster
A downstream cluster is a Kubernetes cluster where user workloads will run, without any Fleet controllers living there. It is a target cluster for Fleet, where only a Fleet agent lives beside user workloads.
## fleet.yaml
A `fleet.yaml` file lives in a git repository and stores options for a bundle and bundle deployments to be generated from that bundle. More information is available [here](https://fleet.rancher.io/ref-fleet-yaml).
## GitOps
GitOps refers to git-triggered operations, where git is the source of truth and changes to a git repository lead to
changes being applied to the state of one or more clusters.
## GitRepo
A `GitRepo` is a Fleet-specific resource, to be used as an entry point to using Fleet.
Creating a `GitRepo` pointing to a set of paths in a git repository enables Fleet to monitor those paths and deploy resources stored or referenced there.
## Label
Refers to a [Kubernetes label](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
## Multi-Cluster
A multi-cluster setup involves more than one cluster: the upstream cluster, needed to manage deployment of workloads, and at least one downstream cluster.
## Namespace
Refers to a [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
## Reconcile
Reconciling is used in the context of states in Kubernetes clusters. Reconciling a resource means updating it so that its _actual_ state matches its _expected_ state, be it from configuration, eg. from a git repository, chart, etc.
When using GitOps, updates to a git repository may translate into new expected states for resources configured through that git repository. As a result, affected resources will be reconciled.
A resource's state may also depend on another resource, leading to additional reconciliation. For instance, a cluster group's status depends on statuses of individual clusters contained in that cluster group. Therefore, a change in a cluster's state will result in any cluster group(s) to which that cluster belongs to being reconciled as well.
## Registration
Cluster registration is the process of getting a Fleet agent, living in a downstream cluster, recognised by Fleet controllers in the upstream cluster.
Once registration is complete for a downstream cluster, Fleet is able to deploy workloads to that cluster.
## Repository
A repository may be:
* a git repository, storing code, configuration or any kind of files and keeping track of changes made to those files
through commits. Fleet can monitor a git repository for new commits pushed to a specific branch or revision, at one or
more paths, through [GitRepo](./gitrepo-add.md) resources.
* a Helm repository, hosting Helm charts and an index file referencing them. Fleet is able to install Helm charts and
apply user-defined configuration to them.
## Resources
This usually refers to Kubernetes resources, which may be:
* core resources defined by Kubernetes itself, such as config maps, deployments, pods, services, etc
* custom resources defined by individual applications, such as Fleet itself, which defines GitRepo, Bundle,
Bundledeployment and a few others.
## Target
Fleet uses this word in the context of determining where a workload will run. This represents a _destination_ cluster for a workload.
## Upstream Cluster
A Kubernetes cluster where Fleet controllers run. This is the cluster where `GitRepo`s, bundles and bundle deployments are created.
Also called _management_ cluster.
## Workload
A workload represents what users want to deploy through Fleet. It may be a set of Helm charts, Kubernetes manifests, kustomize, etc, stored or referenced in a git repository.
When a user creates a `GitRepo` resource pointing to that git repository, and subsequently when relevant changes are found in that repository, Fleet deploys workloads.

View File

@ -0,0 +1,164 @@
# HelmOps
HelmOps is a simplified way of creating bundles by directly pointing to a Helm repository or to an OCI registry, without
needing to set up a git repository.
## Summary
When a `GitRepo` resource is created, Fleet monitors a git repository, creating one or more bundles from paths specified
in the `GitRepo`, following a GitOps, or git-driven, approach to continuous deployment. This requires a git repository
to be available, possibly containing `fleet.yaml` or other configuration files.
HelmOps, on the other hand, relies on a Helm registry as its source of truth, just as GitOps uses a git repository.
Leveraging HelmOps is done by creating a `HelmOp` resource, with similar options to those available in a `GitRepo`
resource and/or in a `fleet.yaml` file for targeting bundles to clusters, configuring chart values, etc.
HelmOps is the concept. A `HelmOp` is a custom Kubernetes resource managed by Fleet.
The Fleet HelmOps controller will create lightweight bundles, pointing to referenced Helm charts, without downloading
them.
However, it will resolve chart versions to ensure that the same, and latest, version of a chart is deployed to all
targeted downstream clusters. This applies to the following cases:
* a wildcard or empty version is specified
* a [semantic versioning](https://semver.org/) constraint is specified, such as `0.1.x`, `< 2.0.0`. More information on
supported constraints [here](https://github.com/Masterminds/semver?tab=readme-ov-file#checking-version-constraints).
When constraints are invalid or no matching version can be found, Fleet will display a descriptive error message.
When using this feature, Helm charts are downloaded from downstream clusters, which must therefore have access to Helm
registries.
## Creating a HelmOp resource
A `HelmOp` resource can be created as follows to start deploying Helm charts directly:
```yaml
apiVersion: fleet.cattle.io/v1alpha1
kind: HelmOp
metadata:
name: my-awesome-helmop
namespace: "fleet-local"
spec:
helm:
releaseName: my-fantastic-chart
repo: https://foo.bar/baz
chart: fantastic-chart
version: ''
namespace: that-amazing-namespace
helmSecretName: my-top-secret-helm-access
insecureSkipTLSVerify: false
```
For private charts, this requires a Helm access secret (referenced by field `helmSecretName`) to be created in the same
namespace as the `HelmOp` resource.
The Fleet HelmOps controller will take care of copying that secret to targeted downstream clusters, enabling the Fleet
agent to access the registry.
## Supported use cases
With 3 fields available to reference a Helm chart, let's clarify a few rules.
As per the Helm install [documentation](https://helm.sh/docs/helm/helm_install/), there are 6 ways of expressing a chart
to install. 3 of them use either repository aliases or the local filesystem, which are not available in Fleet's HelmOps
context. This leaves us with 3 options:
### Absolute URL
Referencing a Helm chart by absolute URL is as simple as providing a URL to a `.tgz` file in the `chart` field. Helm
options would look like:
```yaml
helm:
chart: https://example.com/charts/my-chart-1.2.3.tgz
# can be omitted
repo: ''
version: ''
```
If a non-empty repo, or a non-empty version is specified in this case, an error will appear in the HelmOp status and no
bundle will be created, aborting deployment.
### Chart reference and repo URL
A Helm chart can also be referenced through its repository and chart name, with an optional version, which may be a
static version or a version constraint.
This is where polling can make sense, because referencing the chart using a repository allows Fleet to check the
repository's `index.yaml` for available versions matching the `version` field.
Example:
```yaml
helm:
repo: https://foo.bar/baz
chart: fantastic-chart
version: '1.2.3'
```
In this case, only the `version` field may be empty. If any of the `chart` or `repo` field is empty, Fleet will set an
error in the HelmOp status and no bundle will be created.
### OCI registry
Helm supports OCI registries, which can be referenced in Fleet using the `repo` field.
In this case, Helm options would be similar to this:
```yaml
helm:
repo: oci://foo.bar/baz
version: '1.2.3' # optional
```
When an OCI URL is provided in the `repo` field, a non-empty `chart` field will lead to an error in the HelmOps status,
and no bundle being created.
:::note
In this case, Fleet will be downloading OCI artifacts. This means that:
* the `version` field represents an OCI artifact's tag, which may be different to the actual version of the
chart stored in the OCI artifact.
* an OCI artifact may contain multiple Helm charts. This use case has only been validated with OCI artifacts containing
a single Helm chart.
:::
## Polling
Fleet can poll the referenced Helm registry, periodically checking if new versions are available.
Of course, this only makes sense if the `version` field contains a version constraint, which may resolve to multiple
versions.
### How to enable it
Polling involves a `pollingInterval` field, similar to what exists for GitOps. However, in the HelmOps case, the default
polling interval is 0 seconds, meaning that polling will be disabled.
The following conditions must be met on a HelmOp resource for Fleet to enable polling on it:
* the `pollingInterval` field is set to a non-zero duration (e.g. `10s`, `1m`, etc)
* the `version` field is set to a valid semantic versioning constraint (e.g. `2.x.x`, `< 1.0`), not a static version
(e.g. 1.2.3)
### What it does
When polling is enabled, Fleet does the following at the configured interval:
* checking the referenced Helm registry for the latest version matching the version constraint configured in the
`version` field
* if a new version is found, setting that version on the Bundle created from the HelmOp object, so that the new version
of the chart will be installed on all targeted clusters
* updating the status of the HelmOp resource:
* setting its `Polled` condition:
* with `true` if polling was successful
* with `false` with an error if a failure happened
* updating the `Last Polling Time` field to the starting time of the last polling attempt, even if it failed.
## Status updates
Creating a HelmOp resource leads to a bundle being created, if Helm options are valid and a chart version can be found.
The status of that bundle will evolve over time, as bundle deployments are created from it, for each target cluster, and
as these bundle deployments' statuses themselves evolve and are propagated back to the bundle.
Fleet propagates updates from the bundle status to the status of the HelmOp resource itself.
This includes:
* a display status with a summary, expected and ready cluster counts
* conditions providing more information about the state of the resource, whether it is valid and its deployments are
ready
* resource counts by status
See [status fields](./ref-status-fields.md) for more details on resource counts and conditions.

View File

@ -0,0 +1,122 @@
# Using Image Scan to Update Container Image References
Image scan in fleet allows you to scan your image repository, fetch the desired image and update your git repository,
without the need to manually update your manifests.
:::caution
This feature is considered as experimental feature.
:::
Go to `fleet.yaml` and add the following section.
```yaml
imageScans:
# specify the policy to retrieve images, can be semver or alphabetical order
- policy:
# if range is specified, it will take the latest image according to semver order in the range
# for more details on how to use semver, see https://github.com/Masterminds/semver
semver:
range: "*"
# can use ascending or descending order
alphabetical:
order: asc
# specify images to scan
image: "your.registry.com/repo/image"
# Specify the tag name, it has to be unique in the same bundle
tagName: test-scan
# specify secret to pull image if in private registry
secretRef:
name: dockerhub-secret
# Specify the scan interval
interval: 5m
```
:::info
You can create multiple image scans in fleet.yaml.
:::
:::note
Semver will ignore pre-release versions (for example, 0.0.1-10) unless a pre-release version is explicitly used in the range definition.
For example, the "*" range will ignore pre-releases while ">= 0.0.1-10" will take them into account.
:::
Go to your manifest files and update the field that you want to replace. For example:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-slave
spec:
selector:
matchLabels:
app: redis
role: slave
tier: backend
replicas: 2
template:
metadata:
labels:
app: redis
role: slave
tier: backend
spec:
containers:
- name: slave
image: <image>:<tag> # {"$imagescan": "test-scan"}
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
```
:::note
There are multiple form of tagName you can reference. For example
`{"$imagescan": "test-scan"}`: Use full image name(foo/bar:tag)
`{"$imagescan": "test-scan:name"}`: Only use image name without tag(foo/bar)
`{"$imagescan": "test-scan:tag"}`: Only use image tag
`{"$imagescan": "test-scan:digest"}`: Use full image name with digest(foo/bar:tag@sha256...)
:::
Create a GitRepo that includes your fleet.yaml
```yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: my-repo
namespace: fleet-local
spec:
# change this to be your own repo
repo: https://github.com/rancher/fleet-examples
# define how long it will sync all the images and decide to apply change
imageScanInterval: 5m
# user must properly provide a secret that have write access to git repository
clientSecretName: secret
# specify the commit pattern
imageScanCommit:
authorName: foo
authorEmail: foo@bar.com
messageTemplate: "update image"
```
Try pushing a new image tag, for example, `<image>:<new-tag>`. Wait for a while and there should be a new commit pushed into your git repository to change tag in deployment.yaml.
Once change is made into git repository, fleet will read through the change and deploy the change into your cluster.

View File

@ -0,0 +1,13 @@
# Overview
![](/img/fleet-architecture.svg)
### What is Fleet?
- **Cluster engine**: Fleet is a container management and deployment engine designed to offer users more control on the local cluster and constant monitoring through **GitOps**. Fleet focuses not only on the ability to scale, but it also gives users a high degree of control and visibility to monitor exactly what is installed on the cluster.
- **Deployment management**: Fleet can manage deployments from git of raw Kubernetes YAML, Helm charts, Kustomize, or any combination of the three. Regardless of the source, all resources are dynamically turned into Helm charts, and Helm is used as the engine to deploy all resources in the cluster. As a result, users can enjoy a high degree of control, consistency, and auditability of their clusters.
### Configuration Management
Fleet is fundamentally a set of Kubernetes [custom resource definitions (CRDs)](https://fleet.rancher.io/concepts) and controllers that manage GitOps for a single Kubernetes cluster or a large scale deployment of Kubernetes clusters. It is a distributed initialization system that makes it easy to customize applications and manage HA clusters from a single point.

View File

@ -0,0 +1,344 @@
import {versions} from '@site/src/fleetVersions';
import CodeBlock from '@theme/CodeBlock';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Installation Details
The installation is broken up into two different use cases: single and multi-cluster.
The single cluster install is for if you wish to use GitOps to manage a single cluster,
in which case you do not need a centralized manager cluster. In the multi-cluster use case
you will setup a centralized manager cluster to which you can register clusters.
If you are just learning Fleet the single cluster install is the recommended starting
point. After which you can move from single cluster to multi-cluster setup down the line.
![](/img/single-cluster.png)
Single-cluster is the default installation. The same cluster will run both the Fleet
manager and the Fleet agent. The cluster will communicate with Git server to
deploy resources to this local cluster. This is the simplest setup and very
useful for dev/test and small scale setups. This use case is supported as a valid
use case for production.
## Prerequisites
<Tabs>
<TabItem value="helm" label="Helm 3" default>
Fleet is distributed as a Helm chart. Helm 3 is a CLI, has no server side component, and is
fairly straight forward. To install the Helm 3 CLI follow the <a href="https://helm.sh/docs/intro/install">official install instructions</a>.
</TabItem>
<TabItem value="kubernetes" label="Kubernetes" default>
Fleet is a controller running on a Kubernetes cluster so an existing cluster is required. For the
single cluster use case you will install Fleet to the cluster which you intend to manage with GitOps.
Any Kubernetes community supported version of Kubernetes will work, in practice this means {versions.next.kubernetes} or greater.
</TabItem>
</Tabs>
## Default Install
Install the following two Helm charts.
<Tabs>
<TabItem value="install" label="Install" default>
:::caution Fleet in Rancher
Rancher has separate helm charts for Fleet and uses a different repository.
:::
First add Fleet's Helm repository.
<CodeBlock language="bash">
{`helm repo add fleet https://rancher.github.io/fleet-helm-charts/`}
</CodeBlock>
Second install the Fleet CustomResourcesDefintions.
<CodeBlock language="bash">
{`helm -n cattle-fleet-system install --create-namespace --wait fleet-crd \\
fleet/fleet-crd`}
</CodeBlock>
Third install the Fleet controllers.
<CodeBlock language="bash">
{`helm -n cattle-fleet-system install --create-namespace --wait fleet \\
fleet/fleet`}
</CodeBlock>
</TabItem>
<TabItem value="verify" label="Verify">
Fleet should be ready to use now for single cluster. You can check the status of the Fleet controller pods by
running the below commands.
```bash
kubectl -n cattle-fleet-system logs -l app=fleet-controller
kubectl -n cattle-fleet-system get pods -l app=fleet-controller
```
```
NAME READY STATUS RESTARTS AGE
fleet-controller-64f49d756b-n57wq 1/1 Running 0 3m21s
```
</TabItem>
</Tabs>
You can now [register some git repos](./gitrepo-add.md) in the `fleet-local` namespace to start deploying Kubernetes resources.
## Tweaking your Fleet install
### Controller and agent replicas
Starting with v0.13, Fleet charts expose new Helm values setting replica counts for each type of controller and the
agent:
* `controller.replicas` for the `fleet-controller` deployment reconciling bundles, bundle deployments, clusters and
cluster groups
* `gitjob.replicas` for the gitOps controller reconciling `GitRepo` resources
* `helmops.replicas` for the experimental HelmOps controller
* `agent.replicas` for the agent.
Each of them defaults to 1.
## Multi-controller install: sharding
### Deployment
From 0.10 onwards, Fleet supports static sharding.
Each shard is defined by its shard ID.
Optionally, a shard can have a [node
selector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector), instructing Fleet to
create all controller pods and jobs for that shard on nodes matching that selector.
The Fleet controller chart can be installed with the following arguments:
* `--set shards[$index].id=$shard_id`
* `--set shards[$index].nodeSelector.$key=$value`
This will result in:
* as many Fleet controller and gitjob deployments as specified unique shard IDs,
* plus the usual unsharded Fleet controller pod. That latter pod will be the only one containing agent management and
cleanup containers.
For instance:
```bash
$ helm -n cattle-fleet-system install --create-namespace --wait fleet fleet/fleet \
--set shards[0].id=foo \
--set shards[0].nodeSelector."kubernetes\.io/hostname"=k3d-upstream-server-0 \
--set shards[1].id=bar \
--set shards[1].nodeSelector."kubernetes\.io/hostname"=k3d-upstream-server-1 \
--set shards[2].id=baz \
--set shards[2].nodeSelector."kubernetes\.io/hostname"=k3d-upstream-server-2 \
$ kubectl -n cattle-fleet-system get pods -l app=fleet-controller \
-o=custom-columns='Name:.metadata.name,Shard-ID:.metadata.labels.fleet\.cattle\.io/shard-id,Node:spec.nodeName'
Name Shard-ID Node
fleet-controller-b4c469c85-rj2q8 k3d-upstream-server-2
fleet-controller-shard-bar-5f5999958f-nt4bm bar k3d-upstream-server-1
fleet-controller-shard-baz-75c8587898-2wkk9 baz k3d-upstream-server-2
fleet-controller-shard-foo-55478fb9d8-42q2f foo k3d-upstream-server-0
$ kubectl -n cattle-fleet-system get pods -l app=gitjob \
-o=custom-columns='Name:.metadata.name,Shard-ID:.metadata.labels.fleet\.cattle\.io/shard-id,Node:spec.nodeName'
Name Shard-ID Node
gitjob-8498c6d78b-mdhgh k3d-upstream-server-1
gitjob-shard-bar-8659ffc945-9vtlx bar k3d-upstream-server-1
gitjob-shard-baz-6d67f596dc-fsz9m baz k3d-upstream-server-2
gitjob-shard-foo-8697bb7f67-wzsfj foo k3d-upstream-server-0
```
### How it works
With sharding in place, each Fleet controller will process resources bearing its own shard ID. This also holds for the
unsharded controller, which has no set shard ID and will therefore process all unsharded resources.
To deploy a GitRepo for a specific shard, simply add label `fleet.cattle.io/shard-ref` with your desired shard ID as a
value.
Here is an example:
```bash
$ kubectl apply -n fleet-local -f - <<EOF
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: sharding-test
labels:
fleet.cattle.io/shard-ref: foo
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- single-cluster/helm
EOF
```
A GitRepo with a label ID for which a Fleet controller is deployed (eg. `foo` in the above example) will then be
processed by that controller.
On the other hand, a GitRepo with an unknown label ID (eg. `boo` in the above example) will _not_ be processed by any
Fleet controller, hence no resources other than the GitRepo itself will be created.
Removing or adding supported shard IDs currently requires redeploying Fleet with a new set of shard IDs.
## Configuration for Multi-Cluster
:::caution
Downstream clusters in Rancher are automatically registered in Fleet. Users can access Fleet under `Continuous Delivery` on Rancher.
The multi-cluster install described below is **only** covered in standalone Fleet, which is untested by Rancher QA.
:::
:::info
The setup is the same as for a single cluster.
After installing the Fleet manager, you will then need to register remote downstream clusters with the Fleet manager.
However, to allow for [manager-initiated registration](./cluster-registration.md#manager-initiated) of downstream clusters, a few extra settings are required. Without the API server URL and the CA, only [agent-initiated registration](./cluster-registration.md#agent-initiated) of downstream clusters is possible.
:::
### API Server URL and CA certificate
In order for your Fleet management installation to properly work it is important
the correct API server URL and CA certificates are configured properly. The Fleet agents
will communicate to the Kubernetes API server URL. This means the Kubernetes
API server must be accessible to the downstream clusters. You will also need
to obtain the CA certificate of the API server. The easiest way to obtain this information
is typically from your kubeconfig file (`$HOME/.kube/config`). The `server`,
`certificate-authority-data`, or `certificate-authority` fields will have these values.
```yaml title="$HOME/.kube/config"
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTi...
server: https://example.com:6443
```
#### Extract CA certificate
Please note that the `certificate-authority-data` field is base64 encoded and will need to be
decoded before you save it into a file. This can be done by saving the base64 encoded contents to
a file and then running
```shell
base64 -d encoded-file > ca.pem
```
Next, retrieve the CA certificate from your kubeconfig.
<Tabs>
<TabItem value="extractca" label="Extract First">
If you have `jq` and `base64` available then this one-liners will pull all CA certificates from your
`KUBECONFIG` and place then in a file named `ca.pem`.
```shell
kubectl config view -o json --raw | jq -r '.clusters[].cluster["certificate-authority-data"]' | base64 -d > ca.pem
```
</TabItem>
<TabItem value="extractcas" label="Multiple Entries">
Or, if you have a multi-cluster setup, you can use this command:
```shell
# replace CLUSTERNAME with the name of the cluster according to your KUBECONFIG
kubectl config view -o json --raw | jq -r '.clusters[] | select(.name=="CLUSTERNAME").cluster["certificate-authority-data"]' | base64 -d > ca.pem
```
</TabItem>
</Tabs>
#### Extract API Server
If you have a multi-cluster setup, you can use this command:
```shell
# replace CLUSTERNAME with the name of the cluster according to your KUBECONFIG
API_SERVER_URL=$(kubectl config view -o json --raw | jq -r '.clusters[] | select(.name=="CLUSTER").cluster["server"]')
# Leave empty if your API server is signed by a well known CA
API_SERVER_CA="ca.pem"
```
#### Validate
First validate the server URL is correct.
```shell
curl -fLk "$API_SERVER_URL/version"
```
The output of this command should be JSON with the version of the Kubernetes server or a `401 Unauthorized` error.
If you do not get either of these results than please ensure you have the correct URL. The API server port is typically
6443 for Kubernetes.
Next validate that the CA certificate is proper by running the below command. If your API server is signed by a
well known CA then omit the `--cacert "$API_SERVER_CA"` part of the command.
```shell
curl -fL --cacert "$API_SERVER_CA" "$API_SERVER_URL/version"
```
If you get a valid JSON response or an `401 Unauthorized` then it worked. The Unauthorized error is
only because the curl command is not setting proper credentials, but this validates that the TLS
connection work and the `ca.pem` is correct for this URL. If you get a `SSL certificate problem` then
the `ca.pem` is not correct. The contents of the `$API_SERVER_CA` file should look similar to the below:
```pem title="ca.pem"
-----BEGIN CERTIFICATE-----
MIIBVjCB/qADAgECAgEAMAoGCCqGSM49BAMCMCMxITAfBgNVBAMMGGszcy1zZXJ2
ZXItY2FAMTU5ODM5MDQ0NzAeFw0yMDA4MjUyMTIwNDdaFw0zMDA4MjMyMTIwNDda
MCMxITAfBgNVBAMMGGszcy1zZXJ2ZXItY2FAMTU5ODM5MDQ0NzBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABDXlQNkXnwUPdbSgGz5Rk6U9ldGFjF6y1YyF36cNGk4E
0lMgNcVVD9gKuUSXEJk8tzHz3ra/+yTwSL5xQeLHBl+jIzAhMA4GA1UdDwEB/wQE
AwICpDAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49BAMCA0cAMEQCIFMtZ5gGDoDs
ciRyve+T4xbRNVHES39tjjup/LuN4tAgAiAteeB3jgpTMpZyZcOOHl9gpZ8PgEcN
KDs/pb3fnMTtpA==
-----END CERTIFICATE-----
```
### Install for Multi-Cluster
In the following example it will be assumed the API server URL from the `KUBECONFIG` which is `https://example.com:6443`
and the CA certificate is in the file `ca.pem`. If your API server URL is signed by a well-known CA you can
omit the `apiServerCA` parameter below or just create an empty `ca.pem` file (ie `touch ca.pem`).
Setup the environment with your specific values, e.g.:
```shell
API_SERVER_URL="https://example.com:6443"
API_SERVER_CA="ca.pem"
```
Once you have validated the API server URL and API server CA parameters, install the following two
Helm charts.
<Tabs>
<TabItem value="install2" label="Install" default>
First add Fleet's Helm repository.
<CodeBlock language="bash">
{`helm repo add fleet https://rancher.github.io/fleet-helm-charts/`}
</CodeBlock>
Second install the Fleet CustomResourcesDefintions.
<CodeBlock language="bash">
{`helm -n cattle-fleet-system install --create-namespace --wait \\
fleet-crd fleet/fleet-crd`}
</CodeBlock>
Third install the Fleet controllers.
<CodeBlock language="bash">
{`helm -n cattle-fleet-system install --create-namespace --wait \\
--set apiServerURL="$API_SERVER_URL" \\
--set-file apiServerCA="$API_SERVER_CA" \\
fleet fleet/fleet`}
</CodeBlock>
</TabItem>
<TabItem value="verifiy2" label="Verify">
Fleet should be ready to use. You can check the status of the Fleet controller pods by running the below commands.
```bash
kubectl -n cattle-fleet-system logs -l app=fleet-controller
kubectl -n cattle-fleet-system get pods -l app=fleet-controller
```
```
NAME READY STATUS RESTARTS AGE
fleet-controller-64f49d756b-n57wq 1/1 Running 0 3m21s
```
</TabItem>
</Tabs>
At this point the Fleet manager should be ready. You can now [register clusters](./cluster-registration.md) and [git repos](./gitrepo-add.md#create-gitrepo-instance) with
the Fleet manager.

View File

@ -0,0 +1,214 @@
# Setup Multi User
Fleet uses Kubernetes RBAC where possible.
One addition on top of RBAC is the [`GitRepoRestriction`](./namespaces.md#restricting-gitrepos) resource, which can be used to control GitRepo resources in a namespace.
A multi-user fleet setup looks like this:
* tenants don't share namespaces, each tenant has one or more namespaces on the
upstream cluster, where they can create GitRepo resources
* tenants can't deploy cluster wide resources and are limited to a set of
namespaces on downstream clusters
* clusters are in a separate namespace
![Shared Clusters](/img/FleetSharedClusters.svg)
:::warning important information
The isolation of tenants is not complete and relies on Kubernetes RBAC to be
set up correctly. Without manual setup from an operator tenants can still
deploy cluster wide resources. Even with the available Fleet restrictions,
users are only restricted to namespaces, but namespaces don't provide much
isolation on their own. E.g. they can still consume as many resources as they
like.
However, the existing Fleet restrictions allow users to share clusters, and
deploy resources without conflicts.
:::
## Example Fleet Standalone
This would create a user 'fleetuser', who can only manage GitRepo resources in the 'project1' namespace.
```bash
kubectl create serviceaccount fleetuser
kubectl create namespace project1
kubectl create -n project1 role fleetuser --verb=get --verb=list --verb=create --verb=delete --resource=gitrepos.fleet.cattle.io
kubectl create -n project1 rolebinding fleetuser --serviceaccount=default:fleetuser --role=fleetuser
```
If we want to give access to multiple namespaces, we can use a single cluster role with two role bindings:
```bash
kubectl create clusterrole fleetuser --verb=get --verb=list --verb=create --verb=delete --resource=gitrepos.fleet.cattle.io
kubectl create -n project1 rolebinding fleetuser --serviceaccount=default:fleetuser --clusterrole=fleetuser
kubectl create -n project2 rolebinding fleetuser --serviceaccount=default:fleetuser --clusterrole=fleetuser
```
This makes sure, tenants can't interfere with GitRepo resources from other tenants, since they don't have access to their namespaces.
## Isolated Workspaces in Rancher
Users belonging to a specific group/organization within the company, may want to disable visibility of their clusters to users from other groups/organizations of the same company.
In order to achieve this isolation, Rancher provides `GlobalRoles` to allow permissions to the users on certain Kubernetes resources. `GlobalRoles` have the ability to limit access to specific namespaces present on the cluster, thanks to `NamespacedRules`.
When a new fleet workspace is created, a corresponding namespace with an identical name is automatically generated within the Rancher local cluster.
For a user to see and deploy fleet resources in a specific workspace, they need at least the following permissions:
- list/get the `fleetworkspace` cluster-wide resource in the local cluster
- Permissions to create fleet resources (such as `bundles`, `gitrepos`, ...) in the backing namespace for the workspace in the local cluster.
Let's grant permissions to deploy fleet resources in the `project1` and `project2` fleet workspaces:
- To create the `project1` and `project2` fleet workspaces, you can either do it in the [Rancher UI](https://ranchermanager.docs.rancher.com/integrations-in-rancher/fleet/overview#accessing-fleet-in-the-rancher-ui) or use the following YAML resources:
```yaml
apiVersion: management.cattle.io/v3
kind: FleetWorkspace
metadata:
name: project1
```
```yaml
apiVersion: management.cattle.io/v3
kind: FleetWorkspace
metadata:
name: project2
```
- Create a `GlobalRole` that grants permission to deploy fleet resources in the `project1` and `project2` fleet workspaces:
```yaml
apiVersion: management.cattle.io/v3
kind: GlobalRole
metadata:
name: fleet-projects1and2
namespacedRules:
project1:
- apiGroups:
- fleet.cattle.io
resources:
- gitrepos
- bundles
- clusterregistrationtokens
- gitreporestrictions
- clusters
- clustergroups
verbs:
- '*'
project2:
- apiGroups:
- fleet.cattle.io
resources:
- gitrepos
- bundles
- clusterregistrationtokens
- gitreporestrictions
- clusters
- clustergroups
verbs:
- '*'
rules:
- apiGroups:
- management.cattle.io
resourceNames:
- project1
- project2
resources:
- fleetworkspaces
verbs:
- '*'
```
Assign the `GlobalRole` to users or groups, more info can be found in the [Rancher docs](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions#configuring-global-permissions-for-individual-users)
The user now has access to the `Continuous Delivery` tab in Rancher and can deploy resources to both the `project1` and `project2` workspaces.
In order to have a well organized environment, each workspace should have its own related `GlobalRole` to help with the separation of duties and isolation required by the customer. This way, each user can be assigned to one or more `GlobalRoles`, depending on the needs.
## Allow Access to Clusters
This assumes all GitRepos created by 'fleetuser' have the `team: one` label. Different labels could be used, to select different cluster namespaces.
In each of the user's namespaces, as an admin create a [`BundleNamespaceMapping`](./namespaces.md#cross-namespace-deployments).
```yaml
kind: BundleNamespaceMapping
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: mapping
namespace: project1
# Bundles to match by label.
# The labels are defined in the fleet.yaml # labels field or from the
# GitRepo metadata.labels field
bundleSelector:
matchLabels:
team: one
# or target one repo
#fleet.cattle.io/repo-name: simpleapp
# Namespaces, containing clusters, to match by label
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: fleet-default
# the label is on the namespace
#workspace: prod
```
The [`target` section](./gitrepo-targets.md) in the GitRepo resource can be used to deploy only to a subset of the matched clusters.
## Restricting Access to Downstream Clusters
Admins can further restrict tenants by creating a `GitRepoRestriction` in each of their namespaces.
```yaml
kind: GitRepoRestriction
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: restriction
namespace: project1
allowedTargetNamespaces:
- project1simpleapp
```
This denies the creation of cluster wide resources, which may interfere with other tenants and limit the deployment to the 'project1simpleapp' namespace.
## An Example GitRepo Resource
A GitRepo resource created by a tenant, without admin access could look like this:
```yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: simpleapp
namespace: project1
labels:
team: one
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- bundle-diffs
targetNamespace: project1simpleapp
# do not match the upstream/local cluster, won't work
targets:
- name: dev
clusterSelector:
matchLabels:
env: dev
```
This includes the `team: one` label and the required `targetNamespace`.
Together with the previous `BundleNamespaceMapping` it would target all clusters with a `env: dev` label in the 'fleet-default' namespace.
:::note
`BundleNamespaceMappings` do not work with local clusters, so make sure not to target them.
:::

View File

@ -0,0 +1,180 @@
---
toc_max_heading_level: 4
---
# Namespaces
## Workload Namespaces
### Namespace Creation Behavior in Bundles
When deploying a Fleet bundle, the specified namespace will automatically be
created if it does not already exist.
### Configuring Workload Namespaces
When configuring workload namespaces, it is important to be aware that certain
options are designed to override the values of other options or namespace
definitions in workload resources. In some cases, setting namespaces using some
options may result in errors if the resources to be deployed contain
non-namespaced resources. To get a better understanding of how these options
interact, refer to the diagram below. For more details on a specific option,
please refer to the [GitRepo](./ref-gitrepo.md) or
[fleet.yaml](./ref-fleet-yaml.md) reference.
![Configuring Workload Namespaces](/img/FleetWorkloadNamespaces.png)
### Cross Namespace Deployments
It is possible to create a GitRepo that will deploy across namespaces. The
primary purpose of this is so that a central privileged team can manage common
configuration for many clusters that are managed by different teams. The way
this is accomplished is by creating a `BundleNamespaceMapping` resource in a
cluster.
If you are creating a `BundleNamespaceMapping` resource it is best to do it in a
namespace that only contains `GitRepos` and no `Clusters`. It seems to get
confusing if you have Clusters in the same repo as the cross namespace
`GitRepos` will still always be evaluated against the current namespace. So if
you have clusters in the same namespace you may wish to make them canary
clusters.
A `BundleNamespaceMapping` has only two fields. Which are as below
```yaml
kind: BundleNamespaceMapping
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: not-important
namespace: typically-unique
# Bundles to match by label. The labels are defined in the fleet.yaml
# labels field or from the GitRepo metadata.labels field
bundleSelector:
matchLabels:
foo: bar
# Namespaces to match by label
namespaceSelector:
matchLabels:
foo: bar
```
If the `BundleNamespaceMappings` `bundleSelector` field matches a `Bundles`
labels then that `Bundle` target criteria will be evaluated against all clusters
in all namespaces that match `namespaceSelector`. One can specify labels for the
created bundles from git by putting labels in the `fleet.yaml` file or on the
`metadata.labels` field on the `GitRepo`.
### Restricting GitRepos
A namespace can contain multiple `GitRepoRestriction` resources. All `GitRepos`
created in that namespace will be checked against the list of restrictions. If a
`GitRepo` violates one of the constraints its `BundleDeployment` will be in an
error state and won't be deployed.
This can also be used to set the defaults for GitRepo's `serviceAccount` and
`clientSecretName` fields.
```yaml
kind: GitRepoRestriction
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: restriction
namespace: typically-unique
allowedClientSecretNames: []
allowedRepoPatterns: []
allowedServiceAccounts: []
allowedTargetNamespaces: []
defaultClientSecretName: ""
defaultServiceAccount: ""
```
#### Allowed Target Namespaces
This can be used to limit a deployment to a set of namespaces on a downstream
cluster. If an allowedTargetNamespaces restriction is present, all `GitRepos`
must specify a `targetNamespace` and the specified namespace must be in the
allow list. This also prevents the creation of cluster wide resources.
## Fleet Namespaces
All types in the Fleet manager are namespaced. The namespaces of a custom
resource, e.g. GitRepo, does not influence the namespace of deployed resources.
Understanding how namespaces are used in the Fleet manager
is important to understand the security model and how one can use Fleet in a
multi-tenant fashion.
![Namespace](/img/FleetNamespaces.svg)
### GitRepos, Bundles, Clusters, ClusterGroups
All selectors for `GitRepo` targets will be evaluated against the `Clusters`
and `ClusterGroups` in the same namespaces. This means that if you give
`create` or `update` privileges to a `GitRepo` type in a namespace, that end
user can modify the selector to match any cluster in that namespace. This means
in practice if you want to have two teams self manage their own `GitRepo`
registrations but they should not be able to target each others clusters, they
should be in different namespaces.
The cluster registration namespace, called 'workspace' in Rancher, contains the `Cluster` and the
`ClusterRegistration` resources, as well as any `GitRepos` and `Bundles`.
Rancher will create two Fleet workspaces: **fleet-default** and
**fleet-local**.
- `fleet-default` will contain all the downstream clusters that are already
registered through Rancher.
- `fleet-local` will contain the local cluster by default. Access to
`fleet-local` is limited.
:::warning important information
Deleting the workspace, cluster registration namespace, will delete all the clusters within that namespace.
This will uninstall all deployed bundles, except for the fleet agent, from the deleted clusters.
:::
If you are using Fleet in a [single cluster](./concepts.md) style, the namespace
will always be **fleet-local**. Check
[here](https://fleet.rancher.io/namespaces#fleet-local) for more on the
`fleet-local` namespace.
For a [multi-cluster](./concepts.md) style, please ensure you use the correct
repo that will map to the right target clusters.
### Internal Namespaces
#### Cluster Registration Namespace: fleet-local
The **fleet-local** namespace is a special namespace used for the single cluster
use case or to bootstrap the configuration of the Fleet manager.
Access to the local cluster should be limited to operators.
When fleet is installed the `fleet-local` namespace is created along with one
`Cluster` called `local` and one `ClusterGroup` called `default`. If no targets
are specified on a `GitRepo`, it is by default targeted to the `ClusterGroup`
named `default`. This means that all `GitRepos` created in `fleet-local` will
automatically target the `local` `Cluster`. The `local` `Cluster` refers to the
cluster the Fleet manager is running on.
#### System Namespace: cattle-fleet-system
The Fleet controller and Fleet agent run in this namespace. All service accounts
referenced by `GitRepos` are expected to live in this namespace in the
downstream cluster.
#### System Registration Namespace: cattle-fleet-clusters-system
This namespace holds secrets for the cluster registration process. It should
contain no other resources in it, especially secrets.
#### Cluster Namespaces
For every cluster that is registered a namespace is created by the Fleet manager
for that cluster. These namespaces are named in the form
`cluster-${namespace}-${cluster}-${random}`. The purpose of this namespace is
that all `BundleDeployments` for that cluster are put into this namespace and
then the downstream cluster is given access to watch and update
`BundleDeployments` in that namespace only.

View File

@ -0,0 +1,102 @@
# Observability
## Status Fields
Fleet reports most information via status fields on its custom resources.
These fields are also used by the Rancher UI to display information about the state of the resources.
See [status fields reference](./ref-status-fields.md) for more information on status fields and conditions.
## K8S Events
Fleet will generate k8s events a user can subscribe to. This is the list of events:
* `Created `- a new git cloning job was created
* `GotNewCommit `- a git repository has a new commit
* `JobDeleted `- a successful git cloning job is removed
* `FailedValidatingSecret `- a git cloning job cannot be created, because a required secret is missing
* `FailedToApplyRestrictions `- the GitRepo resource violates the GitRepoRestriction resource's rules
* `FailedToCheckCommit `- cannot get latest commit from the git server
* `FailedToGetGitJob `- cannot retrieve information from the git cloning job
* `Failed `- polling is disabled, triggered via webhook, but cannot get latest commit from the git server
## Metrics
Fleet publishes prometheus metrics. They can be retrieved from these services:
* `monitoring-fleet-controller.cattle-fleet-system.svc.cluster.local:8080/metrics`
* `monitoring-gitjob.cattle-fleet-system.svc.cluster.local:8081/metrics`
The [collection of exported metrics](https://book.kubebuilder.io/reference/metrics-reference) includes all the information from controller-runtime, like the number of reconciled resources, the number of errors, and the time it took to reconcile.
When the Fleet is used by Rancher and the `rancher-monitoring` chart is
installed, Prometheus is automatically configured to scrape the Fleet metrics.
**_NOTE_** Depending on how many resources are handled by Fleet, metrics may
cause performance issues. If you have a lot of resources, you may want to
disable metrics. You can do this by setting `metrics.enabled` in the
`values.yaml` file to `false` when installing Fleet.
### Grafana
When using Grafana and Prometheus, e.g. from https://github.com/prometheus-community/helm-charts, some setup is needed to access Fleet metrics.
1. Create a `ServiceMonitor` resource to scrape Fleet metrics. Here is an
example:
```yaml
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
# Create this in the same namespace as your application
namespace: cattle-fleet-system
name: fleet-controller-monitor
labels:
# This label makes the ServiceMonitor discoverable by the Prometheus Operator
release: monitoring # <-- ADD THIS LABEL!
spec:
selector:
matchLabels:
# This label must exist on the service you want to scrape
app: fleet-controller # Assumed label, verify this
namespaceSelector:
matchNames:
# We are only looking for the service in its own namespace
- cattle-fleet-system
endpoints:
- port: metrics
path: /metrics
interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
# Create this in the same namespace as your application
namespace: cattle-fleet-system
name: fleet-gitjob-monitor
labels:
# This label makes the ServiceMonitor discoverable by the Prometheus Operator
release: monitoring # <-- ADD THIS LABEL!
spec:
selector:
matchLabels:
# This label must exist on the service you want to scrape
app: gitjob
namespaceSelector:
matchNames:
# We are only looking for the service in its own namespace
- cattle-fleet-system
endpoints:
- port: metrics
path: /metrics
interval: 30s
```
And create it in Fleet's namespace, e.g. `cattle-fleet-system`: `kubectl create -f servicemonitor.yaml -n cattle-fleet-system`
2. Build the Grafana dashboards and import them into Grafana. You can find the
dashboards in the [fleet-dashboard
repository](https://github.com/rancher/fleet-dashboards). Follow the README
to build them.

View File

@ -0,0 +1,120 @@
# OCI Storage
Fleet stores Kubernetes bundle resources in etcd by default. However, etcd has strict size limits and is not optimized for large workloads. If your bundle resources exceed the etcd size limits in the target cluster, consider using an OCI registry as the storage backend.
:::note
To reduce bundle size, compress and base64-encode bundle content before uploading to the OCI registry.
:::
Using an OCI registry helps you:
* Reduce etcd load by offloading large bundle content.
* Use a standardized storage backend for large manifests or Helm charts.
![A visual asset displaying the flow of Fleet with OCI Storage.](../static/img/fleet-ociStorage-flow.png)
:::note
Fleet checks for the integrity of OCI artifacts and Fleet tags OCI artifact as `latest`.
:::
## Prerequisites
* A running OCI registry.
* A Kubernetes secret with valid credentials.
* A Fleet installation (v2.12.0 or later) .
## How to enable OCI storage
To enable OCI storage, create a secret that includes the necessary information and access options for the OCI registry. There are two ways of defining secrets:
* **Global secret:** A secret exactly named `ocistorage` in the same namespace as your `GitRepo`s.
* This is the fallback secret. If no `GitRepo`-level secret is specified, Fleet uses this secret for all `GitRepo`s in the namespace.
* **GitRepo-level secret:** A custom secret for specific `GitRepo` resouces.
* This is a user-defined secret can have any name and must be referenced in the `GitRepo` resource.
* Set the `ociRegistrySecret` field in the `GitRepo` spec to the secrets name.
:::note
Fleet does not fall back to etcd if the secret is missing or invalid. Instead, it logs an error and skips the deployment.
:::
Create a Kubernetes Secret that contains the registry address and optional credentials:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: ocistorage
namespace: fleet-local
type: fleet.cattle.io/bundle-oci-storage/v1alpha1
data:
reference: <base64-encoded-registry-url> # Only the reference field is required. All other fields are optional.
username: <base64-encoded-user>
password: <base64-encoded-password>
insecureSkipTLS: <base64-encoded-true/false>
basicHTTP: <base64-encoded-true/false>
agentUsername: <base64-encoded-readonly-user>
agentPassword: <base64-encoded-password>
```
:::note
The secret must have the type: `fleet.cattle.io/bundle-oci-storage/v1alpha1`. Fleet requires this value and rejects any secret with a different type.
:::
Changing the secret does not trigger a redeployment. Fleet uses the new registry only after a Git update or a manual force update.
### Secret Field Reference
The fields you can configure are:
| Field | Description | Format | Notes |
| -- | ---- | -- | ------ |
| `reference` | URL of the OCI registry. | Base64-encoded string | Do not use `oci://` or similar prefixes. |
| `username` | Username with write access to the registry. | Base64-encoded string | If not specified, Fleet accesses the registry without authentication.|
| `password` | Password for the write-access user. | Base64-encoded string | If not specified, Fleet accesses the registry without authentication.|
| `agentUsername` | Read-only username for agents. | Base64-encoded string | Use read-only credentials for agents to enhance security. If you dont set these credentials, the agent uses username. |
| `agentPassword` | Read-only password for agents. | Base64-encoded string | Use read-only credentials for agents to enhance security. If you dont set these credentials, the agent uses user password. |
| `insecureSkipTLS` | Skips TLS certificate validation. | Base64-encoded `true/false` | Use only for development or testing. By default, `InsecureSkipTLS` is set to `false`. |
| `basicHTTP` | Enables HTTP instead of HTTPS. | Base64-encoded `true/false` | Not recommended. Allows insecure traffic. By default, `basicHTTP` is set to `false`. |
## Fleet Example
Consider the following `GitRepo` file:
```yaml
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: frontend-oci
namespace: fleet-local
spec:
repo: https://github.com/your-org/fleet-oci-example.git
branch: main
paths:
- ./frontend
ociRegistrySecret: ocistorage
```
You can either create and apply a YAML file that contains the registry address and optional credentials similar to the example above. Then run `kubectl apply -f secrets/oci-secret.yaml` before applying the `GitRepo`.
Or you can use `kubectl` command to create the secret using unencoded text. Kubernetes converts them to base64 encoded for storing the secret.
```bash
kubectl -n fleet-local create secret generic ocistorage \
--type=fleet.cattle.io/bundle-oci-storage/v1alpha1 \
--from-literal=username=fleet-ci \
--from-literal=password=fleetRocks \
--from-literal=reference=192.168.1.39:8082 \
--from-literal=insecureSkipTLS=true \
--from-literal=basicHTTP=false \
--from-literal=agentUsername=fleet-ci-readonly \
--from-literal=agentPassword=readonlypass
```
To validate your secret, you can run:
`kubectl get secret ocistorage -n fleet-local -o yaml`
To decrypt your secret, you can run:
`kubectl get secret ocistorage -n fleet-local -o json | jq '.data | map_values(@base64d)`
![A screenshot of OCI secrets enabled for Fleet](../static/img/ociStorage-secret-ss.png)

View File

@ -0,0 +1,7 @@
# Key Personas and Their Roles
| Persona | Responsibilities | Key Benefits |
| --- | ----- | ----- |
| Platform Engineers | • Set up and manage Fleet installation across clusters. <br/> • Define and enforce multi-cluster application and policy deployments. <br/> • Manage GitRepo and `fleet.yaml` configurations. <br/> • Perform cluster-wide updates and rollbacks. | • Centralized control across clusters. <br/> • Improved standardization and compliance. |
| Application Developers | • Structure application Helm charts or manifests in Git. <br/> • Define deployment targets and environments in `fleet.yaml`. <br/> • Monitor deployment status via UI or CLI. | • GitOps-based workflows. <br/> • Faster, automated, and consistent deployments across environments. |
| Security Administrators | • Define and enforce security policies as code. <br/> • Deploy security tools and configurations. <br/> • Audit and manage security posture across clusters. | • Reduced risk and centralized compliance management. <br/> • Automated, consistent security configuration. |

View File

@ -0,0 +1,91 @@
import {versions} from '@site/src/fleetVersions';
import CodeBlock from '@theme/CodeBlock';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Quick Start
![](/img/single-cluster.png)
Who needs documentation, lets just run this thing!
## Install
Fleet is distributed as a Helm chart. Helm 3 is a CLI, has no server side component, and its use is
fairly straightforward. To install the Helm 3 CLI follow the <a href="https://helm.sh/docs/intro/install">official install instructions</a>.
:::caution Fleet in Rancher
Rancher has separate helm charts for Fleet and uses a different repository.
:::
<Tabs>
<TabItem value="linux" label="Linux/Mac" default>
<CodeBlock language="bash">
{`brew install helm\n`}
{`helm repo add fleet https://rancher.github.io/fleet-helm-charts/`}
</CodeBlock>
</TabItem>
<TabItem value="windows" label="Windows" default>
<CodeBlock language="bash">
{`choco install kubernetes-helm\n`}
{`helm repo add fleet https://rancher.github.io/fleet-helm-charts/`}
</CodeBlock>
</TabItem>
</Tabs>
Install the Fleet Helm charts (there's two because we separate out CRDs for ultimate flexibility.)
<CodeBlock language="bash">
{`helm -n cattle-fleet-system install --create-namespace --wait fleet-crd \\
fleet/fleet-crd\n`}
{`helm -n cattle-fleet-system install --create-namespace --wait fleet \\
fleet/fleet`}
</CodeBlock>
## Add a Git Repo to Watch
Change `spec.repo` to your git repo of choice. Kubernetes manifest files that should
be deployed should be in `/manifests` in your repo.
```bash
cat > example.yaml << "EOF"
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: sample
# This namespace is special and auto-wired to deploy to the local cluster
namespace: fleet-local
spec:
# Everything from this repo will be run in this cluster. You trust me right?
repo: "https://github.com/rancher/fleet-examples"
paths:
- simple
EOF
kubectl apply -f example.yaml
```
## Get Status
Get status of what fleet is doing
```shell
kubectl -n fleet-local get fleet
```
You should see something like this get created in your cluster.
```
kubectl get deploy frontend
```
```
NAME READY UP-TO-DATE AVAILABLE AGE
frontend 3/3 3 3 116m
```
## Next steps
Do you need to...
* Monitor private git or Helm repositories? Check [this](./gitrepo-add.md).
* Customise your deployments per target cluster? [This page](./gitrepo-targets.md) will help.

View File

@ -0,0 +1,56 @@
# Bundle Lifecycle
A bundle is an internal resource used for the orchestration of resources from git. When a GitRepo is scanned it will produce one or more bundles.
To demonstrate the life cycle of a Fleet bundle, we will use [multi-cluster/helm](https://github.com/rancher/fleet-examples/tree/master/multi-cluster/helm) as a case study.
1. User will create a [GitRepo](./gitrepo-add.md#create-gitrepo-instance) that points to the multi-cluster/helm repository.
2. The `gitjob-controller` will sync changes from the GitRepo and detect changes from the polling or [webhook event](./webhook.md). With every commit change, the `gitjob-controller` will create a job that clones the git repository, reads content from the repo such as `fleet.yaml` and other manifests, and creates the Fleet [bundle](./ref-status-fields.md#bundles).
>**Note:** The job pod with the image name `rancher/tekton-utils` will be under the same namespace as the GitRepo.
3. The `fleet-controller` then syncs changes from the bundle. According to the targets, the `fleet-controller` will create `BundleDeployment` resources, which are a combination of a bundle and a target cluster.
4. The `fleet-agent` will then pull the `BundleDeployment` from the Fleet controlplane. The agent deploys bundle manifests as a [Helm chart](https://helm.sh/docs/intro/install/) from the `BundleDeployment` into the downstream clusters.
5. The `fleet-agent` will continue to monitor the application bundle and report statuses back in the following order: bundledeployment > bundle > GitRepo > cluster.
This diagram shows the different rendering stages a bundle goes through until deployment.
![Bundle Stages](/img/FleetBundleStages.svg)
## Examining the Bundle Lifecycle With the CLI
Several fleet CLI commands help with debugging bundles.
### fleet apply
[Apply](./cli/fleet-cli/fleet_apply.md) renders a folder with Kubernetes resources, such as a Helm chart, manifests, or kustomize folders, into a Fleet bundle resource.
```
git clone https://github.com/rancher/fleet-test-data
cd fleet-test-data
fleet apply -n fleet-local -o bundle.yaml testbundle simple-chart/
```
More information on how to create bundles with `fleet apply` can be found in the [section on bundles](https://fleet.rancher.io/bundle-add).
### fleet target
[Target](./cli/fleet-cli/fleet_target.md) reads a bundle from a file and works with a live cluster to print out the `bundledeployment` & `content` resource, which fleetcontroller would create. It takes a namespace as an argument, so it can look in that namespace for e.g. cluster resources. It can also dump the data structure which is used during "targeting", so decisions taken regarding labels and cluster names can be checked.
### fleet deploy
[Deploy](./cli/fleet-cli/fleet_deploy.md) takes the output of `fleet target`, or a dumped bundledeployment/content resource and deploys it to a cluster, just like fleet-agent would. It supports a dry run mode, to print out the resources which would be created, instead of installing them with helm. Since the command doesn't create the input resources, a running fleet-agent would likely garbage collect the deployment.
The deploy command can be used to bring bundles to air-gapped clusters.
### Lifecycle CLI Example
```
git clone https://github.com/rancher/fleet-test-data
cd fleet-test-data
# for information about apply see https://fleet.rancher.io/bundle-add
fleet apply -n fleet-local -o bundle.yaml testbundle simple-chart/
fleet target --bundle-file bundle.yaml --dump-input-list > bd.yaml
fleet deploy --input-file bd.yaml --dry-run
```

View File

@ -0,0 +1,94 @@
# Bundle Resource
Bundles are automatically created by Fleet when a `GitRepo` is created.
The content of the resource corresponds to the [BundleSpec](./ref-crds#bundlespec).
For more information on how to use the Bundle resource [Create a Bundle Resource](./bundle-add.md).
```yaml
kind: Bundle
apiVersion: fleet.cattle.io/v1alpha1
metadata:
# Any name can be used here
name: my-bundle
# For single cluster use fleet-local, otherwise use the namespace of
# your choosing
namespace: fleet-local
spec:
# Namespace used for resources that do not specify a namespace.
# This field is not used to enforce or lock down the deployment to a specific namespace.
# defaultNamespace: test
# If present will assign all resource to this
# namespace and if any cluster scoped resource exists the deployment will fail.
# targetNamespace: app
# Kustomize options for the deployment, like the dir containing the kustomization.yaml file.
# kustomize: ...
# Helm options for the deployment, like the chart name, repo and values.
# helm: ...
# ServiceAccount which will be used to perform this deployment.
# serviceAccount: sa
# ForceSyncGeneration is used to force a redeployment.
# forceSyncGeneration: 0
# YAML options, if using raw YAML these are names that map to overlays/{name} that will be used to replace or patch a resource.
# yaml: ...
# Diff can be used to ignore the modified state of objects which are amended at runtime.
# A specific commit or tag can also be watched.
#
# diff: ...
# KeepResources can be used to keep the deployed resources when removing the bundle.
# keepResources: false
# If set to true, will stop any BundleDeployments from being updated. It will be marked as out of sync.
# paused: false
# Controls the rollout of bundles, by defining partitions, canaries and percentages for cluster availability.
# rolloutStrategy: ...
# Contain the actual resources from the git repo which will be deployed.
resources:
- content: |
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
name: nginx.yaml
# Target clusters to deploy to if running Fleet in a multi-cluster
# style. Refer to the "Mapping to Downstream Clusters" docs for
# more information.
#
# targets: ...
# This field is used by Fleet internally, and it should not be modified manually.
# Fleet will copy all targets into targetRestrictions when a Bundle is created for a GitRepo.
# targetRestrictions: ...
# Refers to the bundles which must be ready before this bundle can be deployed.
# dependsOn: ...
```

View File

@ -0,0 +1,106 @@
# Configuration
A reference list of, mostly internal, configuration options.
## Configure Fleet Install Options In Rancher
Rancher looks for Helm values in a ConfigMap called `rancher-config` in the `cattle-system` namespace.
Any Helm chart value for Fleet can be specified under the name of the chart. Note the value of the fleet key is a string.
A `rancher-config` could look like this, after adding some options:
```
kind: ConfigMap
apiVersion: v1
metadata:
name: rancher-config
namespace: cattle-system
data:
fleet: |
controller:
reconciler:
workers:
gitrepo: "50"
bundle: "50"
bundledeployment: "50"
priorityClassName: rancher-critical
```
The `rancher-config` resource is created by the Rancher chart and can be patched:
```
kubectl patch cm -n cattle-system rancher-config --type merge \
--patch '{"data":{"fleet": "controller:\n reconciler:\n workers:\n gitrepo: \"200\"\n bundle: \"200\"\n bundledeployment: \"200\"\n"}}'
```
## Helm Charts
The Helm charts accept, at least, the options as shown with their default in `values.yaml`:
* https://github.com/rancher/fleet/blob/main/charts/fleet/values.yaml
* https://github.com/rancher/fleet/blob/main/charts/fleet-crd/values.yaml
* https://github.com/rancher/fleet/blob/main/charts/fleet-agent/values.yaml
## Environment Variables
The controllers can be started with these environment variables:
* `CATTLE_DEV_MODE` - used to debug wrangler, not usable
* `FLEET_CLUSTER_ENQUEUE_DELAY` - tune how often non-ready clusters are checked
* `FLEET_CPU_PPROF_PERIOD` - used to turn on [performance profiling](https://github.com/rancher/fleet/blob/main/docs/performance.md)
## Configuration
In cluster configuration for the agent and fleet manager. Changing these can lead to full re-deployments.
The config [struct](https://github.com/rancher/fleet/blob/main/internal/config/config.go#L57) is used in both config maps:
* cattle-fleet-system/fleet-agent
* cattle-fleet-system/fleet-controller
## Labels
Labels used by fleet:
* `fleet.cattle.io/agent=true` - NodeSelector label for agent's deployment affinity setting
* `fleet.cattle.io/non-managed-agent` - managed agent bundle won't target Clusters with this label
* `fleet.cattle.io/repo-name` - used on Bundle to reference the git repo resource
* `fleet.cattle.io/bundle-namespace` - used on BundleDeployment to reference the Bundle resource
* `fleet.cattle.io/bundle-name` - used on BundleDeployment to reference the Bundle resource
* `fleet.cattle.io/managed=true` - cluster namespaces with this label will be cleaned up. Other resources will be cleaned up if it is in a label. Used in Rancher to identify fleet namespaces.
* `fleet.cattle.io/bootstrap-token` - unused
* `fleet.cattle.io/shard-id=<shard-id>` - The shard ID of a fleet controller pod.
* `fleet.cattle.io/shard-default=true` - true if this is the controller managing resources without a shard reference label.
* `fleet.cattle.io/shard-ref=<shard-id>` - references the Shard ID assigned by
Fleet to resources, inherited from a `GitRepo`, which determines which Fleet controller deployment will reconcile them.
* If this label is not provided or has an empty value, then the unsharded Fleet controller will process the resource.
* If this label has a value which does not match any shard ID for which a Fleet controller is deployed, then the
resource will not be processed.
## Annotations
Annotations used by fleet:
* `fleet.cattle.io/agent-namespace`
* `fleet.cattle.io/bundle-id`
* `fleet.cattle.io/cluster`, `fleet.cattle.io/cluster-namespace` - used on a cluster namespace to reference the cluster registration namespace and cluster name
* `fleet.cattle.io/cluster-group`
* `fleet.cattle.io/cluster-registration-namespace`
* `fleet.cattle.io/cluster-registration`
* `fleet.cattle.io/commit`
* `fleet.cattle.io/managed` - appears unused
* `fleet.cattle.io/service-account`
## Fleet agent configuration
Tolerations, affinity and resources can be customized for the Fleet agent. These fields can be provided when creating a
[Cluster](https://fleet.rancher.io/ref-crds#clusterspec), see [Registering Downstream Cluster](https://fleet.rancher.io/cluster-registration) for more info on how to create
Clusters. Default configuration will be used if these fields are not provided.
If you change the resources limits, make sure the limits allow the fleet-agent to work normally.
Keep in mind that if you downgrade Fleet to a previous version than v0.7.0 Fleet will fallback to the built-in defaults.
Agents will redeploy if they had custom affinity. If Fleet version number does not change, redeployment might not be immediate.

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,476 @@
# fleet.yaml
The `fleet.yaml` file adds options to a bundle. Any directory with a
`fleet.yaml` is automatically turned into bundle.
For more information on how to use the `fleet.yaml` to customize bundles see
[Git Repository Contents](./gitrepo-content.md).
The content of the fleet.yaml corresponds to the `FleetYAML` struct at
[pkg/apis/fleet.cattle.io/v1alpha1/fleetyaml.go](https://github.com/rancher/fleet/blob/main/pkg/apis/fleet.cattle.io/v1alpha1/fleetyaml.go),
which contains the [BundleSpec](./ref-crds#bundlespec).
### Reference
<details>
<summary>Full YAML reference</summary>
```yaml title="fleet.yaml"
# The default namespace to be applied to resources. This field is not used to
# enforce or lock down the deployment to a specific namespace, but instead
# provide the default value of the namespace field if one is not specified in
# the manifests.
#
# Default: default
defaultNamespace: default
# All resources will be assigned to this namespace and if any cluster scoped
# resource exists the deployment will fail.
#
# Default: ""
namespace: default
# namespaceLabels are labels that will be appended to the namespace created by
# Fleet.
namespaceLabels:
key: value
# namespaceAnnotations are annotations that will be appended to the namespace
# created by Fleet.
namespaceAnnotations:
key: value
# Optional map of labels, that are set at the bundle and can be used in a
# dependsOn.selector
labels:
key: value
kustomize:
# Use a custom folder for kustomize resources. This folder must contain a
# kustomization.yaml file.
dir: ./kustomize
helm:
# These options control how "fleet apply" downloads the chart
# (See `Helm Options` below for more details)
#
chart: ./chart
# A https URL to a Helm repo to download the chart from. It's typically easier
# to just use `chart` field and refer to a tgz file. If repo is used the
# value of `chart` will be used as the chart name to lookup in the Helm
# repository.
repo: https://charts.rancher.io
# The version of the chart or semver constraint of the chart to find. If a
# constraint is specified, it is evaluated each time git changes.
# (See `Helm Options` below for more details)
version: 0.1.0
# By default fleet downloads any dependency found in a helm chart. Use
# disableDependencyUpdate: true to disable this feature.
disableDependencyUpdate: false
### These options only work for helm-type bundles.
#
# Any values that should be placed in the `values.yaml` and passed to helm
# during install.
values:
any-custom: value
# All labels on Rancher clusters are available using
# global.fleet.clusterLabels.LABELNAME These can now be accessed directly as
# variables The variable's value will be an empty string if the referenced
# cluster label does not exist on the targeted cluster.
variableName: global.fleet.clusterLabels.LABELNAME
# See Templating notes below for more information on templating.
templatedLabel: "${ .ClusterLabels.LABELNAME }-foo"
valueFromEnv:
"${ .ClusterLabels.ENV }": ${ .ClusterValues.someValue | upper | quote }
# Path to any values files that need to be passed to helm during install.
valuesFiles:
- values1.yaml
- values2.yaml
# Allow to use values files from configmaps or secrets defined in the
# downstream clusters.
valuesFrom:
- configMapKeyRef:
name: configmap-values
# default to namespace of bundle
namespace: default
key: values.yaml
- secretKeyRef:
name: secret-values
namespace: default
key: values.yaml
### These options control how fleet-agent deploys the bundle, they also apply
### for kustomize- and manifest-style bundles.
#
# A custom release name to deploy the chart as. If not specified a release name
# will be generated by combining the invoking GitRepo.name + GitRepo.path.
releaseName: my-release
#
# Makes helm skip the check for its own annotations
takeOwnership: false
#
# Override immutable resources. This could be dangerous.
force: false
#
# Set the Helm --atomic flag when upgrading
atomic: false
#
# Disable go template pre-processing on the fleet values
disablePreProcess: false
#
# Disable DNS resolution in Helm's template functions
disableDNS: false
#
# Skip evaluation of the values.schema.json file
skipSchemaValidation: false
#
# If set and timeoutSeconds provided, will wait until all Jobs have been
# completed before marking the GitRepo as ready. It will wait for as long as
# timeoutSeconds.
waitForJobs: true
# A paused bundle will not update downstream clusters but instead mark the bundle
# as OutOfSync. One can then manually confirm that a bundle should be deployed to
# the downstream clusters.
#
# Default: false
paused: false
# If rolloutStrategy is not defined in the fleet.yaml file, Fleet uses default rollout values.
rolloutStrategy:
# A number or percentage of clusters that can be unavailable during an update
# of a bundle. This follows the same basic approach as a deployment rollout
# strategy. Once the number of clusters meets unavailable state update will be
# paused. Default value is 100% which doesn't take effect on update.
#
# default: 100%
maxUnavailable: 15%
# A number or percentage of cluster partitions that can be unavailable during
# an update of a bundle.
#
# default: 0
maxUnavailablePartitions: 20%
# A number or percentage of how to automatically partition clusters if not
# specific partitioning strategy is configured.
# The default value is defined in rolloutStrategy.maxUnavailable
autoPartitionSize: 10%
# A list of definitions of partitions. If any target clusters do not match
# the configuration they are added to partitions at the end following the
# autoPartitionSize.
partitions:
# A user friend name given to the partition used for Display (optional).
# default: ""
- name: canary
# A number or percentage of clusters that can be unavailable in this
# partition before this partition is treated as done.
# default: 10%
maxUnavailable: 10%
# Selector matching cluster labels to include in this partition
clusterSelector:
matchLabels:
env: prod
# A cluster group name to include in this partition
clusterGroup: agroup
# Selector matching cluster group labels to include in this partition
clusterGroupSelector:
clusterSelector:
matchLabels:
env: prod
# Target customization are used to determine how resources should be modified
# per target Targets are evaluated in order and the first one to match a cluster
# is used for that cluster.
targetCustomizations:
# The name of target. If not specified a default name of the format
# "target000" will be used. This value is mostly for display
- name: prod
# Custom namespace value overriding the value at the root.
namespace: newvalue
# Custom defaultNamespace value overriding the value at the root.
defaultNamespace: newdefaultvalue
# Custom kustomize options overriding the options at the root.
kustomize: {}
# Custom Helm options override the options at the root.
helm: {}
# If using raw YAML these are names that map to overlays/{name} that will be
# used to replace or patch a resource. If you wish to customize the file
# ./subdir/resource.yaml then a file
# ./overlays/myoverlay/subdir/resource.yaml will replace the base file. A
# file named ./overlays/myoverlay/subdir/resource_patch.yaml will patch the
# base file. A patch can in JSON Patch or JSON Merge format or a strategic
# merge patch for builtin Kubernetes types. Refer to "Raw YAML Resource
# Customization" below for more information.
yaml:
overlays:
- custom2
- custom3
# A selector used to match clusters. The structure is the standard
# metav1.LabelSelector format. If clusterGroupSelector or clusterGroup is
# specified, clusterSelector will be used only to further refine the
# selection after clusterGroupSelector and clusterGroup is evaluated.
clusterSelector:
matchLabels:
env: prod
# A selector used to match a specific cluster by name. When using Fleet in
# Rancher, make sure to put the name of the clusters.fleet.cattle.io
# resource.
clusterName: dev-cluster
# A selector used to match cluster groups.
clusterGroupSelector:
matchLabels:
region: us-east
# A specific clusterGroup by name that will be selected.
clusterGroup: group1
# Resources will not be deployed in the matched clusters if doNotDeploy is
# true.
doNotDeploy: false
# Drift correction removes any external change made to resources managed by
# Fleet. It performs a helm rollback, which uses a three-way merge strategy
# by default. It will try to update all resources by doing a PUT request if
# force is enabled. Three-way strategic merge might fail when updating an
# item inside of an array as it will try to add a new item instead of
# replacing the existing one. This can be fixed by using force. Keep in
# mind that resources might be recreated if force is enabled. Failed
# rollback will be removed from the helm history unless keepFailHistory is
# set to true.
correctDrift:
enabled: false
force: false # Warning: it might recreate resources if set to true
keepFailHistory: false
# dependsOn allows you to configure dependencies to other bundles. The current
# bundle will only be deployed, after all dependencies are deployed and in a
# Ready state.
dependsOn:
# Format:
# <GITREPO-NAME>-<BUNDLE_PATH> with all path separators replaced by "-"
#
# Example:
#
# GitRepo name "one", Bundle path "/multi-cluster/hello-world"
# results in "one-multi-cluster-hello-world".
#
# Note:
#
# Bundle names are limited to 53 characters long. If longer they will be
# shortened:
#
# opni-fleet-examples-fleets-opni-ui-plugin-operator-crd becomes
# opni-fleet-examples-fleets-opni-ui-plugin-opera-021f7
- name: one-multi-cluster-hello-world
# Select bundles to depend on based on their label.
- selector:
matchLabels:
app: weak-monkey
# Ignore fields when monitoring a Bundle. This can be used when Fleet thinks
# some conditions in Custom Resources makes the Bundle to be in an error state
# when it shouldn't.
ignore:
# Conditions to be ignored
conditions:
# In this example a condition will be ignored if it contains
# {"type": "Active", "status", "False"}
- type: Active
status: "False"
# Override targets defined in the GitRepo. The Bundle will not have any targets
# from the GitRepo if overrideTargets is provided.
overrideTargets:
- clusterSelector:
matchLabels:
env: dev
```
</details>
### Helm Options
#### Main options
##### chart
This specifies a custom location for the Helm chart. This can refer to any go-getter URL or OCI registry based Helm
chart URL, e.g. `oci://ghcr.io/fleetrepoci/guestbook`.
This allows one to download charts from many different locations. go-getter URLs support adding a digest to validate the
download. If the `repo` field is set, this field is the name of the chart to lookup.
It is possible to download the chart from a Git repository, e.g. by using
`git@github.com:rancher/fleet-examples//single-cluster/helm`. If a secret for the SSH key was defined in the GitRepo via
`helmSecretName`, it will be injected into the chart URL.
:::note chart reference depending on `fleet.yaml` location
If a `fleet.yaml` file is located outside of an embedded chart's directory, then it must explicitly reference the chart
using a `helm.chart` field. Otherwise, Fleet will not install the chart.
This also means that if no `helm.chart` field is specified in such a case, then Helm-specific fields like `valuesFiles`
or `valuesFrom` will not have any effect.
It is not necessary to specify a chart's own `values.yaml` via `valuesFiles:`. It will always be used as a default when the agent installs the chart. See [Using Helm Values](./gitrepo-content#using-helm-values).
See [Using Helm Values](./gitrepo-content#using-helm-values) for more details.
:::note
:::warning Limitation: downloading Helm charts from git with custom CA bundles
Git repositories can be downloaded via unauthenticated http, by using for example:
`git::http://github.com/rancher/fleet-examples/single-cluster/helm`.
However, this does not work with custom CA bundles at this point: if a CA bundle is configured in a secret referenced in
`helmSecretName`, will not be used, which will result in the git job displaying errors such as `SSL certificate problem:
unable to get local issuer certificate` when running `fleet apply` to generate a bundle.
See [fleet#3646](https://github.com/rancher/fleet/issues/3646) for more details.
:::
##### version
The version also determines which chart to download from OCI registries.
:::note `+` character support
OCI registries don't support the `+` character, which is supported by semver. When pushing a Helm chart with a tag
containing the `+` character, Helm automatically replaces `+` with '_' before uploading it.
You should use the version with the `+` in `fleet.yaml`, as the `_` character is not supported by semver and Fleet also
replaces `+` with `_` when accessing the OCI registry.
:::
#### How fleet-agent deploys the bundle
These options also apply to kustomize- and manifest-style bundles. They control
how the fleet-agent deploys the bundle. All bundles are converted into Helm
charts and deployed with the Helm SDK. These options are often similar to the
Helm CLI options for install and update.
- releaseName
- takeOwnership
- force
- atomic
- disablePreProcess
- disableDNS
- skipSchemaValidation
- waitForJobs
#### Helm Chart Download Options
These options are for Helm-style bundles, they specify how to download the
chart.
- chart
- repo
- version
The reference to the chart can be either:
- a local path in the cloned Git repository, specified by `chart`.
- a [go-getter URL](https://github.com/hashicorp/go-getter?tab=readme-ov-file#url-format),
specified by `chart`. This can be used to download a tarball
of the chart. go-getter also allows to download a chart from a Git repo.
- OCI chart URL, specified by `chart`. This can be used to download a chart
directly from a OCI server. It uses the Helm SDK to download the chart.
- a Helm repository, specified by `repo` and optionally `version`.
- an OCI Helm repository, specified by `repo` and optionally `version`.
#### Helm Chart Value Options
Options for the downloaded Helm chart.
- values
- valuesFiles
- valueFrom
### Values
Values are processed in different stages of the lifecycle: https://fleet.rancher.io/ref-bundle-stages
* fleet.yaml `values:` and `valuesFile:` are added to the bundle's values when it is created.
* helm values templating, e.g. with `${ }` happens when the bundle is targeted at a cluster, cluster labels filled in, etc.
* When the agent installs the chart, values from `valuesFrom` are read. Then Helm templating `{{ }}` is processed.
### Templating
It is possible to specify the keys and values as go template strings for
advanced templating needs. Most of the functions from the [sprig templating
library](https://masterminds.github.io/sprig/) are available.
This can be turned off in `fleet.yaml`, by setting `disablePreProcess`, e.g. to
avoid conflicts with other templating languages.
Note that if the functions output changes with every call, e.g. `uuidv4`, the
bundle will get redeployed.
You can [test values templating with the CLI](./ref-bundle-stages#fleet-target) against existing clusters.
The template context has the following keys:
* `.ClusterValues` are retrieved from target cluster's `spec.templateValues`
* `.ClusterLabels` and `.ClusterAnnotations` are the labels and annotations in
the cluster resource.
* `.ClusterName` as the fleet's cluster resource name.
* `.ClusterNamespace` as the namespace in which the cluster resource exists.
To access Labels or Annotations by their key name:
```
${ get .ClusterLabels "management.cattle.io/cluster-display-name" }
```
Note: The fleet.yaml must be valid yaml. Templating uses `${ }` as delims,
unlike Helm which uses `{{ }}`. These fleet.yaml template delimiters can be
escaped using backticks, eg.:
```
foo-bar-${`${PWD}`}
```
will result in the following text:
```
foo-bar-${PWD}
```
:::warning empty values
It is easier to use `global.fleet.clusterLabels.LABELNAME` instead of templating. When using templating, make sure to protect against null values.
Example:
```
${ if hasKey .ClusterLabels "LABELNAME" }${ .ClusterLabels.LABELNAME }${ else }missing${ end}
```
:::

View File

@ -0,0 +1,162 @@
# GitRepo Resource
The GitRepo resource describes git repositories, how to access them and where the bundles are located.
The content of the resource corresponds to the [GitRepoSpec](./ref-crds#gitrepospec).
For more information on how to use GitRepo resource, e.g. how to watch private repositories, see [Create a GitRepo Resource](./gitrepo-add.md).
```yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
# Any name can be used here
name: my-repo
# For single cluster use fleet-local, otherwise use the namespace of
# your choosing
namespace: fleet-local
# Labels are copied to bundles, but not to workloads.
labels:
created-by: fleet
spec:
# This can be a HTTPS or git URL. If you are using a git URL then
# clientSecretName will probably need to be set to supply a credential.
# repo is the only required parameter for a repo to be monitored.
#
repo: https://github.com/rancher/fleet-examples
# Enforce all resources go to this target namespace. If a cluster scoped
# resource is found the deployment will fail.
#
# targetNamespace: app1
# Any branch can be watched, this field is optional. If not specified the
# branch is assumed to be master
#
# branch: master
# A specific commit or tag can also be watched.
#
# revision: v0.3.0
# For a private git repository you must supply a clientSecretName. A default
# secret can be set at the namespace level using the GitRepoRestriction
# type. Secrets must be of the type "kubernetes.io/ssh-auth" or
# "kubernetes.io/basic-auth". The secret is assumed to be in the
# same namespace as the GitRepo
# If no clientSecretName is supplied, Fleet checks for a secret named
# "gitcredential".
#
# clientSecretName: my-ssh-key
# If fleet.yaml contains a private Helm repo that requires authentication,
# provide the credentials in a K8s secret and specify them here.
# Danger: the credentials will be sent to all repositories referenced from
# this gitrepo. See section below for more information.
#
# helmSecretName: my-helm-secret
# Helm credentials from helmSecretName will be used if the helm repository url matches this regular expression.
# Credentials will always be used if it is empty or not provided
#
# helmRepoURLRegex: https://charts.rancher.io/*
# Contains the auth secret for private Helm repository for each path.
# See [Create a GitRepo Resource](.gitrepo-add#use-different-helm-credentials-for-each-path)
#
# helmSecretNameForPaths: multi-helm-secret
# To add additional ca-bundle for self-signed certs, caBundle can be
# filled with base64 encoded pem data. For example:
# `cat /path/to/ca.pem | base64 -w 0`
#
# caBundle: my-ca-bundle
# Disable SSL verification for git repo
#
# insecureSkipTLSVerify: true
# A git repo can read multiple paths in a repo at once.
# The below field is expected to be an array of paths and
# supports path globbing (ex: some/*/path)
#
# Example:
# paths:
# - single-path
# - multiple-paths/*
paths:
- simple
# PollingInterval configures how often fleet checks the git repo. The default
# is 15 seconds.
# Setting this to zero does not disable polling. It results in a 15s
# interval, too.
# As checking a git repo incurs a CPU cost, raising this value can help
# lowering fleetcontroller's CPU usage if tens of git repos are used or more
#
# pollingInterval: 15s
# When disablePolling is set to true the git repo won't be checked periodically.
# It will rely on webhooks only.
# See [Using Webhooks Instead of Polling](https://fleet.rancher.io/webhook)
# disablePolling: false
# When using a webhook, a secret can be defined per GitRepo to validate the received payload.
# webhookSecret is the name of the previously created secret for this purpose.
# See [Using Webhooks Instead of Polling](https://fleet.rancher.io/webhook)
# webhookSecret: webhook-secret-name
# Paused causes changes in Git to not be propagated down to the clusters but
# instead mark resources as OutOfSync
#
# paused: false
# Increment this number to force a redeployment of contents from Git
#
# forceSyncGeneration: 0
# Drift correction removes any external change made to resources managed by Fleet. It performs a helm rollback, which uses
# a three-way merge strategy by default.
# It will try to update all resources by doing a PUT request if force is enabled. Three-way strategic merge might fail when updating
# an item inside of an array as it will try to add a new item instead of replacing the existing one. This can be fixed by using force.
# Keep in mind that resources might be recreated if force is enabled.
# Failed rollback will be removed from the helm history unless keepFailHistory is set to true.
#
# correctDrift:
# enabled: false
# force: false #Warning: it might recreate resources if set to true
# keepFailHistory: false
# The service account that will be used to perform this deployment.
# This is the name of the service account that exists in the
# downstream cluster in the cattle-fleet-system namespace. It is assumed
# this service account already exists so it should be created beforehand,
# for instance coming from another git repo registered with
# the Fleet manager.
# If no service account is configured, Fleet checks for a service account
# named "fleet-default".
#
# serviceAccount: moreSecureAccountThanClusterAdmin
# DeleteNamespace specifies if the namespace created
# must be deleted after deleting the GitRepo.
# deleteNamespace: false
# bundles specifies the user-driven bundle definitions
# With this way of defining Bundles, Fleet will simply load the specified resources
# along with the options file (if defined), or it will attempt to find a
# fleet.yaml file in the defined base.
# See [How Repos are scanned](https://fleet.rancher.io/gitrepo-content#how-repos-are-scanned)
# It is recommended to avoid using the following characters in the base and options paths: :,|?<>
# You can use any of those (or even more than one), but not all of them at once.
# bundles:
# base: basedirectory/to/bundle/resources
# options: path/to/fleet.yaml (optional)
# base: basedirectory/to/bundle2
# Target clusters to deploy to if running Fleet in a multi-cluster
# style. Refer to the "Mapping to Downstream Clusters" docs for
# more information.
# If empty, the "default" cluster group is used.
#
# targets: ...
```

View File

@ -0,0 +1,115 @@
# Cluster Registration Internals
## How does cluster registration work?
This text describes cluster registration with more technical details. The text ignores agent initiated registration, as its not commonly used.
[Agent initiated registration](./cluster-registration.md#agent-initiated) is ["`ClusterRegistrationToken` first"](./cluster-registration.md#create-cluster-registration-tokens), which means pre-creating a cluster is optional.
See "[Register Downstream Clusters](./cluster-registration.md)" to learn how to register clusters.
### Cluster first
`fleet-controller` starts up and may "bootstrap" the local cluster resource. In Rancher creating the local cluster resource is handled by the fleetcluster controller instead, but otherwise the process is identical.
The process is identical for the local cluster or any downstream cluster. It starts by creating a cluster resource, which refers to a kubeconfig secret.
### Creating the Bootstrap Secret for the Downstream Cluster
In this step a `ClusterRegistationToken` and an "import" service account are created based on a `Cluster` resource.
The Fleet controller creates a [`ClusterRegistrationToken`](https://fleet.rancher.io/architecture#security)
and waits for it to be complete. The `ClusterRegistationToken` triggers the creation of the "import" service account, which can create
`ClusterRegistrations` and read any secret in the system registration namespace (eg "cattle-fleet-clusters-system"). The `import.go` controller will
enqueue itself until the "import" service account exists, because that account is needed to create the `fleet-agent-bootstrap` secret.
### Creating the Fleet Agent Deployment
The Fleet controller will now create the Fleet agent deployment and the bootstrap secret on the downstream cluster.
The bootstrap secret contains the API server URL of the upstream cluster and is used to build a kubeconfig to access the upstream cluster. Both values are taken from the Fleet controller config configmap. That configmap is part of the helm chart.
### Fleet Agent Starts Registration, Upgrades to Request Account
The agent uses the "import" account to upgrade to a request account.
Immediately the Fleet agent checks for a `fleet-agent-bootstrap` secret. If the bootstrap secret, which contains the "import" kubeconfig, is present the agent starts registering.
Then agent creates the final `ClusterRegistration` resource in fleet-default on the management cluster, with a random number. The random number will be used for the registration secret's name.
The Fleet controller triggers and tries to grant the `ClusterRegistration` request to create agent's service account and create the 'c-\*' registration secret with the client's new kubeconfig. The registration secret name is `hash("clientID-clientRandom")`.
The new kubeconfig uses the "request" account. The "request" account can access the cluster status, `BundleDeployments` and `Contents`.
![ClusterRegistationToken to ClusterRegistration](/img/FleetRegistrationToken.svg)
### Fleet Agent is Registered, Watches for `BundleDeployments`
At this point the agent is fully registered and will persist the "request" account into a `fleet-agent` secret.
The API server URL and CA are copied from the bootstrap secret, which inherited these values from the Fleet controller's helm chart values.
The bootstrap secret is deleted. When the agent restarts, it will not re-register, since the bootstrap secret is missing.
The agent starts watching its "[Cluster Namespace](https://fleet.rancher.io/namespaces#cluster-namespaces)" for `BundleDeployments`. At this point the agent is ready to deploy workloads.
### Notes
* The registration starts with the "import" account and pivots to the "request" account.
* The fleet-default namespace has all the cluster registrations, the "import" account uses a separate namespace.
* Once the agent is registered, `fleet-controller` will trigger on a cluster or namespace change. The `manageagent` controller will then create a bundle to adopt the existing agent deployment. The agent will update itself to the bundle and since the "generation" environment variable changes, it will restart.
* If no bootstrap secret exists, the agent will not re-register.
## Diagram
### Registration Process and Controllers
Detailed analysis of the registration process for clusters. This shows the interaction of controllers, resources and service accounts during the registration of a new downstream cluster or the local cluster.
It is important to note that there are multiple ways to start this:
* Creating a bootstrap config. Fleet does this for the local agent.
* Creating a `Cluster` resource with a kubeconfig. Rancher does this for downstream clusters. See [manager-initiated registration](./cluster-registration.md#manager-initiated).
* Create a `ClusterRegistrationToken` resource, optionally create a `Cluster` resource for a pre-defined (`clientID`) cluster. See [agent-initiated registration](./cluster-registration.md#agent-initiated).
![Registration](/img/FleetRegistration.svg)
### Secrets during Agent Deployment
This diagram shows the resources created during registration and focuses on the k8s API server configuration.
The `import.go` controller triggers on Cluster creation/update events and deploys the agent.
**This image shows how the API server URL and CA propagates through the secrets during registration:**
The arrows in the diagram show how the API server values are copied from
the Helm values to the cluster registration secret on the upstream
cluster and finally downstream to the bootstrap secret of the agent.
There is one special case, if the agent is for the local/"bootstrap"
cluster, the server values also exist in the kubeconfig secret,
referenced by the Cluster resource. In this case the kubeconfig secret
contains the upstream server URL and CA, next to the downstream's
kubeconfig. If the settings are present in the kubeconfig secret, they
override the configured values.
![Registration Secrets](/img/FleetRegistrationSecrets.svg)
## Fleet Cluster Registration in Rancher
Rancher installs the fleet helm chart. The API server URL and CA are [derived from Rancher's settings](https://github.com/rancher/rancher/blob/main/pkg/controllers/dashboard/fleetcharts/controller.go#L113-L114).
Fleet will pass these values to a Fleet agent, so it can connect back to the Fleet controller.
### Import Cluster into Rancher
When the user runs `curl | kubectl apply`, the applied manifest includes the rancher agent deployment.
The deployment contains a secret `cattle-credentials-` which contains the API URL and a token.
The Rancher agent starts up and reports downstream's kubeconfig to upstream.
Rancher then creates the fleet Cluster resource, which references a [kubeconfig secret](https://github.com/rancher/rancher/blob/871b6d9137246bd93733f01184ea435f40c5d56c/pkg/provisioningv2/kubeconfig/manager.go#L69).
👉Fleet will use this kubeconfig to deploy the agent on the downstream cluster.

View File

@ -0,0 +1,74 @@
# List of Deployed Resources
After installing Fleet in Rancher these resources are created in the upstream cluster.
Also see [Namespaces](namespaces).
## From Helm, Intial Setup
| Type | Name | Namespace |
| ----- | ----------- | --------- |
| ClusterRole | fleet-controller | - |
| ClusterRole | gitjob | - |
| ClusterRoleBinding | fleet-controller | - |
| ClusterRoleBinding | gitjob-binding | - |
| ConfigMap | fleet-controller | cattle-fleet-system |
| Deployment | fleet-controller | cattle-fleet-system |
| Deployment | gitjob | cattle-fleet-system |
| Role | fleet-controller | cattle-fleet-system |
| Role | gitjob | cattle-fleet-system |
| RoleBinding | fleet-controller | cattle-fleet-system |
| RoleBinding | gitjob | cattle-fleet-system |
| Service | gitjob | cattle-fleet-system |
| ServiceAccount | fleet-controller | cattle-fleet-system |
| ServiceAccount | gitjob | cattle-fleet-system |
## Generated By Bootstrap
| Type | Name | Namespace |
| ----- | ----------- | --------- |
| clusters.fleet.cattle.io | local | fleet-local |
| clusters.provisioning.cattle.io | local | fleet-local |
| clusters.management.cattle.io | local | - |
| ClusterGroup | default | fleet-local |
| Bundle | fleet-agent-local | fleet-local |
## For Each Registered Cluster
| Type | Name | Namespace |
| ----- | ----------- | --------- |
| clusters.provisioning.cattle.io | | by default fleet-default |
| clusters.management.cattle.io | generated | - |
| clusters.fleet.cattle.io | | fleet-default |
| Secret, opaque | kubeconfig | fleet-default |
| ClusterRegistrationToken | | fleet-default |
| ServiceAccount | import | fleet-default |
| RoleBinding | request | fleet-default |
| RoleBinding | request | cluster-fleet-default |
| Role | request | cluster-fleet-default |
| Secret, service-account-token | import-token | fleet-default |
| Namespace | cluster-fleet-default | - |
| ClusterRegistration | fleet-default | |
| ServiceAccount | request | fleet-default |
| Secret, agent-credential | c-ID | fleet-default |
| Secret, service-account-token | request-token | fleet-default |
| Bundle | fleet-agent-name | fleet-name |
| BundleDeployment | fleet-agent-name | cluster-fleet-default |
## For Each Bundle, Per Cluster
| Type | Name | Namespace |
| ----- | ----------- | --------- |
| Bundle | name | fleet-default |
| Secret, bundle| name | fleet-default |
| BundleDeployment | name | cluster-fleet-default |
| Secret, bundle-deployment | name | cluster-fleet-default |
| Secret, opaque (content-access) | name | cluster-fleet-default |
| Content | - | - |
## For Each GitRepo
| Type | Name | Namespace |
| ----- | ----------- | --------- |
| ServiceAccount | name | fleet-default |
| Role | name | fleet-default |
| RoleBinding | name | fleet-default |
| Job | name | fleet-default |
| ConfigMap | name-config | fleet-default |
| Secret | name-cabundle | fleet-default |

View File

@ -0,0 +1,387 @@
# Status Fields
<!-- markdownlint-disable MD024 -->
## Display States
GitRepos, HelmOps, Clusters and Bundles have different states in each phase of
applying Bundles.
Since the states from the BundleDeployments are propagated to the Bundle,
GitRepo, Cluster and ClusterGroup, you will find them to be displayed there in
the same way. The difference is the perspective on the resources.
By looking at the GitRepo, the states of all resources related to the GitRepo
are displayed there. When looking at the Cluster, the states of all Bundles in
that Cluster are displayed, which may span over many GitRepos. When looking at
the Bundle, the states of all BundleDeployments in that Bundle are displayed.
### Ready Condition
The `Ready` condition is used to determine if the BundleDeployments are in a
`Ready` state. The `Ready` condition is set to `True` if all BundleDeployments
are in the `Ready` state. If at least one BundleDeployment is not in the `Ready`
state, the `Ready` condition is set to `False`.
All the states of the BundleDeployments are aggregated into the `message` field
of the `Ready` status condition, but to prevent the message from becoming too
long, only the first 10 states are shown. The `message` field contains the
number of BundleDeployments in each state, followed by the Cluster name where
the BundleDeployment is located. `Ready` statuses are excluded from the
`message` field. For example:
```yaml
status:
conditions:
- lastUpdateTime: "2025-06-25T14:59:35Z"
message: WaitApplied(1) [Cluster fleet-default/downstream4]
status: "False"
type: Ready
```
#### Ready Status Determination
Fleet uses the `kstatus` package of `sigs.k8s.io/cli-utils` module to determine
the Ready status of BundleDeployments based on the status of its resources. For
an in-depth explanation of how the Ready status is determined, see the
[README](https://pkg.go.dev/sigs.k8s.io/cli-utils@v0.37.2/pkg/kstatus#section-readme)
of the `kstatus` package.
#### Propagation of Ready Status
```mermaid
flowchart TB
subgraph Upstream Cluster
gr[GitRepo]
c[Cluster]
cg[ClusterGroup]
b[Bundle]
bd[BundleDeployment]
end
subgraph Downstream Clusters
a[Agent]
end
gr -->|triggers creation or update| b
b -->|triggers creation or update| bd
bd -->|triggers deployment| a
a -->|updates status| bd
bd -->|triggers status update| b
bd -->|triggers status update| c
b -->|triggers status update| gr
c -->|triggers status update| cg
linkStyle 0,1,2 stroke:#0000AA,stroke-width:2px;
linkStyle 3,4,5,6,7 stroke:#00AA00,stroke-width:2px;
```
### Display State
The `status.display` field provides a more digestible summary of the state.
States are ranked and the worst possible state is used as the `state` in the
`status.display` field.
#### State Ranking
This is the ranking in which states are displayed. If a Bundle has
BundleDeployments in different states, the worst state is used in the
`status.display.state` field. This is also the state that is propagated from the
Bundles to other Fleet resources (GitRepos, Clusters, ClusterGroups).
The states are ranked from the best to the worst:
- Ready
- NotReady
- Pending
- OutOfSync
- Modified
- WaitApplied
- ErrApplied
### Bundles
#### Bundle Statuses
- **Ready**
If `True`, bundles have been deployed and all resources are ready. If not, the
`message` field of the `Ready` condition contains an aggregation of the states
of the BundleDeployments.
- **NotReady**
BundleDeployments have been deployed and some resources are not ready.
For instance if the container images are being pulled, or the service has
started but not reported to be ready yet.
- **Pending**
Bundles are to be processed by the Fleet controller.
They might be waiting for the rollout to be resumed if it was paused (see
[Rollout Strategy](./rollout)). `Pending` is shown if there was no previously
applied BundleDeployment.
- **OutOfSync**
Bundles have been synced from the Fleet controller, but corresponding updated
BundleDeployments have not been created yet, so that the downstream agent
could not sync the change yet.
BundleDeployment creation might be waiting for the rollout to be resumed if it
was paused (see [Rollout Strategy](./rollout)). `OutOfSync` is shown if a
BundleDeployment was previously applied, but the Bundle has been modified due
to a change in the Git Repository.
- **Modified**
Bundles have been deployed and all resources are ready, but there are some
changes in the deployed resources that were not made from the Git Repository.
This happens when the deployed resources were modified externally.
- **WaitApplied**
Bundles have been synced from Fleet controller and downstream cluster, but are
waiting to be deployed.
If this state is persistently shown, the targeted Cluster may be unreachable.
- **ErrApplied**
Bundles have been synced from the Fleet controller and the downstream cluster,
but there were some errors when deploying the Bundle.
### Clusters
#### Cluster specific states
- **WaitCheckIn**
Waiting for agent to report registration information and cluster status back.
#### States from Bundles
- **Ready**
All bundles in this cluster have been deployed and all resources are ready.
- **NotReady**
There are bundles in this cluster that are in NotReady state.
- **Pending**
There are bundles in this cluster that are in Pending state.
- **OutOfSync**
There are bundles in this cluster that are in OutOfSync state.
- **Modified**
There are bundles in this cluster that are in Modified state.
- **WaitApplied**
There are bundles in this cluster that are in WaitApplied state.
- **ErrApplied**
There are bundles in this cluster that are in ErrApplied state.
### GitRepo
- **Ready**
`True` if the desired state is the current state. If this is `False`, the
message field of this Condition either contains:
- an error message from the GitJob controller, or
- an error from the Bundle (e.g. when templating failed), or
- an aggregated state of the bundles that are not in a `Ready` state.
- **GitPolling**
When the remote git repository is being polled for changes or initial cloning.
Contains an error if it fails otherwise this should be `True`. This value is
also `True` if polling is disabled, since the initial cloning is preceded by
polling for the latest commit revision.
- **Reconciling**
The controller is currently working on reconciling the latest changes.
- **Stalled**
The controller has encountered an error during the reconcile process or it has
made insufficient progress (timeout).
- **Accepted**
All GitRepo restrictions could be applied and external helm secrets exist.
### HelmOp Conditions
- **Ready**
`True` if all BundleDeployments successfully have been deployed. `False` if at
least one BundleDeployments are not ready. If this is `False`, the message
field of this Condition contains an aggregated state of the bundles that are
not in a `Ready` state.
- **Accepted**
`False` if the
- Helm options are invalid
- a chart version cannot be resolved from those options
- an issue occurred when scheduling a polling job
- creation of the Bundle failed
- **Polled**
`True` if polling is enabled and the last polling attempt was
successful, `False` otherwise. If polling is enabled, a `False` condition will
contain an error message explaining the reason for the polling failure.
### `status.display`
The `status.display` fields are shared between GitRepos and GitOps. Both
resources have a `status.display` field that contains a summary of the state of
the resource. The only difference is that `state` may have a different value,
since not all values for `state` are shared by both resources.
- `readyBundleDeployments` is a string in the form "%d/%d", that describes the
number of ready bundle deployments over the total number of bundle
deployments.
- `state` represents the state of the GitRepo, such as "GitUpdating" or the
highest BundleState according to [State Rank](#state-ranking). If the state
would be `Ready`, it is set to an empty value.
- `message` contains the relevant message from the deployment conditions.
- `error` is true if an error message is present.
## Resources List
The resources lists contain the resources deployed to target clusters,
categorized under `GitRepos` and `HelmOps`.
### GitRepos
The deployed resources are listed in `GitRepos` in `status.Resources`. This list
is derived from `bundleDeployments`.
### HelmOps
Similarly to what happens for `GitRepos`, the deployed resources are listed in
`HelmOps` in `status.Resources`. This list is derived from `bundleDeployments`.
## Resource Counts
This shows how resource counts are propagated from one resource to another:
![Status Propagation](/img/FleetStatusSource.png)
### GitRepos
The `status.ResourceCounts` list for GitRepos is derived from
`bundleDeployments`.
### HelmOps
The `status.ResourceCounts` list for HelmOps is derived from
`bundleDeployments`.
### Clusters
In Clusters, the `status.ResourceCounts` list is derived from GitRepos.
### ClusterGroups
In ClusterGroups, the `status.ResourceCounts` list is also derived from
GitRepos.
## Class Diagram
```mermaid
classDiagram
direction TB
class HelmOp {
HelmOpStatus Status
}
class GitRepo {
GitRepoStatus Status
}
class HelmOpStatus {
StatusBase
metav1.Time LastPollingTime
string Version
}
class GitRepoStatus {
StatusBase
int64 ObservedGeneration
int64 UpdateGeneration
string Commit
string WebhookCommit
string GitJobStatus
metav1.Time LastSyncedImageScanTime
metav1.Time LastPollingTime
}
class StatusBase {
int ReadyClusters
int DesiredReadyClusters
BundleSummary Summary
StatusDisplay Display
[]genericcondition.GenericCondition Conditions
[]Resource Resources
ResourceCounts ResourceCounts
map[string]*ResourceCounts PerClusterResourceCounts
}
class StatusDisplay {
string ReadyBundleDeployments
string State
string Message
bool Error
}
class BundleSummary {
int NotReady
int WaitApplied
int ErrApplied
int OutOfSync
int Modified
int Ready
int Pending
int DesiredReady
NonReadyResource[] NonReadyResources
}
class ResourceCounts {
int Ready
int DesiredReady
int WaitApplied
int Modified
int Orphaned
int Missing
int Unknown
int NotReady
}
HelmOp "1" --> "1" HelmOpStatus : Status
GitRepo "1" --> "1" GitRepoStatus : Status
HelmOpStatus "1" --|> "1" StatusBase : embeds StatusBase
GitRepoStatus "1" --|> "1" StatusBase : embeds StatusBase
StatusBase "1" --> "1" StatusDisplay : Display
StatusBase "1" --> "1" BundleSummary : Summary
StatusBase "1" --> "1" ResourceCounts : ResourceCounts
```

View File

@ -0,0 +1,5 @@
# Custom Resources During Deployment
This shows the resources, also the internal ones, involved in creating a deployment from a git repository.
![Resources](/img/FleetResources.svg)

View File

@ -0,0 +1,263 @@
# Rollout Strategy
Fleet uses a rollout strategy to control how apps are deployed across clusters. You can define the order and grouping of cluster deployments using partitions, enabling controlled rollouts and safer updates.
Fleet evaluates the `Ready` status of each `BundleDeployment` to determine when to proceed to the next partition. For more information, refer to [Status Fields](ref-status-fields.md).
During a rollout, the GitRepo status indicates deployment progress. This helps you understand when bundles become `Ready` before continuing:
* For initial deployments:
* One or more clusters may be in a `NotReady` state.
* Remaining clusters are marked as `Pending`, meaning deployment has not started.
* For rollouts:
* One or more cluster may be in a `NotReady` state.
* Remaining clusters are marked `OutOfSync` until the deployment continues.
The rollout configuration options are documented in the [`rolloutStrategy` field of the `fleet.yaml` reference](ref-fleet-yaml.md).
:::note
If `rolloutStrategy` is not specified in `fleet.yaml`, Fleet uses the default values.
:::
## How Does Partitioning Work?
Partitions are solely used for grouping and controlling the rollout of `BundleDeployments` across clusters. They do not affect deployment options in any way.
If targeted clusters are not part of the manual partitioning, they will not be included in the rollout. If a cluster is part of a partition, it will receive a `BundleDeployment` when the partition is processed.
Partitions are considered `NotReady` if they have clusters that exceed the allowed number of `NotReady` clusters. If a cluster is offline, the targeted cluster will not be considered `Ready` and will stay in the `NotReady` state until it comes back online and successfully deploys the `BundleDeployment`.
The threshold is determined by:
* **Manual partitions**: Use `maxUnavailable` value inside each partition to control readiness for that partition, otherwise, if unspecified, it uses `rolloutStrategy.maxUnavailable`.
* **Automatic partitions**: Use `rolloutStrategy.maxUnavailable` value to control when a partition is ready.
Fleet proceeds only if the number of `NotReady` partitions remains below `maxUnavailablePartitions`.
:::note
Fleet rolls out deployments in batches of up to 50 clusters per partition, regardless of partitions having more clusters assigned. After each batch, Fleet checks the `maxUnavailable` threshold before continuing. After all deployments for a partition have been created, `maxUnavailable` is also evaluated. For example:
* If a partition has 25 clusters and `maxUnavailable` is 5, Fleet deploys to all 25 before checking `maxUnavailable`.
* If a partition has 100 clusters, Fleet deploys to the first 50, checks `maxUnavailable`, and proceeds with the remaining 50 only if the threshold is not exceeded.
:::
The following diagram displays how Fleet handles rollout:
![A visual asset displaying flow of rollout in Fleet.](../static/img/flow-rollout-fleet.png)
Various limits that can be configured in Fleet:
| Field | Description | Default |
| -- | ---- | -- |
| maxUnavailable | Maximum number or percentage of clusters that can be `NotReady` before halting rollout. | 100% |
| maxUnavailablePartitions | Number or percentage of partitions that can be `NotReady` at once. | 0 |
| autoPartitionSize | Number or percentage of clusters per auto-created partition. | 25% |
| partitions | Define manual partitions by cluster labels or group. If set, autoPartitionSize is ignored. | |
Fleet supports automatic and manual partitioning. For more information about configuration options, refer to the [`rolloutStrategy` option in the fleet.yaml reference.](ref-fleet-yaml.md)
**Automatic Partitioning**: Fleet automatically creates partitions using `autoPartitionSize`.
For example, you have 200 clusters and set `autoPartitionSize` to 25%, Fleet creates four partitions of 50 clusters each. Rollout proceeds in 50-cluster batches, checking `maxUnavailable` before continuing.
**Manual Partitioning**: You define specific partitions using the `partitions` option. This provides control over cluster selection and rollout order.
:::note
If you specify partitions manually, the `autoPartitionSize` is ignored.
:::
For example, consider:
```yaml
rolloutStrategy:
partitions:
- name: demoRollout
maxUnavailable: 10%
clusterSelector:
matchLabels:
env: staging
- name: stable
maxUnavailable: 5%
clusterSelector:
matchLabels:
env: prod
```
Fleet then:
1. Selects clusters based on `clusterSelector`, `clusterGroup`, or `clusterGroupSelector`.
* Partitions can be specified by `clusterName`, `clusterSelector`, `clusterGroup`, and `clusterGroupSelector`.
2. Starts rollout to the first partition.
3. Waits until the partition is considered `Ready` (depending on the `maxUnavailable` threshold).
4. Proceeds to the next partition.
The following diagram illustrates how Fleet handles rollout across multiple partitions, including readiness checks and deployment flow:
![A visual asset displaying the flow of partition rollout](../static/img/deploy-targets-partition.png)
:::note
MaxNew is always 50. A bundle change can only stage 50 `BundleDeployments` at a time.
:::
Within each partition, Fleet rolls out up to 50 `BundleDeployments` at a time. The diagram below shows how Fleet determines whether to proceed or wait during this process:
![A visual asset displaying the flow of deploying targets in a partition](../static/img/partition-rollout-flow.png)
:::note
Fleet recommends labeling clusters so you can use those labels to assign clusters to specific partitions.
:::
:::note
Fleet processes partitions in the order they appear in the `fleet.yaml` file.
:::
### Single Partition
If you dont define `rolloutStrategy.partitions`, Fleet creates partitions automatically based on the number of targeted clusters:
* For fewer than 200 clusters, Fleet uses a single partition.
* For 200 or more clusters, Fleet uses the default `autoPartitionSize` value (25%) of the total.
For example, consider 200 clusters, Fleet uses the default `autoPartitionSize` of 25%. This means, Fleet creates 4 partitions (25% of 200 = 50 clusters per partition). Fleet processes up to 50 clusters at a time, which means it:
1. Rolls out to the first 50 clusters.
1. Evaluate readiness based on `maxUnavailable`.
1. If the condition is met, proceed to the next 50, and so on.
### Multiple Partitions
If you define multiple partitions, Fleet uses `maxUnavailablePartitions` to limit how many partitions can be `NotReady` at once. If the number of `NotReady` partitions exceeds `maxUnavailablePartitions`, Fleet pauses the rollout.
## Preventing image pull storms
During rollout, each downstream cluster pulls container images. If hundreds of clusters begin pulling images simultaneously, this can overwhelm the registry and behave like a DDoS attack.
To avoid this, Fleet can control how many clusters are updated at a time. You can use the following rollout configuration options to slow down and stage the rollout:
* `autoPartitionSize`
* `partitions`
* `maxUnavailable`
Fleet does not add artificial delays during rollout. Instead, it proceeds based on the `readiness` status of workloads in each cluster. Factors that affect readiness include image pull time, startup time, and readiness probes. Although using readiness probes is recommended, they are not strictly required to control rollout speed.
For example, you have 200 clusters, which are manually partitioned, each with 40 clusters and want to prevent an image pull storm:
* `maxUnavailablePartitions`: Set to 0.
* `maxUnavailable`: Set to 10%.
How rollout proceeds:
1. Fleet begins with the first partition (40 clusters).
1. It deploys up to 50 `BundleDeployments` at once. So it deploys to all 40 clusters in the partition in one batch.
1. Fleet checks the readiness of clusters in the partition.
1. If more than 4 clusters are not ready, then the partition is considered `NotReady` and the rollout is paused.
1. Once ≤4 clusters are `NotReady`, Fleet proceeds with the deployment.
1. When the entire partition is mostly ready (90%), Fleet moves to the next partition.
If you want or need to process fewer than 40 deployments at once, you can put fewer clusters into each partition.
## Use Cases and Behavior
If the number of clusters doesnt divide evenly, Fleet rounds down partition sizes. For example, 230 clusters with `autoPartitionSize: 25%` results in:
* Four partitions of 57 clusters
* One partition of 2 clusters
### Scenario: 50 Clusters (Single Partition)
```yaml
rolloutStrategy:
maxUnavailable: 10%
```
* Fleet creates one partition containing all 50 clusters, since no partitions are defined.
* No requirement to specify `maxUnavailablePartitions`, as only one partition is created.
* Although there is no specified manual partition and `maxUnavailable` is set to 10%, Fleet deploys to all 50 clusters at once (batch behavior overrides `maxUnavailable` initially).
* Evaluation occurs after all deployments are created.
The following diagram illustrates how Fleet handles 50 clusters in a single partition:
![A visual asset displaying 50 clusters](../static/img/deploy-50Clusters.png)
### Scenario: 100 Clusters (Single Partition)
```yaml
rolloutStrategy:
maxUnavailable: 10%
```
* Fleet creates one partition containing all 100 clusters, since no partitions are defined.
* No requirement to specify `maxUnavailablePartitions`, as you have only one.
* Although there is no specified manual partition and `maxUnavailable` is set to 10%, Fleet deploys to 50 clusters at once (batch behavior overrides `maxUnavailable` initially).
If 10 clusters (10% of 100 clusters) are unavailable, the deployment of the remaining 50 clusters is paused until less than 10 clusters are `NotReady`.
### Scenario: 200 Clusters (Multiple Partitions)
```yaml
rolloutStrategy:
maxUnavailablePartitions: 1
autoPartitionSize: 10%
```
* Fleet creates 10 partitions, each with 20 clusters.
* Deployment proceeds sequentially by partition.
* If two or more partitions become `NotReady`, rollout pauses.
* If one partition is `NotReady`, rollout can proceed to the next.
Fleet creates `BundleDeployments` for 20 clusters, waits for them to become `Ready`, then proceeds to the next. This effectively limits the amount of image pulls from downstream clusters to up to ~40 images at a time.
### Scenario: 200 Clusters (Strict Readiness, Manual partitions)
Manual partitioning allows you control over cluster grouping with `maxUnavailablePartitions: 0`.
```yaml
rolloutStrategy:
maxUnavailable: 0
maxUnavailablePartitions: 0
partitions:
- name: demoRollout
clusterSelector:
matchLabels:
stage: demoRollout
- name: stable
clusterSelector:
matchLabels:
stage: stable
```
* You define manual partitions using `clusterSelector` and labels like `stage: demoRollout` and `stage: stable`.
* Fleet creates `BundleDeployments` for clusters in the first partition (for example, `demoRollout`).
* The rollout proceeds strictly in order, Fleet only moves to the next partition when the current one is considered ready.
* With `maxUnavailable: 0` and `maxUnavailablePartitions: 0`, Fleet pauses the rollout if any partition is not considered ready.
The following diagram describes how Fleet handles whether to continue or pause rollout.
![A visual asset displaying the partitions about rollout in Fleet](../static/img/partition-fleet-rollout.png)
This ensures full readiness and staged rollout across all 200 clusters. Use this approach when you need precise rollout sequencing and full cluster readiness before advancing.
## Rollout Strategy Defaults
If partition-level rollout values are not defined, Fleet applies the global values from `rolloutStrategy` in `fleet.yaml`. Partition-specific settings override global values when explicitly set.
By default, Fleet sets:
* `maxUnavailable` to `100%`: All clusters in a partition can be `NotReady` and still be considered Ready.
* `maxUnavailablePartitions` to `0`: Prevents rollout only when one or more partitions are considered `NotReady`. However, this check is ineffective if all partitions appear Ready due to `maxUnavailable: 100%`.
For example, consider 200 clusters with default settings:
* Fleet creates 4 partitions of 50 clusters each (`autoPartitionSize: 25%`).
* Because `maxUnavailable` is `100%`, each partition is treated as `Ready` immediately.
* Fleet proceeds through all partitions regardless of actual readiness.
Fleet recommends you to control rollouts by setting:
* Lower `maxUnavailable`, e.g. 10%.
* Set `maxUnavailablePartitions` to 0 or higher, if desired.
This ensures:
* Partitions meet readiness before rollout continues.
* Fleet pauses rollout if too many partitions are not ready.

View File

@ -0,0 +1,317 @@
# Troubleshooting
This section contains commands and tips to troubleshoot Fleet.
## Where to look for root causes of issues
The first things to check when Fleet behaves unexpectedly would be:
* `fleet-controller` logs on the management cluster: has Fleet failed to reconcile any resource's (bundle, bundle
deployment) current state with its expected state?
* `gitjob` pod logs on the management cluster: has Fleet encountered any issue while creating jobs to generate new
bundles for new commits found in monitored git repositories?
* status of the `GitRepo` for which resources are not properly deployed:
* How many `Ready Bundle Deployments` does it list?
* How many bundle deployments are listed as expected? How many do you expect to see?
* Keep in mind that a `GitRepo` creates a bundle per path; each bundle leads to as many bundle deployments
as there are target clusters. A mismatch between `Desired Ready Clusters` and your own expectation could point to a
targeting issue.
* Which resources are listed in the `GitRepo`'s status?
* Which commit appears in the `GitRepo`'s status?
If the issue is specific to a target cluster, one might want to check Fleet agent logs on that cluster: has Fleet failed
to deploy a bundle deployment on that cluster?
The next section explains how to run all these checks.
## **How Do I...**
### Fetch the log from `fleet-controller`?
In the local management cluster where the `fleet-controller` is deployed, run the following command with your specific `fleet-controller` pod name filled in:
```
$ kubectl logs -l app=fleet-controller -n cattle-fleet-system
```
### Fetch the log from the `fleet-agent`?
Go to each downstream cluster and run the following command for the local cluster with your specific `fleet-agent` pod name filled in:
```
# Downstream cluster
$ kubectl logs -l app=fleet-agent -n cattle-fleet-system
# Local cluster
$ kubectl logs -l app=fleet-agent -n cattle-local-fleet-system
```
### Fetch detailed error logs from `GitRepos` and `Bundles`?
Normally, errors should appear in the Rancher UI. However, if there is not enough information displayed about the error there, you can research further by trying one or more of the following as needed:
- For more information about the bundle, click on `bundle`, and the YAML mode will be enabled.
- For more information about the GitRepo, click on `GitRepo`, then click on `View Yaml` in the upper right of the screen. After viewing the YAML, check `status.conditions`; a detailed error message should be displayed here.
- Check the `fleet-controller` for synching errors.
- Check the `fleet-agent` log in the downstream cluster if you encounter issues when deploying the bundle.
### Fetch detailed status from `GitRepos` and `Bundles`?
For debugging and bug reports the raw JSON of the resources status fields is most useful.
This can be accessed in the Rancher UI, or through `kubectl`:
```
kubectl get bundle -n fleet-local fleet-agent-local -o=jsonpath={.status}
kubectl get gitrepo -n fleet-default gitrepo-name -o=jsonpath={.status}
```
To download more resources, but not the spec fields:
```
kubectl get clusters.fleet.cattle.io -A -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.metadata.labels}{"\t"}{.status}{"\n"}{end}'
kubectl get bundles -A -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.targets}{"\t"}{.status}{"\n"}{end}'
kubectl get gitrepos -A -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.targets}{"\t"}{.status}{"\n"}{end}'
```
### Check a chart rendering error in `Kustomize`?
Check the [`fleet-controller` logs](./troubleshooting.md#fetch-the-log-from-fleet-controller) and the [`fleet-agent` logs](./troubleshooting.md#fetch-the-log-from-the-fleet-agent).
### Check errors about watching or checking out the `GitRepo`, or about the downloaded Helm repo in `fleet.yaml`?
Check the `gitjob-controller` logs using the following command with your specific `gitjob` pod name filled in:
```
$ kubectl logs -f $gitjob-pod-name -n cattle-fleet-system
```
Note that there are two containers inside the pod: the `step-git-source` container that clones the git repo, and the `fleet` container that applies bundles based on the git repo.
The pods will usually have images named `rancher/tekton-utils` with the `gitRepo` name as a prefix. Check the logs for these Kubernetes job pods in the local management cluster as follows, filling in your specific `gitRepoName` pod name and namespace:
```
$ kubectl logs -f $gitRepoName-pod-name -n namespace
```
### Check the status of the `fleet-controller`?
You can check the status of the `fleet-controller` pods by running the commands below:
```bash
kubectl -n cattle-fleet-system logs -l app=fleet-controller
kubectl -n cattle-fleet-system get pods -l app=fleet-controller
```
```bash
NAME READY STATUS RESTARTS AGE
fleet-controller-64f49d756b-n57wq 1/1 Running 0 3m21s
```
### Enable debug logging for `fleet-controller` and `fleet-agent`?
Available in Rancher v2.6.3 (Fleet v0.3.8), the ability to enable debug logging has been added.
- Go to the **Dashboard**, then click on the **local cluster** in the left navigation menu
- Select **Apps & Marketplace**, then **Installed Apps** from the dropdown
- From there, you will upgrade the Fleet chart with the value `debug=true`. You can also set `debugLevel=5` if desired.
#### Via Fleet Install Options
You can create a config map `rancher-config` in the `cattle-system` namespace with [Fleet Installation Options](./ref-configuration#configure-fleet-install-options-in-rancher).
For example, to enable debug logging for `fleet-controller` and `fleet-agent`, you can create a config map with the following content:
```yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: rancher-config
namespace: cattle-system
data:
fleet: |
debug: true
debugLevel: 1
propagateDebugSettingsToAgents: true
```
Modifying the config will re-install Fleet and re-deploy the agents.
### Record Resource Changes Over Time
Sometimes it is useful to record the changes of a resource over time. You can do this by watching the resource and saving the output to files.
```bash
for kind in gitrepos.fleet.cattle.io bundles.fleet.cattle.io bundledeployments.fleet.cattle.io; do
{
kubectl get -A --show-managed-fields -w --output-watch-events -o yaml $kind > $kind-watch.yaml &
pid=$!
sleep 60
kill $pid
} &
done ; wait
```
## **Additional Solutions for Other Fleet Issues**
### Naming conventions for CRDs
1. For CRD terms like `clusters` and `gitrepos`, you must reference the full CRD name. For example, the cluster CRD's complete name is `cluster.fleet.cattle.io`, and the gitrepo CRD's complete name is `gitrepo.fleet.cattle.io`.
1. `Bundles`, which are created from the `GitRepo`, follow the pattern `$gitrepoName-$path` in the same workspace/namespace where the `GitRepo` was created. Note that `$path` is the path directory in the git repository that contains the `bundle` (`fleet.yaml`).
1. `BundleDeployments`, which are created from the `bundle`, follow the pattern `$bundleName-$clusterName` in the namespace `clusters-$workspace-$cluster-$generateHash`. Note that `$clusterName` is the cluster to which the bundle will be deployed.
### HTTP secrets in Github
When testing Fleet with private git repositories, you will notice that HTTP secrets are no longer supported in Github. To work around this issue, follow these steps:
1. Create a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) in Github.
1. In Rancher, create an HTTP [secret](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-resources-setup/secrets) with your Github username.
1. Use your token as the secret.
### Fleet fails with bad response code: 403
If your GitJob returns the error below, the problem may be that Fleet cannot access the Helm repo you specified in your [`fleet.yaml`](./ref-fleet-yaml.md):
```
time="2021-11-04T09:21:24Z" level=fatal msg="bad response code: 403"
```
Perform the following steps to assess:
- Check that your repo is accessible from your dev machine, and that you can download the Helm chart successfully
- Check that your credentials for the git repo are valid
### Helm chart repo: certificate signed by unknown authority
If your GitJob returns the error below, you may have added the wrong certificate chain:
```
time="2021-11-11T05:55:08Z" level=fatal msg="Get \"https://helm.intra/virtual-helm/index.yaml\": x509: certificate signed by unknown authority"
```
Please verify your certificate with the following command:
```bash
context=playground-local
kubectl get secret -n fleet-default helm-repo -o jsonpath="{['data']['cacerts']}" --context $context | base64 -d | openssl x509 -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
7a:1e:df:79:5f:b0:e0:be:49:de:11:5e:d9:9c:a9:71
Signature Algorithm: sha512WithRSAEncryption
Issuer: C = CH, O = MY COMPANY, CN = NOP Root CA G3
...
```
### Fleet deployment stuck in modified state
When you deploy bundles to Fleet, some of the components are modified, and this causes the "modified" flag in the Fleet environment.
To ignore the modified flag for the differences between the Helm install generated by `fleet.yaml` and the resource in your cluster, add a `diff.comparePatches` to the `fleet.yaml` for your Deployment, as shown in this example:
```yaml
defaultNamespace: <namespace name>
helm:
releaseName: <release name>
repo: <repo name>
chart: <chart name>
diff:
comparePatches:
- apiVersion: apps/v1
kind: Deployment
operations:
- {"op":"remove", "path":"/spec/template/spec/hostNetwork"}
- {"op":"remove", "path":"/spec/template/spec/nodeSelector"}
jsonPointers: # jsonPointers allows to ignore diffs at certain json path
- "/spec/template/spec/priorityClassName"
- "/spec/template/spec/tolerations"
```
To determine which operations should be removed, observe the logs from `fleet-agent` on the target cluster. You should see entries similar to the following:
```text
level=error msg="bundle monitoring-monitoring: deployment.apps monitoring/monitoring-monitoring-kube-state-metrics modified {\"spec\":{\"template\":{\"spec\":{\"hostNetwork\":false}}}}"
```
Based on the above log, you can add the following entry to remove the operation:
```json
{"op":"remove", "path":"/spec/template/spec/hostNetwork"}
```
### `GitRepo` or `Bundle` stuck in modified state
**Modified** means that there is a mismatch between the actual state and the desired state, the source of truth, which lives in the git repository.
1. Check the [bundle diffs documentation](./bundle-diffs.md) for more information.
1. You can also force update the `gitrepo` to perform a manual resync. Select **GitRepo** on the left navigation bar, then select **Force Update**.
### Bundle has a Horizontal Pod Autoscaler (HPA) in modified state
For bundles with an HPA, the expected state is `Modified`, as the bundle contains fields that differ from the state of the Bundle at deployment - usually `ReplicaSet`.
You must define a patch in the `fleet.yaml` to ignore this field according to [`GitRepo` or `Bundle` stuck in modified state](#gitrepo-or-bundle-stuck-in-modified-state).
Here is an example of such a patch for the deployment `nginx` in namespace `default`:
```yaml
diff:
comparePatches:
- apiVersion: apps/v1
kind: Deployment
name: nginx
namespace: default
operations:
- {"op": "remove", "path": "/spec/replicas"}
```
### What if the cluster is unavailable, or is in a `WaitCheckIn` state?
You will need to re-import and restart the registration process: Select **Cluster** on the left navigation bar, then select **Force Update**
:::caution
__WaitCheckIn status for Rancher v2.5__:
The cluster will show in `WaitCheckIn` status because the `fleet-controller` is attempting to communicate with Fleet using the Rancher service IP. However, Fleet must communicate directly with Rancher via the Kubernetes service DNS using service discovery, not through the proxy. For more, see the [Rancher docs](https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy).
:::
### GitRepo complains with `gzip: invalid header`
When you see an error like the one below ...
```sh
Error opening a gzip reader for /tmp/getter154967024/archive: gzip: invalid header
```
... the content of the helm chart is incorrect. Manually download the chart to your local machine and check the content.
### Agent is no longer registered
You can force a redeployment of an agent for a given cluster by setting `redeployAgentGeneration`.
```sh
kubectl patch clusters.fleet.cattle.io -n fleet-local local --type=json -p '[{"op": "add", "path": "/spec/redeployAgentGeneration", "value": -1}]'
```
### Migrate the local cluster to the Fleet default cluster workspace?
Users can create new workspaces and move clusters across workspaces.
It's currently not possible to move the local cluster from `fleet-local` to another workspace.
### Bundle failed to deploy: "resource already exists" Error
If your bundle encounters the following error message during deployment:
```sh
not installed: rendered manifests contain a resource that already
exists. Unable to continue with install: ClusterRole "grafana-clusterrole"
in namespace "" exists and cannot be imported into the current release: invalid
ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace"
must equal "ns-2": current value is "ns-1"
```
This error occurs because a Helm resource with the same `releaseName` already exists in the cluster. To resolve this issue, you need to change the `releaseName` of the resource you want to create to avoid the conflict.

View File

@ -0,0 +1,465 @@
import CodeBlock from '@theme/CodeBlock';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Creating a Deployment
To deploy workloads onto downstream clusters, first create a Git repo, then create a GitRepo resource and apply it.
This tutorial uses the [fleet-examples](https://github.com/rancher/fleet-examples) repository.
:::note
For more details on how to structure the repository and configure the deployment of each bundle see [GitRepo Contents](./gitrepo-content.md).
For more details on the options that are available per Git repository see [Adding a GitRepo](./gitrepo-add.md).
:::
## Single-Cluster Examples
All examples will deploy content to clusters with no per-cluster customizations. This is a good starting point to understand the basics of structuring Git repos for Fleet.
<Tabs groupId="examples">
<TabItem value="helm" label="Helm" default>
An example using Helm. We are deploying the <a href="https://github.com/rancher/fleet-examples/tree/master/single-cluster/helm">helm example</a> to the local cluster.
The repository contains a helm chart and a `fleet.yaml` to configure the deployment:
```yaml title="fleet.yaml"
namespace: fleet-helm-example
# Custom helm options
helm:
# The release name to use. If empty a generated release name will be used
releaseName: guestbook
# The directory of the chart in the repo. Also any valid go-getter supported
# URL can be used there is specify where to download the chart from.
# If repo below is set this value if the chart name in the repo
chart: ""
# An https to a valid Helm repository to download the chart from
repo: ""
# Used if repo is set to look up the version of the chart
version: ""
# Force recreate resource that can not be updated
force: false
# How long for helm to wait for the release to be active. If the value
# is less that or equal to zero, we will not wait in Helm
timeoutSeconds: 0
# Custom values that will be passed as values.yaml to the installation
values:
replicas: 2
```
To create the deployment, we apply the custom resource to the upstream cluster. The `fleet-local` namespace contains the local cluster resource. The local fleet-agent will create the deployment in the `fleet-helm-example` namespace.
```bash
kubectl apply -n fleet-local -f - <<EOF
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- single-cluster/helm
EOF
```
</TabItem>
<TabItem value="helm-multi-chart" label="Helm Multi Chart" default>
An <a href="https://github.com/rancher/fleet-examples/blob/master/single-cluster/helm-multi-chart">example deploying multiple charts</a> from a single repo. This is similar to the previous example, but will deploy three helm charts from the sub folders, each configured by its own `fleet.yaml`.
```bash
kubectl apply -n fleet-local -f - <<EOF
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- single-cluster/helm-multi-chart
EOF
```
</TabItem>
<TabItem value="helm-kustomize" label="Helm & Kustomize" default>
An example using <a href="https://github.com/rancher/fleet-examples/blob/master/single-cluster/helm-kustomize">Kustomize to modify a third party Helm chart</a>.
It will deploy the Kubernetes sample guestbook application as packaged as a Helm chart downloaded from a third party source and will modify the helm chart using Kustomize. The app will be deployed into the fleet-helm-kustomize-example namespace.
```bash
kubectl apply -n fleet-local -f - <<EOF
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- single-cluster/helm-kustomize
EOF
```
</TabItem>
<TabItem value="kustomize" label="Kustomize" default>
An <a href="https://github.com/rancher/fleet-examples/blob/master/single-cluster/kustomize">example using Kustomize</a>.
Note that the `fleet.yaml` has a `kustomize:` key to specify the path to the required `kustomization.yaml`:
```yaml title="fleet.yaml"
kustomize:
# To use a kustomization.yaml different from the one in the root folder
dir: ""
```
```bash
kubectl apply -n fleet-local -f - <<EOF
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- single-cluster/kustomize
EOF
```
</TabItem>
<TabItem value="manifests" label="Manifests" default>
An <a href="https://github.com/rancher/fleet-examples/tree/master/single-cluster/manifests">example using raw Kubernetes YAML</a>.
```bash
kubectl apply -n fleet-local -f - <<EOF
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- single-cluster/manifests
EOF
```
</TabItem>
</Tabs>
## Multi-Cluster Examples
The examples below will deploy a multi git repo to multiple clusters at once and configure the app differently for each target.
<Tabs groupId="examples">
<TabItem value="helm" label="Helm" default>
An example using Helm. We are deploying the <a href="https://github.com/rancher/fleet-examples/tree/master/multi-cluster/helm">helm example</a> and customizing it per target cluster
The repository contains a helm chart and a `fleet.yaml` to configure the deployment. The `fleet.yaml` is used to configure different deployment options, depending on the cluster's labels:
```yaml title="fleet.yaml"
namespace: fleet-mc-helm-example
targetCustomizations:
- name: dev
helm:
values:
replication: false
clusterSelector:
matchLabels:
env: dev
- name: test
helm:
values:
replicas: 3
clusterSelector:
matchLabels:
env: test
- name: prod
helm:
values:
serviceType: LoadBalancer
replicas: 3
clusterSelector:
matchLabels:
env: prod
```
To create the deployment, we apply the custom resource to the upstream cluster. The `fleet-default` namespace, by default, contains the downstream cluster resources. The chart will be deployed to all clusters in the fleet-default namespace, which have a labeled cluster resources that matches any entry under `targets:`.
```yaml title="gitrepo.yaml"
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm
namespace: fleet-default
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- multi-cluster/helm
targets:
- name: dev
clusterSelector:
matchLabels:
env: dev
- name: test
clusterSelector:
matchLabels:
env: test
- name: prod
clusterSelector:
matchLabels:
env: prod
```
By applying the gitrepo resource to the upstream cluster, fleet will start to monitor the repository and create deployments:
<CodeBlock language="bash">
{`kubectl apply -n fleet-default -f gitrepo.yaml`}
</CodeBlock>
</TabItem>
<TabItem value="helm-external" label="Helm External" default>
An <a href="https://github.com/rancher/fleet-examples/blob/master/multi-cluster/helm-external">example using a Helm chart that is downloaded from a third party source and customizing it per target cluster</a>. The customization is similar to the previous example.
To create the deployment, we apply the custom resource to the upstream cluster. The `fleet-default` namespace, by default, contains the downstream cluster resources. The chart will be deployed to all clusters in the fleet-default namespace, which have a labeled cluster resources that matches any entry under `targets:`.
```yaml title="gitrepo.yaml"
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm-external
namespace: fleet-default
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- multi-cluster/helm-external
targets:
- name: dev
clusterSelector:
matchLabels:
env: dev
- name: test
clusterSelector:
matchLabels:
env: test
- name: prod
clusterSelector:
matchLabels:
env: prod
```
By applying the gitrepo resource to the upstream cluster, fleet will start to monitor the repository and create deployments:
<CodeBlock language="bash">
{`kubectl apply -n fleet-default -f gitrepo.yaml`}
</CodeBlock>
</TabItem>
<TabItem value="helm-kustomize" label="Helm & Kustomize" default>
An example using <a href="https://github.com/rancher/fleet-examples/blob/master/multi-cluster/helm-kustomize">kustomize to modify a third party Helm chart</a>.
It will deploy the Kubernetes sample guestbook application as packaged as a Helm chart downloaded from a third party source and will modify the helm chart using Kustomize. The app will be deployed into the fleet-helm-kustomize-example namespace.
The application will be customized as follows per environment:
* Dev clusters: Only the redis leader is deployed and not the followers.
* Test clusters: Scale the front deployment to 3
* Prod clusters: Scale the front deployment to 3 and set the service type to LoadBalancer
The `fleet.yaml` is used to control which overlays are used, depending on the cluster's labels:
```yaml title="fleet.yaml"
namespace: fleet-mc-kustomize-example
targetCustomizations:
- name: dev
clusterSelector:
matchLabels:
env: dev
kustomize:
dir: overlays/dev
- name: test
clusterSelector:
matchLabels:
env: test
kustomize:
dir: overlays/test
- name: prod
clusterSelector:
matchLabels:
env: prod
kustomize:
dir: overlays/prod
```
To create the deployment, we apply the custom resource to the upstream cluster. The `fleet-default` namespace, by default, contains the downstream cluster resources. The chart will be deployed to all clusters in the fleet-default namespace, which have a labeled cluster resources that matches any entry under `targets:`.
```yaml title="gitrepo.yaml"
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: helm-kustomize
namespace: fleet-default
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- multi-cluster/helm-kustomize
targets:
- name: dev
clusterSelector:
matchLabels:
env: dev
- name: test
clusterSelector:
matchLabels:
env: test
- name: prod
clusterSelector:
matchLabels:
env: prod
```
By applying the gitrepo resource to the upstream cluster, fleet will start to monitor the repository and create deployments:
<CodeBlock language="bash">
{`kubectl apply -n fleet-default -f gitrepo.yaml`}
</CodeBlock>
</TabItem>
<TabItem value="kustomize" label="Kustomize" default>
An <a href="https://github.com/rancher/fleet-examples/blob/master/multi-cluster/kustomize">example using Kustomize</a> and customizing it per target cluster.
The customization in `fleet.yaml` is identical to the "Helm & Kustomize" example.
To create the deployment, we apply the custom resource to the upstream cluster. The `fleet-default` namespace, by default, contains the downstream cluster resources. The chart will be deployed to all clusters in the fleet-default namespace, which have a labeled cluster resources that matches any entry under `targets:`.
```bash
kubectl apply -n fleet-default -f - <<EOF
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: kustomize
namespace: fleet-default
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- multi-cluster/kustomize
targets:
- name: dev
clusterSelector:
matchLabels:
env: dev
- name: test
clusterSelector:
matchLabels:
env: test
- name: prod
clusterSelector:
matchLabels:
env: prod
EOF
```
By applying the gitrepo resource to the upstream cluster, fleet will start to monitor the repository and create deployments:
</TabItem>
<TabItem value="manifests" label="Manifests" default>
An <a href="https://github.com/rancher/fleet-examples/tree/master/multi-cluster/manifests">example using raw Kubernetes YAML and customizing it per target cluster</a>.
The application will be customized as follows per environment:
* Dev clusters: Only the redis leader is deployed and not the followers.
* Test clusters: Scale the front deployment to 3
* Prod clusters: Scale the front deployment to 3 and set the service type to LoadBalancer
The `fleet.yaml` is used to control which 'yaml' overlays are used, depending on the cluster's labels:
```yaml title="fleet.yaml"
namespace: fleet-mc-manifest-example
targetCustomizations:
- name: dev
clusterSelector:
matchLabels:
env: dev
yaml:
overlays:
# Refers to overlays/noreplication folder
- noreplication
- name: test
clusterSelector:
matchLabels:
env: test
yaml:
overlays:
# Refers to overlays/scale3 folder
- scale3
- name: prod
clusterSelector:
matchLabels:
env: prod
yaml:
# Refers to overlays/servicelb, scale3 folders
overlays:
- servicelb
- scale3
```
To create the deployment, we apply the custom resource to the upstream cluster. The `fleet-default` namespace, by default, contains the downstream cluster resources. The chart will be deployed to all clusters in the fleet-default namespace, which have a labeled cluster resources that matches any entry under `targets:`.
```yaml title="gitrepo.yaml"
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: manifests
namespace: fleet-default
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- multi-cluster/manifests
targets:
- name: dev
clusterSelector:
matchLabels:
env: dev
- name: test
clusterSelector:
matchLabels:
env: test
- name: prod
clusterSelector:
matchLabels:
env: prod
```
<CodeBlock language="bash">
{`kubectl apply -n fleet-default -f gitrepo.yaml`}
</CodeBlock>
</TabItem>
</Tabs>

View File

@ -0,0 +1,63 @@
# Uninstall
:::caution
Removing the CRDs will remove all deployed workloads.
:::
## Fleet Standalone
Fleet is packaged as two Helm charts so uninstall is accomplished by
uninstalling the appropriate Helm charts.
However Fleet uses [finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/),
so uninstall workloads first to give the Fleet controllers time to clean up.
```
kubectl delete clusters.fleet.cattle.io -A
kubectl delete gitrepos -A
kubectl delete bundles -A
```
Note, it can be easier to remove the namespaces containing the workloads, e.g.:
```
kubectl delete ns fleet-local
kubectl delete ns fleet-default
```
The custom resources are not uninstalled by Helm 3, remove them first:
```
kubectl delete crd gitrepos.fleet.cattle.io bundles.fleet.cattle.io contents.fleet.cattle.io \
bundledeployments.fleet.cattle.io imagescans.fleet.cattle.io \
bundlenamespacemappings.fleet.cattle.io gitreporestrictions.fleet.cattle.io \
clusters.fleet.cattle.io clustergroups.fleet.cattle.io \
clusterregistrations.fleet.cattle.io clusterregistrationtokens.fleet.cattle.io
```
Then, to uninstall Fleet, run the following commands:
```shell
helm -n cattle-fleet-system uninstall fleet
helm -n cattle-fleet-system uninstall fleet-crd
```
:::note
If uninstall gets stuck it is likely due to finalizers. Resource status fields, e.g. on a namespace, will list the resources waiting for their finalizers to be removed. The finalizers can be removed manually with kubectl, if their controllers are no longer running.
:::
Afterwards one can also remove Fleet-related namespaces and their content:
```
kubectl delete ns cattle-fleet-system
kubectl delete ns cattle-fleet-clusters-system
kubectl delete ns -l "fleet.cattle.io/managed=true"
```
## Fleet in Rancher
Fleet in Rancher cannot be uninstalled, without uninstalling Rancher. However, the GitOps feature can be disabled.
When disabled, the gitjob deployment is no longer deployed into the Rancher servers local cluster, and continuous-delivery is not shown in the Rancher UI.
For more information, refer to [Continuous Delivery in Rancher](https://documentation.suse.com/cloudnative/rancher-manager/latest/en/rancher-admin/experimental-features/continuous-delivery.html)

View File

@ -0,0 +1,133 @@
# Using Webhooks Instead of Polling
By default, Fleet utilizes polling (default: every 15 seconds) to pull from a Git repo. This is a convenient default that works reasonably well for a small number of repos (up to a few tens).
For installations with multiple tens up to hundreds of Git repos, and in general to reduce latency (the time between a push to Git and fleet reacting to it), configuring webhooks is recommended instead of polling.
Fleet currently supports Azure DevOps, GitHub, GitLab, Bitbucket, Bitbucket Server, and Gogs.
### 1. Configure the webhook service. Fleet uses a gitjob service to handle webhook requests. Create an ingress that points to the gitjob service.
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: webhook-ingress
namespace: cattle-fleet-system
spec:
rules:
- host: your.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gitjob
port:
number: 80
```
If you want to have the webhook available using the same host name as your Rancher or another service, you can use the following YAML with the URL http://your.domain.com/gitjob. The below YAML is specific for the Nginx Ingress Controller:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: webhook-ingress
namespace: cattle-fleet-system
spec:
rules:
- host: your.domain.com
http:
paths:
- path: /gitjob(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: gitjob
port:
number: 80
```
:::info
You can configure [TLS](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls) on ingress.
:::
### 2. Go to your webhook provider and configure the webhook callback url. Here is a Github example.
![](/img/webhook.png)
Configuring a secret is optional. This is used to validate the webhook payload as the payload should not be trusted by default.
If your webhook server is publicly accessible to the Internet, then it is recommended to configure the secret. If you do configure the
secret, follow step 3.
:::note
only application/json is supported due to the limitation of webhook library.
:::
:::caution
If you configured the webhook the polling interval will be automatically adjusted to 1 hour.
:::
### 3. (Optional) Configure a webhook secret.
The secret is for validating the webhook payload.
The secret must contain the designated key for the desired provider. The list of providers with their corresponding keys is as follows.
| Provider | K8s Secret Key |
|-----------------|--------------------|
| GitHub | `github` |
| GitLab | `gitlab` |
| BitBucket | `bitbucket` |
| BitBucketServer | `bitbucket-server` |
| Gogs | `gogs` |
| Azure DevOps | `azure-username` |
| Azure DevOps | `azure-password` |
#### Option 1: Configure a cluster secret.
In this case, the secret is unique per cluster, and all GitRepos will use the same one. The user does not need to reference it at all; when a payload is received for a specific provider, the system checks if the global secret exists, and if so, whether the key for that provider also exists. If the key is present, the secret will be used to validate the payload.
Make sure to put it in a k8s secret called `gitjob-webhook` in `cattle-fleet-system`.
For example, to create a secret containing a GitHub secret to validate the webhook payload, run:
```shell
kubectl create secret generic gitjob-webhook -n cattle-fleet-system --from-literal=github=webhooksecretvalue
```
For Azure DevOps:
- Enable basic authentication in Azure
- Create a secret containing the credentials for the basic authentication
```shell
kubectl create secret generic gitjob-webhook -n cattle-fleet-system --from-literal=azure-username=user --from-literal=azure-password=pass123
```
#### Option 2. Define a secret for each GitRepo.
Alternatively, you can define a Webhook secret for each GitRepo. The secret must be created in the same namespace as the GitRepo, with the desired name, and then you can reference it in the `webhookSecret` field in the GitRepo specification.
Example:
```yaml
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: simple
namespace: fleet-local
spec:
repo: "https://github.com/rancher/fleet-examples"
paths:
- simple
disablePolling: true
webhookSecret: webhook-secret-name
```
If both exist — the global secret for the cluster and a secret defined for the GitRepo — the latter will take precedence.
### 4. Go to your git provider and test the connection. You should get a HTTP response code.

View File

@ -0,0 +1,184 @@
{
"docs": [
"index",
"persona",
{
"type": "category",
"label": "Tutorials",
"collapsed": false,
"items": [
"quickstart",
"tut-deployment",
{
"type": "doc",
"id": "uninstall"
}
]
},
{
"type": "category",
"label": "Explanations",
"collapsed": false,
"items": [
"concepts",
"glossary",
"architecture",
"ref-bundle-stages",
"gitrepo-content",
"namespaces",
"resources-during-deployment"
]
},
{
"type": "category",
"label": "How-tos for Operators",
"collapsed": false,
"items": [
{
"type": "doc",
"id": "installation"
},
{
"type": "doc",
"id": "cluster-registration"
},
{
"type": "doc",
"id": "cluster-group"
},
"multi-user"
]
},
{
"type": "category",
"label": "How-tos for Users",
"collapsed": false,
"items": [
{
"type": "doc",
"id": "gitrepo-add"
},
{
"type": "doc",
"id": "gitrepo-targets"
},
{
"type": "doc",
"id": "bundle-diffs"
},
{
"type": "doc",
"id": "oci-storage"
},
{
"type": "doc",
"id": "webhook"
},
{
"type": "doc",
"id": "imagescan"
},
{
"type": "doc",
"id": "bundle-add"
},
{
"type": "doc",
"id": "observability"
},
{
"type": "doc",
"id": "helm-ops"
},
{
"type": "doc",
"id": "rollout"
}
]
},
{
"type": "category",
"label": "Reference",
"collapsed": false,
"items": [
{
"CLI": [
"cli/fleet-agent/fleet-agent",
"cli/fleet-agent/fleet-agent_clusterstatus",
"cli/fleet-agent/fleet-agent_register",
{
"type": "doc",
"id": "cli/fleet-cli/fleet"
},
{
"type": "doc",
"id": "cli/fleet-cli/fleet_apply"
},
{
"type": "doc",
"id": "cli/fleet-cli/cleanup/fleet_cleanup"
},
{
"type": "doc",
"id": "cli/fleet-cli/cleanup/fleet_cleanup_clusterregistration"
},
{
"type": "doc",
"id": "cli/fleet-cli/cleanup/fleet_cleanup_gitjob"
},
{
"type": "doc",
"id": "cli/fleet-cli/fleet_deploy"
},
{
"type": "doc",
"id": "cli/fleet-cli/fleet_target"
},
{
"type": "doc",
"id": "cli/fleet-cli/fleet_test"
},
{
"type": "doc",
"id": "cli/fleet-controller/fleet-controller"
},
{
"type": "doc",
"id": "cli/fleet-controller/fleet-controller_agentmanagement"
},
{
"type": "doc",
"id": "cli/fleet-controller/fleet-controller_cleanup"
},
{
"type": "doc",
"id": "cli/fleet-controller/fleet-controller_gitjob"
}
]
},
{
"type": "doc",
"id": "ref-status-fields"
},
"ref-registration",
"ref-configuration",
"ref-resources",
"ref-crds",
"ref-fleet-yaml",
"ref-gitrepo",
"ref-bundle"
]
},
"troubleshooting",
{
"type": "category",
"label": "Changelog",
"items": [
{
"type": "autogenerated",
"dirName": "changelogs/changelogs"
}
]
}
]
}

View File

@ -1,4 +1,5 @@
[ [
"0.13",
"0.12", "0.12",
"0.11", "0.11",
"0.10", "0.10",