Add health checks and dependencies to API spec
This commit is contained in:
stefanprodan
parent
759fd5d6f5
commit
91fa7b856d
81
README.md
81
README.md
|
|
@ -2,18 +2,22 @@
|
|||
|
||||
[](https://github.com/fluxcd/kustomize-controller/actions)
|
||||
|
||||
The kustomize-controller is a Kubernetes operator that applies [kustomizations](docs/spec/v1alpha1/README.md) in-cluster.
|
||||
The kustomize-controller is a continuous delivery (CD) tool for Kubernetes.
|
||||
The controller runs CD pipelines inside the cluster for workloads and infrastructure manifests
|
||||
coming from source control systems that are generated with Kustomize.
|
||||
|
||||
|
||||
|
||||
Features:
|
||||
* watches for `Kustomization` objects
|
||||
* watches for [Kustomization](docs/spec/v1alpha1/README.md) objects
|
||||
* fetches artifacts produced by [source-controller](https://github.com/fluxcd/source-controller) from `Source` objects
|
||||
* watches `Source` objects for revision changes
|
||||
* builds the kustomization using the latest fetched artifact
|
||||
* validates the Kubernetes objects with client-side or APIServer dry-run
|
||||
* applies the resulting Kubernetes manifests on the cluster
|
||||
* prunes the Kubernetes objects removed from source based on a label selector
|
||||
* generates Kubernetes manifests with kustomize build
|
||||
* validates the build output with client-side or APIServer dry-run
|
||||
* applies the generated manifests on the cluster
|
||||
* prunes the Kubernetes objects removed from source
|
||||
* checks the health of the deployed workloads
|
||||
* runs `Kustomizations` in a specific order, taking into account the depends-on relationship
|
||||
|
||||
## Usage
|
||||
|
||||
|
|
@ -73,7 +77,7 @@ kubectl annotate --overwrite gitrepository/podinfo source.fluxcd.io/syncAt="$(da
|
|||
|
||||
### Define a kustomization
|
||||
|
||||
Create a [kustomization](docs/spec/v1alpha1/README.md) object that uses the git repository defined above:
|
||||
Create a kustomization object that uses the git repository defined above:
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
|
|
@ -88,18 +92,23 @@ spec:
|
|||
kind: GitRepository
|
||||
name: podinfo
|
||||
validation: client
|
||||
healthChecks:
|
||||
- kind: Deployment
|
||||
name: podinfo
|
||||
namespace: dev
|
||||
timeout: 80s
|
||||
```
|
||||
|
||||
With `spec.interval` we tell the controller how often it should reconcile the cluster state.
|
||||
With `spec.path` we tell the controller where to look for the `kustomization.yaml` file.
|
||||
With `spec.prune` we configure [garbage collection](docs/spec/v1alpha1/README.md#garbage-collection).
|
||||
With `spec.validation` we instruct the controller to validate the Kubernetes objects before applying them in-cluster.
|
||||
When setting the validation to `server`, the controller will perform an
|
||||
[APIServer dry-run](https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/)
|
||||
(requires Kubernetes >= 1.16).
|
||||
A detailed explanation of the Kustomization object and its fields
|
||||
can be found in the [specification doc](docs/spec/v1alpha1/README.md).
|
||||
|
||||
Save the above file and apply it on the cluster.
|
||||
You can wait for the kustomize controller to apply the manifest corresponding to the dev overlay with:
|
||||
Based on the above definition, the kustomize-controller fetches the Git repository content from source-controller,
|
||||
generates Kubernetes manifests by running kustomize build inside `./overlays/dev/`,
|
||||
and validates them with a dry-run apply. If the manifests pass validation, the controller will apply them
|
||||
on the cluster and starts the health assessment of the deployed workload. If the health checks are passing, the
|
||||
Kustomization object status transitions to a ready state.
|
||||
|
||||
You can wait for the kustomize controller to complete the deployment with:
|
||||
|
||||
```bash
|
||||
kubectl wait kustomization/podinfo-dev --for=condition=ready
|
||||
|
|
@ -155,6 +164,46 @@ status:
|
|||
}
|
||||
```
|
||||
|
||||
### Define the order for kustomizations
|
||||
|
||||
When defining a kustomization, you may need to make sure other kustomizations have been
|
||||
successfully applied beforehand. A kustomization can specify a list of dependencies with `spec.dependsOn`.
|
||||
When combined with health assessment, a kustomization will run after all its dependencies health checks are passing.
|
||||
|
||||
For example, a service mesh proxy injector should be running before deploying applications inside the mesh:
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: istio
|
||||
spec:
|
||||
interval: 10m
|
||||
path: "./profiles/default/"
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: istio
|
||||
healthChecks:
|
||||
- kind: Deployment
|
||||
name: istiod
|
||||
namespace: istio-system
|
||||
timeout: 2m
|
||||
---
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: podinfo-dev
|
||||
spec:
|
||||
dependsOn:
|
||||
- istio
|
||||
interval: 5m
|
||||
path: "./overlays/dev/"
|
||||
prune: "env=dev"
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: podinfo
|
||||
```
|
||||
|
||||
### Deploy releases to production
|
||||
|
||||
For production deployments, instead of synchronizing with a branch you can use a semver range to target stable releases:
|
||||
|
|
|
|||
|
|
@ -1,14 +1,22 @@
|
|||
# Kustomization API
|
||||
# kustomize.fluxcd.io/v1alpha1
|
||||
|
||||
This is the v1alpha1 API specification for defining continuous delivery pipelines
|
||||
of Kubernetes objects generated with Kustomize.
|
||||
|
||||
## Specification
|
||||
|
||||
A **kustomization** object defines the source of Kubernetes manifests by referencing an object
|
||||
managed by [source-controller](https://github.com/fluxcd/source-controller),
|
||||
the path to the kustomization file within that source,
|
||||
and the interval at which the kustomization is applied on the cluster.
|
||||
and the interval at which the kustomize build output is applied on the cluster.
|
||||
|
||||
```go
|
||||
type KustomizationSpec struct {
|
||||
// A list of kustomization that must be ready before this
|
||||
// kustomization can be applied.
|
||||
// +optional
|
||||
DependsOn []string `json:"dependsOn,omitempty"`
|
||||
|
||||
// The interval at which to apply the kustomization.
|
||||
// +required
|
||||
Interval metav1.Duration `json:"interval"`
|
||||
|
|
@ -23,6 +31,11 @@ type KustomizationSpec struct {
|
|||
// +optional
|
||||
Prune string `json:"prune,omitempty"`
|
||||
|
||||
// A list of workloads (Deployments, DaemonSets and StatefulSets)
|
||||
// to be included in the health assessment.
|
||||
// +optional
|
||||
HealthChecks []WorkloadReference `json:"healthChecks,omitempty"`
|
||||
|
||||
// Reference of the source where the kustomization file is.
|
||||
// +required
|
||||
SourceRef corev1.TypedLocalObjectReference `json:"sourceRef"`
|
||||
|
|
@ -32,6 +45,11 @@ type KustomizationSpec struct {
|
|||
// +optional
|
||||
Suspend bool `json:"suspend,omitempty"`
|
||||
|
||||
// Timeout for validation, apply and health checking operations.
|
||||
// Defaults to 'Interval' duration.
|
||||
// +optional
|
||||
Timeout *metav1.Duration `json:"timeout,omitempty"`
|
||||
|
||||
// Validate the Kubernetes objects before applying them on the cluster.
|
||||
// The validation strategy can be 'client' (local dry-run) or 'server' (APIServer dry-run).
|
||||
// +kubebuilder:validation:Enum=client;server
|
||||
|
|
@ -40,63 +58,7 @@ type KustomizationSpec struct {
|
|||
}
|
||||
```
|
||||
|
||||
### Kustomization execution
|
||||
|
||||
The kustomization `spec.interval` tells the controller at which interval to fetch the Kubernetes manifest for the source,
|
||||
build the kustomization and apply it on the cluster.
|
||||
The interval time units are `s`, `m` and `h` e.g. `interval: 5m`, the minimum value should be over 60 seconds.
|
||||
|
||||
The kustomization execution can be suspended by setting `spec.susped` to `true`.
|
||||
|
||||
The controller can be told to execute the kustomization outside of the specified interval
|
||||
by annotating the kustomization object with:
|
||||
|
||||
```go
|
||||
const (
|
||||
// SyncAtAnnotation is the annotation used for triggering a
|
||||
// sync outside of the specified schedule.
|
||||
SyncAtAnnotation string = "kustomize.fluxcd.io/syncAt"
|
||||
)
|
||||
```
|
||||
|
||||
On-demand execution example:
|
||||
|
||||
```bash
|
||||
kubectl annotate --overwrite kustomization/podinfo kustomize.fluxcd.io/syncAt="$(date +%s)"
|
||||
```
|
||||
|
||||
### Kustomization source
|
||||
|
||||
The kustomization `spec.sourceRef` is a reference to an object managed by
|
||||
[source-controller](https://github.com/fluxcd/source-controller). When the source
|
||||
[revision](https://github.com/fluxcd/source-controller/blob/master/docs/spec/v1alpha1/common.md#source-status)
|
||||
changes, it creates a Kubernetes event that triggers a kustomization apply outside of the specified interval.
|
||||
|
||||
Source supported types:
|
||||
|
||||
* [GitRepository](https://github.com/fluxcd/source-controller/blob/master/docs/spec/v1alpha1/gitrepositories.md)
|
||||
|
||||
### Garbage collection
|
||||
|
||||
Garbage collection means that the Kubernetes objects that were previously applied on the cluster
|
||||
but are missing from the current apply, will be removed. Garbage collection is also performed when a Kustomization
|
||||
object is deleted, triggering a removal of all Kubernetes objects previously applied on the cluster.
|
||||
|
||||
When garbage collection is enabled, all Kubernetes objects must have a common label that matches the `spec.prune`
|
||||
[label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
|
||||
|
||||
For example, `prune: env=dev,app=frontend` requires a `kustomization.yaml` with `commonLabels`:
|
||||
```yaml
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
commonLabels:
|
||||
env: dev
|
||||
app: frontend
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
The status sub-resource describes the result of the last kustomization execution.
|
||||
The status sub-resource describes the result of the last kustomization execution:
|
||||
|
||||
```go
|
||||
type KustomizationStatus struct {
|
||||
|
|
@ -139,11 +101,181 @@ const (
|
|||
)
|
||||
```
|
||||
|
||||
Wait for condition example:
|
||||
## Kustomization source
|
||||
|
||||
```bash
|
||||
kubectl wait kustomization/podinfo --for=condition=ready --timeout=1m
|
||||
The kustomization `spec.sourceRef` is a reference to an object managed by
|
||||
[source-controller](https://github.com/fluxcd/source-controller). When the source
|
||||
[revision](https://github.com/fluxcd/source-controller/blob/master/docs/spec/v1alpha1/common.md#source-status)
|
||||
changes, it generates a Kubernetes event that triggers a kustomize build and apply.
|
||||
|
||||
Source supported types:
|
||||
|
||||
* [GitRepository](https://github.com/fluxcd/source-controller/blob/master/docs/spec/v1alpha1/gitrepositories.md)
|
||||
|
||||
> **Note** that the source must contain the kustomization.yaml and all the Kubernetes manifests and configuration files
|
||||
> referenced in the kustomization.yaml.
|
||||
|
||||
## Kustomization execution
|
||||
|
||||
The kustomization `spec.interval` tells the controller at which interval to fetch the
|
||||
Kubernetes manifest for the source, build the kustomization and apply it on the cluster.
|
||||
The interval time units are `s`, `m` and `h` e.g. `interval: 5m`, the minimum value should be over 60 seconds.
|
||||
|
||||
The kustomization execution can be suspended by setting `spec.susped` to `true`.
|
||||
|
||||
The controller can be told to execute the kustomization outside of the specified interval
|
||||
by annotating the kustomization object with:
|
||||
|
||||
```go
|
||||
const (
|
||||
// SyncAtAnnotation is the annotation used for triggering a
|
||||
// sync outside of the specified schedule.
|
||||
SyncAtAnnotation string = "kustomize.fluxcd.io/syncAt"
|
||||
)
|
||||
```
|
||||
|
||||
On-demand execution example:
|
||||
|
||||
```bash
|
||||
kubectl annotate --overwrite kustomization/podinfo kustomize.fluxcd.io/syncAt="$(date +%s)"
|
||||
```
|
||||
|
||||
## Garbage collection
|
||||
|
||||
Garbage collection means that the Kubernetes objects that were previously applied on the cluster
|
||||
but are missing from the current apply, are removed from cluster automatically.
|
||||
Garbage collection is also performed when a Kustomization object is deleted,
|
||||
triggering a removal of all Kubernetes objects previously applied on the cluster.
|
||||
|
||||
Tpo enable garbage collection, all Kubernetes objects must have a common label that matches the `spec.prune`
|
||||
[label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
|
||||
|
||||
For example, `prune: env=dev,app=frontend` requires a `kustomization.yaml` with `commonLabels`:
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
commonLabels:
|
||||
env: dev
|
||||
app: frontend
|
||||
```
|
||||
|
||||
> **Note** that each kustomization must have a unique combination of labels values, otherwise the
|
||||
> garbage collection will remove resources outside of the kustomization scope.
|
||||
|
||||
## Health assessment
|
||||
|
||||
A kustomization can contain a series of health checks used to determine the
|
||||
[rollout status](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#deployment-status)
|
||||
of the deployed workloads. A health check entry can reference one of the following Kubernetes types:
|
||||
Deployment, DaemonSet or StatefulSet.
|
||||
|
||||
Assuming the kustomization source contains a Kubernetes Deployment named `backend`,
|
||||
a health check can be defined as follows:
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: backend
|
||||
spec:
|
||||
interval: 5m
|
||||
path: "./webapp/backend/"
|
||||
prune: "component=backend"
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: webapp
|
||||
healthChecks:
|
||||
- kind: Deployment
|
||||
name: backend
|
||||
namespace: dev
|
||||
timeout: 2m
|
||||
```
|
||||
|
||||
After applying the kustomize build output, the controller verifies if the rollout completed successfully.
|
||||
If the deployment was successful, the kustomization ready condition is marked as `true`,
|
||||
if the rollout failed, or if it takes more than the specified timeout to complete, then the
|
||||
kustomization ready condition is set to `false`. If the deployment becomes healthy on the next
|
||||
execution, then the kustomization is marked as ready.
|
||||
|
||||
## Kustomization dependencies
|
||||
|
||||
When running a kustomization, you may need to make sure other resources exist before the
|
||||
workloads defined in your kustomization are deployed.
|
||||
For example, a namespace must exist before applying resources to it.
|
||||
|
||||
With `spec.dependsOn` you can specify that the execution of a kustomization follows another.
|
||||
When you add `dependsOn` entries to a kustomization, that kustomization is applied
|
||||
only after all of its dependencies are ready. The readiness state of a kustomization is determined by
|
||||
its last apply status condition.
|
||||
|
||||
Assuming two kustomizations: a `common` one, that contains a namespace definition and a `backend` one, that
|
||||
contains the workload to be deployed in that namespace. You can instruct the controller to apply the `common`
|
||||
kustomization first with:
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: common
|
||||
spec:
|
||||
interval: 5m
|
||||
path: "./webapp/common/"
|
||||
prune: "part-of=webapp"
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: webapp
|
||||
---
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: backend
|
||||
spec:
|
||||
dependsOn:
|
||||
- common
|
||||
interval: 5m
|
||||
path: "./webapp/backend/"
|
||||
prune: "part-of=webapp,component=backend"
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: webapp
|
||||
```
|
||||
|
||||
When combined with health assessment, a kustomization will run after all its dependencies health checks are passing.
|
||||
For example, a service mesh proxy injector should be running before deploying applications inside the mesh.
|
||||
|
||||
```yaml
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: istio
|
||||
spec:
|
||||
interval: 5m
|
||||
path: "./profiles/default/"
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: istio
|
||||
healthChecks:
|
||||
- kind: Deployment
|
||||
name: istiod
|
||||
namespace: istio-system
|
||||
timeout: 2m
|
||||
---
|
||||
apiVersion: kustomize.fluxcd.io/v1alpha1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: backend
|
||||
spec:
|
||||
dependsOn:
|
||||
- common
|
||||
- istio
|
||||
interval: 5m
|
||||
path: "./webapp/backend/"
|
||||
prune: "part-of=webapp,component=backend"
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: webapp
|
||||
```
|
||||
|
||||
> **Note** that circular dependencies between kustomizations must be avoided, otherwise the
|
||||
> interdependent kustomizations will never be applied on the cluster.
|
||||
|
|
|
|||
Loading…
Reference in New Issue