fluxcd
/
source-controller
mirror of https://github.com/fluxcd/source-controller.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3 from fluxcd/spec-v1alpha1 

Add source API spec v1
This commit is contained in:
Stefan Prodan 2020-04-16 13:27:40 +03:00 committed by GitHub
parent 96c7ad502c 000bf41dc2
commit ecc2c62ed7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 859 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

65
docs/spec/README.md Normal file
View File

@ -0,0 +1,65 @@
# Source Controller Proposal
The main goal is to define a set of Kubernetes objects that cluster
admins and various automated operators can interact with to offload
the sources (e.g. Git and Helm repositories) registration, authentication,
verification and resource fetching to a dedicated controller.
## Motivation
Each Flux and each Helm operator mirrors the Git repositories they are
using, in the same way, using the same code. But other components
might benefit from access to the source mirrors, and Flux and the Helm
operator could work more in sympathy with Kubernetes by factoring it out.
If "sources" (usually git repos, but also Helm charts and potentially
other things) existed in their own right as Kubernetes resources,
components like Flux and Helm operator could use standard Kubernetes
mechanisms to build on them; and, they could be managed independently
of the components using them.
## API Specification
* [v1alpha1](v1alpha1/README.md)
## Implementation
The controller implementation will watch for source objects in a cluster and act on them.
The actions performed by the source controller could be:
* validate source definitions
* authenticate to sources and validate authenticity
* detect source changes based on update policies (semver)
* fetch resources on-demand and on-a-schedule
* package the fetched resources into a well known format (tar.gz, yaml)
* store the artifacts locally
* make the artifacts addressable by their source identifier (sha, version, ts)
* make the artifacts available in-cluster to interested 3rd parties
* notify interested 3rd parties of source changes and availability (status conditions, events, hooks)
## Impact to Flux
Having a dedicated controller that manages Git repositories defined with Kubernetes custom resources would:
* simplify Flux configuration as fluxd could subscribe to Git sources in-cluster and pull the artifacts
automatically without manual intervention from users to reconfigure and redeploy FLux
* improve the installation experience as users will not have to patch fluxd's deployment to inject
the HTTPS basic auth credentials, change the source URL or other Git and PGP related settings
* enable fluxd to compose the desired state of a cluster from multiple sources by applying all artifacts present in flux namespace
* enable fluxd to apply manifests coming from other sources than Git, e.g. S3 buckets
* allow fluxd to run under a non-root user as it wouldn't need to shell out to ssh-keygen, git or pgp
* enable fluxd to apply manifests coming from the most recent semver tag of a Git repository
* allow user to pin the cluster desired state to a specific Git commit or Git tag
## Impact to Helm Operator
Having a dedicated controller that manages Helm repositories and charts defined with Kubernetes custom
resources would:
* simplify the Helm Operator configuration as repository and chart definitions can be re-used across
`HelmRelease` resources (see [fluxcd/helm-operator#142](https://github.com/fluxcd/helm-operator/issues/142))
* improve the user experience as repositories requiring authentication will no longer require a
`repositories.yaml` import / file mount
* simplify the architecture of the Helm Operator as it allows the operator to work with a single
source type (`HelmChart`) and way of preparing and executing installations and/or upgrades
* allow the Helm Operator to run under a non-root user as it wouldn't need to shell out to git

15
docs/spec/v1alpha1/README.md Normal file
View File

@ -0,0 +1,15 @@
# source.fluxcd.io/v1alpha1
The is the v1alpha1 API specification for defining the desired state sources of Kubernetes clusters.
## Specification
* [Common](common.md)
* Source kinds:
+ [GitRepository](gitrepositories.md)
+ [HelmRepository](helmrepositories.md)
+ [HelmChart](helmcharts.md)
## Implementation
* source-controller [v0.0.1-alpha.1](https://github.com/fluxcd/source-controller/releases)

146
docs/spec/v1alpha1/common.md Normal file
View File

@ -0,0 +1,146 @@
# Common
Common defines resources used across all source types.
## Specification
### Source interface
Source objects should adhere to the `Source` interface. This interface exposes the [interval](#source-synchronization)
and [artifact](#source-status) of the source to clients without the prerequisite of knowing the source kind:
````go
type Source interface {
// GetInterval returns the interval at which the source is updated.
GetInterval() metav1.Duration
// GetArtifact returns the latest artifact from the source, or nil.
GetArtifact() *Artifact
}
````
### Source synchronization
Source objects should contain a `spec.interval` field that tells the controller at which interval to check for updates:
```go
type SourceSpec struct {
// The interval at which to check for source updates.
// +required
Interval metav1.Duration `json:"interval"`
}
```
Valid time units are `s`, `m` and `h` e.g. `interval: 5m`.
The controller can be told to check for updates right away by setting an annotation on source objects:
```go
const (
// ForceSyncAnnotation is the timestamp corresponding to an on-demand source sync.
ForceSyncAnnotation string = "source.fluxcd.io/syncAt"
)
```
Force sync example:
```bash
kubectl annotate --overwrite gitrepository/podinfo source.fluxcd.io/syncAt="$(date +%s)"
```
### Source status
Source objects should contain a status sub-resource that embeds an artifact object:
```go
// Artifact represents the output of a source synchronisation
type Artifact struct {
// Path is the local file path of this artifact.
// +required
Path string `json:"path"`
// URL is the HTTP address of this artifact.
// +required
URL string `json:"url"`
// Revision is a human readable identifier traceable in the origin source system.
// It can be a commit sha, git tag, a helm index timestamp,
// a helm chart version, a checksum, etc.
// +optional
Revision string `json:"revision"`
// LastUpdateTime is the timestamp corresponding to the last
// update of this artifact.
// +required
LastUpdateTime metav1.Time `json:"lastUpdateTime,omitempty"`
}
```
### Source condition
> **Note:** to be replaced with <https://github.com/kubernetes/enhancements/pull/1624>
> once made available.
```go
// SourceCondition contains condition information for a source.
type SourceCondition struct {
// Type of the condition, currently ('Ready').
// +required
Type string `json:"type"`
// Status of the condition, one of ('True', 'False', 'Unknown').
// +required
Status corev1.ConditionStatus `json:"status"`
// LastTransitionTime is the timestamp corresponding to the last status
// change of this condition.
// +required
LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`
// Reason is a brief machine readable explanation for the condition's last
// transition.
// +required
Reason string `json:"reason,omitempty"`
// Message is a human readable description of the details of the last
// transition, complementing reason.
// +optional
Message string `json:"message,omitempty"`
}
```
#### Types
```go
const (
// ReadyCondition represents the fact that a given source is in ready state.
ReadyCondition string = "Ready"
)
```
#### Reasons
```go
const (
// InitializingReason represents the fact that a given source is being initialized.
InitializingReason string = "Initializing"
// URLInvalidReason represents the fact that a given source has an invalid URL.
URLInvalidReason string = "URLInvalid"
// StorageOperationFailedReason signals a failure caused by a storage operation.
StorageOperationFailedReason string = "StorageOperationFailed"
// AuthenticationFailedReason represents the fact that a given secret does not
// have the required fields or the provided credentials do not match.
AuthenticationFailedReason string = "AuthenticationFailed"
// VerificationFailedReason represents the fact that the cryptographic provenance
// verification for the source failed.
VerificationFailedReason string = "VerificationFailed"
)
```
## Examples
See the [Git repository](gitrepositories.md) and [Helm chart](helmrepositories.md) APIs.

342
docs/spec/v1alpha1/gitrepositories.md Normal file
View File

@ -0,0 +1,342 @@
# Git Repositories
The `GitRepository` API defines a source for artifacts coming from Git. The
resource exposes the latest synchronized state from Git as an artifact in
an archive.
## Specification
Git repository:
```go
// GitRepositorySpec gives the specification for fetching a Git repository as
// a source.
type GitRepositorySpec struct {
// The repository URL, can be a HTTP or SSH address.
// +kubebuilder:validation:Pattern="^(http|https|ssh)://"
URL string `json:"url"`
// The secret name containing the Git credentials.
// For HTTPS repositories the secret must contain username and password
// fields.
// For SSH repositories the secret must contain identity, identity.pub and
// known_hosts fields.
// +optional
SecretRef *v1.LocalObjectReference `json:"secretRef,omitempty"`
// The interval at which to check for repository updates.
Interval metav1.Duration `json:"interval"`
// The git reference to checkout and monitor for changes, defaults to
// master branch.
// +optional
Reference *GitRepositoryRef `json:"ref,omitempty"`
// Verify OpenPGP signature for the commit that HEAD points to.
// +optional
Verification *GitRepositoryVerification `json:"verify,omitempty"`
}
```
Git repository reference:
```go
// GitRepositoryRef defines the git ref used for pull and checkout operations.
type GitRepositoryRef struct {
// The git branch to checkout, defaults to master.
// +optional
Branch string `json:"branch"`
// The git tag to checkout, takes precedence over branch.
// +optional
Tag string `json:"tag"`
// The git tag semver expression, takes precedence over tag.
// +optional
SemVer string `json:"semver"`
// The git commit sha to checkout, if specified branch and tag filters will
// ignored.
// +optional
Commit string `json:"commit"`
}
```
Git repository cryptographic provenance verification:
```go
// GitRepositoryVerification defines the OpenPGP signature verification process.
type GitRepositoryVerification struct {
// Mode describes what git object should be verified, currently ('head').
// +kubebuilder:validation:Enum=head
Mode string `json:"mode"`
// The secret name containing the public keys of all trusted git authors.
SecretRef corev1.LocalObjectReference `json:"secretRef"`
}
```
### Status
```go
// GitRepositoryStatus defines the observed state of the GitRepository.
type GitRepositoryStatus struct {
// +optional
Conditions []SourceCondition `json:"conditions,omitempty"`
// URL is the download link for the artifact output of the last repository
// sync.
// +optional
URL string `json:"url,omitempty"`
// Artifact represents the output of the last successful repository sync.
// +optional
Artifact *Artifact `json:"artifact,omitempty"`
}
```
### Condition reasons
```go
const (
// GitOperationSucceedReason represents the fact that the git
// clone, pull and checkout operations succeeded.
GitOperationSucceedReason string = "GitOperationSucceed"
// GitOperationFailedReason represents the fact that the git
// clone, pull or checkout operations failed.
GitOperationFailedReason string = "GitOperationFailed"
)
```
## Spec examples
Pull the master branch of a public repository every minute:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 1m
url: https://github.com/stefanprodan/podinfo
```
Pull a specific branch:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 1m
url: https://github.com/stefanprodan/podinfo
ref:
branch: v3.x
```
Checkout a specific commit from a branch:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 1m
url: https://github.com/stefanprodan/podinfo
ref:
branch: master
commit: 363a6a8fe6a7f13e05d34c163b0ef02a777da20a
```
Pull a specific tag:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 1m
url: https://github.com/stefanprodan/podinfo
ref:
tag: 3.2.0
```
Pull tag based on a [semver range](https://github.com/blang/semver#ranges):
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 1m
url: https://github.com/stefanprodan/podinfo
ref:
semver: ">=3.1.0-rc.1 <3.2.0"
```
HTTPS authentication (requires a secret with `username` and `password` fields):
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
url: https://github.com/stefanprodan/podinfo
secretRef:
name: https-credentials
---
apiVersion: v1
kind: Secret
metadata:
name: https-credentials
namespace: default
type: Opaque
data:
username: <BASE64>
password: <BASE64>
```
SSH authentication (requires a secret with `identity` and `known_hosts` fields):
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
url: ssh://git@github.com/stefanprodan/podinfo
secretRef:
name: ssh-credentials
---
apiVersion: v1
kind: Secret
metadata:
name: ssh-credentials
namespace: default
type: Opaque
data:
identity: <BASE64>
identity.pub: <BASE64>
known_hosts: <BASE64>
```
> **Note:** that the SSH address does not support SCP syntax. The URL format is
> `ssh://user@host:port/org/repository`.
Example of generating the SSH credentials secret:
```bash
ssh-keygen -q -N "" -f ./identity
ssh-keyscan github.com > ./known_hosts
kubectl create secret generic ssh-credentials \
--from-file=./identity \
--from-file=./identity.pub \
--from-file=./known_hosts
```
Verify the OpenPGP signature for the commit that master branch HEAD points to:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 1m
url: https://github.com/stefanprodan/podinfo
ref:
branch: master
verify:
mode: head
secretRef:
name: pgp-public-keys
---
apiVersion: v1
kind: Secret
metadata:
name: pgp-public-keys
namespace: default
type: Opaque
data:
author1.asc: <BASE64>
author2.asc: <BASE64>
```
Example of generating the PGP public keys secret:
```bash
gpg --export --armor 3CB12BA185C47B67 > author1.asc
gpg --export --armor 6A7436E8790F8689 > author2.asc
kubectl create secret generic pgp-public-keys \
--from-file=author1.asc \
--from-file=author2.asc
```
## Status examples
Successful sync:
```yaml
status:
artifact:
lastUpdateTime: "2020-04-07T06:59:23Z"
path: /data/gitrepository/podinfo-default/363a6a8fe6a7f13e05d34c163b0ef02a777da20a.tar.gz
revision: master/363a6a8fe6a7f13e05d34c163b0ef02a777da20a
url: http://<host>/gitrepository/podinfo-default/363a6a8fe6a7f13e05d34c163b0ef02a777da20a.tar.gz
conditions:
- lastTransitionTime: "2020-04-07T06:59:23Z"
message: 'Fetched artifacts are available at
/data/gitrepository/podinfo-default/363a6a8fe6a7f13e05d34c163b0ef02a777da20a.tar.gz'
reason: GitOperationSucceed
status: "True"
type: Ready
url: http://<host>/gitrepository/podinfo-default/latest.tar.gz
```
Failed authentication:
```yaml
status:
conditions:
- lastTransitionTime: "2020-04-06T06:48:59Z"
message: 'git clone error ssh: handshake failed: ssh: unable to authenticate,
attempted methods [none publickey], no supported methods remain'
reason: AuthenticationFailed
status: "False"
type: Ready
```
Failed PGP signature verification:
```yaml
status:
conditions:
- lastTransitionTime: "2020-04-06T06:48:59Z"
message: 'PGP signature of {Stefan Prodan 2020-04-04 13:36:58 +0300 +0300} can not be verified'
reason: VerificationFailed
status: "False"
type: Ready
```
Wait for condition:
```bash
kubectl wait gitrepository/podinfo --for=condition=ready --timeout=1m
```

143
docs/spec/v1alpha1/helmcharts.md Normal file
View File

@ -0,0 +1,143 @@
# Helm Charts
The `HelmChart` API defines a source for Helm chart artifacts coming
from [`HelmRepository` sources](helmrepositories.md). The resource
exposes the latest pulled chart for the defined version as an artifact.
## Specification
Helm chart:
```go
// HelmChartSpec defines the desired state of a Helm chart source.
type HelmChartSpec struct {
// The name of the Helm chart, as made available by the referenced
// Helm repository.
// +required
Name string `json:"name"`
// The chart version semver expression, defaults to latest when
// omitted.
// +optional
Version string `json:"version,omitempty"`
// The name of the HelmRepository the chart is available at.
// +required
HelmRepositoryRef v1.LocalObjectReference `json:"helmRepositoryRef"`
// The interval at which to check the referenced HelmRepository index
// for updates.
// +required
Interval metav1.Duration `json:"interval"`
}
```
### Status
```go
// HelmChartStatus defines the observed state of the HelmChart.
type HelmChartStatus struct {
// +optional
Conditions []SourceCondition `json:"conditions,omitempty"`
// URL is the download link for the last chart fetched.
// +optional
URL string `json:"url,omitempty"`
// Artifact represents the output of the last successful chart sync.
// +optional
Artifact *Artifact `json:"artifact,omitempty"`
}
```
### Condition reasons
```go
const (
// ChartPullFailedReason represents the fact that the pull of the
// given Helm chart failed.
ChartPullFailedReason string = "ChartPullFailed"
// ChartPullSucceededReason represents the fact that the pull of
// the given Helm chart succeeded.
ChartPullSucceededReason string = "ChartPullSucceeded"
)
```
## Spec examples
Pinned version:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: HelmChart
metadata:
name: redis
namespace: default
annotations:
# force sync trigger
source.fluxcd.io/syncAt: "2020-04-06T15:39:52+03:00"
spec:
name: redis
version: 10.5.7
helmRepositoryRef:
name: stable
```
Semver range:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: HelmChart
metadata:
name: redis
namespace: default
spec:
name: redis
version: ^10.0.0
helmRepositoryRef:
name: stable
```
Interval:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: HelmChart
metadata:
name: redis
namespace: default
spec:
name: redis
version: ^10.0.0
helmRepositoryRef:
name: stable
interval: 30m
```
## Status examples
Successful chart pull:
```yaml
status:
url: http://<host>/helmcharts/redis-default/redis-10.5.7.tgz
conditions:
- lastTransitionTime: "2020-04-10T09:34:45Z"
message: Fetched artifact are available at /data/helmcharts/redis-default/redis-10.5.7.tgz
reason: ChartPullSucceeded
status: "True"
type: Ready
```
Failed chart pull:
```yaml
status:
conditions:
- lastTransitionTime: "2020-04-10T09:34:45Z"
message: ''
reason: ChartPullFailed
status: "False"
type: Ready
```

148
docs/spec/v1alpha1/helmrepositories.md Normal file
View File

@ -0,0 +1,148 @@
# Helm Repositories
The `HelmRepository` API defines a source for Helm repositories.
The resource exposes the latest synchronized repository index as
an artifact.
## Specification
Helm repository:
```go
// HelmRepositorySpec defines the reference to a Helm repository.
type HelmRepositorySpec struct {
// The Helm repository URL, a valid URL contains at least a
// protocol and host.
// +required
URL string `json:"url"`
// The name of the secret containing authentication credentials
// for the Helm repository.
// For HTTP/S basic auth the secret must contain username and password
// fields.
// For TLS the secret must contain caFile, keyFile and caCert fields.
// +optional
SecretRef *v1.LocalObjectReference `json:"secretRef,omitempty"`
// The interval at which to check the upstream for updates.
// +required
Interval metav1.Duration `json:"interval"`
}
```
### Status
```go
// HelmRepositoryStatus defines the observed state of the HelmRepository.
type HelmRepositoryStatus struct {
// +optional
Conditions []SourceCondition `json:"conditions,omitempty"`
// URL is the download link for the last index fetched.
// +optional
URL string `json:"url,omitempty"`
// Artifact represents the output of the last successful repository sync.
// +optional
Artifact *Artifact `json:"artifact,omitempty"`
}
```
### Condition reasons
```go
const (
// IndexationFailedReason represents the fact that the indexation
// of the given Helm repository failed.
IndexationFailedReason string = "IndexationFailed"
// IndexationSucceededReason represents the fact that the indexation
// of the given Helm repository succeeded.
IndexationSucceedReason string = "IndexationSucceed"
)
```
## Spec examples
Public Helm repository:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: HelmRepository
metadata:
name: stable
namespace: default
annotations:
# force sync trigger
source.fluxcd.io/syncAt: "2020-04-06T15:39:52+03:00"
spec:
url: https://kubernetes-charts.storage.googleapis.com/
interval: 1m
```
Private Helm repository:
```yaml
apiVersion: source.fluxcd.io/v1alpha1
kind: HelmRepository
metadata:
name: private
namespace: default
spec:
url: https://charts.example.com
secretRef:
name: https-credentials
interval: 1m
---
apiVersion: v1
kind: Secret
metadata:
name: https-credentials
namespace: default
type: Opaque
data:
username: <BASE64>
password: <BASE64>
certFile: <BASE64>
keyFile: <BASE64>
caFile: <BASE64>
```
## Status examples
Successful indexation:
```yaml
status:
url: http://<host>/helmrepository/podinfo-default/index.yaml
conditions:
- lastTransitionTime: "2020-04-10T09:34:45Z"
message: Fetched artifact are available at /data/helmrepositories/podinfo-default/index-21c195d78e699e4b656e2885887d019627838993.yaml
reason: IndexationSucceeded
status: "True"
type: Ready
```
Failed indexation:
```yaml
status:
conditions:
- lastTransitionTime: "2020-04-10T09:27:21Z"
message: 'failed to fetch https://invalid.example.com/index.yaml : 404 Not Found'
reason: IndexationFailed
status: "False"
type: Ready
```
Invalid repository URL:
```yaml
status:
conditions:
- lastTransitionTime: "2020-04-10T09:27:21Z"
message: scheme "invalid" not supported
reason: URLInvalid
status: "False"
type: Ready
```