diff --git a/docs/spec/README.md b/docs/spec/README.md new file mode 100644 index 00000000..4e43ca52 --- /dev/null +++ b/docs/spec/README.md @@ -0,0 +1,52 @@ +# Source Controller Proposal + +## Context + +The desired state of a cluster is made out of Kubernetes objects, these objects are expressed in `.yaml` format and +are applied on the cluster by operators running inside the cluster. An operator's role is to fetch the Kubernetes +objects, run transformations on them and reconcile the cluster state with the resulting manifest. + +For an operator to acquire the resources that make up the desired state it needs to understand the communication +protocol and the authentication scheme, verify the authenticity of a source and deal with rate limits and retries. +In the FluxCD organization there are currently two operators that perform such operations. Both Flux and +Helm Operator connect to Git repositories to fetch Kubernetes objects, they need to maintain an up-to-date mirror +of one or several repos. Besides Git, Helm Operator needs to connect to Helm repositories hosted on public or +private HTTPS servers. + +## Motivation + +Each Flux or Helm Operator instance maintains its own Git repository mirror even if all of them +point to the same source. If the Git repository host becomes unavailable, the cluster state will diverge from the last +know desired state since the operators will stop the reconciliation due to pull errors. + +Decoupling the Kubernetes objects acquisition from the reconciliation process with an in-cluster +source manager would make Flux and Helm Operator resilient to outbound connectivity issues and would +simplify the state machine(s) that these controllers operate. + +Managing the source operations in a dedicated controller could allow Flux to compose the desire state of a cluster +from multiple source. +Further more the manifests transformation process could be performed by 3rd party tools +(e.g. kustomize, jk, tanka, cue run by Tekton pipelines or Kubernetes Jobs) +that would output the final state of the Kubernetes objects into an artifacts repository maintained by the source controller. + +## Goals + +The main goal is to define a set of Kubernetes objects that cluster admins and various automated operators +can interact with to offload the sources (e.g. Git and Helm repositories) +registration, authentication and resource fetching to a dedicated controller. + +The controller implementation will watch for source objects in a cluster and act on them. +The actions performed by the source controller could be: +* validate source definitions +* authenticate to sources and validate authenticity +* detect source changes based on update policies (semver) +* fetch resources on-demand and on-a-schedule +* package the fetched resources into a well known format (tar.gz) +* store the artifacts locally +* make the artifacts addressable by their source identifier (sha, version, ts) +* make the artifacts available in-cluster to interested 3rd parties +* notify interested 3rd parties of source changes and availability (status conditions, events, hooks) + +## API Specification + +* [v1alpha1](v1alpha1/README.md) diff --git a/docs/spec/v1alpha1/README.md b/docs/spec/v1alpha1/README.md new file mode 100644 index 00000000..23491974 --- /dev/null +++ b/docs/spec/v1alpha1/README.md @@ -0,0 +1,10 @@ +# source.fluxcd.io/v1alpha1 + +The is the v1alpha1 API specification for defining the desired state sources of Kubernetes clusters. + +Source kinds: +* [GitRepository](gitrepositories.md) +* [HelmRepository](helmrepositories.md) + +Implementations: +* source-controller [v0.0.1-alpha.1](https://github.com/fluxcd/source-controller/releases) diff --git a/docs/spec/v1alpha1/gitrepositories.md b/docs/spec/v1alpha1/gitrepositories.md new file mode 100644 index 00000000..eb8004e8 --- /dev/null +++ b/docs/spec/v1alpha1/gitrepositories.md @@ -0,0 +1,212 @@ +# Git Repositories + +The `GitReposiory` API defines a source for artifacts coming from Git. + +## Specification + +Git repository spec: + +```go +// GitRepositorySpec defines the desired state of GitRepository +type GitRepositorySpec struct { + // +kubebuilder:validation:Pattern="^(http|https|ssh)://" + + // The repository URL, can be a HTTP or SSH address. + Url string `json:"url"` + + // The secret name containing the Git credentials + // +optional + SecretRef *v1.LocalObjectReference `json:"secretRef,omitempty"` + + // The git reference to checkout and monitor for changes. + // +optional + Reference *GitRepositoryRef `json:"ref,omitempty"` + + // The interval at which to check for repository updates. + Interval metav1.Duration `json:"interval"` +} + +// GitRepositoryRef defines the git ref used for pull and checkout operations +type GitRepositoryRef struct { + // The git branch to checkout, defaults to ('master'). + // +optional + Branch string `json:"branch"` + + // The git tag to checkout, takes precedence over branch. + // +optional + Tag string `json:"tag"` + + // The git tag semver expression, takes precedence over tag. + // +optional + SemVer string `json:"semver"` +} +``` + +Git repository status: + +```go +// GitRepositoryStatus defines the observed state of GitRepository +type GitRepositoryStatus struct { + // +optional + Conditions []RepositoryCondition `json:"conditions,omitempty"` + + // LastUpdateTime is the timestamp corresponding to the last status + // change of this repository. + // +optional + LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"` + + // URI for the artifacts of the last successful repository sync. + // +optional + Artifacts string `json:"artifacts,omitempty"` +} +``` + +Git repository status conditions: + +```go +// RepositoryCondition contains condition information for a repository +type RepositoryCondition struct { + // Type of the condition, currently ('Ready'). + Type RepositoryConditionType `json:"type"` + + // Status of the condition, one of ('True', 'False', 'Unknown'). + Status corev1.ConditionStatus `json:"status"` + + // LastTransitionTime is the timestamp corresponding to the last status + // change of this condition. + // +optional + LastTransitionTime *metav1.Time `json:"lastTransitionTime,omitempty"` + + // Reason is a brief machine readable explanation for the condition's last + // transition. + // +optional + Reason string `json:"reason,omitempty"` + + // Message is a human readable description of the details of the last + // transition, complementing reason. + // +optional + Message string `json:"message,omitempty"` +} + +// RepositoryConditionType represents an repository condition value +type RepositoryConditionType string + +const ( + // RepositoryConditionReady represents the fact that a given repository condition + // is in ready state. + RepositoryConditionReady RepositoryConditionType = "Ready" +) +``` + +## Spec examples + +Public repository: + +```yaml +apiVersion: source.fluxcd.io/v1alpha1 +kind: GitRepository +metadata: + name: podinfo + namespace: default + annotations: + # force sync trigger + source.fluxcd.io/syncAt: "2020-04-06T15:39:52+03:00" +spec: + interval: 1m + url: https://github.com/stefanprodan/podinfo + ref: + branch: master + tag: "3.2.0" + semver: ">= 3.2.0 <3.3.0" +``` + +HTTPS authentication: + +```yaml +apiVersion: source.fluxcd.io/v1alpha1 +kind: GitRepository +metadata: + name: podinfo + namespace: default +spec: + url: https://github.com/stefanprodan/podinfo + secretRef: + name: https-credentials +--- +apiVersion: v1 +kind: Secret +metadata: + name: https-credentials + namespace: default +type: Opaque +data: + username: + password: +``` + +SSH authentication: + +```yaml +apiVersion: source.fluxcd.io/v1alpha1 +kind: GitRepository +metadata: + name: podinfo + namespace: default +spec: + url: ssh://git@github.com:stefanprodan/podinfo + secretRef: + name: ssh-credentials +--- +apiVersion: v1 +kind: Secret +metadata: + name: ssh-credentials + namespace: default +type: Opaque +data: + identity: + identity.pub: + know_hosts: +``` + +Example of generating the SSH credentials secret: + +```bash +ssh-keygen -q -N "" -f ./identity +ssh-keyscan github.com > ./know_hosts + +kubectl create secret generic ssh-credentials \ + --from-file=./identity \ + --from-file=./identity.pub \ + --from-file=./know_hosts +``` + +## Status examples + +Successful sync: + +```yaml +status: + artifacts: http://source-controller.source-system/repositories/podinfo-default/5e747d3e088cd7a34ace4abc8cf7f3c3696e402f.tar.gz + conditions: + - lastTransitionTime: "2020-04-07T06:59:23Z" + message: 'Fetched artifacts are available at + /data/repositories/podinfo-default/5e747d3e088cd7a34ace4abc8cf7f3c3696e402f.tar.gz' + reason: GitCloneSucceed + status: "True" + type: Ready + lastUpdateTime: "2020-04-07T06:59:23Z" +``` + +Failed sync: + +```yaml +status: + conditions: + - lastTransitionTime: "2020-04-06T06:48:59Z" + message: 'git clone error ssh: handshake failed: ssh: unable to authenticate, + attempted methods [none publickey], no supported methods remain' + reason: GitCloneFailed + status: "False" + type: Ready +``` diff --git a/docs/spec/v1alpha1/helmrepositories.md b/docs/spec/v1alpha1/helmrepositories.md new file mode 100644 index 00000000..4e71d089 --- /dev/null +++ b/docs/spec/v1alpha1/helmrepositories.md @@ -0,0 +1,15 @@ +# Helm Repositories + +The `HelmReposiory` and `HelmChart` API defines a source for artifacts coming from Helm repositories. + +## Specification + +TODO + +## Spec examples + +TODO + +## Status examples + +TODO \ No newline at end of file