Merge pull request #448 from weaveworks/docs-dev

docs: Add development and release guide

.gitbook.yaml

@@ -1 +1,12 @@
-root: ./docs/gitbook
+root: ./docs/gitbook
+
+redirects:
+  usage/progressive-delivery: tutorials/istio-progressive-delivery.md
+  usage/ab-testing: tutorials/istio-ab-testing.md
+  usage/blue-green: tutorials/kubernetes-blue-green.md
+  usage/appmesh-progressive-delivery: tutorials/appmesh-progressive-delivery.md
+  usage/linkerd-progressive-delivery: tutorials/linkerd-progressive-delivery.md
+  usage/contour-progressive-delivery: tutorials/contour-progressive-delivery.md
+  usage/gloo-progressive-delivery: tutorials/gloo-progressive-delivery.md
+  usage/nginx-progressive-delivery: tutorials/nginx-progressive-delivery.md
+  usage/crossover-progressive-delivery: tutorials/crossover-progressive-delivery.md

README.md

@@ -25,7 +25,6 @@ Flagger documentation can be found at [docs.flagger.app](https://docs.flagger.ap
   * [Flagger install on Kubernetes](https://docs.flagger.app/install/flagger-install-on-kubernetes)
   * [Flagger install on GKE Istio](https://docs.flagger.app/install/flagger-install-on-google-cloud)
   * [Flagger install on EKS App Mesh](https://docs.flagger.app/install/flagger-install-on-eks-appmesh)
-  * [Flagger install with SuperGloo](https://docs.flagger.app/install/flagger-install-with-supergloo)
 * How it works
   * [Canary custom resource](https://docs.flagger.app/how-it-works#canary-custom-resource)
   * [Routing](https://docs.flagger.app/how-it-works#istio-routing)
@@ -37,18 +36,20 @@ Flagger documentation can be found at [docs.flagger.app](https://docs.flagger.ap
   * [Load testing](https://docs.flagger.app/how-it-works#load-testing)
   * [Manual gating](https://docs.flagger.app/how-it-works#manual-gating)
   * [FAQ](https://docs.flagger.app/faq)
+  * [Development guide](https://docs.flagger.app/dev-guide)
 * Usage
-  * [Istio canary deployments](https://docs.flagger.app/usage/progressive-delivery)
-  * [Linkerd canary deployments](https://docs.flagger.app/usage/linkerd-progressive-delivery)
-  * [App Mesh canary deployments](https://docs.flagger.app/usage/appmesh-progressive-delivery)
-  * [NGINX ingress controller canary deployments](https://docs.flagger.app/usage/nginx-progressive-delivery)
-  * [Gloo ingress controller canary deployments](https://docs.flagger.app/usage/gloo-progressive-delivery)
-  * [Contour Canary Deployments](https://docs.flagger.app/usage/contour-progressive-delivery)
-  * [Crossover canary deployments](https://docs.flagger.app/usage/crossover-progressive-delivery)
-  * [Blue/Green deployments](https://docs.flagger.app/usage/blue-green)
+  * [Deployment Strategies](https://docs.flagger.app/usage/deployment-strategies)
   * [Monitoring](https://docs.flagger.app/usage/monitoring)
   * [Alerting](https://docs.flagger.app/usage/alerting)
 * Tutorials
+  * [Istio Canary Deployments](https://docs.flagger.app/tutorials/istio-progressive-delivery)
+  * [Istio A/B Testing](https://docs.flagger.app/tutorials/istio-ab-testing)
+  * [Linkerd Canary Deployments](https://docs.flagger.app/tutorials/linkerd-progressive-delivery)
+  * [App Mesh Canary Deployments](https://docs.flagger.app/tutorials/appmesh-progressive-delivery)
+  * [NGINX Canary Deployments](https://docs.flagger.app/tutorials/nginx-progressive-delivery)
+  * [Gloo Canary Deployments](https://docs.flagger.app/tutorials/gloo-progressive-delivery)
+  * [Contour Canary Deployments](https://docs.flagger.app/tutorials/contour-progressive-delivery)
+  * [Kubernetes Blue/Green Deployments](https://docs.flagger.app/tutorials/kubernetes-blue-green)
   * [Canary deployments with Helm charts and Weave Flux](https://docs.flagger.app/tutorials/canary-helm-gitops)
 
 ## Canary CRD

docs/gitbook/SUMMARY.md

@@ -3,6 +3,7 @@
 * [Introduction](README.md)
 * [How it works](how-it-works.md)
 * [FAQ](faq.md)
+* [Development guide](dev-guide.md)
 
 ## Install
 
@@ -13,20 +14,21 @@
 
 ## Usage
 
-* [Istio Canary Deployments](usage/progressive-delivery.md)
-* [Istio A/B Testing](usage/ab-testing.md)
-* [Linkerd Canary Deployments](usage/linkerd-progressive-delivery.md)
-* [App Mesh Canary Deployments](usage/appmesh-progressive-delivery.md)
-* [NGINX Canary Deployments](usage/nginx-progressive-delivery.md)
-* [Gloo Canary Deployments](usage/gloo-progressive-delivery.md)
-* [Contour Canary Deployments](usage/contour-progressive-delivery.md)
-* [Crossover Canary Deployments](usage/crossover-progressive-delivery.md)
-* [Blue/Green Deployments](usage/blue-green.md)
-* [Monitoring](usage/monitoring.md)
+* [Deployment Strategies](usage/deployment-strategies.md)
 * [Alerting](usage/alerting.md)
+* [Monitoring](usage/monitoring.md)
 
 ## Tutorials
 
+* [Istio Canary Deployments](tutorials/istio-progressive-delivery.md)
+* [Istio A/B Testing](tutorials/istio-ab-testing.md)
+* [Linkerd Canary Deployments](tutorials/linkerd-progressive-delivery.md)
+* [App Mesh Canary Deployments](tutorials/appmesh-progressive-delivery.md)
+* [NGINX Canary Deployments](tutorials/nginx-progressive-delivery.md)
+* [Gloo Canary Deployments](tutorials/gloo-progressive-delivery.md)
+* [Contour Canary Deployments](tutorials/contour-progressive-delivery.md)
+* [Blue/Green Deployments](tutorials/kubernetes-blue-green.md)
+* [Crossover Canary Deployments](tutorials/crossover-progressive-delivery.md)
 * [SMI Istio Canary Deployments](tutorials/flagger-smi-istio.md)
 * [Canaries with Helm charts and GitOps](tutorials/canary-helm-gitops.md)
 * [Zero downtime deployments](tutorials/zero-downtime-deployments.md)

docs/gitbook/dev-guide.md

@@ -0,0 +1,182 @@
+# Flagger Development Guide
+
+This document describes how to build, test and run Flagger from source.
+
+### Setup dev environment
+
+Flagger is written in Go and uses Go modules for dependency management.
+
+On your dev machine install the following tools:
+* go >= 1.13
+* git >= 2.20
+* bash >= 5.0
+* make >= 3.81
+* kubectl >= 1.16
+* kustomize >= 3.5
+* helm >= 3.0
+* docker >= 19.03
+
+You'll also need a Kubernetes cluster for testing Flagger.
+You can use Minikube, Kind, Docker desktop or any remote cluster
+(AKS/EKS/GKE/etc) Kubernetes version 1.14 or newer.
+
+### Build
+
+To start contributing to Flagger, fork the repository and clone it locally:
+
+```bash
+git clone https://github.com/<YOUR-USERNAME>/flagger
+cd flagger
+```
+
+Download Go modules:
+
+```bash
+go mod download
+```
+
+Build Flagger binary:
+
+```bash
+CGO_ENABLED=0 go build -o ./bin/flagger ./cmd/flagger/
+```
+
+Build Flagger container image:
+
+```bash
+make build
+```
+
+### Unit testing
+
+Make a change to the source code and run the linter and unit tests:
+
+```bash
+make test
+```
+
+If you made changes to `go.mod` run:
+
+```bash
+go mod tidy
+```
+
+If you made changes to `pkg/apis` regenerate Kubernetes client sets with:
+
+```bash
+./hack/update-codegen.sh
+```
+
+### Manual testing
+
+Install a service mesh and/or an ingress controller on your cluster and deploy Flagger
+using one of the install options [listed here](https://docs.flagger.app/install/flagger-install-on-kubernetes).
+
+If you made changes to the CRDs, apply your local copy with:
+
+```bash
+kubectl apply -f artifacts/flagger/crd.yaml
+```
+
+Shutdown the Flagger instance installed on your cluster (replace the namespace with your mesh/ingress one):
+
+```bash
+kubectl -n istio-system scale deployment/flagger --replicas=0
+```
+
+Port forward to your Prometheus instance:
+
+```bash
+kubectl -n istio-system port-forward svc/prometheus 9090:9090
+```
+
+Run Flagger locally against your remote cluster by specifying a kubeconfig path:
+
+```bash
+go run cmd/flagger/ -kubeconfig=$HOME/.kube/config \
+-log-level=info \
+-mesh-provider=istio \
+-metrics-server=http://localhost:9090
+```
+
+Another option to manually test your changes is to build and push the image to your container registry:
+
+```bash
+make build
+docker tag weaveworks/flagger:latest <YOUR-DOCKERHUB-USERNAME>/flagger:<YOUR-TAG>
+docker push <YOUR-DOCKERHUB-USERNAME>/flagger:<YOUR-TAG>
+```
+
+Deploy your image on the cluster and scale up Flagger:
+
+```bash
+kubectl -n istio-system set image deployment/flagger flagger=<YOUR-DOCKERHUB-USERNAME>/flagger:<YOUR-TAG>
+kubectl -n istio-system scale deployment/flagger --replicas=1
+```
+
+Now you can use one of the [tutorials]() to manually test your changes.
+
+### Integration testing
+
+Flagger end-to-end tests can be run locally with [Kubernetes Kind](https://github.com/kubernetes-sigs/kind).
+
+Create a Kind cluster:
+
+```bash
+kind create cluster
+```
+
+Install a service mesh and/or an ingress controller in Kind.
+
+Linkerd example:
+
+```bash
+linkerd install | kubectl apply -f -
+linkerd check
+```
+
+Build Flagger container image and load it on the cluster:
+
+```bash
+make build
+docker tag weaveworks/flagger:latest test/flagger:latest
+kind load docker-image test/flagger:latest
+```
+
+Install Flagger on the cluster and set the test image:
+
+```bash
+kubectl apply -k ./kustomize/linkerd
+kubectl -n linkerd set image deployment/flagger flagger=test/flagger:latest
+kubectl -n linkerd rollout status deployment/flagger
+```
+
+Run the Linkerd e2e tests:
+
+```bash
+./test/e2e-linkerd-tests.sh
+```
+
+For each service mesh and ingress controller there is a dedicated e2e test suite,
+chose one that matches your changes from this [list](https://github.com/weaveworks/flagger/tree/master/test).
+
+When you open a pull request on Flagger repo, the unit and integration tests will be run in CI.
+
+### Release
+
+To release a new Flagger version (e.g. `2.0.0`) follow these steps:
+* create a branch `git checkout -b prep-2.0.0`
+* set the version in code and manifests `TAG=2.0.0 make version-set`
+* commit changes and merge PR
+* checkout master `git checkout master && git pull`
+* tag master `make release`
+
+After the tag has been pushed to GitHub, the CI release pipeline does the following:
+* creates a GitHub release
+* pushes the Flagger binary and change log to GitHub release
+* pushes the Flagger container image to Docker Hub
+* pushes the Helm chart to github-pages branch
+* GitHub pages publishes the new chart version on the Helm repository
+
+
+

docs/gitbook/usage/alerting.md

@@ -53,55 +53,3 @@ Besides Slack, you can use Alertmanager to trigger alerts when a canary deployme
       description: "Workload {{ $labels.name }} namespace {{ $labels.namespace }}"
 ```
 
-### Event Webhook
-
-Flagger can be configured to send event payloads to a specified webhook:
-
-```bash
-helm upgrade -i flagger flagger/flagger \
---set eventWebhook=https://example.com/flagger-canary-event-webhook
-```
-
-The environment variable *EVENT_WEBHOOK_URL* can be used for activating the event-webhook, too.
-This is handy for using a secret to store a sensible value that could contain api keys for example.
-
-When configured, every action that Flagger takes during a canary deployment will be sent as JSON via an HTTP POST
-request. The JSON payload has the following schema:
-
-```json
-{
-  "name": "string (canary name)",
-  "namespace": "string (canary namespace)",
-  "phase": "string (canary phase)",
-  "metadata": {
-    "eventMessage": "string (canary event message)",
-    "eventType": "string (canary event type)",
-    "timestamp": "string (unix timestamp ms)"
-  }
-}
-```
-
-Example:
-
-```json
-{
-  "name": "podinfo",
-  "namespace": "default",
-  "phase": "Progressing",
-  "metadata": {
-    "eventMessage": "New revision detected! Scaling up podinfo.default",
-    "eventType": "Normal",
-    "timestamp": "1578607635167"
-  }
-}
-```
-
-The event webhook can be overwritten at canary level with:
-
-```yaml
-  canaryAnalysis:
-    webhooks:
-      - name: "send to Slack"
-        type: event
-        url: http://event-recevier.notifications/slack
-```

docs/gitbook/usage/deployment-strategies.md

@@ -0,0 +1,235 @@
+# Deployment Strategies
+
+Flagger can run automated application analysis, promotion and rollback for the following deployment strategies:
+* Canary release (progressive traffic shifting)
+    * Istio, Linkerd, App Mesh, NGINX, Contour, Gloo
+* A/B Testing (HTTP headers and cookies traffic routing)
+    * Istio, App Mesh, NGINX, Contour
+* Blue/Green (traffic switch)
+    * Kubernetes CNI, Istio, Linkerd, App Mesh, NGINX, Contour, Gloo
+* Blue/Green (traffic mirroring)
+    * Istio
+
+For Canary releases and A/B testing you'll need a Layer 7 traffic management solution like a service mesh or an ingress controller.
+For Blue/Green deployments no service mesh or ingress controller is required.
+
+A canary analysis is triggered by changes in any of the following objects:
+
+* Deployment PodSpec (container image, command, ports, env, resources, etc)
+* ConfigMaps mounted as volumes or mapped to environment variables
+* Secrets mounted as volumes or mapped to environment variables
+
+### Canary Release
+
+Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance
+indicators like HTTP requests success rate, requests average duration and pod health.
+Based on analysis of the KPIs a canary is promoted or aborted.
+
+![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
+
+The canary analysis runs periodically until it reaches the maximum traffic weight or the failed checks threshold. 
+
+Spec:
+
+```yaml
+  canaryAnalysis:
+    # schedule interval (default 60s)
+    interval: 1m
+    # max number of failed metric checks before rollback
+    threshold: 10
+    # max traffic percentage routed to canary
+    # percentage (0-100)
+    maxWeight: 50
+    # canary increment step
+    # percentage (0-100)
+    stepWeight: 2
+  # deploy straight to production without
+  # the metrics and webhook checks
+  skipAnalysis: false
+```
+
+The above analysis, if it succeeds, will run for 25 minutes while validating the HTTP metrics and webhooks every minute.
+You can determine the minimum time that it takes to validate and promote a canary deployment using this formula:
+
+```
+interval * (maxWeight / stepWeight)
+```
+
+And the time it takes for a canary to be rollback when the metrics or webhook checks are failing:
+
+```
+interval * threshold 
+```
+
+In emergency cases, you may want to skip the analysis phase and ship changes directly to production. 
+At any time you can set the `spec.skipAnalysis: true`. 
+When skip analysis is enabled, Flagger checks if the canary deployment is healthy and 
+promotes it without analysing it. If an analysis is underway, Flagger cancels it and runs the promotion.
+
+### A/B Testing
+
+For frontend applications that require session affinity you should use HTTP headers or cookies match conditions
+to ensure a set of users will stay on the same version for the whole duration of the canary analysis.
+
+![Flagger A/B Testing Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-abtest-steps.png)
+
+You can enable A/B testing by specifying the HTTP match conditions and the number of iterations.
+If Flagger finds a HTTP match condition, it will ignore the `maxWeight` and `stepWeight` settings.
+
+Istio example:
+
+```yaml
+  canaryAnalysis:
+    # schedule interval (default 60s)
+    interval: 1m
+    # total number of iterations
+    iterations: 10
+    # max number of failed iterations before rollback
+    threshold: 2
+    # canary match condition
+    match:
+      - headers:
+          x-canary:
+            regex: ".*insider.*"
+      - headers:
+          cookie:
+            regex: "^(.*?;)?(canary=always)(;.*)?$"
+```
+
+The above configuration will run an analysis for ten minutes targeting the Safari users and those that have a test cookie.
+You can determine the minimum time that it takes to validate and promote a canary deployment using this formula:
+
+```
+interval * iterations
+```
+
+And the time it takes for a canary to be rollback when the metrics or webhook checks are failing:
+
+```
+interval * threshold 
+```
+
+App Mesh example:
+
+```yaml
+  canaryAnalysis:
+    interval: 1m
+    threshold: 10
+    iterations: 2
+    match:
+      - headers:
+          user-agent:
+            regex: ".*Chrome.*"
+```
+
+Note that App Mesh supports a single condition.
+
+Contour example:
+
+```yaml
+  canaryAnalysis:
+    interval: 1m
+    threshold: 10
+    iterations: 2
+    match:
+      - headers:
+          user-agent:
+            prefix: "Chrome"
+```
+
+Note that Contour does not support regex, you can use prefix, suffix or exact.
+
+NGINX example:
+
+```yaml
+  canaryAnalysis:
+    interval: 1m
+    threshold: 10
+    iterations: 2
+    match:
+      - headers:
+          x-canary:
+            exact: "insider"
+      - headers:
+          cookie:
+            exact: "canary"
+```
+
+Note that the NGINX ingress controller supports only exact matching for a single header and the cookie value is set to `always`.
+
+The above configurations will route users with the x-canary header or canary cookie to the canary instance during analysis:
+
+```bash
+curl -H 'X-Canary: insider' http://app.example.com
+curl -b 'canary=always' http://app.example.com
+```
+
+### Blue/Green Deployments
+
+For applications that are not deployed on a service mesh, Flagger can orchestrate blue/green style deployments 
+with Kubernetes L4 networking. When using Istio you have the option to mirror traffic between blue and green.
+
+![Flagger Blue/Green Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-bluegreen-steps.png)
+
+You can use the blue/green deployment strategy by replacing `stepWeight/maxWeight` with `iterations` in the `canaryAnalysis` spec:
+
+```yaml
+  canaryAnalysis:
+    # schedule interval (default 60s)
+    interval: 1m
+    # total number of iterations
+    iterations: 10
+    # max number of failed iterations before rollback
+    threshold: 2
+```
+
+With the above configuration Flagger will run conformance and load tests on the canary pods for ten minutes. 
+If the metrics analysis succeeds, live traffic will be switched from the old version to the new one when the
+canary is promoted.
+
+The blue/green deployment strategy is supported for all service mesh providers.
+
+Blue/Green rollout steps for service mesh:
+* scale up the canary (green)
+* run conformance tests for the canary pods
+* run load tests and metric checks for the canary pods
+* route traffic to canary
+* promote canary spec over primary (blue)
+* wait for primary rollout
+* route traffic to primary
+* scale down canary
+
+After the analysis finishes, the traffic is routed to the canary (green) before triggering the primary (blue)
+rolling update, this ensures a smooth transition to the new version avoiding dropping in-flight requests during
+the Kubernetes deployment rollout.
+
+### Blue/Green with Traffic Mirroring
+
+Traffic Mirroring is a pre-stage in a Canary (progressive traffic shifting) or
+Blue/Green deployment strategy. Traffic mirroring will copy each incoming
+request, sending one request to the primary and one to the canary service.
+The response from the primary is sent back to the user. The response from the canary
+is discarded.  Metrics are collected on both requests so that the deployment will
+only proceed if the canary metrics are healthy.
+
+Mirroring must only be used for requests that are **idempotent** or capable of
+being processed twice (once by the primary and once by the canary). Reads are
+idempotent. Before using mirroring on requests that may be writes, you should
+consider what will happen if a write is duplicated and handled by the primary
+and canary.
+
+To use mirroring, set `spec.canaryAnalysis.mirror` to `true`. 
+
+Istio example:
+
+```yaml
+  canaryAnalysis:
+    # schedule interval (default 60s)
+    interval: 1m
+    # total number of iterations
+    iterations: 10
+    # max number of failed iterations before rollback
+    threshold: 2
+    # Traffic shadowing (compatible with Istio only)
+    mirror: true
+```

docs/gitbook/usage/monitoring.md

@@ -40,6 +40,59 @@ Scaling down podinfo.test
 Promotion completed! podinfo.test
 ```
 
+### Event Webhook
+
+Flagger can be configured to send event payloads to a specified webhook:
+
+```bash
+helm upgrade -i flagger flagger/flagger \
+--set eventWebhook=https://example.com/flagger-canary-event-webhook
+```
+
+The environment variable *EVENT_WEBHOOK_URL* can be used for activating the event-webhook, too.
+This is handy for using a secret to store a sensible value that could contain api keys for example.
+
+When configured, every action that Flagger takes during a canary deployment will be sent as JSON via an HTTP POST
+request. The JSON payload has the following schema:
+
+```json
+{
+  "name": "string (canary name)",
+  "namespace": "string (canary namespace)",
+  "phase": "string (canary phase)",
+  "metadata": {
+    "eventMessage": "string (canary event message)",
+    "eventType": "string (canary event type)",
+    "timestamp": "string (unix timestamp ms)"
+  }
+}
+```
+
+Example:
+
+```json
+{
+  "name": "podinfo",
+  "namespace": "default",
+  "phase": "Progressing",
+  "metadata": {
+    "eventMessage": "New revision detected! Scaling up podinfo.default",
+    "eventType": "Normal",
+    "timestamp": "1578607635167"
+  }
+}
+```
+
+The event webhook can be overwritten at canary level with:
+
+```yaml
+  canaryAnalysis:
+    webhooks:
+      - name: "send to Slack"
+        type: event
+        url: http://event-recevier.notifications/slack
+```
+
 ### Metrics
 
 Flagger exposes Prometheus metrics that can be used to determine the canary analysis status and