diff --git a/docs/diagrams/flagger-canary-traffic-mirroring.png b/docs/diagrams/flagger-canary-traffic-mirroring.png new file mode 100644 index 00000000..dd3ab1c9 Binary files /dev/null and b/docs/diagrams/flagger-canary-traffic-mirroring.png differ diff --git a/docs/gitbook/faq.md b/docs/gitbook/faq.md index 9bf2e75b..10a7a6cc 100644 --- a/docs/gitbook/faq.md +++ b/docs/gitbook/faq.md @@ -105,9 +105,9 @@ If the analysis result is positive, Flagger will promote the canary (green versi **When can I use traffic mirroring?** Traffic Mirroring is a pre-stage in a Canary (progressive traffic shifting) or -Blue/Green deployment strategy. Traffic mirroring will copy each incoming -request, sending one request to the primary and one to the canary service. The -response from the primary is sent back to the user. The response from the canary +Blue/Green deployment strategy. Traffic mirroring will copy each incoming +request, sending one request to the primary and one to the canary service. +The response from the primary is sent back to the user. The response from the canary is discarded. Metrics are collected on both requests so that the deployment will only proceed if the canary metrics are healthy. @@ -115,12 +115,11 @@ Mirroring is supported by Istio only. In Istio, mirrored requests have `-shadow` appended to the `Host` (HTTP) or `Authority` (HTTP/2) header; for example requests to `podinfo.test` that are -mirrored will be reported in telemetry with a destination host -`podinfo.test-shadow`. +mirrored will be reported in telemetry with a destination host `podinfo.test-shadow`. Mirroring must only be used for requests that are **idempotent** or capable of -being processed twice (once by the primary and once by the canary). Reads are -idempotent. Before using mirroring on requests that may be writes, you should +being processed twice (once by the primary and once by the canary). Reads are +idempotent. Before using mirroring on requests that may be writes, you should consider what will happen if a write is duplicated and handled by the primary and canary. diff --git a/docs/gitbook/usage/progressive-delivery.md b/docs/gitbook/usage/progressive-delivery.md index d2ae025f..9d0bfcdc 100644 --- a/docs/gitbook/usage/progressive-delivery.md +++ b/docs/gitbook/usage/progressive-delivery.md @@ -2,8 +2,14 @@ This guide shows you how to use Istio and Flagger to automate canary deployments. +![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png) + ### Bootstrap +Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), +then creates a series of objects (Kubernetes deployments, ClusterIP services, Istio destination rules and virtual services). +These objects expose the application inside the mesh and drive the canary analysis and promotion. + Create a test namespace with Istio sidecar injection enabled: ```bash @@ -57,6 +63,11 @@ spec: # Istio virtual service host names (optional) hosts: - app.example.com + # Istio traffic policy (optional) + trafficPolicy: + tls: + # use ISTIO_MUTUAL when mTLS is enabled + mode: DISABLE canaryAnalysis: # schedule interval (default 60s) interval: 1m @@ -164,6 +175,11 @@ Events: **Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis. +A canary deployment is triggered by changes in any of the following objects: +* Deployment PodSpec (container image, command, ports, env, resources, etc) +* ConfigMaps mounted as volumes or mapped to environment variables +* Secrets mounted as volumes or mapped to environment variables + You can monitor all canaries with: ```bash @@ -229,3 +245,88 @@ Events: Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10 Warning Synced 1m flagger Canary failed! Scaling down podinfo.test ``` + +### Traffic mirroring + +![Flagger Canary Traffic Shadowing](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-traffic-mirroring.png) + +For applications that perform read operations, Flagger can be configured to drive a canary release with traffic mirroring. +Istio traffic mirroring will copy each incoming request, sending one request to the primary and one to the canary service. +The response from the primary is sent back to the user and the response from the canary is discarded. +Metrics are collected on both requests so that the deployment will only proceed if the canary metrics are healthy. + +Note that mirroring should be used for requests that are **idempotent** or capable of being processed twice +(once by the primary and once by the canary). + +You can enable mirroring by replacing `stepWeight/maxWeight` with `iterations` and +by setting `canaryAnalysis.mirror` to `true`: + +```yaml +apiVersion: flagger.app/v1alpha3 +kind: Canary +metadata: + name: podinfo + namespace: test +spec: + canaryAnalysis: + # schedule interval + interval: 1m + # max number of failed metric checks before rollback + threshold: 5 + # total number of iterations + iterations: 10 + # enable traffic shadowing + mirror: true + metrics: + - name: request-success-rate + threshold: 99 + interval: 1m + - name: request-duration + threshold: 500 + interval: 1m + webhooks: + - name: acceptance-test + type: pre-rollout + url: http://flagger-loadtester.test/ + timeout: 30s + metadata: + type: bash + cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token" + - name: load-test + url: http://flagger-loadtester.test/ + timeout: 5s + metadata: + cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/" +``` + +With the above configuration, Flagger will run a canary release with the following steps: +* detect new revision (deployment spec, secrets or configmaps changes) +* scale from zero the canary deployment +* wait for the HPA to set the canary minimum replicas +* check canary pods health +* run the acceptance tests +* abort the canary release if tests fail +* start the load tests +* mirror traffic from primary to canary +* check request success rate and request duration every minute +* abort the canary release if the metrics check failure threshold is reached +* stop traffic mirroring after the number of iterations is reached +* route live traffic to the canary pods +* promote the canary (update the primary secrets, configmaps and deployment spec) +* wait for the primary deployment rollout to finish +* wait for the HPA to set the primary minimum replicas +* check primary pods health +* switch live traffic back to primary +* scale to zero the canary +* send notification with the canary analysis result + +The above procedure can be extended with [custom metrics](https://docs.flagger.app/how-it-works#custom-metrics) checks, +[webhooks](https://docs.flagger.app/how-it-works#webhooks), +[manual promotion](https://docs.flagger.app/how-it-works#manual-gating) approval and +[Slack or MS Teams](https://docs.flagger.app/usage/alerting) notifications. + + + + + + diff --git a/pkg/controller/scheduler.go b/pkg/controller/scheduler.go index fa00b39c..7d9b909c 100644 --- a/pkg/controller/scheduler.go +++ b/pkg/controller/scheduler.go @@ -380,7 +380,7 @@ func (c *Controller) advanceCanary(name string, namespace string, skipLivenessCh c.recordEventWarningf(cd, "%v", err) } c.logger.With("canary", fmt.Sprintf("%s.%s", name, namespace)). - Infof("Enabling mirroring for Blue/Green") + Infof("Start traffic mirroring") } if err := c.deployer.SetStatusIterations(cd, cd.Status.Iterations+1); err != nil { c.recordEventWarningf(cd, "%v", err) @@ -399,7 +399,11 @@ func (c *Controller) advanceCanary(name string, namespace string, skipLivenessCh // route all traffic to canary - max iterations reached if cd.Spec.CanaryAnalysis.Iterations == cd.Status.Iterations { if provider != "kubernetes" { - c.recordEventInfof(cd, "Routing all traffic to canary") + if cd.Spec.CanaryAnalysis.Mirror { + c.recordEventInfof(cd, "Stop traffic mirroring and route all traffic to canary") + } else { + c.recordEventInfof(cd, "Routing all traffic to canary") + } if err := meshRouter.SetRoutes(cd, 0, 100, false); err != nil { c.recordEventWarningf(cd, "%v", err) return