Merge pull request #7292 from FairwindsOps/update-the-hammer

Adding documentation to mitigate workload outages on AWS

docs/etcd3-migration.md

@@ -9,10 +9,11 @@
     - [Adopt etcd-manager with kops 1.11 / kubernetes 1.11](#adopt-etcd-manager-with-kops-111--kubernetes-111)
     - [Delay adopting etcd-manager with kops 1.12](#delay-adopting-etcd-manager-with-kops-112)
     - [Delay adopting etcd3 with kops 1.12](#delay-adopting-etcd3-with-kops-112)
+  - [Mitigating Workload Downtime](#mitigating-workload-downtime)
 
 ## Background Info
 
-kubernetes is moving from etcd2 to etcd3, which is an upgrade that involves
+Kubernetes is moving from etcd2 to etcd3, which is an upgrade that involves Kubernetes API Server
 downtime. Technically there is no usable upgrade path from etcd2 to etcd3 that
 supports HA scenarios, but kops has enabled it using etcd-manager.
 
@@ -50,6 +51,9 @@ When upgrading to kubernetes 1.12 with kops 1.12, by default:
 
 The upgrade is therefore disruptive to the masters.  The recommended procedure is to quickly roll your masters, and then do a normal roll of your nodes:
 
+> **DANGER:** Using the procedure to quickly roll your masters can result in downtime for any workloads using Service LoadBalancers. (The "Hammer 🔨" Method)  
+> Any time you kill off all three masters with `--cloudonly` and `--master-interval=1s`, you may experience worker nodes go into a `NotReady` state when the new masters come online and reconcile the cluster state. This can lead to Kubernetes Service LoadBalancers removing nodes in a `NotReady` state. In some cases, larger clusters have all nodes in a `NotReady` state, causing a cluster-wide Service LoadBalancer disruption. See [Mitigating Workload Downtime](#mitigating-workload-downtime) for workarounds.
+
 ```bash
 # Roll masters as quickly as possible
 kops rolling-update cluster --cloudonly --instance-group-roles master --master-interval=1s
@@ -65,6 +69,8 @@ kops rolling-update cluster --yes
 
 If you are using calico the switch to CRDs will effectively cause a network partition during the rolling-update.  Your application might tolerate this, but it probably won't.  We therefore recommend rolling your nodes as fast as possible also:
 
+> **DANGER:** Using the procedure to quickly roll your masters can result in downtime for any workloads using Service LoadBalancers. (The "Hammer 🔨" Method)  
+> Any time you kill off all three masters with `--cloudonly` and `--master-interval=1s`, you may experience worker nodes go into a `NotReady` state when the new masters come online and reconcile the cluster state. This can lead to Kubernetes Service LoadBalancers removing nodes in a `NotReady` state. In some cases, larger clusters have all nodes in a `NotReady` state, causing a cluster-wide Service LoadBalancer disruption. See [Mitigating Workload Downtime](#mitigating-workload-downtime) for workarounds.
 
 ```bash
 # Roll masters and nodes as quickly as possible
@@ -107,4 +113,32 @@ To delay adopting etcd3 with kops 1.12, specify the etcd version as 2.2.1
 kops set cluster cluster.spec.etcdClusters[*].version=2.2.1
 ```
 
-To remove, `kops edit` your cluster and delete the `version: 2.2.1` lines from both etcdCluster blocks.
+To remove, `kops edit` your cluster and delete the `version: 2.2.1` lines from both etcdCluster blocks.
+
+## Mitigating Workload Downtime
+### AWS ELB Mitigation
+When quickly rolling all your masters, you can hit conditions which lead to nodes entering a `NotReady` state. Kubernetes, by default, will remove any `NotReady` nodes from ELBs managed by Services. To avoid possible ELB service interruption, you can add a temporary IAM policy which blocks the masters from removing `NotReady` nodes from LoadBalancer type services. This policy only needs to be in play while you are performing this upgrade and can be removed once the nodes (masters and workers) are all in a `Ready` state. Make sure you remove the policy once the cluster is upgraded and stable, otherwise Kubernetes will not be able to effectively manage your nodes in ELBs.
+
+**Adding the Temporary Policy**
+```bash
+# Configure your master_node_role_name (Generally "masters.your.cluster.name")
+masters_role_name="masters.<your.cluster.name>"
+
+# Install a temporary IAM policy. This avoids nodes being removed from LoadBalancer type services while masters reconcile the state of the cluster.
+aws iam put-role-policy \
+  --role-name "${masters_role_name}" \
+  --policy-name temporary-etcd-upgrade-deny-lb-changes \
+  --policy-document \
+    '{"Version": "2012-10-17", "Statement": [{"Action": ["elasticloadbalancing:DeregisterInstancesFromLoadBalancer", "elasticloadbalancing:DeregisterTargets"], "Resource": ["*"], "Effect": "Deny"}]}'
+```
+
+**Removing the Temporary Policy**
+```bash
+# Configure your master_node_role_name
+masters_role_name="masters.<your.cluster.name>"
+
+# Once your cluster node states have stabilized from `NotReady` to `Ready` you can remove the temporary policy from your master nodes
+aws iam delete-role-policy \
+  --role-name "${masters_role_name}" \
+  --policy-name temporary-etcd-upgrade-deny-lb-changes
+```