kubernetes
/
kops
mirror of https://github.com/kubernetes/kops.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1544 from justinsb/update_ha_doc 

Update HA doc
This commit is contained in:
Kris Nova 2017-01-21 07:43:15 -07:00 committed by GitHub
parent ef5e2fdae8 4aff63f651
commit 45ad8aaa13
2 changed files with 59 additions and 21 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
HA.md
View File

@ -1,21 +0,0 @@
# High Availability (HA)
## Introduction
When we create kubernetes cluster using kops we are able to create a multiple K8S masters and by doing so we can assure that in a case of
a master instance failure our K8S functionality won't damaged.
In scenario of a single master instance that fails our minions/pods will keep running but there will be no new pods schedualing, kubectl
won't work (since the api-server will be down).
## Kops HA
We can create HA cluster using kops only in the first creation of the cluster by using the "--master-zones" flag
example: https://github.com/kubernetes/kops/blob/master/docs/advanced_create.md
K8S relay on a key value DB named "etcd", which is using the the Qurom concept that the cluster need at least 51% of nodes to be
available, for the cluster to work.
Kops currently doesn't support cross region HA
As a result there are few considerations that need to be taken into account when using kops with HA:
* Only odd number of masters instances can be created.
* Currently Kops can't create more than 1 master in a single AZ.
* Kops can't create HA cluster on a region with 2 AZ. (There is no point to create 2 masters due to the fact the we need at least
51% of the nodes to be avilable so failure of one of the master will cause the whole HA to fail, thus running 2 masters only
increase the chance of the cluster failures).

59
docs/high_availability.md Normal file
View File

@ -0,0 +1,59 @@
# High Availability (HA)
## Introduction
Kubernetes has two strategies for high availability:
* Run multiple independent clusters and combine them behind one management plane: [federation](https://kubernetes.io/docs/user-guide/federation/)
* Run a single cluster in multiple cloud zones, with redundant components
kops has experimental early support for federation, but it already has good support for a cluster than runs
with redundant components. kops is able to create multiple kubernetes masters, so in the event of
a master instance failure, the kubernetes API will continue to operate.
However, when running kubernetes with a single master, if the master fails, the kubernetes API will be unavailable, but pods and services that are running on the (unaffected) nodes should continue to operate. In this situation, we won't be able to do anything that involves the API (adding nodes, scaling pods, replacing
terminated pods), and kubectl won't work. However your application should continue to run, and most applications
could probably tolerate an API outage of an hour or more.
Moreover, kops runs the masters in an automatic replacement mode. Masters are run in auto-scaling groups, with
the data on an EBS volume. If a master node is terminated, the ASG will launch a new master instance, and kops
will mount the master volume and replace the master.
In short:
* A single master kops cluster is still reasonably available; if the master instance terminates it will be automatically
replaced. But the use of EBS binds us to a single AZ, and in the event of a prolonged AZ outage, we might experience
downtime.
* A multi-node kops cluster can tolerate the outage of a single AZ
* Federation will allow you to create "uber-clusters" that can tolerate a regional outage
## Using Kops HA
We can create HA clusters using kops, but only it's important to note that you must plan for this at time of cluster creation. Currently it is not possible to change
the etcd cluster size (i.e. we cannot change an HA cluster to be non-HA, or a non-HA cluster to be HA.) [Issue #1512](https://github.com/kubernetes/kops/issues/1512)
When you first call `kops create cluster`, you specify the `--master-zones` flag listing the zones you want your masters
to run in, for example:
```
kops create cluster \
--node-count 3 \
--zones us-west-2a,us-west-2b,us-west-2c \
--master-zones us-west-2a,us-west-2b,us-west-2c \
--node-size t2.medium \
--master-size t2.medium \
--topology private \
--networking kopeio-vxlan \
hacluster.example.com
```
Kubernetes relies on a key-value store called "etcd", which uses the Quorum approach to consistency,
so it is available if 51% of the nodes are available.
As a result there are a few considerations that need to be taken into account when using kops with HA:
* Only odd number of masters instances should be created, as an even number is likely _less_ reliable than the lower odd number.
* Kops has experimental support for running multiple masters in the same AZ, but it should be used carefully.
If we create 2 (or more) masters in the same AZ, then failure of the AZ will likely cause etcd to lose quorum
and stop operating (with 3 nodes). Running in the same AZ therefore increases the risk of cluster disruption,
though it can be a valid scenario, particularly if combined with [federation](https://kubernetes.io/docs/user-guide/federation/).