kubernetes
/
website
mirror of https://github.com/kubernetes/website.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Improvements to etcd set up (#9088) 

Signed-off-by: Chuck Ha <ha.chuck@gmail.com>
This commit is contained in:
Chuck Ha 2018-06-18 11:49:33 -04:00 committed by k8s-ci-robot
parent 5bfb804e4e
commit a2c651f60c
1 changed files with 168 additions and 133 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

239
content/en/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md
View File

@ -1,14 +1,14 @@
--- ---
reviewers: reviewers:
- chuckha - sig-cluster-lifecycle
title: Set up a High-Availability Etcd Cluster With Kubeadm title: Set up a Highly Availabile etcd Cluster With kubeadm
content_template: templates/task content_template: templates/task
--- ---
{{% capture overview %}} {{% capture overview %}}
Kubeadm defaults to running a single member etcd cluster in a static pod managed Kubeadm defaults to running a single member etcd cluster in a static pod managed
by the kubelet on the control plane node. This is not a highly-available setup by the kubelet on the control plane node. This is not a highly available setup
as the the etcd cluster contains only one member and cannot sustain any members as the the etcd cluster contains only one member and cannot sustain any members
becoming unavailable. This task walks through the process of creating a highly becoming unavailable. This task walks through the process of creating a highly
available etcd cluster of three members that can be used as an external etcd available etcd cluster of three members that can be used as an external etcd
@ -30,55 +30,60 @@ when using kubeadm to set up a kubernetes cluster.
{{% capture steps %}} {{% capture steps %}}
## Setting up the cluster
The general approach is to generate all certs on one node and only distribute The general approach is to generate all certs on one node and only distribute
the *necessary* files to the other nodes. Note that kubeadm contains all the necessary the *necessary* files to the other nodes. Note that kubeadm contains all the
crytographic machinery to generate the certificates described below; no other cryptographic tooling necessary crytographic machinery to generate the certificates described below;
is required for this exercise. no other cryptographic tooling is required for this example.
## Create configuration files for kubeadm 1. Create configuration files for kubeadm.
Using the template provided below, create one kubeadm configuration file for Generate one kubeadm configuration file for each host that will have an etcd
each host that will have an etcd member running on it. Update the value of member running on it using the following script.
`CURRENT_HOST` and `NAME` before running the `cat` command.
``` ```sh
export HOST0=10.0.0.1 # Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts # Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
export HOST1=10.0.0.2 export HOST0=10.0.0.6
export HOST2=10.0.0.3 export HOST1=10.0.0.7
export HOST2=10.0.0.8
# Create temp directories to store files that will end up on other hosts. # Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/certs /tmp/${HOST1}/certs /tmp/${HOST2}/certs mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
export CURRENT_HOST="${HOST0}" # Update on each ranging through HOST0, HOST1 and HOST2 ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
export NAME=infra0 # Update to use infra0 for HOST0, infra1 for HOST1 and infra2 for HOST2 NAMES=("infra0" "infra1" "infra2")
cat << EOF > /tmp/${CURRENT_HOST}/kubeadmcfg.yaml for i in "${!ETCDHOSTS[@]}"; do
apiVersion: "kubeadm.k8s.io/v1alpha1" HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1alpha2"
kind: MasterConfiguration kind: MasterConfiguration
etcd: etcd:
localEtcd:
serverCertSANs: serverCertSANs:
- "${CURRENT_HOST}" - "${HOST}"
peerCertSANs: peerCertSANs:
- "${CURRENT_HOST}" - "${HOST}"
extraArgs: extraArgs:
initial-cluster: infra0=https://${HOST0}:2380,infra1=https://${HOST1}:2380,infra2=https://${HOST2}:2380 initial-cluster: infra0=https://${ETCDHOST0}:2380,infra1=https://${ETCDHOST1}:2380,infra2=https://${ETCDHOST2:2380
initial-cluster-state: new initial-cluster-state: new
name: ${NAME} name: ${NAME}
listen-peer-urls: https://${CURRENT_HOST}:2380 listen-peer-urls: https://${HOST}:2380
listen-client-urls: https://${CURRENT_HOST}:2379 listen-client-urls: https://${HOST}:2379
advertise-client-urls: https://${CURRENT_HOST}:2379 advertise-client-urls: https://${HOST}:2379
initial-advertise-peer-urls: https://${CURRENT_HOST}:2380 initial-advertise-peer-urls: https://${HOST}:2380
EOF EOF
done
``` ```
## Generate certificates needed for the etcd cluster 1. Generate the certificate authority
### Certificate Authority
If you already have a CA then the only action that is copying the CA's `crt` and If you already have a CA then the only action that is copying the CA's `crt` and
`key` file to `/etc/kubernetes/pki/etcd/ca.crt` and `key` file to `/etc/kubernetes/pki/etcd/ca.crt` and
`/etc/kubernetes/pki/etcd/ca.key`. After those files have been copied, please `/etc/kubernetes/pki/etcd/ca.key`. After those files have been copied, please
skip to the Certificate Swizzling section below. skip this step.
If you do not already have a CA then run this command on `$HOST0` (where you If you do not already have a CA then run this command on `$HOST0` (where you
generated the configuration files for kubeadm). generated the configuration files for kubeadm).
@ -89,105 +94,135 @@ kubeadm alpha phase certs etcd-ca
This creates two files This creates two files
1. `/etc/kubernetes/pki/etcd/ca.crt` - `/etc/kubernetes/pki/etcd/ca.crt`
2. `/etc/kubernetes/pki/etcd/ca.key` - `/etc/kubernetes/pki/etcd/ca.key`
### Create certificates for each member 1. Create certificates for each member
In this step we create all the certs for each host in our cluster. ```sh
```
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
# Move the generated certs out of the generated directory kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
find /etc/kubernetes/pki/etcd -not -name ca.crt -not -name ca.key -type f -exec mv {} /tmp/${HOST2}/certs \; cp -R /etc/kubernetes/pki /tmp/${HOST2}/
cp /etc/kubernetes/pki/etcd/ca.crt /tmp/${HOST2}/certs # cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
# Move the generated certs out of the generated directory kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
find /etc/kubernetes/pki/etcd -not -name ca.crt -not -name ca.key -type f -exec mv {} /tmp/${HOST1}/certs \; cp -R /etc/kubernetes/pki /tmp/${HOST2}/
cp /etc/kubernetes/pki/etcd/ca.crt /tmp/${HOST1}/certs find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0 # No need to move the certs because they are for HOST0
# clean up certs that should not be copied off this host
find /tmp/${HOST2} -name ca.key -type f -delete
find /tmp/${HOST1} -name ca.key -type f -delete
``` ```
### Copy certs and configs to other hosts 1. Copy certificates and kubeadm configs
Copy the certs and configs in each tmp directory to the respective hosts and put The certificates have been generated and now they must be moved to their
the certs owned by root:root in `/etc/kubernetes/pki/etcd/`. respective hosts.
The steps to get these files on `$HOST1` might look like this if you can ssh ```sh
between hosts: USER=ubuntu
HOST=${HOST1}
``` scp -r /tmp/${HOST}/* ${USER}@${HOST}:
root@HOST0 $ scp -i /home/ubuntu/.ssh/id_rsa -r /tmp/${HOST1}/* ubuntu@${HOST1}:/home/ubuntu ssh ${USER}@${HOST}
root@HOST0 $ ssh -i /home/ubuntu/.ssh/id_rsa ubuntu@${HOST1} USER@HOST $ sudo -Es
ubuntu@HOST1 $ sudo -s root@HOST $ chown -R root:root pki
root@HOST1 $ chown -R root:root certs root@HOST $ mv pki /etc/kubernetes/
root@HOST1 $ mv certs/* /etc/kubernetes/pki/etcd/
# Repeat for HOST2
``` ```
### List of all generated certs 1. Ensure all expected files exist
This is a list of all the files you have generated and where on which host they The complete list of required files on `$HOST0` is:
should live.
#### Host 0
1. `/etc/kubernetes/pki/etcd/ca.crt`
1. `/etc/kubernetes/pki/etcd/ca.key`
1. `/etc/kubernetes/pki/etcd/server.crt`
1. `/etc/kubernetes/pki/etcd/server.key`
1. `/etc/kubernetes/pki/etcd/peer.crt`
1. `/etc/kubernetes/pki/etcd/peer.key`
1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt`
1. `/etc/kubernetes/pki/etcd/healthcheck-client.key`
1. `/tmp/${HOST0}/kubeadmcfg.yaml`
#### Host 1
1. `/etc/kubernetes/pki/etcd/ca.crt`
1. `/etc/kubernetes/pki/etcd/server.crt`
1. `/etc/kubernetes/pki/etcd/server.key`
1. `/etc/kubernetes/pki/etcd/peer.crt`
1. `/etc/kubernetes/pki/etcd/peer.key`
1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt`
1. `/etc/kubernetes/pki/etcd/healthcheck-client.key`
1. `/home/ubuntu/kubeadmcfg.yaml`
#### Host 2
1. `/etc/kubernetes/pki/etcd/ca.crt`
1. `/etc/kubernetes/pki/etcd/server.crt`
1. `/etc/kubernetes/pki/etcd/server.key`
1. `/etc/kubernetes/pki/etcd/peer.crt`
1. `/etc/kubernetes/pki/etcd/peer.key`
1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt`
1. `/etc/kubernetes/pki/etcd/healthcheck-client.key`
1. `/home/ubuntu/kubeadmcfg.yaml`
## Manifests
Now that the certs and configs are in place we can create the manifest. On each
host run the `kubeadm` command to generate a static manifest for etcd.
``` ```
/tmp/${HOST0}
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
├── ca.crt
├── ca.key
├── healthcheck-client.crt
├── healthcheck-client.key
├── peer.crt
├── peer.key
├── server.crt
└── server.key
```
On `$HOST1`:
```
$HOME
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
├── ca.crt
├── healthcheck-client.crt
├── healthcheck-client.key
├── peer.crt
├── peer.key
├── server.crt
└── server.key
```
On `$HOST2`
```
$HOME
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
├── ca.crt
├── healthcheck-client.crt
├── healthcheck-client.key
├── peer.crt
├── peer.key
├── server.crt
└── server.key
```
1. Create the static pod manifests
Now that the certificates and configs are in place it's time to create the
manifests. On each host run the `kubeadm` command to generate a static manifest
for etcd.
```sh
root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
root@HOST1 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml root@HOST1 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
root@HOST2 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml root@HOST2 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
``` ```
## Optional: Check the cluster health 1. Optional: Check the cluster health
``` ```sh
docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.14 etcdctl --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --ca-file /etc/kubernetes/pki/etcd/ca.crt --endpoints https://${HOST0}:2379 cluster-health docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.18 etcdctl \
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
--key-file /etc/kubernetes/pki/etcd/peer.key \
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://${HOST0}:2379 cluster-health
... ...
cluster is healthy cluster is healthy
``` ```
@ -196,8 +231,8 @@ cluster is healthy
{{% capture whatsnext %}} {{% capture whatsnext %}}
Once your have a working 3 member etcd cluster, you can continue [setting up an Once your have a working 3 member etcd cluster, you can continue setting up a
HA control plane using highly available control plane using the [external etcd method with
kubeadm](/docs/setup/independent/high-availability/). kubeadm](/docs/setup/independent/high-availability/).
{{% /capture %}} {{% /capture %}}