k8s-ci-robot
GitHub
commit
c5f3779040
|
|
@ -0,0 +1,211 @@
|
|||
---
|
||||
kep-number: 31
|
||||
title: etcdadm
|
||||
authors:
|
||||
- "@justinsb"
|
||||
owning-sig: sig-cluster-lifecycle
|
||||
#participating-sigs:
|
||||
#- sig-apimachinery
|
||||
reviewers:
|
||||
- @roberthbailey
|
||||
- @timothysc
|
||||
approvers:
|
||||
- @roberthbailey
|
||||
- @timothysc
|
||||
editor: TBD
|
||||
creation-date: 2018-10-22
|
||||
last-updated: 2018-10-22
|
||||
status: provisional
|
||||
#see-also:
|
||||
# - KEP-1
|
||||
# - KEP-2
|
||||
#replaces:
|
||||
# - KEP-3
|
||||
#superseded-by:
|
||||
# - KEP-100
|
||||
---
|
||||
|
||||
# etcdadm - automation for etcd clusters
|
||||
|
||||
## Table of Contents
|
||||
|
||||
* [Table of Contents](#table-of-contents)
|
||||
* [Summary](#summary)
|
||||
* [Motivation](#motivation)
|
||||
* [Goals](#goals)
|
||||
* [Non-Goals](#non-goals)
|
||||
* [Proposal](#proposal)
|
||||
* [User Stories](#user-stories)
|
||||
* [Manual Cluster Creation](#manual-cluster-creation)
|
||||
* [Automatic Cluster Creation](#automatic-cluster-creation)
|
||||
* [Automatic Cluster Creation with EBS volumes](#automatic-cluster-creation-with-ebs-volumes)
|
||||
* [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
|
||||
* [Risks and Mitigations](#risks-and-mitigations)
|
||||
* [Graduation Criteria](#graduation-criteria)
|
||||
* [Implementation History](#implementation-history)
|
||||
* [Infrastructure Needed](#infrastructure-needed)
|
||||
|
||||
## Summary
|
||||
|
||||
etcdadm makes operation of etcd for the Kubernetes control plane easy, on clouds
|
||||
and on bare-metal, including both single-node and HA configurations.
|
||||
|
||||
It is able to perform cluster reconfigurations, upgrades / downgrades, and
|
||||
backups / restores.
|
||||
|
||||
## Motivation
|
||||
|
||||
Today each installation tool must reimplement etcd operation, and this is
|
||||
difficult. It also leads to ecosystem fragmentation - e.g. etcd backups from
|
||||
one tool are not necessarily compatible with the backups from other tools. The
|
||||
failure modes are subtle and rare, and thus the kubernetes project benefits from
|
||||
having more collaboration.
|
||||
|
||||
|
||||
### Goals
|
||||
|
||||
The following key tasks are in scope:
|
||||
|
||||
* Cluster creation
|
||||
* Cluster teardown
|
||||
* Cluster resizing / membership changes
|
||||
* Cluster backups
|
||||
* Disaster recovery or restore from backup
|
||||
* Cluster upgrades
|
||||
* Cluster downgrades
|
||||
* PKI management
|
||||
|
||||
We will implement this functionality both as a base layer of imperative (manual
|
||||
CLI) operation, and a self-management layer which should enable automated
|
||||
in "safe" scenarios (with fallback to manual operation).
|
||||
|
||||
We'll also optionally support limited interaction with cloud infrastructure, for
|
||||
example for mounting volumes and peer-discovery. This is primarily for the
|
||||
self-management layer, but we'll expose it via etcdadm for consistency and for
|
||||
power-users. The tasks are limited today to listing & mounting a persistent
|
||||
volume, and listing instances to find peers. A full solution for management of
|
||||
machines or networks (for example) is out of scope, though we might share some
|
||||
example configurations for exposition. We expect kubernetes installation
|
||||
tooling to configure the majority of the cloud infrastructure here, because both
|
||||
the configurations and the configuration tooling varies widely.
|
||||
|
||||
The big reason that volume mounting is in scope is that volume mounting acts as
|
||||
a simple mutex on most clouds - it is a cheap way to boost the safety of our
|
||||
leader/gossip algorithms, because we have an external source of truth.
|
||||
|
||||
We'll also support reading & writing backups to S3 / GCS etc.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
* The project is not targeted at operation of an etcd cluster for use other than
|
||||
by Kubernetes apiserver. We are not building a general-purpose etcd operation
|
||||
toolkit. Likely it will work well for other use-cases, but other tools may be
|
||||
more suitable.
|
||||
* As described above, we aren't building a full "turn up an etcd cluster on a
|
||||
cloud solution"; we expect this to be a building block for use by kubernetes
|
||||
installation tooling (e.g. cluster API solutions).
|
||||
|
||||
## Proposal
|
||||
|
||||
We will combine the [etcdadm](https://github.com/platform9/etcdadm) from
|
||||
Platform9 with the [etcd-manager](https://github.com/kopeio/etcd-manager)
|
||||
project from kopeio / @justinsb.
|
||||
|
||||
etcdadm gives us easy to use CLI commands, which will form the base layer of
|
||||
operation. Automation should ideally describe what it is doing in terms of
|
||||
etcdadm commands, though we will also expose etcdadm as a go-library for easier
|
||||
consumption, following the kubectl pattern of a `cmd/` layer calling into a
|
||||
`pkg/` layer. This means the end-user can understand the operation of the
|
||||
tooling, and advanced users can feel confident that they can use the CLI tooling
|
||||
for advanced operations.
|
||||
|
||||
etcd-manager provides automation of the common scenarios, particularly when
|
||||
running on a cloud. It will be rebased to work in terms of etcdadm CLI
|
||||
operations (which will likely require some functionality to be added to etcdadm
|
||||
itself). Where automation is not known to be safe, etcd-manager can stop and
|
||||
allow for manual intervention using the CLI.
|
||||
|
||||
kops is currently using etcd-manager, and we aim to switch to the (new) etcadm asap.
|
||||
|
||||
We expect other tooling (e.g. cluster-api implementations) to adopt this project
|
||||
for etcd management going forwards, and do a first integration or two if it
|
||||
hasn't happened already.
|
||||
|
||||
### User Stories
|
||||
|
||||
#### Manual Cluster Creation
|
||||
|
||||
A cluster operator setting up a cluster manually will be able to do so using etcdadm and kubeadm.
|
||||
|
||||
The basic flow looks like:
|
||||
|
||||
* On a master machine, run `etcdadm init`, making note of the `etcdadm join
|
||||
<endpoint>` command
|
||||
* On each other master machine, copy the CA certificate and key from one of the
|
||||
other masters, then run the `etcdadm join <endpoint>` command.
|
||||
* Run kubeadm following the [external etcd procedure](https://kubernetes.io/docs/setup/independent/high-availability/#external-etcd)
|
||||
|
||||
This results in an multi-node ("HA") etcd cluster.
|
||||
|
||||
#### Automatic Cluster Creation
|
||||
|
||||
etcd-manager works by coordinating via a shared filesystem-like store (e.g. S3
|
||||
or GCS) and/or via cloud APIs (e.g. EC2 or GCE). In doing so it is able to
|
||||
automate the manual commands, which is very handy for running in a cloud
|
||||
environment like AWS or GCE.
|
||||
|
||||
The basic flow would look like:
|
||||
|
||||
* The user writes a configuration file to GCS using `etcdadm seed
|
||||
gs://mybucket/cluster1/etcd1 version=3.2.12 nodes=3`
|
||||
* On each master machine, run `etcdadm auto gs://mybucket/cluster1/etcd1`.
|
||||
(Likely the user will have to run that persistently, either as a systemd
|
||||
service or a static pod.)
|
||||
|
||||
`etcdadm auto` downloads the target configuration from GCS, discovers other
|
||||
peers also running etcdadm, gossips with them to do basic leader election. When
|
||||
sufficient nodes are available to form a quorum, it starts etcd.
|
||||
|
||||
#### Automatic Cluster Creation with EBS volumes
|
||||
|
||||
etcdadm can also automatically mount EBS volumes. The workflow looks like this:
|
||||
|
||||
* As before, write a configuration file using `etcadm seed ...`, but this time
|
||||
passing additional arguments "--volume-tag cluster=mycluster"
|
||||
* Create EBS volumes with the matching tags
|
||||
* On each master machine, run `etcdadm auto ...` as before. Now etcdadm will
|
||||
try to mount a volume with the correct tags before acting as a member of the
|
||||
cluster.
|
||||
|
||||
### Implementation Details/Notes/Constraints
|
||||
|
||||
* There will be some changes needed to both platform9/etcdadm (e.g. etcd2
|
||||
support) and kopeio/etcd-manager (to rebase on top of etcdadm).
|
||||
* It is unlikely that e.g. GKE / EKS will use etcdadm (at least initially),
|
||||
which limits the pool of contributors.
|
||||
|
||||
### Risks and Mitigations
|
||||
|
||||
* Automatic mode may make incorrect decisions and break a cluster. Mitigation:
|
||||
automated backups, and a willingness to stop and wait for a fix / operator
|
||||
intervention (CLI mode).
|
||||
* Automatic mode relies on peer-to-peer discovery and gossiping, which is less
|
||||
reliable than Raft. Mitigation: rely on Raft as much as possible, be very
|
||||
conservative in automated operations (favor correctness over availability or
|
||||
speed). etcd non-voting members will make this much more reliable.
|
||||
|
||||
## Graduation Criteria
|
||||
|
||||
etcdadm will be considered successful when it is used by the majority of OSS
|
||||
cluster installations.
|
||||
|
||||
## Implementation History
|
||||
|
||||
* Much SIG discussion
|
||||
* Initial proposal to SIG 2018-10-09
|
||||
* Initial KEP draft 2018-10-22
|
||||
* Added clarification of cloud interaction 2018-10-23
|
||||
|
||||
## Infrastructure Needed
|
||||
|
||||
* etcdadm will be a subproject under sig-cluster-lifecycle
|
||||
Loading…
Reference in New Issue