Merge pull request #33791 from sftim/20200109_reword_replicated_stateful_app_task
Reword replicated stateful app task
This commit is contained in:
commit
10ba719413
|
|
@ -14,7 +14,7 @@ weight: 30
|
||||||
<!-- overview -->
|
<!-- overview -->
|
||||||
|
|
||||||
This page shows how to run a replicated stateful application using a
|
This page shows how to run a replicated stateful application using a
|
||||||
[StatefulSet](/docs/concepts/workloads/controllers/statefulset/) controller.
|
{{< glossary_tooltip term_id="statefulset" >}}.
|
||||||
This application is a replicated MySQL database. The example topology has a
|
This application is a replicated MySQL database. The example topology has a
|
||||||
single primary server and multiple replicas, using asynchronous row-based
|
single primary server and multiple replicas, using asynchronous row-based
|
||||||
replication.
|
replication.
|
||||||
|
|
@ -24,12 +24,10 @@ replication.
|
||||||
on general patterns for running stateful applications in Kubernetes.
|
on general patterns for running stateful applications in Kubernetes.
|
||||||
{{< /note >}}
|
{{< /note >}}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## {{% heading "prerequisites" %}}
|
## {{% heading "prerequisites" %}}
|
||||||
|
|
||||||
|
|
||||||
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
* {{< include "task-tutorial-prereqs.md" >}}
|
||||||
* {{< include "default-storage-class-prereqs.md" >}}
|
* {{< include "default-storage-class-prereqs.md" >}}
|
||||||
* This tutorial assumes you are familiar with
|
* This tutorial assumes you are familiar with
|
||||||
[PersistentVolumes](/docs/concepts/storage/persistent-volumes/)
|
[PersistentVolumes](/docs/concepts/storage/persistent-volumes/)
|
||||||
|
|
@ -46,7 +44,7 @@ on general patterns for running stateful applications in Kubernetes.
|
||||||
## {{% heading "objectives" %}}
|
## {{% heading "objectives" %}}
|
||||||
|
|
||||||
|
|
||||||
* Deploy a replicated MySQL topology with a StatefulSet controller.
|
* Deploy a replicated MySQL topology with a StatefulSet.
|
||||||
* Send MySQL client traffic.
|
* Send MySQL client traffic.
|
||||||
* Observe resistance to downtime.
|
* Observe resistance to downtime.
|
||||||
* Scale the StatefulSet up and down.
|
* Scale the StatefulSet up and down.
|
||||||
|
|
@ -60,7 +58,7 @@ on general patterns for running stateful applications in Kubernetes.
|
||||||
The example MySQL deployment consists of a ConfigMap, two Services,
|
The example MySQL deployment consists of a ConfigMap, two Services,
|
||||||
and a StatefulSet.
|
and a StatefulSet.
|
||||||
|
|
||||||
### ConfigMap
|
### Create a ConfigMap {#configmap}
|
||||||
|
|
||||||
Create the ConfigMap from the following YAML configuration file:
|
Create the ConfigMap from the following YAML configuration file:
|
||||||
|
|
||||||
|
|
@ -71,7 +69,7 @@ kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
This ConfigMap provides `my.cnf` overrides that let you independently control
|
This ConfigMap provides `my.cnf` overrides that let you independently control
|
||||||
configuration on the primary MySQL server and replicas.
|
configuration on the primary MySQL server and its replicas.
|
||||||
In this case, you want the primary server to be able to serve replication logs to replicas
|
In this case, you want the primary server to be able to serve replication logs to replicas
|
||||||
and you want replicas to reject any writes that don't come via replication.
|
and you want replicas to reject any writes that don't come via replication.
|
||||||
|
|
||||||
|
|
@ -80,7 +78,7 @@ portions to apply to different Pods.
|
||||||
Each Pod decides which portion to look at as it's initializing,
|
Each Pod decides which portion to look at as it's initializing,
|
||||||
based on information provided by the StatefulSet controller.
|
based on information provided by the StatefulSet controller.
|
||||||
|
|
||||||
### Services
|
### Create Services {#services}
|
||||||
|
|
||||||
Create the Services from the following YAML configuration file:
|
Create the Services from the following YAML configuration file:
|
||||||
|
|
||||||
|
|
@ -90,23 +88,24 @@ Create the Services from the following YAML configuration file:
|
||||||
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-services.yaml
|
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-services.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
The Headless Service provides a home for the DNS entries that the StatefulSet
|
The headless Service provides a home for the DNS entries that the StatefulSet
|
||||||
controller creates for each Pod that's part of the set.
|
{{< glossary_tooltip text="controllers" term_id="controller" >}} creates for each
|
||||||
Because the Headless Service is named `mysql`, the Pods are accessible by
|
Pod that's part of the set.
|
||||||
|
Because the headless Service is named `mysql`, the Pods are accessible by
|
||||||
resolving `<pod-name>.mysql` from within any other Pod in the same Kubernetes
|
resolving `<pod-name>.mysql` from within any other Pod in the same Kubernetes
|
||||||
cluster and namespace.
|
cluster and namespace.
|
||||||
|
|
||||||
The Client Service, called `mysql-read`, is a normal Service with its own
|
The client Service, called `mysql-read`, is a normal Service with its own
|
||||||
cluster IP that distributes connections across all MySQL Pods that report
|
cluster IP that distributes connections across all MySQL Pods that report
|
||||||
being Ready. The set of potential endpoints includes the primary MySQL server and all
|
being Ready. The set of potential endpoints includes the primary MySQL server and all
|
||||||
replicas.
|
replicas.
|
||||||
|
|
||||||
Note that only read queries can use the load-balanced Client Service.
|
Note that only read queries can use the load-balanced client Service.
|
||||||
Because there is only one primary MySQL server, clients should connect directly to the
|
Because there is only one primary MySQL server, clients should connect directly to the
|
||||||
primary MySQL Pod (through its DNS entry within the Headless Service) to execute
|
primary MySQL Pod (through its DNS entry within the headless Service) to execute
|
||||||
writes.
|
writes.
|
||||||
|
|
||||||
### StatefulSet
|
### Create the StatefulSet {#statefulset}
|
||||||
|
|
||||||
Finally, create the StatefulSet from the following YAML configuration file:
|
Finally, create the StatefulSet from the following YAML configuration file:
|
||||||
|
|
||||||
|
|
@ -122,7 +121,7 @@ You can watch the startup progress by running:
|
||||||
kubectl get pods -l app=mysql --watch
|
kubectl get pods -l app=mysql --watch
|
||||||
```
|
```
|
||||||
|
|
||||||
After a while, you should see all 3 Pods become Running:
|
After a while, you should see all 3 Pods become `Running`:
|
||||||
|
|
||||||
```
|
```
|
||||||
NAME READY STATUS RESTARTS AGE
|
NAME READY STATUS RESTARTS AGE
|
||||||
|
|
@ -132,8 +131,11 @@ mysql-2 2/2 Running 0 1m
|
||||||
```
|
```
|
||||||
|
|
||||||
Press **Ctrl+C** to cancel the watch.
|
Press **Ctrl+C** to cancel the watch.
|
||||||
|
|
||||||
|
{{< note >}}
|
||||||
If you don't see any progress, make sure you have a dynamic PersistentVolume
|
If you don't see any progress, make sure you have a dynamic PersistentVolume
|
||||||
provisioner enabled as mentioned in the [prerequisites](#before-you-begin).
|
provisioner enabled, as mentioned in the [prerequisites](#before-you-begin).
|
||||||
|
{{< /note >}}
|
||||||
|
|
||||||
This manifest uses a variety of techniques for managing stateful Pods as part of
|
This manifest uses a variety of techniques for managing stateful Pods as part of
|
||||||
a StatefulSet. The next section highlights some of these techniques to explain
|
a StatefulSet. The next section highlights some of these techniques to explain
|
||||||
|
|
@ -155,10 +157,10 @@ properties to perform orderly startup of MySQL replication.
|
||||||
### Generating configuration
|
### Generating configuration
|
||||||
|
|
||||||
Before starting any of the containers in the Pod spec, the Pod first runs any
|
Before starting any of the containers in the Pod spec, the Pod first runs any
|
||||||
[Init Containers](/docs/concepts/workloads/pods/init-containers/)
|
[init containers](/docs/concepts/workloads/pods/init-containers/)
|
||||||
in the order defined.
|
in the order defined.
|
||||||
|
|
||||||
The first Init Container, named `init-mysql`, generates special MySQL config
|
The first init container, named `init-mysql`, generates special MySQL config
|
||||||
files based on the ordinal index.
|
files based on the ordinal index.
|
||||||
|
|
||||||
The script determines its own ordinal index by extracting it from the end of
|
The script determines its own ordinal index by extracting it from the end of
|
||||||
|
|
@ -166,8 +168,7 @@ the Pod name, which is returned by the `hostname` command.
|
||||||
Then it saves the ordinal (with a numeric offset to avoid reserved values)
|
Then it saves the ordinal (with a numeric offset to avoid reserved values)
|
||||||
into a file called `server-id.cnf` in the MySQL `conf.d` directory.
|
into a file called `server-id.cnf` in the MySQL `conf.d` directory.
|
||||||
This translates the unique, stable identity provided by the StatefulSet
|
This translates the unique, stable identity provided by the StatefulSet
|
||||||
controller into the domain of MySQL server IDs, which require the same
|
into the domain of MySQL server IDs, which require the same properties.
|
||||||
properties.
|
|
||||||
|
|
||||||
The script in the `init-mysql` container also applies either `primary.cnf` or
|
The script in the `init-mysql` container also applies either `primary.cnf` or
|
||||||
`replica.cnf` from the ConfigMap by copying the contents into `conf.d`.
|
`replica.cnf` from the ConfigMap by copying the contents into `conf.d`.
|
||||||
|
|
@ -187,7 +188,7 @@ logs might not go all the way back to the beginning of time.
|
||||||
These conservative assumptions are the key to allow a running StatefulSet
|
These conservative assumptions are the key to allow a running StatefulSet
|
||||||
to scale up and down over time, rather than being fixed at its initial size.
|
to scale up and down over time, rather than being fixed at its initial size.
|
||||||
|
|
||||||
The second Init Container, named `clone-mysql`, performs a clone operation on
|
The second init container, named `clone-mysql`, performs a clone operation on
|
||||||
a replica Pod the first time it starts up on an empty PersistentVolume.
|
a replica Pod the first time it starts up on an empty PersistentVolume.
|
||||||
That means it copies all existing data from another running Pod,
|
That means it copies all existing data from another running Pod,
|
||||||
so its local state is consistent enough to begin replicating from the primary server.
|
so its local state is consistent enough to begin replicating from the primary server.
|
||||||
|
|
@ -202,7 +203,7 @@ Ready before starting Pod `N+1`.
|
||||||
|
|
||||||
### Starting replication
|
### Starting replication
|
||||||
|
|
||||||
After the Init Containers complete successfully, the regular containers run.
|
After the init containers complete successfully, the regular containers run.
|
||||||
The MySQL Pods consist of a `mysql` container that runs the actual `mysqld`
|
The MySQL Pods consist of a `mysql` container that runs the actual `mysqld`
|
||||||
server, and an `xtrabackup` container that acts as a
|
server, and an `xtrabackup` container that acts as a
|
||||||
[sidecar](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns).
|
[sidecar](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns).
|
||||||
|
|
@ -291,13 +292,13 @@ endpoint might be selected upon each connection attempt:
|
||||||
You can press **Ctrl+C** when you want to stop the loop, but it's useful to keep
|
You can press **Ctrl+C** when you want to stop the loop, but it's useful to keep
|
||||||
it running in another window so you can see the effects of the following steps.
|
it running in another window so you can see the effects of the following steps.
|
||||||
|
|
||||||
## Simulating Pod and Node downtime
|
## Simulate Pod and Node failure {#simulate-pod-and-node-downtime}
|
||||||
|
|
||||||
To demonstrate the increased availability of reading from the pool of replicas
|
To demonstrate the increased availability of reading from the pool of replicas
|
||||||
instead of a single server, keep the `SELECT @@server_id` loop from above
|
instead of a single server, keep the `SELECT @@server_id` loop from above
|
||||||
running while you force a Pod out of the Ready state.
|
running while you force a Pod out of the Ready state.
|
||||||
|
|
||||||
### Break the Readiness Probe
|
### Break the Readiness probe
|
||||||
|
|
||||||
The [readiness probe](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes)
|
The [readiness probe](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes)
|
||||||
for the `mysql` container runs the command `mysql -h 127.0.0.1 -e 'SELECT 1'`
|
for the `mysql` container runs the command `mysql -h 127.0.0.1 -e 'SELECT 1'`
|
||||||
|
|
@ -371,14 +372,18 @@ NAME READY STATUS RESTARTS AGE IP NODE
|
||||||
mysql-2 2/2 Running 0 15m 10.244.5.27 kubernetes-node-9l2t
|
mysql-2 2/2 Running 0 15m 10.244.5.27 kubernetes-node-9l2t
|
||||||
```
|
```
|
||||||
|
|
||||||
Then drain the Node by running the following command, which cordons it so
|
Then, drain the Node by running the following command, which cordons it so
|
||||||
no new Pods may schedule there, and then evicts any existing Pods.
|
no new Pods may schedule there, and then evicts any existing Pods.
|
||||||
Replace `<node-name>` with the name of the Node you found in the last step.
|
Replace `<node-name>` with the name of the Node you found in the last step.
|
||||||
|
|
||||||
This might impact other applications on the Node, so it's best to
|
{{< caution >}}
|
||||||
**only do this in a test cluster**.
|
Draining a Node can impact other workloads and applications
|
||||||
|
running on the same node. Only perform the following step in a test
|
||||||
|
cluster.
|
||||||
|
{{< /caution >}}
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
|
# See above advice about impact on other workloads
|
||||||
kubectl drain <node-name> --force --delete-emptydir-data --ignore-daemonsets
|
kubectl drain <node-name> --force --delete-emptydir-data --ignore-daemonsets
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -413,8 +418,9 @@ kubectl uncordon <node-name>
|
||||||
|
|
||||||
## Scaling the number of replicas
|
## Scaling the number of replicas
|
||||||
|
|
||||||
With MySQL replication, you can scale your read query capacity by adding replicas.
|
When you use MySQL replication, you can scale your read query capacity by
|
||||||
With StatefulSet, you can do this with a single command:
|
adding replicas.
|
||||||
|
For a StatefulSet, you can achieve this with a single command:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
kubectl scale statefulset mysql --replicas=5
|
kubectl scale statefulset mysql --replicas=5
|
||||||
|
|
@ -453,10 +459,13 @@ Scaling back down is also seamless:
|
||||||
kubectl scale statefulset mysql --replicas=3
|
kubectl scale statefulset mysql --replicas=3
|
||||||
```
|
```
|
||||||
|
|
||||||
Note, however, that while scaling up creates new PersistentVolumeClaims
|
{{< note >}}
|
||||||
|
Although scaling up creates new PersistentVolumeClaims
|
||||||
automatically, scaling down does not automatically delete these PVCs.
|
automatically, scaling down does not automatically delete these PVCs.
|
||||||
|
|
||||||
This gives you the choice to keep those initialized PVCs around to make
|
This gives you the choice to keep those initialized PVCs around to make
|
||||||
scaling back up quicker, or to extract data before deleting them.
|
scaling back up quicker, or to extract data before deleting them.
|
||||||
|
{{< /note >}}
|
||||||
|
|
||||||
You can see this by running:
|
You can see this by running:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,7 @@ metadata:
|
||||||
name: mysql
|
name: mysql
|
||||||
labels:
|
labels:
|
||||||
app: mysql
|
app: mysql
|
||||||
|
app.kubernetes.io/name: mysql
|
||||||
data:
|
data:
|
||||||
primary.cnf: |
|
primary.cnf: |
|
||||||
# Apply this config only on the primary.
|
# Apply this config only on the primary.
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@ metadata:
|
||||||
name: mysql
|
name: mysql
|
||||||
labels:
|
labels:
|
||||||
app: mysql
|
app: mysql
|
||||||
|
app.kubernetes.io/name: mysql
|
||||||
spec:
|
spec:
|
||||||
ports:
|
ports:
|
||||||
- name: mysql
|
- name: mysql
|
||||||
|
|
@ -21,6 +22,8 @@ metadata:
|
||||||
name: mysql-read
|
name: mysql-read
|
||||||
labels:
|
labels:
|
||||||
app: mysql
|
app: mysql
|
||||||
|
app.kubernetes.io/name: mysql
|
||||||
|
readonly: "true"
|
||||||
spec:
|
spec:
|
||||||
ports:
|
ports:
|
||||||
- name: mysql
|
- name: mysql
|
||||||
|
|
|
||||||
|
|
@ -5,13 +5,15 @@ metadata:
|
||||||
spec:
|
spec:
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app: mysql
|
app: mysql-server
|
||||||
|
app.kubernetes.io/name: mysql
|
||||||
serviceName: mysql
|
serviceName: mysql
|
||||||
replicas: 3
|
replicas: 3
|
||||||
template:
|
template:
|
||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app: mysql
|
app: mysql
|
||||||
|
app.kubernetes.io/name: mysql
|
||||||
spec:
|
spec:
|
||||||
initContainers:
|
initContainers:
|
||||||
- name: init-mysql
|
- name: init-mysql
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue