Replace master/slave with primary/secondary where possible
This commit is contained in:
parent
8cb0b199ae
commit
044d11de32
|
@ -15,7 +15,7 @@ weight: 30
|
||||||
|
|
||||||
This page shows how to run a replicated stateful application using a
|
This page shows how to run a replicated stateful application using a
|
||||||
[StatefulSet](/docs/concepts/workloads/controllers/statefulset/) controller.
|
[StatefulSet](/docs/concepts/workloads/controllers/statefulset/) controller.
|
||||||
The example is a MySQL single-master topology with multiple slaves running
|
The example is a MySQL single-primary topology with multiple secondaries running
|
||||||
asynchronous replication.
|
asynchronous replication.
|
||||||
|
|
||||||
{{< note >}}
|
{{< note >}}
|
||||||
|
@ -69,9 +69,9 @@ kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
This ConfigMap provides `my.cnf` overrides that let you independently control
|
This ConfigMap provides `my.cnf` overrides that let you independently control
|
||||||
configuration on the MySQL master and slaves.
|
configuration on the MySQL primary and secondaries.
|
||||||
In this case, you want the master to be able to serve replication logs to slaves
|
In this case, you want the primary to be able to serve replication logs to secondaries
|
||||||
and you want slaves to reject any writes that don't come via replication.
|
and you want secondaries to reject any writes that don't come via replication.
|
||||||
|
|
||||||
There's nothing special about the ConfigMap itself that causes different
|
There's nothing special about the ConfigMap itself that causes different
|
||||||
portions to apply to different Pods.
|
portions to apply to different Pods.
|
||||||
|
@ -96,12 +96,12 @@ cluster and namespace.
|
||||||
|
|
||||||
The Client Service, called `mysql-read`, is a normal Service with its own
|
The Client Service, called `mysql-read`, is a normal Service with its own
|
||||||
cluster IP that distributes connections across all MySQL Pods that report
|
cluster IP that distributes connections across all MySQL Pods that report
|
||||||
being Ready. The set of potential endpoints includes the MySQL master and all
|
being Ready. The set of potential endpoints includes the MySQL primary and all
|
||||||
slaves.
|
secondaries.
|
||||||
|
|
||||||
Note that only read queries can use the load-balanced Client Service.
|
Note that only read queries can use the load-balanced Client Service.
|
||||||
Because there is only one MySQL master, clients should connect directly to the
|
Because there is only one MySQL primary, clients should connect directly to the
|
||||||
MySQL master Pod (through its DNS entry within the Headless Service) to execute
|
MySQL primary Pod (through its DNS entry within the Headless Service) to execute
|
||||||
writes.
|
writes.
|
||||||
|
|
||||||
### StatefulSet
|
### StatefulSet
|
||||||
|
@ -167,33 +167,33 @@ This translates the unique, stable identity provided by the StatefulSet
|
||||||
controller into the domain of MySQL server IDs, which require the same
|
controller into the domain of MySQL server IDs, which require the same
|
||||||
properties.
|
properties.
|
||||||
|
|
||||||
The script in the `init-mysql` container also applies either `master.cnf` or
|
The script in the `init-mysql` container also applies either `primary.cnf` or
|
||||||
`slave.cnf` from the ConfigMap by copying the contents into `conf.d`.
|
`secondary.cnf` from the ConfigMap by copying the contents into `conf.d`.
|
||||||
Because the example topology consists of a single MySQL master and any number of
|
Because the example topology consists of a single MySQL primary and any number of
|
||||||
slaves, the script simply assigns ordinal `0` to be the master, and everyone
|
secondaries, the script simply assigns ordinal `0` to be the primary, and everyone
|
||||||
else to be slaves.
|
else to be secondaries.
|
||||||
Combined with the StatefulSet controller's
|
Combined with the StatefulSet controller's
|
||||||
[deployment order guarantee](/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees/),
|
[deployment order guarantee](/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees/),
|
||||||
this ensures the MySQL master is Ready before creating slaves, so they can begin
|
this ensures the MySQL primary is Ready before creating secondaries, so they can begin
|
||||||
replicating.
|
replicating.
|
||||||
|
|
||||||
### Cloning existing data
|
### Cloning existing data
|
||||||
|
|
||||||
In general, when a new Pod joins the set as a slave, it must assume the MySQL
|
In general, when a new Pod joins the set as a secondary, it must assume the MySQL
|
||||||
master might already have data on it. It also must assume that the replication
|
primary might already have data on it. It also must assume that the replication
|
||||||
logs might not go all the way back to the beginning of time.
|
logs might not go all the way back to the beginning of time.
|
||||||
These conservative assumptions are the key to allow a running StatefulSet
|
These conservative assumptions are the key to allow a running StatefulSet
|
||||||
to scale up and down over time, rather than being fixed at its initial size.
|
to scale up and down over time, rather than being fixed at its initial size.
|
||||||
|
|
||||||
The second Init Container, named `clone-mysql`, performs a clone operation on
|
The second Init Container, named `clone-mysql`, performs a clone operation on
|
||||||
a slave Pod the first time it starts up on an empty PersistentVolume.
|
a secondary Pod the first time it starts up on an empty PersistentVolume.
|
||||||
That means it copies all existing data from another running Pod,
|
That means it copies all existing data from another running Pod,
|
||||||
so its local state is consistent enough to begin replicating from the master.
|
so its local state is consistent enough to begin replicating from the primary.
|
||||||
|
|
||||||
MySQL itself does not provide a mechanism to do this, so the example uses a
|
MySQL itself does not provide a mechanism to do this, so the example uses a
|
||||||
popular open-source tool called Percona XtraBackup.
|
popular open-source tool called Percona XtraBackup.
|
||||||
During the clone, the source MySQL server might suffer reduced performance.
|
During the clone, the source MySQL server might suffer reduced performance.
|
||||||
To minimize impact on the MySQL master, the script instructs each Pod to clone
|
To minimize impact on the MySQL primary, the script instructs each Pod to clone
|
||||||
from the Pod whose ordinal index is one lower.
|
from the Pod whose ordinal index is one lower.
|
||||||
This works because the StatefulSet controller always ensures Pod `N` is
|
This works because the StatefulSet controller always ensures Pod `N` is
|
||||||
Ready before starting Pod `N+1`.
|
Ready before starting Pod `N+1`.
|
||||||
|
@ -206,15 +206,15 @@ server, and an `xtrabackup` container that acts as a
|
||||||
[sidecar](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns).
|
[sidecar](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns).
|
||||||
|
|
||||||
The `xtrabackup` sidecar looks at the cloned data files and determines if
|
The `xtrabackup` sidecar looks at the cloned data files and determines if
|
||||||
it's necessary to initialize MySQL replication on the slave.
|
it's necessary to initialize MySQL replication on the secondary.
|
||||||
If so, it waits for `mysqld` to be ready and then executes the
|
If so, it waits for `mysqld` to be ready and then executes the
|
||||||
`CHANGE MASTER TO` and `START SLAVE` commands with replication parameters
|
`CHANGE MASTER TO` and `START SLAVE` commands with replication parameters
|
||||||
extracted from the XtraBackup clone files.
|
extracted from the XtraBackup clone files.
|
||||||
|
|
||||||
Once a slave begins replication, it remembers its MySQL master and
|
Once a secondary begins replication, it remembers its MySQL primary and
|
||||||
reconnects automatically if the server restarts or the connection dies.
|
reconnects automatically if the server restarts or the connection dies.
|
||||||
Also, because slaves look for the master at its stable DNS name
|
Also, because secondaries look for the primary at its stable DNS name
|
||||||
(`mysql-0.mysql`), they automatically find the master even if it gets a new
|
(`mysql-0.mysql`), they automatically find the primary even if it gets a new
|
||||||
Pod IP due to being rescheduled.
|
Pod IP due to being rescheduled.
|
||||||
|
|
||||||
Lastly, after starting replication, the `xtrabackup` container listens for
|
Lastly, after starting replication, the `xtrabackup` container listens for
|
||||||
|
@ -224,7 +224,7 @@ case the next Pod loses its PersistentVolumeClaim and needs to redo the clone.
|
||||||
|
|
||||||
## Sending client traffic
|
## Sending client traffic
|
||||||
|
|
||||||
You can send test queries to the MySQL master (hostname `mysql-0.mysql`)
|
You can send test queries to the MySQL primary (hostname `mysql-0.mysql`)
|
||||||
by running a temporary container with the `mysql:5.7` image and running the
|
by running a temporary container with the `mysql:5.7` image and running the
|
||||||
`mysql` client binary.
|
`mysql` client binary.
|
||||||
|
|
||||||
|
@ -291,7 +291,7 @@ it running in another window so you can see the effects of the following steps.
|
||||||
|
|
||||||
## Simulating Pod and Node downtime
|
## Simulating Pod and Node downtime
|
||||||
|
|
||||||
To demonstrate the increased availability of reading from the pool of slaves
|
To demonstrate the increased availability of reading from the pool of secondaries
|
||||||
instead of a single server, keep the `SELECT @@server_id` loop from above
|
instead of a single server, keep the `SELECT @@server_id` loop from above
|
||||||
running while you force a Pod out of the Ready state.
|
running while you force a Pod out of the Ready state.
|
||||||
|
|
||||||
|
@ -409,9 +409,9 @@ Now uncordon the Node to return it to a normal state:
|
||||||
kubectl uncordon <node-name>
|
kubectl uncordon <node-name>
|
||||||
```
|
```
|
||||||
|
|
||||||
## Scaling the number of slaves
|
## Scaling the number of secondaries
|
||||||
|
|
||||||
With MySQL replication, you can scale your read query capacity by adding slaves.
|
With MySQL replication, you can scale your read query capacity by adding secondaries.
|
||||||
With StatefulSet, you can do this with a single command:
|
With StatefulSet, you can do this with a single command:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
|
|
|
@ -5,12 +5,12 @@ metadata:
|
||||||
labels:
|
labels:
|
||||||
app: mysql
|
app: mysql
|
||||||
data:
|
data:
|
||||||
master.cnf: |
|
primary.cnf: |
|
||||||
# Apply this config only on the master.
|
# Apply this config only on the primary.
|
||||||
[mysqld]
|
[mysqld]
|
||||||
log-bin
|
log-bin
|
||||||
slave.cnf: |
|
secondary.cnf: |
|
||||||
# Apply this config only on slaves.
|
# Apply this config only on secondaries.
|
||||||
[mysqld]
|
[mysqld]
|
||||||
super-read-only
|
super-read-only
|
||||||
|
|
||||||
|
|
|
@ -14,7 +14,7 @@ spec:
|
||||||
app: mysql
|
app: mysql
|
||||||
---
|
---
|
||||||
# Client service for connecting to any MySQL instance for reads.
|
# Client service for connecting to any MySQL instance for reads.
|
||||||
# For writes, you must instead connect to the master: mysql-0.mysql.
|
# For writes, you must instead connect to the primary: mysql-0.mysql.
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: Service
|
kind: Service
|
||||||
metadata:
|
metadata:
|
||||||
|
|
|
@ -29,9 +29,9 @@ spec:
|
||||||
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
|
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
|
||||||
# Copy appropriate conf.d files from config-map to emptyDir.
|
# Copy appropriate conf.d files from config-map to emptyDir.
|
||||||
if [[ $ordinal -eq 0 ]]; then
|
if [[ $ordinal -eq 0 ]]; then
|
||||||
cp /mnt/config-map/master.cnf /mnt/conf.d/
|
cp /mnt/config-map/primary.cnf /mnt/conf.d/
|
||||||
else
|
else
|
||||||
cp /mnt/config-map/slave.cnf /mnt/conf.d/
|
cp /mnt/config-map/secondary.cnf /mnt/conf.d/
|
||||||
fi
|
fi
|
||||||
volumeMounts:
|
volumeMounts:
|
||||||
- name: conf
|
- name: conf
|
||||||
|
@ -47,7 +47,7 @@ spec:
|
||||||
set -ex
|
set -ex
|
||||||
# Skip the clone if data already exists.
|
# Skip the clone if data already exists.
|
||||||
[[ -d /var/lib/mysql/mysql ]] && exit 0
|
[[ -d /var/lib/mysql/mysql ]] && exit 0
|
||||||
# Skip the clone on master (ordinal index 0).
|
# Skip the clone on primary (ordinal index 0).
|
||||||
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
|
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
|
||||||
ordinal=${BASH_REMATCH[1]}
|
ordinal=${BASH_REMATCH[1]}
|
||||||
[[ $ordinal -eq 0 ]] && exit 0
|
[[ $ordinal -eq 0 ]] && exit 0
|
||||||
|
@ -108,12 +108,12 @@ spec:
|
||||||
# Determine binlog position of cloned data, if any.
|
# Determine binlog position of cloned data, if any.
|
||||||
if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
|
if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
|
||||||
# XtraBackup already generated a partial "CHANGE MASTER TO" query
|
# XtraBackup already generated a partial "CHANGE MASTER TO" query
|
||||||
# because we're cloning from an existing slave. (Need to remove the tailing semicolon!)
|
# because we're cloning from an existing secondary. (Need to remove the tailing semicolon!)
|
||||||
cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
|
cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
|
||||||
# Ignore xtrabackup_binlog_info in this case (it's useless).
|
# Ignore xtrabackup_binlog_info in this case (it's useless).
|
||||||
rm -f xtrabackup_slave_info xtrabackup_binlog_info
|
rm -f xtrabackup_slave_info xtrabackup_binlog_info
|
||||||
elif [[ -f xtrabackup_binlog_info ]]; then
|
elif [[ -f xtrabackup_binlog_info ]]; then
|
||||||
# We're cloning directly from master. Parse binlog position.
|
# We're cloning directly from primary. Parse binlog position.
|
||||||
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
|
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
|
||||||
rm -f xtrabackup_binlog_info xtrabackup_slave_info
|
rm -f xtrabackup_binlog_info xtrabackup_slave_info
|
||||||
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
|
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
|
||||||
|
|
Loading…
Reference in New Issue