Add a CockroachDB PetSet example
The example starts a simple five-node cluster with otherwise default setting (in particular, 3x replication).
This commit is contained in:
Tobias Schottdorf
parent
afb4d74fe5
commit
abc07f328e
|
|
@ -0,0 +1,97 @@
|
||||||
|
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
|
||||||
|
|
||||||
|
<!-- BEGIN STRIP_FOR_RELEASE -->
|
||||||
|
|
||||||
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||||
|
width="25" height="25">
|
||||||
|
|
||||||
|
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
|
||||||
|
|
||||||
|
If you are using a released version of Kubernetes, you should
|
||||||
|
refer to the docs that go with that version.
|
||||||
|
|
||||||
|
Documentation for other releases can be found at
|
||||||
|
[releases.k8s.io](http://releases.k8s.io).
|
||||||
|
</strong>
|
||||||
|
--
|
||||||
|
|
||||||
|
<!-- END STRIP_FOR_RELEASE -->
|
||||||
|
|
||||||
|
<!-- END MUNGE: UNVERSIONED_WARNING -->
|
||||||
|
|
||||||
|
# CockroachDB on Kubernetes as a PetSet
|
||||||
|
|
||||||
|
This example deploys [CockroachDB](https://cockroachlabs.com) on Kubernetes as
|
||||||
|
a PetSet. CockroachDB is a distributed, scalable NewSQL database. Please see
|
||||||
|
[the homepage](https://cockroachlabs.com) and the
|
||||||
|
[documentation](https://www.cockroachlabs.com/docs/) for details.
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
### PetSet limitations
|
||||||
|
|
||||||
|
Standard PetSet limitations apply: There is currently no possibility to use
|
||||||
|
node-local storage (outside of single-node tests), and so there is likely
|
||||||
|
a performance hit associated with running CockroachDB on some external storage.
|
||||||
|
Note that CockroachDB already does replication and thus should not be deployed on
|
||||||
|
a persistent volume which already replicates internally.
|
||||||
|
High-performance use cases on a private Kubernetes cluster should consider
|
||||||
|
a DaemonSet deployment.
|
||||||
|
|
||||||
|
### Recovery after persistent storage failure
|
||||||
|
|
||||||
|
A persistent storage failure (e.g. losing the hard drive) is gracefully handled
|
||||||
|
by CockroachDB as long as enough replicas survive (two out of three by
|
||||||
|
default). Due to the bootstrapping in this deployment, a storage failure of the
|
||||||
|
first node is special in that the administrator must manually prepopulate the
|
||||||
|
"new" storage medium by running an instance of CockroachDB with the `--join`
|
||||||
|
parameter. If this is not done, the first node will bootstrap a new cluster,
|
||||||
|
which will lead to a lot of trouble.
|
||||||
|
|
||||||
|
### Dynamic provisioning
|
||||||
|
|
||||||
|
The deployment is written for a use case in which dynamic provisioning is
|
||||||
|
available. When that is not the case, the persistent volume claims need
|
||||||
|
to be created manually. See [minikube.sh](minikube.sh) for the necessary
|
||||||
|
steps.
|
||||||
|
|
||||||
|
## Testing locally on minikube
|
||||||
|
|
||||||
|
Follow the steps in [minikube.sh](minikube.sh) (or simply run that file).
|
||||||
|
|
||||||
|
## Simulating failures
|
||||||
|
|
||||||
|
When all (or enough) nodes are up, simulate a failure like this:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl exec cockroachdb-0 -- /bin/bash -c "while true; do kill 1; done"
|
||||||
|
```
|
||||||
|
|
||||||
|
On one of the other pods, run `./cockroach sql --host $(hostname)` and use
|
||||||
|
(mostly) Postgres-flavor SQL. The example runs with three-fold replication,
|
||||||
|
so it can tolerate one failure of any given node at a time.
|
||||||
|
Note also that there is a brief period of time immediately after the creation
|
||||||
|
of the cluster during which the three-fold replication is established, and
|
||||||
|
during which killing a node may lead to unavailability.
|
||||||
|
|
||||||
|
There is also a [demo script](demo.sh).
|
||||||
|
|
||||||
|
## Scaling up or down
|
||||||
|
|
||||||
|
Simply edit the PetSet (but note that you may need to create a new persistent
|
||||||
|
volume claim first). If you ran `minikube.sh`, there's a spare volume so you
|
||||||
|
can immediately scale up by one. Convince yourself that the new node
|
||||||
|
immediately serves reads and writes.
|
||||||
|
|
||||||
|
|
||||||
|
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||||||
|
[]()
|
||||||
|
<!-- END MUNGE: GENERATED_ANALYTICS -->
|
||||||
|
|
@ -0,0 +1,96 @@
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
annotations:
|
||||||
|
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
|
||||||
|
name: cockroachdb
|
||||||
|
labels:
|
||||||
|
app: cockroachdb
|
||||||
|
spec:
|
||||||
|
ports:
|
||||||
|
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
|
||||||
|
# traffic and the cli.
|
||||||
|
- port: 26257
|
||||||
|
targetPort: 26257
|
||||||
|
name: grpc
|
||||||
|
# The secondary port serves the UI as well as health and debug endpoints.
|
||||||
|
- port: 8080
|
||||||
|
targetPort: 8080
|
||||||
|
name: http
|
||||||
|
clusterIP: None
|
||||||
|
selector:
|
||||||
|
app: cockroachdb
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1alpha1
|
||||||
|
kind: PetSet
|
||||||
|
metadata:
|
||||||
|
name: cockroachdb
|
||||||
|
spec:
|
||||||
|
serviceName: "cockroachdb"
|
||||||
|
replicas: 5
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: cockroachdb
|
||||||
|
annotations:
|
||||||
|
pod.alpha.kubernetes.io/initialized: "true"
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: cockroachdb
|
||||||
|
# Runs the master branch. Not recommended for production, but since
|
||||||
|
# CockroachDB is in Beta, you don't want to run it in production
|
||||||
|
# anyway. See
|
||||||
|
# https://hub.docker.com/r/cockroachdb/cockroach/tags/
|
||||||
|
# if you prefer to run a beta release.
|
||||||
|
image: cockroachdb/cockroach
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
ports:
|
||||||
|
- containerPort: 26257
|
||||||
|
name: grpc
|
||||||
|
- containerPort: 8080
|
||||||
|
name: http
|
||||||
|
volumeMounts:
|
||||||
|
- name: datadir
|
||||||
|
mountPath: /cockroach/cockroach-data
|
||||||
|
command:
|
||||||
|
- "/bin/bash"
|
||||||
|
- "-ecx"
|
||||||
|
- |
|
||||||
|
# The use of qualified `hostname -f` is crucial:
|
||||||
|
# Other nodes aren't able to look up the unqualified hostname.
|
||||||
|
CRARGS=("start" "--logtostderr" "--insecure" "--host" "$(hostname -f)")
|
||||||
|
# TODO(tschottdorf): really want to use an init container to do
|
||||||
|
# the bootstrapping. The idea is that the container would know
|
||||||
|
# whether it's on the first node and could check whether there's
|
||||||
|
# already a data directory. If not, it would bootstrap the cluster.
|
||||||
|
# We will need some version of `cockroach init` back for this to
|
||||||
|
# work. For now, just do the same in a shell snippet.
|
||||||
|
# Of course this isn't without danger - if node0 loses its data,
|
||||||
|
# upon restarting it will simply bootstrap a new cluster and smack
|
||||||
|
# it into our existing cluster.
|
||||||
|
# There are likely ways out. For example, the init container could
|
||||||
|
# query the kubernetes API and see whether any other nodes are
|
||||||
|
# around, etc. Or, of course, the admin can pre-seed the lost
|
||||||
|
# volume somehow (and in that case we should provide a better way,
|
||||||
|
# for example a marker file).
|
||||||
|
if [ ! "$(hostname)" == "cockroachdb-0" ] || \
|
||||||
|
[ -e "/cockroach/cockroach-data/COCKROACHDB_VERSION" ]
|
||||||
|
then
|
||||||
|
CRARGS+=("--join" "cockroachdb")
|
||||||
|
fi
|
||||||
|
/cockroach/cockroach ${CRARGS[*]}
|
||||||
|
volumes:
|
||||||
|
- name: datadir
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: datadir
|
||||||
|
volumeClaimTemplates:
|
||||||
|
- metadata:
|
||||||
|
name: datadir
|
||||||
|
annotations:
|
||||||
|
volume.alpha.kubernetes.io/storage-class: anything
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- "ReadWriteOnce"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 1Gi
|
||||||
|
|
@ -0,0 +1,47 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Copyright 2016 The Kubernetes Authors.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
function sql() {
|
||||||
|
# TODO(knz): Why does the more idiomatic read from stdin not produce any
|
||||||
|
# output?
|
||||||
|
kubectl exec "cockroachdb-${1}" -- /cockroach/cockroach sql \
|
||||||
|
--host "cockroachdb-${1}.cockroachdb" \
|
||||||
|
-e "$(cat /dev/stdin)"
|
||||||
|
}
|
||||||
|
|
||||||
|
function kill() {
|
||||||
|
! kubectl exec -t "cockroachdb-${1}" -- /bin/bash -c "while true; do kill 1; done" &> /dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create database on second node (idempotently for convenience).
|
||||||
|
cat <<EOF | sql 1
|
||||||
|
CREATE DATABASE IF NOT EXISTS foo;
|
||||||
|
CREATE TABLE IF NOT EXISTS foo.bar (k STRING PRIMARY KEY, v STRING);
|
||||||
|
UPSERT INTO foo.bar VALUES ('Kuber', 'netes'), ('Cockroach', 'DB');
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Kill the node we just created the table on.
|
||||||
|
kill 1
|
||||||
|
|
||||||
|
# Read the data from all other nodes (we could also read from the one we just
|
||||||
|
# killed, but it's awkward to wait for it to respawn).
|
||||||
|
for i in 0 2 3 4; do
|
||||||
|
cat <<EOF | sql "${i}"
|
||||||
|
SELECT CONCAT(k, v) FROM foo.bar;
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
@ -0,0 +1,66 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# Copyright 2016 The Kubernetes Authors.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
# Run the CockroachDB PetSet example on a minikube instance.
|
||||||
|
#
|
||||||
|
# For a fresh start, run the following first:
|
||||||
|
# minikube delete
|
||||||
|
# minikube start
|
||||||
|
#
|
||||||
|
# To upgrade minikube & kubectl on OSX, the following should suffice:
|
||||||
|
# brew reinstall kubernetes-cli --devel
|
||||||
|
# url -Lo minikube \
|
||||||
|
# https://storage.googleapis.com/minikube/releases/v0.4.0/minikube-darwin-amd64 && \
|
||||||
|
# chmod +x minikube && sudo mv minikube /usr/local/bin/
|
||||||
|
|
||||||
|
set -exuo pipefail
|
||||||
|
|
||||||
|
# Make persistent volumes and (correctly named) claims. We must create the
|
||||||
|
# claims here manually even though that sounds counter-intuitive. For details
|
||||||
|
# see https://github.com/kubernetes/contrib/pull/1295#issuecomment-230180894.
|
||||||
|
# Note that we make an extra volume here so you can manually test scale-up.
|
||||||
|
for i in $(seq 0 5); do
|
||||||
|
cat <<EOF | kubectl create -f -
|
||||||
|
kind: PersistentVolume
|
||||||
|
apiVersion: v1
|
||||||
|
metadata:
|
||||||
|
name: pv${i}
|
||||||
|
labels:
|
||||||
|
type: local
|
||||||
|
spec:
|
||||||
|
capacity:
|
||||||
|
storage: 1Gi
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
hostPath:
|
||||||
|
path: "/tmp/${i}"
|
||||||
|
EOF
|
||||||
|
|
||||||
|
cat <<EOF | kubectl create -f -
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
apiVersion: v1
|
||||||
|
metadata:
|
||||||
|
name: datadir-cockroachdb-${i}
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 1Gi
|
||||||
|
EOF
|
||||||
|
done;
|
||||||
|
|
||||||
|
kubectl create -f cockroachdb-petset.yaml
|
||||||
Loading…
Reference in New Issue