Add a CockroachDB PetSet example
The example starts a simple five-node cluster with otherwise default setting (in particular, 3x replication).
This commit is contained in:
Tobias Schottdorf
parent
afb4d74fe5
commit
abc07f328e
|
|
@ -0,0 +1,97 @@
|
|||
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
|
||||
|
||||
<!-- BEGIN STRIP_FOR_RELEASE -->
|
||||
|
||||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
|
||||
width="25" height="25">
|
||||
|
||||
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
|
||||
|
||||
If you are using a released version of Kubernetes, you should
|
||||
refer to the docs that go with that version.
|
||||
|
||||
Documentation for other releases can be found at
|
||||
[releases.k8s.io](http://releases.k8s.io).
|
||||
</strong>
|
||||
--
|
||||
|
||||
<!-- END STRIP_FOR_RELEASE -->
|
||||
|
||||
<!-- END MUNGE: UNVERSIONED_WARNING -->
|
||||
|
||||
# CockroachDB on Kubernetes as a PetSet
|
||||
|
||||
This example deploys [CockroachDB](https://cockroachlabs.com) on Kubernetes as
|
||||
a PetSet. CockroachDB is a distributed, scalable NewSQL database. Please see
|
||||
[the homepage](https://cockroachlabs.com) and the
|
||||
[documentation](https://www.cockroachlabs.com/docs/) for details.
|
||||
|
||||
## Limitations
|
||||
|
||||
### PetSet limitations
|
||||
|
||||
Standard PetSet limitations apply: There is currently no possibility to use
|
||||
node-local storage (outside of single-node tests), and so there is likely
|
||||
a performance hit associated with running CockroachDB on some external storage.
|
||||
Note that CockroachDB already does replication and thus should not be deployed on
|
||||
a persistent volume which already replicates internally.
|
||||
High-performance use cases on a private Kubernetes cluster should consider
|
||||
a DaemonSet deployment.
|
||||
|
||||
### Recovery after persistent storage failure
|
||||
|
||||
A persistent storage failure (e.g. losing the hard drive) is gracefully handled
|
||||
by CockroachDB as long as enough replicas survive (two out of three by
|
||||
default). Due to the bootstrapping in this deployment, a storage failure of the
|
||||
first node is special in that the administrator must manually prepopulate the
|
||||
"new" storage medium by running an instance of CockroachDB with the `--join`
|
||||
parameter. If this is not done, the first node will bootstrap a new cluster,
|
||||
which will lead to a lot of trouble.
|
||||
|
||||
### Dynamic provisioning
|
||||
|
||||
The deployment is written for a use case in which dynamic provisioning is
|
||||
available. When that is not the case, the persistent volume claims need
|
||||
to be created manually. See [minikube.sh](minikube.sh) for the necessary
|
||||
steps.
|
||||
|
||||
## Testing locally on minikube
|
||||
|
||||
Follow the steps in [minikube.sh](minikube.sh) (or simply run that file).
|
||||
|
||||
## Simulating failures
|
||||
|
||||
When all (or enough) nodes are up, simulate a failure like this:
|
||||
|
||||
```shell
|
||||
kubectl exec cockroachdb-0 -- /bin/bash -c "while true; do kill 1; done"
|
||||
```
|
||||
|
||||
On one of the other pods, run `./cockroach sql --host $(hostname)` and use
|
||||
(mostly) Postgres-flavor SQL. The example runs with three-fold replication,
|
||||
so it can tolerate one failure of any given node at a time.
|
||||
Note also that there is a brief period of time immediately after the creation
|
||||
of the cluster during which the three-fold replication is established, and
|
||||
during which killing a node may lead to unavailability.
|
||||
|
||||
There is also a [demo script](demo.sh).
|
||||
|
||||
## Scaling up or down
|
||||
|
||||
Simply edit the PetSet (but note that you may need to create a new persistent
|
||||
volume claim first). If you ran `minikube.sh`, there's a spare volume so you
|
||||
can immediately scale up by one. Convince yourself that the new node
|
||||
immediately serves reads and writes.
|
||||
|
||||
|
||||
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||||
[]()
|
||||
<!-- END MUNGE: GENERATED_ANALYTICS -->
|
||||
|
|
@ -0,0 +1,96 @@
|
|||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
annotations:
|
||||
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
|
||||
name: cockroachdb
|
||||
labels:
|
||||
app: cockroachdb
|
||||
spec:
|
||||
ports:
|
||||
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
|
||||
# traffic and the cli.
|
||||
- port: 26257
|
||||
targetPort: 26257
|
||||
name: grpc
|
||||
# The secondary port serves the UI as well as health and debug endpoints.
|
||||
- port: 8080
|
||||
targetPort: 8080
|
||||
name: http
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: cockroachdb
|
||||
---
|
||||
apiVersion: apps/v1alpha1
|
||||
kind: PetSet
|
||||
metadata:
|
||||
name: cockroachdb
|
||||
spec:
|
||||
serviceName: "cockroachdb"
|
||||
replicas: 5
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: cockroachdb
|
||||
annotations:
|
||||
pod.alpha.kubernetes.io/initialized: "true"
|
||||
spec:
|
||||
containers:
|
||||
- name: cockroachdb
|
||||
# Runs the master branch. Not recommended for production, but since
|
||||
# CockroachDB is in Beta, you don't want to run it in production
|
||||
# anyway. See
|
||||
# https://hub.docker.com/r/cockroachdb/cockroach/tags/
|
||||
# if you prefer to run a beta release.
|
||||
image: cockroachdb/cockroach
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 26257
|
||||
name: grpc
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
volumeMounts:
|
||||
- name: datadir
|
||||
mountPath: /cockroach/cockroach-data
|
||||
command:
|
||||
- "/bin/bash"
|
||||
- "-ecx"
|
||||
- |
|
||||
# The use of qualified `hostname -f` is crucial:
|
||||
# Other nodes aren't able to look up the unqualified hostname.
|
||||
CRARGS=("start" "--logtostderr" "--insecure" "--host" "$(hostname -f)")
|
||||
# TODO(tschottdorf): really want to use an init container to do
|
||||
# the bootstrapping. The idea is that the container would know
|
||||
# whether it's on the first node and could check whether there's
|
||||
# already a data directory. If not, it would bootstrap the cluster.
|
||||
# We will need some version of `cockroach init` back for this to
|
||||
# work. For now, just do the same in a shell snippet.
|
||||
# Of course this isn't without danger - if node0 loses its data,
|
||||
# upon restarting it will simply bootstrap a new cluster and smack
|
||||
# it into our existing cluster.
|
||||
# There are likely ways out. For example, the init container could
|
||||
# query the kubernetes API and see whether any other nodes are
|
||||
# around, etc. Or, of course, the admin can pre-seed the lost
|
||||
# volume somehow (and in that case we should provide a better way,
|
||||
# for example a marker file).
|
||||
if [ ! "$(hostname)" == "cockroachdb-0" ] || \
|
||||
[ -e "/cockroach/cockroach-data/COCKROACHDB_VERSION" ]
|
||||
then
|
||||
CRARGS+=("--join" "cockroachdb")
|
||||
fi
|
||||
/cockroach/cockroach ${CRARGS[*]}
|
||||
volumes:
|
||||
- name: datadir
|
||||
persistentVolumeClaim:
|
||||
claimName: datadir
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: datadir
|
||||
annotations:
|
||||
volume.alpha.kubernetes.io/storage-class: anything
|
||||
spec:
|
||||
accessModes:
|
||||
- "ReadWriteOnce"
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Copyright 2016 The Kubernetes Authors.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
function sql() {
|
||||
# TODO(knz): Why does the more idiomatic read from stdin not produce any
|
||||
# output?
|
||||
kubectl exec "cockroachdb-${1}" -- /cockroach/cockroach sql \
|
||||
--host "cockroachdb-${1}.cockroachdb" \
|
||||
-e "$(cat /dev/stdin)"
|
||||
}
|
||||
|
||||
function kill() {
|
||||
! kubectl exec -t "cockroachdb-${1}" -- /bin/bash -c "while true; do kill 1; done" &> /dev/null
|
||||
}
|
||||
|
||||
# Create database on second node (idempotently for convenience).
|
||||
cat <<EOF | sql 1
|
||||
CREATE DATABASE IF NOT EXISTS foo;
|
||||
CREATE TABLE IF NOT EXISTS foo.bar (k STRING PRIMARY KEY, v STRING);
|
||||
UPSERT INTO foo.bar VALUES ('Kuber', 'netes'), ('Cockroach', 'DB');
|
||||
EOF
|
||||
|
||||
# Kill the node we just created the table on.
|
||||
kill 1
|
||||
|
||||
# Read the data from all other nodes (we could also read from the one we just
|
||||
# killed, but it's awkward to wait for it to respawn).
|
||||
for i in 0 2 3 4; do
|
||||
cat <<EOF | sql "${i}"
|
||||
SELECT CONCAT(k, v) FROM foo.bar;
|
||||
EOF
|
||||
done
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Copyright 2016 The Kubernetes Authors.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Run the CockroachDB PetSet example on a minikube instance.
|
||||
#
|
||||
# For a fresh start, run the following first:
|
||||
# minikube delete
|
||||
# minikube start
|
||||
#
|
||||
# To upgrade minikube & kubectl on OSX, the following should suffice:
|
||||
# brew reinstall kubernetes-cli --devel
|
||||
# url -Lo minikube \
|
||||
# https://storage.googleapis.com/minikube/releases/v0.4.0/minikube-darwin-amd64 && \
|
||||
# chmod +x minikube && sudo mv minikube /usr/local/bin/
|
||||
|
||||
set -exuo pipefail
|
||||
|
||||
# Make persistent volumes and (correctly named) claims. We must create the
|
||||
# claims here manually even though that sounds counter-intuitive. For details
|
||||
# see https://github.com/kubernetes/contrib/pull/1295#issuecomment-230180894.
|
||||
# Note that we make an extra volume here so you can manually test scale-up.
|
||||
for i in $(seq 0 5); do
|
||||
cat <<EOF | kubectl create -f -
|
||||
kind: PersistentVolume
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: pv${i}
|
||||
labels:
|
||||
type: local
|
||||
spec:
|
||||
capacity:
|
||||
storage: 1Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
hostPath:
|
||||
path: "/tmp/${i}"
|
||||
EOF
|
||||
|
||||
cat <<EOF | kubectl create -f -
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: datadir-cockroachdb-${i}
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
done;
|
||||
|
||||
kubectl create -f cockroachdb-petset.yaml
|
||||
Loading…
Reference in New Issue