docs/creating-a-kubernetes-cluster.md

Creating a Kubernetes Cluster for Elafros

Two options:

• Setup a GKE cluster
• Run minikube locally

GKE

To use a k8s cluster running in GKE:

1. Install gcloud using the instructions for your platform.

2. Create a GCP project (or use an existing project if you've already created one) at http://console.cloud.google.com/home/dashboard. Set the ID of the project in an environment variable (e.g. PROJECT_ID) along with the email of your GCP user (GCP_USER).

3. Enable the k8s API:

   gcloud --project=$PROJECT_ID services enable container.googleapis.com
   

4. Create a k8s cluster (version 1.9 or greater):

   gcloud --project=$PROJECT_ID container clusters create \
     --cluster-version=1.9.2-gke.1 \
     --zone=us-east1-d \
     --scopes=cloud-platform \
     --enable-autoscaling --min-nodes=1 --max-nodes=3 \
     elafros-demo
   

   • Version 1.9+ is required
   • Change this to whichever zone you choose
   • cloud-platform scope is required to access GCB
   • Autoscale from 1 to 3 nodes. Adjust this for your use case
   • Change this to your preferred cluster name

   You can see the list of supported cluster versions in a particular zone by running:

   # Get the list of valid versions in us-east1-d
   gcloud container get-server-config --zone us-east1-d
   

5. If you haven't installed kubectl yet, you can install it now with gcloud:

   gcloud components install kubectl
   

6. Give your gcloud user cluster-admin privileges:

   kubectl create clusterrolebinding gcloud-admin-binding \
     --clusterrole=cluster-admin \
     --user=$GCP_USER
   

Minikube

1. Install and configure minikube with a VM driver, e.g. kvm on Linux or xhyve on macOS.

2. Create a cluster with version 1.9 or greater and your chosen VM driver:

   Until minikube enables it by default,the MutatingAdmissionWebhook plugin must be manually enabled.

minikube start \
  --kubernetes-version=v1.9.0 \
  --vm-driver=kvm \
  --extra-config=apiserver.Admission.PluginNames=DenyEscalatingExec,LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,SecurityContextDeny,MutatingAdmissionWebhook

Minikube with GCR

You can use Google Container Registry as the registry for a Minikube cluster.

1. Set up a GCR repo. Export the environment variable PROJECT_ID as the name of your project. Also export GCR_DOMAIN as the domain name of your GCR repo. This will be either gcr.io or a region-specific variant like us.gcr.io.

   export PROJECT_ID=elafros-demo-project
   export GCR_DOMAIN=gcr.io
   

   To have Bazel builds push to GCR, set DOCKER_REPO_OVERRIDE to the GCR repo's url.

   export DOCKER_REPO_OVERRIDE="${GCR_DOMAIN}/${PROJECT_ID}"
   

2. Create a GCP service account:

   gcloud iam service-accounts create minikube-gcr \
     --display-name "Minikube GCR Pull" \
     --project $PROJECT_ID
   

3. Give your service account the storage.objectViewer role:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
     --member "serviceAccount:minikube-gcr@${PROJECT_ID}.iam.gserviceaccount.com" \
     --role roles/storage.objectViewer
   

4. Create a key credential file for the service account:

   gcloud iam service-accounts keys create \
     --iam-account "minikube-gcr@${PROJECT_ID}.iam.gserviceaccount.com" \
     minikube-gcr-key.json
   

Now you can use the minikube-gcr-key.json file to create image pull secrets and link them to Kubernetes service accounts. A secret must be created and linked to a service account in each namespace that will pull images from GCR.

For example, use these steps to allow Minikube to pull Elafros and Build images from GCR as built by Bazel (bazel run :everything.create). This is only necessary if you are not using public Elafros and Build images.

1. Create a Kubernetes secret in the ela-system and build-system namespace:

   for prefix in ela build; do
     kubectl create secret docker-registry "gcr" \
       --docker-server=$GCR_DOMAIN \
       --docker-username=_json_key \
       --docker-password="$(cat minikube-gcr-key.json)" \
       --docker-email=your.email@here.com \
       -n "${prefix}-system"
   done
   

   The secret must be created in the same namespace as the pod or service account.

2. Add the secret as an imagePullSecret to the ela-controller and build-controller service accounts:

   for prefix in ela build; do
     kubectl patch serviceaccount "${prefix}-controller" \
       -p '{"imagePullSecrets": [{"name": "gcr"}]}' \
       -n "${prefix}-system"
   done
   

Use the same procedure to add imagePullSecrets to service accounts in any namespace. Use the default service account for pods that do not specify a service account.

_See also the private-repo sample README.