docs/creating-a-kubernetes-cluster.md

# Creating a Kubernetes Cluster for Knative Serving

Two options:

*   Setup a [GKE cluster](#gke)
*   Run [minikube](#minikube) locally

## GKE

To use a k8s cluster running in GKE:

1.  Install `gcloud` using [the instructions for your
    platform](https://cloud.google.com/sdk/downloads).

1.  Create a GCP project (or use an existing project if you've already created
    one) at http://console.cloud.google.com/home/dashboard. Set the ID of the
    project in an environment variable (e.g. `PROJECT_ID`).

    _If you are a new GCP user, you might be eligible for a trial credit making
    your GKE cluster and other resources free for a short time. Otherwise, any
    GCP resources you create will cost money._

1.  Enable the k8s API:

    ```shell
    gcloud --project=$PROJECT_ID services enable container.googleapis.com
    ```

1.  Create a k8s cluster (version 1.10 or greater):

    ```shell
    gcloud --project=$PROJECT_ID container clusters create \
      --cluster-version=latest \
      --zone=us-east1-d \
      --scopes=cloud-platform \
      --machine-type=n1-standard-4 \
      --enable-autoscaling --min-nodes=1 --max-nodes=3 \
      knative-demo
    ```

    *   Version 1.10+ is required
    *   Change this to whichever zone you choose
    *   cloud-platform scope is required to access GCB
    *   Knative Serving currently requires 4-cpu nodes to run conformance tests.
        Changing the machine type from the default may cause failures.
    *   Autoscale from 1 to 3 nodes. Adjust this for your use case
    *   Change this to your preferred cluster name

    You can see the list of supported cluster versions in a particular zone by
    running:

    ```shell
    # Get the list of valid versions in us-east1-d
    gcloud container get-server-config --zone us-east1-d
    ```

1.  **Alternately**, if you wish to re-use an already-created cluster,
    you can fetch the credentials to your local machine with:

    ```shell
    # Load credentials for the new cluster in us-east1-d
    gcloud container clusters get-credentials --zone us-east1-d knative-demo
    ```

1.  If you haven't installed `kubectl` yet, you can install it now with
    `gcloud`:

    ```shell
    gcloud components install kubectl
    ```

1.  Add to your .bashrc:
    ```shell
    # When using GKE, the K8s user is your GCP user.
    export K8S_USER_OVERRIDE=$(gcloud config get-value core/account)
    ```

## Minikube

1.  [Install and configure
    minikube](https://github.com/kubernetes/minikube#minikube) with a [VM
    driver](https://github.com/kubernetes/minikube#requirements), e.g. `kvm2` on
    Linux or `hyperkit` on macOS.

1.  [Create a cluster](https://github.com/kubernetes/minikube#quickstart) with
    version 1.10 or greater and your chosen VM driver.

    The following commands will setup a cluster with `8192 MB` of memory and `4`
    CPUs. If you want to [enable metric and log collection](./DEVELOPMENT.md#enable-log-and-metric-collection),
    bump the memory to `12288 MB`.

    _Providing any admission control pluins overrides the default set provided
    by minikube so we must explicitly list all plugins we want enabled._

    _Until minikube [makes this the
    default](https://github.com/kubernetes/minikube/issues/1647), the
    certificate controller must be told where to find the cluster CA certs on
    the VM._

    For Linux use:

    ```shell
    minikube start --memory=8192 --cpus=4 \
    --kubernetes-version=v1.10.4 \
    --vm-driver=kvm2 \
    --bootstrapper=kubeadm \
    --extra-config=controller-manager.cluster-signing-cert-file="/var/lib/localkube/certs/ca.crt" \
    --extra-config=controller-manager.cluster-signing-key-file="/var/lib/localkube/certs/ca.key" \
    --extra-config=apiserver.admission-control="DenyEscalatingExec,LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook"
    ```
    For macOS use:

    ```shell
    minikube start --memory=8192 --cpus=4 \
    --kubernetes-version=v1.10.4 \
    --vm-driver=hyperkit \
    --bootstrapper=kubeadm \
    --extra-config=controller-manager.cluster-signing-cert-file="/var/lib/localkube/certs/ca.crt" \
    --extra-config=controller-manager.cluster-signing-key-file="/var/lib/localkube/certs/ca.key" \
    --extra-config=apiserver.admission-control="DenyEscalatingExec,LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook"
    ```
1.  [Configure your shell environment](../DEVELOPMENT.md#environment-setup)
    to use your minikube cluster:

    ```shell
    export K8S_CLUSTER_OVERRIDE='minikube'
    # When using Minikube, the K8s user is your local user.
    export K8S_USER_OVERRIDE=$USER
    ```

1.  [Start Knative Serving](../DEVELOPMENT.md#starting-knative-serving):

    You need to deploy Knative itself a bit differently if you would like to have
    routeable `Route` endpoints, and if you would like to setup secrets for
    accessing an image registry from within your cluster, e.g.
    [GCR](#minikube-with-gcr):

    1. [Deploy istio](../DEVELOPMENT.md#deploy-istio):

       By default istio uses a `LoadBalancer` which is not available for Minikube
       but is required for Knative to function properly (`Route` endpoints MUST
       be available via a routable IP before they will be marked as ready)
       so deploy istio with `LoadBalancer` replaced by `NodePort`:

       ```bash
       sed 's/LoadBalancer/NodePort/' third_party/istio-0.8.0/istio.yaml | kubectl apply -f -
       ```

       (Then optionally [enable istio injection](../DEVELOPMENT.md#deploy-istio).)

    1. [Deploy build](../DEVELOPMENT.md#deploy-build):
        
        ```shell
        kubectl apply -f ./third_party/config/build/release.yaml
        ```

    1. [Deploy Knative Serving](../DEVELOPMENT.md#deploy-knative-serving):

        1. Create the namespaces and service accounts you'll be modifying:
           ```bash
           ko apply -f config/100-namespace.yaml
           ko apply -f config/200-serviceaccount.yaml
           ```
        1. Create the secrets you need and patch the service accounts to use them, e.g.
           [to setup access to GCR](#minikube-with-gcr).
        1. Deploy the rest of Knative:
           ```bash
           # Use the minikube docker daemon (among other things)
           eval $(minikube docker-env)

           # Switch the current kubectl context to minikube
           kubectl config use-context minikube

           # Deploy to minikube w/o registry.
           ko apply -L -f config/
           ```

    1. [Enable log and metric collection](../DEVELOPMENT.md#enable-log-and-metric-collection)
       (Note you should be using a cluster with 12 GB of memory, see commands above.)


### Enabling Knative to Use Images in Minikube

In order to have Knative access an image in Minikube's Docker daemon you
should prefix your image name with the `dev.local` registry. This will cause
Knative to use the cached image. You must not tag your image as `latest` since
this causes Kubernetes to [always attempt a pull](https://kubernetes.io/docs/concepts/containers/images/#updating-images).

For example:

```shell
eval $(minikube docker-env)
docker pull gcr.io/knative-samples/primer:latest
docker tag gcr.io/knative-samples/primer:latest dev.local/knative-samples/primer:v1
```

### Minikube locally with `ko`

You can instruct `ko` to sideload images into your Docker daemon instead of
publishing them to a registry via the `-L` (local) flag:

```shell
# Use the minikube docker daemon (among other things)
eval $(minikube docker-env)

# Switch the current kubectl context to minikube
kubectl config use-context minikube

# Deploy to minikube w/o registry.
ko apply -L -f config/
```

Alternatively (if you don't like flags), set `KO_DOCKER_REPO` to `ko.local`:

```shell
# Use the minikube docker daemon (among other things)
eval $(minikube docker-env)

# Switch the current kubectl context to minikube
kubectl config use-context minikube

# Set KO_DOCKER_REPO to a sentinel value for ko to sideload into the daemon.
export KO_DOCKER_REPO="ko.local"

# Deploy to minikube w/o registry.
ko apply -f config/
```

### Minikube with GCR

You can use Google Container Registry as the registry for a Minikube cluster.

1.  [Set up a GCR repo](setting-up-a-docker-registry.md). Export the environment
    variable `PROJECT_ID` as the name of your project. Also export `GCR_DOMAIN`
    as the domain name of your GCR repo. This will be either `gcr.io` or a
    region-specific variant like `us.gcr.io`.

    ```shell
    export PROJECT_ID=knative-demo-project
    export GCR_DOMAIN=gcr.io
    ```

    To publish builds push to GCR, set `KO_DOCKER_REPO` or
    `DOCKER_REPO_OVERRIDE` to the GCR repo's url.

    ```shell
    export KO_DOCKER_REPO="${GCR_DOMAIN}/${PROJECT_ID}"
    export DOCKER_REPO_OVERRIDE="${KO_DOCKER_REPO}"
    ```

1.  Create a GCP service account:

    ```shell
    gcloud iam service-accounts create minikube-gcr \
      --display-name "Minikube GCR Pull" \
      --project $PROJECT_ID
    ```

1.  Give your service account the `storage.objectViewer` role:

    ```shell
    gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member "serviceAccount:minikube-gcr@${PROJECT_ID}.iam.gserviceaccount.com" \
      --role roles/storage.objectViewer
    ```

1.  Create a key credential file for the service account:

    ```shell
    gcloud iam service-accounts keys create \
      --iam-account "minikube-gcr@${PROJECT_ID}.iam.gserviceaccount.com" \
      minikube-gcr-key.json
    ```

Now you can use the `minikube-gcr-key.json` file to create image pull secrets
and link them to Kubernetes service accounts. _A secret must be created and
linked to a service account in each namespace that will pull images from GCR._

For example, use these steps to allow Minikube to pull Knative Serving and Build images
from GCR as published in our development flow (`ko apply -f config/`).
_This is only necessary if you are not using public Knative Serving and Build images._

1.  Create a Kubernetes secret in the `knative-serving` and `knative-build` namespace:

    ```shell
    export DOCKER_EMAIL=your.email@here.com
    kubectl create secret docker-registry "knative-serving-gcr" \
      --docker-server=$GCR_DOMAIN \
      --docker-username=_json_key \
      --docker-password="$(cat minikube-gcr-key.json)" \
      --docker-email=$DOCKER_EMAIL \
      -n "knative-serving"
    kubectl create secret docker-registry "build-gcr" \
      --docker-server=$GCR_DOMAIN \
      --docker-username=_json_key \
      --docker-password="$(cat minikube-gcr-key.json)" \
      --docker-email=$DOCKER_EMAIL \
      -n "knative-build"
    ```

    _The secret must be created in the same namespace as the pod or service
    account._

1.  Add the secret as an imagePullSecret to the `controller` and
    `build-controller` service accounts:

    ```shell
    kubectl patch serviceaccount "build-controller" \
      -p '{"imagePullSecrets": [{"name": "build-gcr"}]}' \
      -n "knative-build"
    kubectl patch serviceaccount "controller" \
      -p '{"imagePullSecrets": [{"name": "knative-serving-gcr"}]}' \
      -n "knative-serving"
    ```
Use the same procedure to add imagePullSecrets to service accounts in any
namespace. Use the `default` service account for pods that do not specify a
service account.

See also the [private-repo sample README](./../sample/private-repos/README.md).