Upgrade demo to KF v0.3.1 (#278)

* Upgrade demo to KF v0.3.1 Update env variable names and values in base file Cleanup ambassador metadata for UI component Add kfctl installation instructions Tighten minikube setup instructions and update k8s version Move environment variable setup to very beginning Replace cluster creation commands with links to the appropriate section in demo_setup/README.md Replace deploy.sh with kfctl Replace kubeflow-core component with individual components Remove connection to UI pod directly & connect via ambassador instead Add cleanup commands * Clarify wording * Update parameter file Resolve python error with file open Consolidate kubeflow install command
2018-10-26 12:58:01 -07:00 · 2018-10-26 12:58:01 -07:00 · dde7d3ee8e
parent 5c38c96fae
commit dde7d3ee8e
6 changed files with 170 additions and 120 deletions
--- a/demos/yelp_demo/README.md
+++ b/demos/yelp_demo/README.md
@ -18,51 +18,27 @@ Before completing any of the below steps, follow the instructions in
 [./demo_setup](https://github.com/kubeflow/examples/blob/master/demos/yelp_demo/demo_setup/README.md)
 to prepare for demonstrating Kubeflow:

-* [Create a minikube cluster](https://github.com/kubeflow/examples/blob/master/demos/yelp_demo/demo_setup/README.md#4-create-a-minikube-cluster)
-* [Create a GKE cluster](https://github.com/kubeflow/examples/blob/master/demos/yelp_demo/demo_setup/README.md#5-create-a-gke-cluster)
-
-## 1. Create clusters
-
 [Setup your environment](https://github.com/kubeflow/examples/tree/master/demos/yelp_demo/demo_setup#2-set-environment-variables)
 or source the base file:

 ```
 cd demo_setup
 source kubeflow-demo-base.env
+cd ..
 ```

-Create a minikube cluster:
-
-```
-minikube start \
-  --cpus 4 \
-  --memory 8096 \
-  --disk-size=50g \
-  --kubernetes-version v1.10.6
-```
-
-Create a GKE cluster with access to GPUs and TPUs:
-
-```
-gcloud beta container clusters create ${CLUSTER} \
-  --project ${DEMO_PROJECT} \
-  --zone ${ZONE} \
-  --accelerator type=nvidia-tesla-k80,count=2 \
-  --cluster-version 1.10.6-gke.2 \
-  --enable-ip-alias \
-  --enable-tpu \
-  --machine-type n1-highmem-8 \
-  --scopes cloud-platform,compute-rw,storage-rw \
-  --verbosity error
-```
+* [Create a minikube cluster](https://github.com/kubeflow/examples/blob/master/demos/yelp_demo/demo_setup/README.md#4-create-a-minikube-cluster)
+* [Create a GKE cluster](https://github.com/kubeflow/examples/blob/master/demos/yelp_demo/demo_setup/README.md#5-create-a-gke-cluster)

 ## 1. Install kubeflow locally

 Run the following script to create a ksonnet app for Kubeflow and deploy it:

 ```
-export KUBEFLOW_VERSION=0.2.5
-curl https://raw.githubusercontent.com/kubeflow/kubeflow/v${KUBEFLOW_VERSION}/scripts/deploy.sh | bash
+kfctl init localapp --platform minikube
+cd localapp
+kfctl generate k8s
+kfctl apply k8s
 ```

 View the installed components:
@ -76,9 +52,9 @@ kubectl get pod
 Retrieve the following files for the t2tcpu & t2ttpu jobs:

 ```
-cd kubeflow_ks_app
-cp ${REPO_PATH}/demo/components/t2t[ct]pu.* components
-cp ${REPO_PATH}/demo/components/params.* components
+cd ks_app
+cp ${DEMO_REPO}/demo/components/t2t[ct]pu.* components
+cp ${DEMO_REPO}/demo/components/params.* components
 ```

 Set parameter values for training:
@ -119,13 +95,13 @@ kubectl config use gke
 Create an environment:

 ```
-ks env add gke
+ks env add gke --namespace=kubeflow
 ```

 Install kubeflow on the cluster:

 ```
-ks apply gke -c kubeflow-core
+ks apply gke -c ambassador -c centraldashboard -c jupyterhub -c spartakus -c tf-job-operator
 ```

 View the installed components:
@ -199,8 +175,8 @@ kubectl logs -f t2ttpu-master-0 | grep INFO:tensorflow
 Retrieve the following files for the serving & UI components:

 ```
-cp ${REPO_PATH}/demo/components/serving.* components
-cp ${REPO_PATH}/demo/components/ui.* components
+cp ${DEMO_REPO}/demo/components/serving.* components
+cp ${DEMO_REPO}/demo/components/ui.* components
 ```

 Create the serving and UI components:
@ -209,31 +185,7 @@ Create the serving and UI components:
 ks apply gke -c serving -c ui
 ```

-Forward the port for viewing from a local browser:
-```
-UI_POD=$(kubectl get po -l app=kubeflow-demo-ui | \
-  grep kubeflow-demo-ui | \
-  head -n 1 | \
-  cut -d " " -f 1 \
-)
-kubectl port-forward ${UI_POD} 8080:80
-```
-
-Optional: If necessary, setup an SSH tunnel from your local laptop into the
-compute instance connecting to GKE:
-
-```
-ssh $HOST -L 8080:localhost:8080
-```
-
-To show the naive version, navigate to [localhost:8080](localhost:8080) from a browser.
-
-To show the ML version, navigate to
-[localhost:8080/kubeflow](localhost:8080/kubeflow) from a browser.
-
-## 6. Bring up a notebook
-
-Connect to the Central Dashboard by forwarding a port to one of the ambassador
+Connect to the UI by forwarding a port to one of the ambassador
 pods:

 ```
@ -242,10 +194,25 @@ AMBASSADOR_POD=$(kubectl get po -l service=ambassador | \
  head -n 1 | \
  cut -d " " -f 1 \
 )
-kubectl port-forward ${AMBASSADOR_POD} 8081:80
+kubectl port-forward ${AMBASSADOR_POD} 8080:80
 ```

-Open a browser and connect to [localhost:8081](localhost:8081).
+Optional: If necessary, setup an SSH tunnel from your local laptop into the
+compute instance connecting to GKE:
+
+```
+ssh ${HOST} -L 8080:localhost:8080
+```
+
+To show the naive version, navigate to
+[localhost:8080/kubeflow_demo](localhost:8080/kubeflow_demo) from a browser.
+
+To show the ML version, navigate to
+[localhost:8080/kubeflow_demo/kubeflow](localhost:8080/kubeflow_demo/kubeflow) from a browser.
+
+## 6. Bring up a notebook
+
+Open a browser and connect to the Central Dashboard at [localhost:8080/](localhost:8080/).
 Show the TF-job dashboard, then click on Jupyterhub.
 Log in with any username and password combination and wait until the page
 refreshes. Spawn a new pod with these resource requirements:
@ -263,3 +230,12 @@ available, open a new terminal and upload this

 Execute the notebook to show that it works.

+## 7. Cleanup
+
+```
+kubectl config use gke
+kubectl delete namespace kubeflow
+kubectl config use minikube
+kubectl delete namespace kubeflow
+```
+
--- a/demos/yelp_demo/demo/components/params.libsonnet
+++ b/demos/yelp_demo/demo/components/params.libsonnet
@ -1,32 +1,86 @@
 {
-  global: {
-    // User-defined global parameters; accessible to all component and environments, Ex:
-    // replicas: 4,
-  },
+  global: {},
  components: {
    // Component-level parameters, defined initially from 'ks prototype use ...'
    // Each object below should correspond to a component in the components/ directory
-    "kubeflow-core": {
-      AmbassadorImage: "quay.io/datawire/ambassador:0.30.1",
-      AmbassadorServiceType: "ClusterIP",
-      StatsdImage: "quay.io/datawire/statsd:0.30.1",
-      centralUiImage: "gcr.io/kubeflow-images-public/centraldashboard:v20180618-v0.2.0-rc.0-5-g715aafc8-e3b0c4",
-      cloud: "gke",
-      disks: "null",
-      jupyterHubAuthenticator: "null",
-      jupyterHubImage: "gcr.io/kubeflow/jupyterhub-k8s:v20180531-3bb991b1",
-      jupyterHubServiceType: "ClusterIP",
-      jupyterNotebookPVCMount: "null",
-      jupyterNotebookRegistry: "gcr.io",
-      jupyterNotebookRepoName: "kubeflow-images-public",
-      name: "kubeflow-core",
-      namespace: "null",
-      reportUsage: "false",
-      tfDefaultImage: "null",
-      tfJobImage: "gcr.io/kubeflow-images-public/tf_operator:v20180809-d2509aa",
-      tfJobUiServiceType: "ClusterIP",
-      tfJobVersion: "v1alpha2",
-      usageId: "unknown_cluster",
+    "pytorch-operator": {
+      cloud: 'null',
+      deploymentNamespace: 'null',
+      deploymentScope: 'cluster',
+      disks: 'null',
+      name: 'pytorch-operator',
+      pytorchDefaultImage: 'null',
+      pytorchJobImage: 'gcr.io/kubeflow-images-public/pytorch-operator:v0.3.0',
+      pytorchJobVersion: 'v1alpha2',
+    },
+    ambassador: {
+      ambassadorImage: 'quay.io/datawire/ambassador:0.37.0',
+      ambassadorServiceType: 'ClusterIP',
+      cloud: 'null',
+      name: 'ambassador',
+      platform: 'minikube',
+    },
+    jupyterhub: {
+      accessLocalFs: 'false',
+      cloud: 'null',
+      disks: 'null',
+      gcpSecretName: 'user-gcp-sa',
+      image: 'gcr.io/kubeflow/jupyterhub-k8s:v20180531-3bb991b1',
+      jupyterHubAuthenticator: 'null',
+      name: 'jupyterhub',
+      notebookGid: '-1',
+      notebookPVCMount: '/home/jovyan',
+      notebookUid: '-1',
+      platform: 'minikube',
+      registry: 'gcr.io',
+      repoName: 'kubeflow-images-public',
+      serviceType: 'ClusterIP',
+      useJupyterLabAsDefault: 'false',
+    },
+    centraldashboard: {
+      image: 'gcr.io/kubeflow-images-public/centraldashboard:v0.3.0',
+      name: 'centraldashboard',
+    },
+    "tf-job-operator": {
+      cloud: 'null',
+      deploymentNamespace: 'null',
+      deploymentScope: 'cluster',
+      name: 'tf-job-operator',
+      tfDefaultImage: 'null',
+      tfJobImage: 'gcr.io/kubeflow-images-public/tf_operator:v0.3.0',
+      tfJobUiServiceType: 'ClusterIP',
+      tfJobVersion: 'v1alpha2',
+    },
+    argo: {
+      executorImage: 'argoproj/argoexec:v2.2.0',
+      name: 'argo',
+      uiImage: 'argoproj/argoui:v2.2.0',
+      workflowControllerImage: 'argoproj/workflow-controller:v2.2.0',
+    },
+    katib: {
+      modeldbDatabaseImage: 'mongo:3.4',
+      modeldbFrontendImage: 'gcr.io/kubeflow-images-public/katib/katib-frontend:v0.1.2-alpha-45-g3dce496',
+      modeldbImage: 'gcr.io/kubeflow-images-public/modeldb-backend:v0.2.0',
+      name: 'katib',
+      studyJobControllerImage: 'katib/studyjob-controller',
+      suggestionBayesianOptimizationImage: 'gcr.io/kubeflow-images-public/katib/suggestion-bayesianoptimization:v0.1.2-alpha-45-g3dce496',
+      suggestionGridImage: 'gcr.io/kubeflow-images-public/katib/suggestion-grid:v0.1.2-alpha-45-g3dce496',
+      suggestionHyperbandImage: 'gcr.io/kubeflow-images-public/katib/suggestion-hyperband:v0.1.2-alpha-45-g3dce496',
+      suggestionRandomImage: 'gcr.io/kubeflow-images-public/katib/suggestion-random:v0.1.2-alpha-45-g3dce496',
+      vizierCoreImage: 'gcr.io/kubeflow-images-public/katib/vizier-core:v0.1.2-alpha-45-g3dce496',
+      vizierDbImage: 'mysql:8.0.3',
+    },
+    spartakus: {
+      name: 'spartakus',
+      reportUsage: 'true',
+      usageId: '819152435',
+    },
+    application: {
+      components: [],
+      emitCRD: 'true',
+      name: 'application',
+      type: 'kubeflow',
+      version: '0.3',
    },
    "t2tcpu": {
      workerGpu: 0,
@ -54,23 +108,6 @@
      modelServerImage: "gcr.io/kubeflow-images-public/tf-model-server-cpu:v20180523-2a68f293",
      name: "serving"
    },
-    "cert-manager": {
-      acmeEmail: "google-kubeflow-team@google.com",
-      acmeUrl: "https://acme-v01.api.letsencrypt.org/directory",
-      name: "cert-manager",
-      namespace: "kubeflow",
-    },
-    "iap-ingress": {
-      disableJwtChecking: "false",
-      envoyImage: "gcr.io/kubeflow-images-staging/envoy:v20180309-0fb4886b463698702b6a08955045731903a18738",
-      hostname: "kubecon-keynote-demo.kubeflow.dev",
-      ipName: "static-ip",
-      issuer: "letsencrypt-prod",
-      name: "iap-ingress",
-      namespace: "kubeflow",
-      oauthSecretName: "kubeflow-oauth",
-      secretName: "envoy-ingress-tls",
-    },
    "ui": {
      image: "gcr.io/kubeflow-demo-base/kubeflow-yelp-demo-ui:latest",
    },
--- a/demos/yelp_demo/demo/components/ui.libsonnet
+++ b/demos/yelp_demo/demo/components/ui.libsonnet
@ -7,7 +7,16 @@
        name: "kubeflow-demo-ui",
        namespace: env.namespace,
        annotations: {
-          "getambassador.io/config": "---\napiVersion: ambassador/v0\nkind:  Mapping\nname:  kubeflow_demo_ui\nprefix: /kubeflow-demo/\nrewrite: /\nservice: kubeflow-demo-ui:80\n",
+          "getambassador.io/config":
+            std.join("\n", [
+              "---",
+              "apiVersion: ambassador/v0",
+              "kind: Mapping",
+              "name: kubeflow_demo_ui",
+              "prefix: /kubeflow_demo/",
+              "rewrite: /",
+              "service: kubeflow-demo-ui:80",
+            ]),
        },
      },
      spec: {
--- a/demos/yelp_demo/demo_setup/README.md
+++ b/demos/yelp_demo/demo_setup/README.md
@ -19,6 +19,7 @@ Ensure that you have at least the below versions of these tools (latest as of

 * [docker](#install-docker) v18.03.1-ce
 * [gcloud](#install-gcloud) v202.0.0
+* [kfctl](#install-kfctl) v0.3.1
 * [ksonnet](#install-ksonnet) v0.12.0
 * [kubectl](#install-kubectl) v1.10.3
 * [miniconda](#install-miniconda) v4.4.10
@ -37,6 +38,20 @@ The latest version for MacOS can be found
 The Google Cloud SDK can be found
 [here](https://cloud.google.com/sdk/downloads).

+### Install kfctl
+
+Clone the Kubeflow GitHub repository, create a symlink to `kfctl.sh`, and add the
+directory to your $PATH:
+
+```
+export KUBEFLOW_TAG=v0.3.1
+git clone git@github.com:kubeflow/kubeflow.git
+cd kubeflow/scripts
+git checkout ${KUBEFLOW_TAG}
+ln -s kfctl.sh kfctl
+export PATH=${PATH}:`pwd`
+```
+
 ### Install ksonnet

 Download the correct binary based on your OS distro. The latest release can be found
@ -335,9 +350,9 @@ To start a minikube instance:
 ```
 minikube start \
  --cpus 4 \
-  --memory 8096 \
+  --memory 8192 \
  --disk-size=50g \
-  --kubernetes-version v1.10.6
+  --kubernetes-version v1.10.7
 ```

 ### Create k8s secrets
@ -348,16 +363,26 @@ credentials. One of type `docker-registry` for pulling images from GCR and one
 one of type `generic` for accessing private assets.

 ```
-kubectl -n $NAMESPACE create secret docker-registry gcp-registry-credentials \
+kubectl create namespace ${NAMESPACE}
+
+kubectl -n ${NAMESPACE} create secret docker-registry gcp-registry-credentials \
  --docker-server=gcr.io \
  --docker-username=_json_key \
-  --docker-password="$(cat $HOME/.ssh/minikube_key.json)" \
+  --docker-password="$(cat ${HOME}/.ssh/minikube_key.json)" \
  --docker-email=minikube@${DEMO_PROJECT}.iam.gserviceaccount.com

-kubectl -n $NAMESPACE create secret generic gcp-credentials \
+kubectl -n ${NAMESPACE} create secret generic gcp-credentials \
  --from-file=key.json="${HOME}/.ssh/minikube_key.json"
 ```

+### Setup context to include namespace
+
+This allows the use of `kubectl` without needing to specify `-n ${NAMESPACE}`
+
+```
+./create_context.sh minikube ${NAMESPACE}
+```
+
 ### Prepare the ksonnet app

 Create the minikube environment:
@ -473,13 +498,13 @@ credentials. One of type `docker-registry` for pulling images from GCR and one
 one of type `generic` for accessing private assets.

 ```
-kubectl -n $NAMESPACE create secret docker-registry gcp-registry-credentials \
+kubectl -n ${NAMESPACE} create secret docker-registry gcp-registry-credentials \
  --docker-server=gcr.io \
  --docker-username=_json_key \
-  --docker-password="$(cat $HOME/.ssh/${CLUSTER}_key.json)" \
+  --docker-password="$(cat ${HOME}/.ssh/${CLUSTER}_key.json)" \
  --docker-email=${CLUSTER}@${DEMO_PROJECT}.iam.gserviceaccount.com

-kubectl -n $NAMESPACE create secret generic gcp-credentials \
+kubectl -n ${NAMESPACE} create secret generic gcp-credentials \
  --from-file=key.json="${HOME}/.ssh/${CLUSTER}_key.json"
 ```

--- a/demos/yelp_demo/demo_setup/kubeflow-demo-base.env
+++ b/demos/yelp_demo/demo_setup/kubeflow-demo-base.env
@ -1,13 +1,14 @@
 export DEMO_PROJECT=kubeflow-demo-base
-export NAMESPACE=default
+export NAMESPACE=kubeflow
 export ZONE=us-central1-a
-export CLUSTER=demo
+export CLUSTER=kfdemo
 # Makefile uses project.
 export PROJECT=${DEMO_PROJECT}
 export ENV=gke
 export SVC_ACCT=minikube
-export KUBEFLOW_VERSION=v0.2.5
-export REPO_PATH=${HOME}/repos/kubeflow/examples/demos/yelp_demo
+export KUBEFLOW_TAG=v0.3.1
+export KUBEFLOW_REPO=${HOME}/repos/kubeflow/kubeflow
+export DEMO_REPO=${HOME}/repos/kubeflow/examples/demos/yelp_demo

 export MAX_CASES=1000000
 export DATA_DIR=featurization/yelp-data-${MAX_CASES}
--- a/demos/yelp_demo/notebooks/yelp.ipynb
+++ b/demos/yelp_demo/notebooks/yelp.ipynb
@ -12,6 +12,8 @@
    "# Grab the data and progress bar\n",
    "# We only need to do this once.\n",
    "!pip install bokeh\n",
+    "import codecs\n",
+    "from io import open\n",
    "!wget https://storage.googleapis.com/aai17/yelp_dataset.tar\n",
    "!tar xfvz yelp_dataset.tar\n",
    "!mv dataset/review.json yelp_reviews.json\n"