5.9 KiB
Using the command line to deploy the application
Only For Development
This CLI installation guide is ONLY for AI Platform Pipelines development. It's not suitable for users of Kubeflow Pipelines.
For users, please refer to Kubeflow Pipelines Standalone on how to install via commandline using kustomize manifests. The installation is almost the same as AI Platform Pipelines.
Refer to the Installation Options for Kubeflow Pipelines doc for all installation options and their feature comparison.
Prerequisites
You can use Google Cloud Shell or a local workstation to complete these steps.
Set up command-line tools
You'll need the following tools in your development environment. If you are using Cloud Shell, these tools are installed in your environment by default.
Configure gcloud as a Docker credential helper:
gcloud auth configure-docker
You can install Kubeflow Pipelines in an existing GKE cluster or create a new GKE cluster.
-
If you want to create a new Google GKE cluster, follow the instructions from the section Create a GKE cluster onwards.
-
If you have an existing GKE cluster, ensure that the cluster nodes have a minimum 3 node cluster with each node having a minimum of 2 vCPU and running k8s version 1.9 and follow the instructions from section Install the application resource definition onwards.
Create a GKE cluster
Kubeflow Pipelines requires a minimum 3 node cluster with each node having a minimum of 2 vCPU and k8s version 1.9. Available machine types can be seen here.
Create a new cluster from the command line:
# The following parameters can be customized based on your needs.
export CLUSTER=kubeflow-pipelines-cluster
export ZONE=us-west1-a
export MACHINE_TYPE=n1-standard-2
export SCOPES="cloud-platform" # This scope is needed for running some pipeline samples. Read the warning below for its security implications.
gcloud config set project <your_project>
gcloud container clusters create "$CLUSTER" --zone "$ZONE" --machine-type "$MACHINE_TYPE" \
--scopes $SCOPES
Configure kubectl to connect to the new cluster:
gcloud container clusters get-credentials "$CLUSTER" --zone "$ZONE"
Warning: Using
SCOPES="cloud-platform"grants all GCP permissions to the cluster. For a more secure cluster setup, refer to Authenticating Pipelines to GCP.
Install the application resource definition
An application resource is a collection of individual Kubernetes components, such as services, stateful sets, deployments, and so on, that you can manage as a group.
To set up your cluster to understand application resources, run the following command:
kubectl apply -f "https://raw.githubusercontent.com/GoogleCloudPlatform/marketplace-k8s-app-tools/master/crd/app-crd.yaml"
You need to run this command once.
The application resource is defined by the Kubernetes SIG-apps community. The source code can be found on github.com/kubernetes-sigs/application.
Prerequisites for using Role-Based Access Control
You must grant your user the ability to create roles in Kubernetes by running the following command.
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user $(gcloud config get-value account)
You need to run this command once.
Install the Application
Run installer script
Set your application instance name and the Kubernetes namespace to deploy:
# set the application instance name
export APP_INSTANCE_NAME=kubeflow-pipelines-app
# set the Kubernetes namespace the application was originally installed
export NAMESPACE=<namespace>
Creat the namespace
kubectl create namespace $NAMESPACE
Follow the instruction and install mpdev TODO: The official mpdev won't work because it doesn't have permission to deploy CRD. The latest unofficial build will have right permission. Remove following instruction when change is in prod.
BIN_FILE="$HOME/bin/mpdev"
docker run gcr.io/cloud-marketplace-staging/marketplace-k8s-app-tools/k8s/dev:remove-ui-ownerrefs cat /scripts/dev > "$BIN_FILE"
chmod +x "$BIN_FILE"
export MARKETPLACE_TOOLS_TAG=remove-ui-ownerrefs
export MARKETPLACE_TOOLS_IMAGE=gcr.io/cloud-marketplace-staging/marketplace-k8s-app-tools/k8s/dev
Run the install script
mpdev scripts/install --deployer=gcr.io/ml-pipeline/google/pipelines/deployer:0.1 --parameters='{"name": "'$APP_INSTANCE_NAME'", "namespace": "'$NAMESPACE'"}'
Watch the deployment come up with
kubectl get pods -n $NAMESPACE --watch
Get public endpoint
kubectl describe configmap inverse-proxy-config -n $NAMESPACE | grep googleusercontent.com
Delete the Application
kubectl delete applications -n $NAMESPACE $APP_INSTANCE_NAME