kubeflow
/
pipelines
mirror of https://github.com/kubeflow/pipelines.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
pipelines/manifests/kustomize/sample
History
Yuan (Bob) Gong be64b32798
feat: update argo image to v2.12.9 and automate update process. Fixes #5232 (#5266) 
* add notices and licenses for argo 2.12

* feat: upgrade argo images to v2.12.9

* update all refs to argo image version

* add NOTICES generation script

* upgrade argo cli to latest

* fix

* fix

* add license_info.csv back

* make release process safer

* add back third_party/license.txt

* refactor(deployment): move argo manifests to third-party, updates for 2.12.9

* update marketplace snapshots

* set up marketplace presubmit test

* add comment
 		2021-03-10 12:52:42 +08:00
..
cluster-scoped-resources [Manifest] Apply kustomize best practices to standalone manifest (#3978) 2020-06-15 19:09:57 -07:00
README.md feat(deployment): GCP managed storage - detailed instructions to set up workload identity bindings before deployment (#4232) 2020-07-16 23:13:00 -07:00
kustomization.yaml feat: update argo image to v2.12.9 and automate update process. Fixes #5232 (#5266) 2021-03-10 12:52:42 +08:00
params-db-secret.env OSS 1.0 Kustomize part-2 parameterize & fix CloudSQL (#3540) 2020-04-20 18:46:35 +08:00
params.env use better sample name (#3558) 2020-04-20 18:57:35 +08:00

README.md

Sample installation

  1. Prepare a cluster and setup kubectl context Do whatever you want to customize your cluster. You can use existing cluster or create a new one.
  • ML Usage GPU normally is required for deep learning task. You may consider create zero-sized GPU node-pool with autoscaling. Please reference GPU Tutorial.
  • Security You may consider use Workload Identity in GCP cluster.

Here for simplicity, we create a small cluster with --scopes=cloud-platform which grants all the GCP permissions to the cluster.

gcloud container clusters create mycluster \
  --zone us-central1-a \
  --machine-type n1-standard-2 \
  --scopes cloud-platform \
  --enable-autoscaling \
  --min-nodes 1 \
  --max-nodes 5 \
  --num-nodes 3
  1. Prepare CloudSQL

Create CloudSQL instance. Console.

Here is a sample for demo.

gcloud beta sql instances create mycloudsqlname \
  --database-version=MYSQL_5_7 \
  --tier=db-n1-standard-1 \
  --region=us-central1 \
  --root-password=password123

You may use Private IP to well protect your CloudSQL. If you use Private IP, please go to VPC network peering to double check whether the "cloudsql-mysql-googleais-com" is created and the "Exchange custom routes" is enabled. You are expected to see "Peer VPC network is connected".

  1. Prepare GCS Bucket

Create Cloud Storage bucket. Console.

gsutil mb -p myProjectId gs://myBucketName/
  1. Customize your values
  • Edit params.env, params-db-secret.env and cluster-scoped-resources/params.env
  • Edit kustomization.yaml to set your namespace, e.x. "kubeflow"
  1. (Optional.) If the cluster is on Workload Identity, please run gcp-workload-identity-setup.sh The script prints usage documentation when calling without argument. Note, you should call it with USE_GCP_MANAGED_STORAGE=true env var.
  • make sure the Google Service Account (GSA) can access the CloudSQL instance and GCS bucket
  • if your workload calls other GCP APIs, make sure the GSA can access them
  1. Install
kubectl apply -k sample/cluster-scoped-resources/

kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s

kubectl apply -k sample/
# If upper one action got failed, e.x. you used wrong value, try delete, fix and apply again
# kubectl delete -k sample/

kubectl wait applications/mypipeline -n kubeflow --for condition=Ready --timeout=1800s

Now you can find the installation in Console