kubeflow
/
examples
mirror of https://github.com/kubeflow/examples.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
examples/kfp-spark
History
Vedant Padwal 468c330bf3
Fix image path in kfp-spark 		2022-01-14 17:36:21 +05:30
..
images Add kfp-spark example 2022-01-10 21:13:49 +05:30
LICENSE Add kfp-spark example 2022-01-10 21:13:49 +05:30
OWNERS Add Owners in KFP-SPARK example 2022-01-14 16:58:11 +05:30
README.md Fix image path in kfp-spark 2022-01-14 17:36:21 +05:30
k8s-apply-component.yaml Add kfp-spark example 2022-01-10 21:13:49 +05:30
k8s-get-component.yaml Add kfp-spark example 2022-01-10 21:13:49 +05:30
kubeflow-pipeline.ipynb Add kfp-spark example 2022-01-10 21:13:49 +05:30
spark-job.yaml Add kfp-spark example 2022-01-10 21:13:49 +05:30
spark-rbac.yaml Add kfp-spark example 2022-01-10 21:13:49 +05:30
spark_job_pipeline.yaml Add kfp-spark example 2022-01-10 21:13:49 +05:30

README.md

KFP version: 1.7.0+ Kubernetes version: 1.17+

Orchestrate Spark Jobs using Kubeflow pipelines

Install kubeflow pipelines standalone or full kubeflow

for standalone kubeflow pipelines installation

https://www.kubeflow.org/docs/components/pipelines/installation/

for full kubeflow installation

https://www.kubeflow.org/docs/started/installing-kubeflow/

Install Spark Operator

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator#installation

Create Spark Service Account and add permissions

kubectl apply -f ./scripts/spark-rbac.yaml

Run the notebok kubeflow-pipeline.ipynb

Access Kubflow/KFP UI

image

OR

image

Upload pipeline

Upload the spark_job_pipeline.yaml file

image

Create Run

image

Start Pipeline add service account spark-sa

image

Wait till the execution is finished. check the print-message logs to view the result

image