* Adds secretscope and workspaceitem Ops to Azure Databricks for KFP
* Adds dbfsblock Op to Azure Databricks for KFP
* Adds code coverage to Azure Databricks for KFP tests
* Changes email address to generic one
* Sets single source of truth for package version
* Removes all warnings from tests
* Removes deprecated calls to Compiler().compile in pipelines
* Removes unnecessary .gitignore line
Co-authored-by: creddy123 <31089923+creddy123@users.noreply.github.com>
* Add samples to manage Azure Databricks in Kubeflow Pipelines.
azure-samples/kfp-azure-databricks folder contains a set of Ops which
allow us to manage Azure Databricks resources directly from our Kubeflow
Pipelines.
These Ops leverage Azure Databricks Operator for Kubernetes to manage
those resources. They wrap ResourceOp and abstract pipeline users from
Kubernetes Operator details.
Supported Ops:
* CreateClusterOp, to create a cluster in Databricks.
* DeleteClusterOp, to delete an existing cluster from Databricks.
* CreateJobOp, to create a Spark Job in Databricks.
* DeleteJobOp, to delete an existing Spark Job from Databricks.
* SubmitRunOp, to submit a Job Run in Databricks.
* DeleteRunOp, to delete an existing Run.
azure-samples/databricks-pipelines folder contains several pipeline
samples:
* databricks_operator_*.py samples show how to use ResourceOp to
manage Databricks resources using Azure Databricks Operator.
* databricks_pkg_*.py samples show how to use Ops in
kfp-azure-databricks package to do the same.
* Fix small bug on script to compile, upload and run azure pipeline samples
* Removes azure databricks operator pipeline samples
* Added new tests for Azure Databricks Delete Ops plus minor tweaks to those Ops
* Removes bash script and adds Python CLI to compile and run pipelines
* Changes the way kfp host is handled in pipeline cli
Alejandro Campos Magencio