mirror of https://github.com/kubeflow/website.git
doc(kfp): connecting KFP SDK client to API generic introduction (#2729)
* docs(kfp): connect to api doc * doc(kfp): generic introduction for connecting KFP SDK client to API * address comments * doc(kfp): update all usages for KFP SDK client * fix a minor issue + add a script to batch convert notebooks * fix url * python-function-components.ipynb * fix all notebooks * fix pipeline_root
This commit is contained in:
Yuan (Bob) Gong
GitHub
parent
88961e7725
commit
4636ab5dbe
|
|
@ -11,3 +11,6 @@ resources/
|
|||
|
||||
# IntelliJ
|
||||
.idea/
|
||||
|
||||
# Notebooks
|
||||
.ipynb_checkpoints
|
||||
|
|
|
|||
|
|
@ -181,3 +181,5 @@ notebooks or cron tasks, current suggested workaround is to connect through
|
|||
public endpoint and follow platform specific documentation to authenticate
|
||||
programmatically using user credentials. For Google Cloud, you can refer to
|
||||
[Connecting to Kubeflow Pipelines in a full Kubeflow deployment on Google Cloud](/docs/gke/pipelines/authentication-sdk/#connecting-to-kubeflow-pipelines-in-a-full-kubeflow-deployment).
|
||||
|
||||
There is work-in-progress to support this use-case, refer to [github issue #5138](https://github.com/kubeflow/pipelines/issues/5138).
|
||||
|
|
|
|||
|
|
@ -54,36 +54,6 @@
|
|||
"import kfp.components as comp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "IARdgYktshE8"
|
||||
},
|
||||
"source": [
|
||||
"3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your\n",
|
||||
" Kubeflow Pipelines cluster's hostname and URL scheme, open the Kubeflow\n",
|
||||
" Pipelines user interface in your browser. The URL of the Kubeflow\n",
|
||||
" Pipelines user interface is something like \n",
|
||||
" `https://my-cluster.my-organization.com/pipelines`. In this case, the\n",
|
||||
" host name and URL scheme are `https://my-cluster.my-organization.com`. \n",
|
||||
"\n",
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "XJ4YU9N_sos6"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you run this command on a Jupyter notebook running on Kubeflow, you can\n",
|
||||
"# exclude the host parameter.\n",
|
||||
"# client = kfp.Client()\n",
|
||||
"client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
|
|
@ -519,7 +489,9 @@
|
|||
"source": [
|
||||
"### Compile and run your pipeline\n",
|
||||
"\n",
|
||||
"After defining the pipeline in Python as described in the preceding section, use the following instructions to compile the pipeline and submit it to the Kubeflow Pipelines service. \n",
|
||||
"After defining the pipeline in Python as described in the preceding section, use one of the following options to compile the pipeline and submit it to the Kubeflow Pipelines service.\n",
|
||||
"\n",
|
||||
"#### Option 1: Compile and then upload in UI\n",
|
||||
"\n",
|
||||
"1. Run the following to compile your pipeline and save it as `pipeline.yaml`. \n"
|
||||
]
|
||||
|
|
@ -537,20 +509,44 @@
|
|||
" package_path='pipeline.yaml')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"2. Upload and run your `pipeline.yaml` using the Kubeflow Pipelines user interface.\n",
|
||||
"See the guide to [getting started with the UI][quickstart].\n",
|
||||
"\n",
|
||||
"[quickstart]: https://www.kubeflow.org/docs/components/pipelines/pipelines-quickstart"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "jNLI1-_bfEky"
|
||||
},
|
||||
"source": [
|
||||
"2. Run the following to submit the compiled workflow specification\n",
|
||||
" (`pipeline.yaml`) using the Kubeflow Pipelines SDK. \n",
|
||||
" \n",
|
||||
" You can also use the Kubeflow Pipelines user interface to upload and run\n",
|
||||
" your `pipeline.yaml`. See the guide to [getting started with the\n",
|
||||
" UI][quickstart].\n",
|
||||
"#### Option 2: run the pipeline using Kubeflow Pipelines SDK client\n",
|
||||
"\n",
|
||||
"[quickstart]: https://www.kubeflow.org/docs/components/pipelines/pipelines-quickstart"
|
||||
"1. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].\n",
|
||||
"\n",
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client\n",
|
||||
"[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = kfp.Client() # change arguments accordingly"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"2. Run the pipeline using the `kfp.Client` instance:"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -561,8 +557,8 @@
|
|||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.create_run_from_pipeline_package(\n",
|
||||
" pipeline_file='pipeline.yaml',\n",
|
||||
"client.create_run_from_pipeline_func(\n",
|
||||
" my_pipeline,\n",
|
||||
" arguments={\n",
|
||||
" 'url': 'https://storage.googleapis.com/ml-pipeline-playground/iris-csv-files.tar.gz'\n",
|
||||
" })"
|
||||
|
|
@ -616,9 +612,9 @@
|
|||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.8"
|
||||
"version": "3.7.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
}
|
||||
|
|
@ -56,23 +56,6 @@ import kfp
|
|||
import kfp.components as comp
|
||||
```
|
||||
|
||||
3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your
|
||||
Kubeflow Pipelines cluster's hostname and URL scheme, open the Kubeflow
|
||||
Pipelines user interface in your browser. The URL of the Kubeflow
|
||||
Pipelines user interface is something like
|
||||
`https://my-cluster.my-organization.com/pipelines`. In this case, the
|
||||
host name and URL scheme are `https://my-cluster.my-organization.com`.
|
||||
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client
|
||||
|
||||
|
||||
```python
|
||||
# If you run this command on a Jupyter notebook running on Kubeflow, you can
|
||||
# exclude the host parameter.
|
||||
# client = kfp.Client()
|
||||
client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')
|
||||
```
|
||||
|
||||
## Understanding pipelines
|
||||
|
||||
A Kubeflow pipeline is a portable and scalable definition of an ML workflow,
|
||||
|
|
@ -415,7 +398,9 @@ def my_pipeline(url):
|
|||
|
||||
### Compile and run your pipeline
|
||||
|
||||
After defining the pipeline in Python as described in the preceding section, use the following instructions to compile the pipeline and submit it to the Kubeflow Pipelines service.
|
||||
After defining the pipeline in Python as described in the preceding section, use one of the following options to compile the pipeline and submit it to the Kubeflow Pipelines service.
|
||||
|
||||
#### Option 1: Compile and then upload in UI
|
||||
|
||||
1. Run the following to compile your pipeline and save it as `pipeline.yaml`.
|
||||
|
||||
|
|
@ -427,19 +412,29 @@ kfp.compiler.Compiler().compile(
|
|||
package_path='pipeline.yaml')
|
||||
```
|
||||
|
||||
2. Run the following to submit the compiled workflow specification
|
||||
(`pipeline.yaml`) using the Kubeflow Pipelines SDK.
|
||||
|
||||
You can also use the Kubeflow Pipelines user interface to upload and run
|
||||
your `pipeline.yaml`. See the guide to [getting started with the
|
||||
UI][quickstart].
|
||||
2. Upload and run your `pipeline.yaml` using the Kubeflow Pipelines user interface.
|
||||
See the guide to [getting started with the UI][quickstart].
|
||||
|
||||
[quickstart]: https://www.kubeflow.org/docs/components/pipelines/pipelines-quickstart
|
||||
|
||||
#### Option 2: run the pipeline using Kubeflow Pipelines SDK client
|
||||
|
||||
1. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].
|
||||
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client
|
||||
[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api
|
||||
|
||||
|
||||
```python
|
||||
client.create_run_from_pipeline_package(
|
||||
pipeline_file='pipeline.yaml',
|
||||
client = kfp.Client() # change arguments accordingly
|
||||
```
|
||||
|
||||
2. Run the pipeline using the `kfp.Client` instance:
|
||||
|
||||
|
||||
```python
|
||||
client.create_run_from_pipeline_func(
|
||||
my_pipeline,
|
||||
arguments={
|
||||
'url': 'https://storage.googleapis.com/ml-pipeline-playground/iris-csv-files.tar.gz'
|
||||
})
|
||||
|
|
@ -462,4 +457,4 @@ client.create_run_from_pipeline_package(
|
|||
<div class="notebook-links">
|
||||
<a class="colab-link" href="https://colab.research.google.com/github/kubeflow/website/blob/master/content/en/docs/components/pipelines/sdk/build-pipeline.ipynb">Run in Google Colab</a>
|
||||
<a class="github-link" href="https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/sdk/build-pipeline.ipynb">View source on GitHub</a>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
+++
|
||||
title = "Connecting to Kubeflow Pipelines using the SDK client"
|
||||
description = "How to connect to Kubeflow Pipelines using the SDK client and configure the SDK client using environment variables"
|
||||
weight = 25
|
||||
+++
|
||||
|
||||
This guide demonstrates how to connect to Kubeflow Pipelines using [the Kubeflow Pipelines SDK client](/docs/components/pipelines/sdk/sdk-overview/), and how to [configure the SDK client using environment variables](#configure-sdk-client-by-environment-variables).
|
||||
|
||||
## Before you begin
|
||||
|
||||
* You need a Kubeflow Pipelines deployment using one of the [installation options](/docs/components/pipelines/installation/overview/).
|
||||
* [Install the Kubeflow Pipelines SDK](/docs/components/pipelines/sdk/install-sdk/).
|
||||
|
||||
## Connect to Kubeflow Pipelines using SDK
|
||||
|
||||
The Kubeflow Pipelines REST API is available at the same endpoint as the Kubeflow Pipelines user interface (UI).
|
||||
The SDK client can send requests to this endpoint to upload pipelines, create pipeline runs, schedule recurring runs and more.
|
||||
|
||||
### Connect to Kubeflow Pipelines from outside your cluster
|
||||
|
||||
Kubeflow distributions secure the Kubeflow Pipelines public endpoint with authentication and authorization.
|
||||
Since Kubeflow distributions can have different authentication and authorization requirements, the steps needed to connect to your Kubeflow Pipelines instance might be different depending on the Kubeflow distribution you installed. Refer to documentation for [your Kubeflow distribution](/docs/started/installing-kubeflow/):
|
||||
|
||||
* [Connecting to Kubeflow Pipelines on Google Cloud using the SDK](/docs/distributions/gke/pipelines/authentication-sdk/)
|
||||
* [Kubeflow Pipelines on AWS](https://www.kubeflow.org/docs/distributions/aws/pipeline/#authenticate-kubeflow-pipeline-using-sdk-inside-cluster)
|
||||
* [Authentication using OIDC in Azure](/docs/distributions/azure/authentication-oidc/)
|
||||
* [Pipelines on IBM Cloud Kubernetes Service (IKS)](/docs/distributions/ibm/pipelines/)
|
||||
|
||||
For [Kubeflow Pipelines standalone](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/) and [Google Cloud AI Platform Pipelines](/docs/components/pipelines/installation/overview/#google-cloud-ai-platform-pipelines), you can also connect to the API via `kubectl port-forward`.
|
||||
|
||||
Kubeflow Pipelines standalone deploys a Kubernetes service named `ml-pipeline-ui` in your Kubernetes cluster without extra authentication.
|
||||
|
||||
You can use [kubectl port-forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to port forward the Kubernetes service locally to your laptop outside of the cluster:
|
||||
|
||||
```bash
|
||||
# Change the namespace if you deployed Kubeflow Pipelines in a different
|
||||
# namespace.
|
||||
$ kubectl port-forward svc/ml-pipeline-ui 3000:80 --namespace kubeflow
|
||||
```
|
||||
|
||||
You can verify that port forwarding is working properly by visiting [http://localhost:3000](http://localhost:3000) in your browser. If port forwarding is working properly, the Kubeflow Pipelines UI appears.
|
||||
|
||||
Run the following python code to instantiate the `kfp.Client`:
|
||||
|
||||
```python
|
||||
import kfp
|
||||
client = kfp.Client(host='http://localhost:3000')
|
||||
print(client.list_experiments())
|
||||
```
|
||||
|
||||
Note, for Kubeflow Pipelines in multi-user mode, you cannot access the API using kubectl port-forward
|
||||
because it requires authentication. Refer to distribution specific documentation as recommended above.
|
||||
|
||||
### Connect to Kubeflow Pipelines from the same cluster
|
||||
|
||||
Note, this is not supported right now for multi-user Kubeflow Pipelines, refer to [Multi-User Isolation for Pipelines -- Current Limitations](/docs/components/pipelines/multi-user/#current-limitations).
|
||||
|
||||
As mentioned above, the Kubeflow Pipelines API Kubernetes service is `ml-pipeline-ui`.
|
||||
|
||||
Using [Kubernetes standard mechanisms to discover the service](https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services), you can access `ml-pipeline-ui` service from a Pod in the same namespace by DNS name:
|
||||
|
||||
```python
|
||||
import kfp
|
||||
client = kfp.Client(host='http://ml-pipeline-ui:80')
|
||||
print(client.list_experiments())
|
||||
```
|
||||
|
||||
Or, you can access `ml-pipeline-ui` service by using environment variables:
|
||||
|
||||
```python
|
||||
import kfp
|
||||
import os
|
||||
host = os.getenv('ML_PIPELINE_UI_SERVICE_HOST')
|
||||
port = os.getenv('ML_PIPELINE_UI_SERVICE_PORT')
|
||||
client = kfp.Client(host=f'http://{host}:{port}')
|
||||
print(client.list_experiments())
|
||||
```
|
||||
|
||||
When accessing Kubeflow Pipelines from a Pod in a different namespace, you must access by the service name and the namespace:
|
||||
|
||||
```python
|
||||
import kfp
|
||||
namespace = 'kubeflow' # or the namespace you deployed Kubeflow Pipelines
|
||||
client = kfp.Client(host=f'http://ml-pipeline-ui.{namespace}:80')
|
||||
print(client.list_experiments())
|
||||
```
|
||||
|
||||
## Configure SDK client by environment variables
|
||||
|
||||
It's usually beneficial to configure the Kubeflow Pipelines SDK client using Kubeflow Pipelines environment variables,
|
||||
so that you can initiate `kfp.Client` instances without any explicit arguments.
|
||||
|
||||
For example, when the API endpoint is [http://localhost:3000](http://localhost:3000), run the following to configure environment variables in bash:
|
||||
|
||||
```bash
|
||||
export KF_PIPELINES_ENDPOINT=http://localhost:3000
|
||||
```
|
||||
|
||||
Or configure in a Jupyter Notebook by using the [IPython built-in `%env` magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-env):
|
||||
|
||||
```python
|
||||
%env KF_PIPELINES_ENDPOINT=http://localhost:3000
|
||||
```
|
||||
|
||||
Then you can use the SDK client without explicit arguments.
|
||||
|
||||
```python
|
||||
import kfp
|
||||
# When not specified, host defaults to env var KF_PIPELINES_ENDPOINT.
|
||||
# This is now equivalent to `client = kfp.Client(host='http://localhost:3000')`
|
||||
client = kfp.Client()
|
||||
print(client.list_experiments())
|
||||
```
|
||||
|
||||
Refer to [more configurable environment variables here](https://github.com/kubeflow/pipelines/blob/54ac9a6a7173aecbbb30a043b2077e790cac6953/sdk/python/kfp/_client.py#L84-L90).
|
||||
|
||||
## Next Steps
|
||||
|
||||
* [Using the Kubeflow Pipelines SDK](/docs/components/pipelines/tutorials/sdk-examples/)
|
||||
* [Kubeflow Pipelines SDK Reference](https://kubeflow-pipelines.readthedocs.io/en/stable/)
|
||||
* [Experiment with the Kubeflow Pipelines API](/docs/components/pipelines/tutorials/api-pipelines/)
|
||||
|
|
@ -1,9 +1,9 @@
|
|||
+++
|
||||
title = "Python Based Visualizations"
|
||||
title = "Python Based Visualizations (Deprecated)"
|
||||
description = "Predefined and custom visualizations of pipeline outputs"
|
||||
weight = 80
|
||||
|
||||
weight = 1400
|
||||
+++
|
||||
|
||||
{{% alert title="Deprecated" color="warning" %}}
|
||||
Python based visualization is deprecated. We recommend fetching data via
|
||||
Kubeflow Pipelines SDK and visualizing from your own notebook instead.
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"2. Import the `kfp` and `kfp.components` packages."
|
||||
"2. Import the `kfp` package."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -54,21 +54,17 @@
|
|||
"outputs": [],
|
||||
"source": [
|
||||
"import kfp\n",
|
||||
"import kfp.components as comp"
|
||||
"from kfp.components import create_component_from_func"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your\n",
|
||||
" Kubeflow Pipelines cluster's hostname, open the Kubeflow Pipelines user\n",
|
||||
" interface in your browser. The URL of the Kubeflow Pipelines user\n",
|
||||
" interface is something like \n",
|
||||
" `https://my-cluster.my-organization.com/pipelines`. In this case, the\n",
|
||||
" hostname is `my-cluster.my-organization.com`. \n",
|
||||
"3. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].\n",
|
||||
"\n",
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.client.html#kfp.Client"
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client\n",
|
||||
"[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -77,10 +73,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you run this command on a Jupyter notebook running on Kubeflow, you can\n",
|
||||
"# exclude the host parameter.\n",
|
||||
"# client = kfp.Client()\n",
|
||||
"client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')"
|
||||
"client = kfp.Client() # change arguments accordingly"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -128,7 +121,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"add_op = comp.create_component_from_func(\n",
|
||||
"add_op = create_component_from_func(\n",
|
||||
" add, output_component_file='add_component.yaml')"
|
||||
]
|
||||
},
|
||||
|
|
@ -652,9 +645,9 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"5. Create and run your pipeline. [Learn more about creating and running pipelines][build-pipelines].\n",
|
||||
"5. Compile and run your pipeline. [Learn more about compiling and running pipelines][build-pipelines].\n",
|
||||
"\n",
|
||||
"[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/build-component/"
|
||||
"[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/build-pipeline/#compile-and-run-your-pipeline"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -692,7 +685,7 @@
|
|||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.8"
|
||||
"version": "3.7.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
|
|
|||
|
|
@ -59,29 +59,22 @@ This document describes how to build Python function-based components and use th
|
|||
$ pip3 install kfp --upgrade
|
||||
```
|
||||
|
||||
2. Import the `kfp` and `kfp.components` packages.
|
||||
2. Import the `kfp` package.
|
||||
|
||||
|
||||
```python
|
||||
import kfp
|
||||
import kfp.components as comp
|
||||
from kfp.components import create_component_from_func
|
||||
```
|
||||
|
||||
3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your
|
||||
Kubeflow Pipelines cluster's hostname, open the Kubeflow Pipelines user
|
||||
interface in your browser. The URL of the Kubeflow Pipelines user
|
||||
interface is something like
|
||||
`https://my-cluster.my-organization.com/pipelines`. In this case, the
|
||||
hostname is `my-cluster.my-organization.com`.
|
||||
3. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].
|
||||
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.client.html#kfp.Client
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client
|
||||
[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api
|
||||
|
||||
|
||||
```python
|
||||
# If you run this command on a Jupyter notebook running on Kubeflow, you can
|
||||
# exclude the host parameter.
|
||||
# client = kfp.Client()
|
||||
client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')
|
||||
client = kfp.Client() # change arguments accordingly
|
||||
```
|
||||
|
||||
For more information about the Kubeflow Pipelines SDK, see the [SDK reference guide][sdk-ref].
|
||||
|
|
@ -111,7 +104,7 @@ def add(a: float, b: float) -> float:
|
|||
|
||||
|
||||
```python
|
||||
add_op = comp.create_component_from_func(
|
||||
add_op = create_component_from_func(
|
||||
add, output_component_file='add_component.yaml')
|
||||
```
|
||||
|
||||
|
|
@ -568,9 +561,9 @@ def calc_pipeline(
|
|||
result_task = add_op(divmod_task.outputs['quotient'], c)
|
||||
```
|
||||
|
||||
5. Create and run your pipeline. [Learn more about creating and running pipelines][build-pipelines].
|
||||
5. Compile and run your pipeline. [Learn more about compiling and running pipelines][build-pipelines].
|
||||
|
||||
[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/build-component/
|
||||
[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/build-pipeline/#compile-and-run-your-pipeline
|
||||
|
||||
|
||||
```python
|
||||
|
|
@ -585,4 +578,4 @@ client.create_run_from_pipeline_func(calc_pipeline, arguments=arguments)
|
|||
<div class="notebook-links">
|
||||
<a class="colab-link" href="https://colab.research.google.com/github/kubeflow/website/blob/master/content/en/docs/components/pipelines/sdk/python-function-components.ipynb">Run in Google Colab</a>
|
||||
<a class="github-link" href="https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/sdk/python-function-components.ipynb">View source on GitHub</a>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -26,7 +26,7 @@
|
|||
"\n",
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"1. Run the following command to install the Kubeflow Pipelines SDK v1.6 or higher.\n",
|
||||
"1. Run the following command to install the Kubeflow Pipelines SDK v1.6.2 or higher.\n",
|
||||
" If you run this command in a Jupyter notebook, restart the kernel after\n",
|
||||
" installing the SDK. "
|
||||
]
|
||||
|
|
@ -39,7 +39,7 @@
|
|||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install kfp==1.6.0rc0"
|
||||
"!pip install --upgrade kfp"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -48,7 +48,7 @@
|
|||
"id": "8KExWR1i_7Ur"
|
||||
},
|
||||
"source": [
|
||||
"2. Import the `kfp` and `kfp.components` packages."
|
||||
"2. Import the `kfp` packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -60,9 +60,7 @@
|
|||
"outputs": [],
|
||||
"source": [
|
||||
"import kfp\n",
|
||||
"import kfp.components as comp\n",
|
||||
"import kfp.vs.dsl as dsl\n",
|
||||
"from kfp.v2 import compiler\n",
|
||||
"from kfp.v2 import dsl\n",
|
||||
"from kfp.v2.dsl import component\n",
|
||||
"from kfp.v2.dsl import (\n",
|
||||
" Input,\n",
|
||||
|
|
@ -72,36 +70,6 @@
|
|||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "IARdgYktshE8"
|
||||
},
|
||||
"source": [
|
||||
"3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your\n",
|
||||
" Kubeflow Pipelines cluster's hostname and URL scheme, open the Kubeflow\n",
|
||||
" Pipelines user interface in your browser. The URL of the Kubeflow\n",
|
||||
" Pipelines user interface is something like \n",
|
||||
" `https://my-cluster.my-organization.com/pipelines`. In this case, the\n",
|
||||
" host name and URL scheme are `https://my-cluster.my-organization.com`. \n",
|
||||
"\n",
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "XJ4YU9N_sos6"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you run this command on a Jupyter notebook running on Kubeflow, you can\n",
|
||||
"# exclude the host parameter.\n",
|
||||
"# client = kfp.Client()\n",
|
||||
"client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
|
|
@ -543,8 +511,9 @@
|
|||
"source": [
|
||||
"# Define a pipeline and create a task from a component:\n",
|
||||
"@dsl.pipeline(\n",
|
||||
" \n",
|
||||
" pipeline_root='gs://my-pipeline-root/example-pipeline',\n",
|
||||
" name='my-pipeline',\n",
|
||||
" # You can optionally specify your own pipeline_root\n",
|
||||
" # pipeline_root='gs://my-pipeline-root/example-pipeline',\n",
|
||||
")\n",
|
||||
"def my_pipeline(url: str):\n",
|
||||
" web_downloader_task = web_downloader_op(url=url)\n",
|
||||
|
|
@ -561,7 +530,9 @@
|
|||
"source": [
|
||||
"### Compile and run your pipeline\n",
|
||||
"\n",
|
||||
"After defining the pipeline in Python as described in the preceding section, use the following instructions to compile the pipeline and submit it to the Kubeflow Pipelines service. \n",
|
||||
"After defining the pipeline in Python as described in the preceding section, use one of the following options to compile the pipeline and submit it to the Kubeflow Pipelines service.\n",
|
||||
"\n",
|
||||
"#### Option 1: Compile and then upload in UI\n",
|
||||
"\n",
|
||||
"1. Run the following to compile your pipeline and save it as `pipeline.yaml`. \n"
|
||||
]
|
||||
|
|
@ -574,10 +545,19 @@
|
|||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"compiler.Compiler().compile(\n",
|
||||
"kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(\n",
|
||||
" pipeline_func=my_pipeline,\n",
|
||||
" package_path='pipeline.yaml',\n",
|
||||
" mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE)"
|
||||
" package_path='pipeline.yaml')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"2. Upload and run your `pipeline.yaml` using the Kubeflow Pipelines user interface.\n",
|
||||
"See the guide to [getting started with the UI][quickstart].\n",
|
||||
"\n",
|
||||
"[quickstart]: https://www.kubeflow.org/docs/components/pipelines/pipelines-quickstart"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -586,14 +566,28 @@
|
|||
"id": "jNLI1-_bfEky"
|
||||
},
|
||||
"source": [
|
||||
"2. Run the following to submit the compiled workflow specification\n",
|
||||
" (`pipeline.yaml`) using the Kubeflow Pipelines SDK. \n",
|
||||
" \n",
|
||||
" You can also use the Kubeflow Pipelines user interface to upload and run\n",
|
||||
" your `pipeline.yaml`. See the guide to [getting started with the\n",
|
||||
" UI][quickstart].\n",
|
||||
"#### Option 2: run the pipeline using Kubeflow Pipelines SDK client\n",
|
||||
"\n",
|
||||
"[quickstart]: https://www.kubeflow.org/docs/components/pipelines/pipelines-quickstart"
|
||||
"1. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].\n",
|
||||
"\n",
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client\n",
|
||||
"[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = kfp.Client() # change arguments accordingly"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"2. Run the pipeline using the `kfp.Client` instance:"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -604,8 +598,11 @@
|
|||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.create_run_from_pipeline_package(\n",
|
||||
" pipeline_file='pipeline.yaml',\n",
|
||||
"client.create_run_from_pipeline_func(\n",
|
||||
" my_pipeline,\n",
|
||||
" mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE,\n",
|
||||
" # You can optionally override your pipeline_root when submitting the run too:\n",
|
||||
" # pipeline_root='gs://my-pipeline-root/example-pipeline',\n",
|
||||
" arguments={\n",
|
||||
" 'url': 'https://storage.googleapis.com/ml-pipeline-playground/iris-csv-files.tar.gz'\n",
|
||||
" })"
|
||||
|
|
@ -656,7 +653,7 @@
|
|||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.8"
|
||||
"version": "3.7.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
|
|
|||
|
|
@ -48,23 +48,21 @@ building and running pipelines that are compatible with the Pipelines SDK v2.
|
|||
|
||||
## Before you begin
|
||||
|
||||
1. Run the following command to install the Kubeflow Pipelines SDK v1.6 or higher.
|
||||
1. Run the following command to install the Kubeflow Pipelines SDK v1.6.2 or higher.
|
||||
If you run this command in a Jupyter notebook, restart the kernel after
|
||||
installing the SDK.
|
||||
|
||||
|
||||
```python
|
||||
$ pip install kfp==1.6.0rc0
|
||||
$ pip install --upgrade kfp
|
||||
```
|
||||
|
||||
2. Import the `kfp` and `kfp.components` packages.
|
||||
2. Import the `kfp` packages.
|
||||
|
||||
|
||||
```python
|
||||
import kfp
|
||||
import kfp.components as comp
|
||||
import kfp.v2.dsl as dsl
|
||||
from kfp.v2 import compiler
|
||||
from kfp.v2 import dsl
|
||||
from kfp.v2.dsl import component
|
||||
from kfp.v2.dsl import (
|
||||
Input,
|
||||
|
|
@ -74,23 +72,6 @@ from kfp.v2.dsl import (
|
|||
)
|
||||
```
|
||||
|
||||
3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your
|
||||
Kubeflow Pipelines cluster's hostname and URL scheme, open the Kubeflow
|
||||
Pipelines user interface in your browser. The URL of the Kubeflow
|
||||
Pipelines user interface is something like
|
||||
`https://my-cluster.my-organization.com/pipelines`. In this case, the
|
||||
host name and URL scheme are `https://my-cluster.my-organization.com`.
|
||||
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client
|
||||
|
||||
|
||||
```python
|
||||
# If you run this command on a Jupyter notebook running on Kubeflow, you can
|
||||
# exclude the host parameter.
|
||||
# client = kfp.Client()
|
||||
client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')
|
||||
```
|
||||
|
||||
## Understanding pipelines
|
||||
|
||||
A Kubeflow pipeline is a portable and scalable definition of an ML workflow,
|
||||
|
|
@ -459,8 +440,9 @@ web_downloader_op = kfp.components.load_component_from_url(
|
|||
```python
|
||||
# Define a pipeline and create a task from a component:
|
||||
@dsl.pipeline(
|
||||
|
||||
pipeline_root='gs://my-pipeline-root/example-pipeline',
|
||||
name='my-pipeline',
|
||||
# You can optionally specify your own pipeline_root
|
||||
# pipeline_root='gs://my-pipeline-root/example-pipeline',
|
||||
)
|
||||
def my_pipeline(url: str):
|
||||
web_downloader_task = web_downloader_op(url=url)
|
||||
|
|
@ -471,32 +453,46 @@ def my_pipeline(url: str):
|
|||
|
||||
### Compile and run your pipeline
|
||||
|
||||
After defining the pipeline in Python as described in the preceding section, use the following instructions to compile the pipeline and submit it to the Kubeflow Pipelines service.
|
||||
After defining the pipeline in Python as described in the preceding section, use one of the following options to compile the pipeline and submit it to the Kubeflow Pipelines service.
|
||||
|
||||
#### Option 1: Compile and then upload in UI
|
||||
|
||||
1. Run the following to compile your pipeline and save it as `pipeline.yaml`.
|
||||
|
||||
|
||||
|
||||
```python
|
||||
compiler.Compiler().compile(
|
||||
kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
|
||||
pipeline_func=my_pipeline,
|
||||
package_path='pipeline.yaml',
|
||||
mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE)
|
||||
package_path='pipeline.yaml')
|
||||
```
|
||||
|
||||
2. Run the following to submit the compiled workflow specification
|
||||
(`pipeline.yaml`) using the Kubeflow Pipelines SDK.
|
||||
|
||||
You can also use the Kubeflow Pipelines user interface to upload and run
|
||||
your `pipeline.yaml`. See the guide to [getting started with the
|
||||
UI][quickstart].
|
||||
2. Upload and run your `pipeline.yaml` using the Kubeflow Pipelines user interface.
|
||||
See the guide to [getting started with the UI][quickstart].
|
||||
|
||||
[quickstart]: https://www.kubeflow.org/docs/components/pipelines/pipelines-quickstart
|
||||
|
||||
#### Option 2: run the pipeline using Kubeflow Pipelines SDK client
|
||||
|
||||
1. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].
|
||||
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client
|
||||
[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api
|
||||
|
||||
|
||||
```python
|
||||
client.create_run_from_pipeline_package(
|
||||
pipeline_file='pipeline.yaml',
|
||||
client = kfp.Client() # change arguments accordingly
|
||||
```
|
||||
|
||||
2. Run the pipeline using the `kfp.Client` instance:
|
||||
|
||||
|
||||
```python
|
||||
client.create_run_from_pipeline_func(
|
||||
my_pipeline,
|
||||
mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE,
|
||||
# You can optionally override your pipeline_root when submitting the run too:
|
||||
# pipeline_root='gs://my-pipeline-root/example-pipeline',
|
||||
arguments={
|
||||
'url': 'https://storage.googleapis.com/ml-pipeline-playground/iris-csv-files.tar.gz'
|
||||
})
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@
|
|||
"\n",
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"1. Run the following command to install the Kubeflow Pipelines SDK v1.6 or higher. If you run this command in a Jupyter\n",
|
||||
"1. Run the following command to install the Kubeflow Pipelines SDK v1.6.2 or higher. If you run this command in a Jupyter\n",
|
||||
" notebook, restart the kernel after installing the SDK. "
|
||||
]
|
||||
},
|
||||
|
|
@ -45,14 +45,14 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install kfp==1.6.0rc0"
|
||||
"!pip install --upgrade kfp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"2. Import the `kfp`, `kfp.components`, `kfp.dsl`, and `kfp.v2.dsl` packages."
|
||||
"2. Import the `kfp`, `kfp.dsl`, and `kfp.v2.dsl` packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -62,9 +62,7 @@
|
|||
"outputs": [],
|
||||
"source": [
|
||||
"import kfp\n",
|
||||
"import kfp.components as comp\n",
|
||||
"import kfp.dsl as dsl\n",
|
||||
"from kfp.v2 import compiler\n",
|
||||
"from kfp.v2.dsl import (\n",
|
||||
" component,\n",
|
||||
" Input,\n",
|
||||
|
|
@ -78,14 +76,10 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your\n",
|
||||
" Kubeflow Pipelines cluster's hostname, open the Kubeflow Pipelines user\n",
|
||||
" interface in your browser. The URL of the Kubeflow Pipelines user\n",
|
||||
" interface is something like \n",
|
||||
" `https://my-cluster.my-organization.com/pipelines`. In this case, the\n",
|
||||
" hostname is `my-cluster.my-organization.com`. \n",
|
||||
"3. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].\n",
|
||||
"\n",
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.client.html#kfp.Client"
|
||||
"[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client\n",
|
||||
"[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -94,10 +88,7 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you run this command on a Jupyter notebook running on Kubeflow, you can\n",
|
||||
"# exclude the host parameter.\n",
|
||||
"# client = kfp.Client()\n",
|
||||
"client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')"
|
||||
"client = kfp.Client() # change arguments accordingly"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -140,7 +131,7 @@
|
|||
"source": [
|
||||
"2. Create and run your pipeline. [Learn more about creating and running pipelines][build-pipelines].\n",
|
||||
"\n",
|
||||
"[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/"
|
||||
"[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/#compile-and-run-your-pipeline"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -168,18 +159,29 @@
|
|||
" # `task.outputs['output_name']`.\n",
|
||||
" second_add_task = add(first_add_task.output, b)\n",
|
||||
"\n",
|
||||
"# Compile the pipeline using the kfp.v2.compiler.Compiler\n",
|
||||
"compiler.Compiler().compile(\n",
|
||||
" pipeline_func=add_pipeline,\n",
|
||||
" package_path='addition-pipeline.json')\n",
|
||||
"\n",
|
||||
"# Specify pipeline argument values\n",
|
||||
"arguments = {'a': 7, 'b': 8}\n",
|
||||
"arguments = {'a': 7, 'b': 8}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"3. Compile and run your pipeline. [Learn more about compiling and running pipelines][build-pipelines].\n",
|
||||
"\n",
|
||||
"# Submit a pipeline run\n",
|
||||
"client.create_run_from_pipeline_package(\n",
|
||||
" pipeline_file='addition-pipeline.json',\n",
|
||||
" arguments=arguments\n",
|
||||
"[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/#compile-and-run-your-pipeline"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Submit a pipeline run using the v2 compatible mode\n",
|
||||
"client.create_run_from_pipeline_func(\n",
|
||||
" add_pipeline,\n",
|
||||
" arguments=arguments,\n",
|
||||
" mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE)"
|
||||
]
|
||||
},
|
||||
|
|
@ -470,6 +472,8 @@
|
|||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example Python function-based component\n",
|
||||
"\n",
|
||||
|
|
@ -483,9 +487,7 @@
|
|||
" By adding the `@component` annotation, you convert your function into a factory function\n",
|
||||
" that creates pipeline steps that execute this function. This example also specifies the\n",
|
||||
" base container image to run you component in."
|
||||
],
|
||||
"cell_type": "markdown",
|
||||
"metadata": {}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
|
|
@ -570,9 +572,9 @@
|
|||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"3. Create and run your pipeline. [Learn more about creating and running pipelines][build-pipelines].\n",
|
||||
"3. Compile and run your pipeline. [Learn more about compiling and running pipelines][build-pipelines].\n",
|
||||
"\n",
|
||||
"[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/"
|
||||
"[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/#compile-and-run-your-pipeline"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -581,18 +583,13 @@
|
|||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Compile the pipeline using the kfp.v2.compiler.Compiler\n",
|
||||
"compiler.Compiler().compile(\n",
|
||||
" pipeline_func=calc_pipeline,\n",
|
||||
" package_path='calc-pipeline.json')\n",
|
||||
"\n",
|
||||
"# Specify pipeline argument values\n",
|
||||
"arguments = {'a': 7, 'b': 8}\n",
|
||||
"\n",
|
||||
"# Submit a pipeline run\n",
|
||||
"client.create_run_from_pipeline_package(\n",
|
||||
" pipeline_file='cal-pipeline.json',\n",
|
||||
" arguments=arguments\n",
|
||||
"client.create_run_from_pipeline_func(\n",
|
||||
" calc_pipeline,\n",
|
||||
" arguments=arguments,\n",
|
||||
" mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE)"
|
||||
]
|
||||
}
|
||||
|
|
@ -618,9 +615,9 @@
|
|||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.8"
|
||||
"version": "3.7.10"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -59,22 +59,20 @@ building and running pipelines that are compatible with the Pipelines SDK v2.
|
|||
|
||||
## Before you begin
|
||||
|
||||
1. Run the following command to install the Kubeflow Pipelines SDK v1.6 or higher. If you run this command in a Jupyter
|
||||
1. Run the following command to install the Kubeflow Pipelines SDK v1.6.2 or higher. If you run this command in a Jupyter
|
||||
notebook, restart the kernel after installing the SDK.
|
||||
|
||||
|
||||
```python
|
||||
$ pip install kfp==1.6.0rc0
|
||||
$ pip install --upgrade kfp
|
||||
```
|
||||
|
||||
2. Import the `kfp`, `kfp.components`, `kfp.dsl`, and `kfp.v2.dsl` packages.
|
||||
2. Import the `kfp`, `kfp.dsl`, and `kfp.v2.dsl` packages.
|
||||
|
||||
|
||||
```python
|
||||
import kfp
|
||||
import kfp.components as comp
|
||||
import kfp.dsl as dsl
|
||||
from kfp.v2 import compiler
|
||||
from kfp.v2.dsl import (
|
||||
component,
|
||||
Input,
|
||||
|
|
@ -84,21 +82,14 @@ from kfp.v2.dsl import (
|
|||
)
|
||||
```
|
||||
|
||||
3. Create an instance of the [`kfp.Client` class][kfp-client]. To find your
|
||||
Kubeflow Pipelines cluster's hostname, open the Kubeflow Pipelines user
|
||||
interface in your browser. The URL of the Kubeflow Pipelines user
|
||||
interface is something like
|
||||
`https://my-cluster.my-organization.com/pipelines`. In this case, the
|
||||
hostname is `my-cluster.my-organization.com`.
|
||||
3. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].
|
||||
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.client.html#kfp.Client
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client
|
||||
[connect-api]: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api
|
||||
|
||||
|
||||
```python
|
||||
# If you run this command on a Jupyter notebook running on Kubeflow, you can
|
||||
# exclude the host parameter.
|
||||
# client = kfp.Client()
|
||||
client = kfp.Client(host='<your-kubeflow-pipelines-host-name>')
|
||||
client = kfp.Client() # change arguments accordingly
|
||||
```
|
||||
|
||||
For more information about the Kubeflow Pipelines SDK, see the [SDK reference guide][sdk-ref].
|
||||
|
|
@ -128,7 +119,7 @@ def add(a: float, b: float) -> float:
|
|||
|
||||
2. Create and run your pipeline. [Learn more about creating and running pipelines][build-pipelines].
|
||||
|
||||
[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/
|
||||
[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/#compile-and-run-your-pipeline
|
||||
|
||||
|
||||
```python
|
||||
|
|
@ -151,18 +142,20 @@ def add_pipeline(
|
|||
# `task.outputs['output_name']`.
|
||||
second_add_task = add(first_add_task.output, b)
|
||||
|
||||
# Compile the pipeline using the kfp.v2.compiler.Compiler
|
||||
compiler.Compiler().compile(
|
||||
pipeline_func=add_pipeline,
|
||||
package_path='addition-pipeline.json')
|
||||
|
||||
# Specify pipeline argument values
|
||||
arguments = {'a': 7, 'b': 8}
|
||||
```
|
||||
|
||||
# Submit a pipeline run
|
||||
client.create_run_from_pipeline_package(
|
||||
pipeline_file='addition-pipeline.json',
|
||||
arguments=arguments
|
||||
3. Compile and run your pipeline. [Learn more about compiling and running pipelines][build-pipelines].
|
||||
|
||||
[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/#compile-and-run-your-pipeline
|
||||
|
||||
|
||||
```python
|
||||
# Submit a pipeline run using the v2 compatible mode
|
||||
client.create_run_from_pipeline_func(
|
||||
add_pipeline,
|
||||
arguments=arguments,
|
||||
mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE)
|
||||
```
|
||||
|
||||
|
|
@ -509,24 +502,19 @@ def calc_pipeline(
|
|||
result_task = add(divmod_task.outputs['quotient'], c)
|
||||
```
|
||||
|
||||
3. Create and run your pipeline. [Learn more about creating and running pipelines][build-pipelines].
|
||||
3. Compile and run your pipeline. [Learn more about compiling and running pipelines][build-pipelines].
|
||||
|
||||
[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/
|
||||
[build-pipelines]: https://www.kubeflow.org/docs/components/pipelines/sdk/v2/build-pipeline/#compile-and-run-your-pipeline
|
||||
|
||||
|
||||
```python
|
||||
# Compile the pipeline using the kfp.v2.compiler.Compiler
|
||||
compiler.Compiler().compile(
|
||||
pipeline_func=calc_pipeline,
|
||||
package_path='calc-pipeline.json')
|
||||
|
||||
# Specify pipeline argument values
|
||||
arguments = {'a': 7, 'b': 8}
|
||||
|
||||
# Submit a pipeline run
|
||||
client.create_run_from_pipeline_package(
|
||||
pipeline_file='cal-pipeline.json',
|
||||
arguments=arguments
|
||||
client.create_run_from_pipeline_func(
|
||||
calc_pipeline,
|
||||
arguments=arguments,
|
||||
mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE)
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -97,12 +97,9 @@ introduces the following changes:
|
|||
|
||||
* All pipeline parameters must be annotated with their data type.
|
||||
|
||||
## Running pipelines in v2 compatibility mode
|
||||
## Compiling and running pipelines in v2 compatibility mode
|
||||
|
||||
To run your pipeline in v2 compatibility mode, specify that
|
||||
`mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE` when you create a pipeline
|
||||
run using `create_run_from_pipeline_func`. The following example demonstrates
|
||||
how to run a pipeline using v2 compatibility mode.
|
||||
First we define a v2 compatible pipeline:
|
||||
|
||||
```python
|
||||
import kfp
|
||||
|
|
@ -121,7 +118,27 @@ def add(a: float, b: float) -> float:
|
|||
)
|
||||
def add_pipeline(a: float=1, b: float=7):
|
||||
add_task = add(a, b)
|
||||
```
|
||||
|
||||
To compile your pipeline in v2 compatibility mode, specify that
|
||||
`mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE` when you initiate the compiler:
|
||||
|
||||
```python
|
||||
from kfp import compiler
|
||||
compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE)
|
||||
.compile(pipeline_func=add_pipeline, package_path='pipeline.yaml')
|
||||
```
|
||||
|
||||
To run your pipeline in v2 compatibility mode:
|
||||
|
||||
1. Create an instance of the [`kfp.Client` class][kfp-client] following steps in [connecting to Kubeflow Pipelines using the SDK client][connect-api].
|
||||
2. Specify that `mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE` when you create a pipeline
|
||||
run using `create_run_from_pipeline_func`.
|
||||
|
||||
The following example demonstrates how to run a pipeline using v2 compatibility mode.
|
||||
|
||||
```python
|
||||
client = kfp.Client()
|
||||
# run the pipeline in v2 compatibility mode
|
||||
client.create_run_from_pipeline_func(
|
||||
add_pipeline,
|
||||
|
|
@ -144,6 +161,8 @@ Kubeflow Pipelines v2 compatible mode is currently in Beta stage. It is under ac
|
|||
[build-pipeline]: /docs/components/pipelines/sdk/v2/build-pipeline/
|
||||
[build-component]: /docs/components/pipelines/sdk/v2/component-development/
|
||||
[python-component]: /docs/components/pipelines/sdk/v2/python-function-components/
|
||||
[kfp-client]: https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html#kfp.Client
|
||||
[connect-api]: /docs/components/pipelines/sdk/connect-api.md
|
||||
[the full Kubeflow]: /docs/components/pipelines/installation/overview/#full-kubeflow
|
||||
[v2 compatible mode github project]: https://github.com/kubeflow/pipelines/projects/13
|
||||
[#5680]: https://github.com/kubeflow/pipelines/issues/5680
|
||||
|
|
|
|||
|
|
@ -0,0 +1,21 @@
|
|||
#!/usr/bin/env bash
|
||||
# Copyright 2021 The Kubeflow Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Convert all Kubeflow Pipelines notebook documentation to markdown.
|
||||
# This script is expected to be run from website root folder.
|
||||
# cd kubeflow/website
|
||||
# ./scripts/kfp_nb_to_md.sh
|
||||
|
||||
find content/en/docs/components/pipelines -name '*.ipynb' -exec python scripts/nb_to_md.py --notebook {} \;
|
||||
Loading…
Reference in New Issue