Commit Graph

3 Commits

Author SHA1 Message Date
Jiaxiao Zheng fb15223f7e
chore: Add doc strings marking the feature stages for SDK. (#4575)
* add doc strings

* Simplify the docstring

* fix unittest

* recover cli.py

* recover cli.py

* substitute docstring in resource ops with TODOs

* revert stable labels
2020-11-24 00:19:00 -08:00
Alexey Volkov c128b2a7b4 SDK - Compiler - Make it possible to create more portable pipelines (#2271)
* SDK - Compiler - Allow creating portable pipelines

This change allows directly passing the PipelineConf instance to compiler or launcher which makes it easier to create portable pipelines by allowing the environment-specific configuration to be directly passed to the environment-specific launcher.

Background:
PipelineConf holds all pipeline-level configuration including `op_transformers`, `image_pull_secrets` etc. Some of these are specific to particular execution environment (e.g. GCP secret or Argo artifact location or Kubernetes-specific options).
Previously, the only way to modify `PipelineConf` was to do it inside the piepline function. That tied the pipeline function to specific execution environment (e.g. GCP, Argo or Kubernetes)

Solution: This change allows directly passing the PipelineConf instance to compiler or launcher. This allows writing portable enlauncher and environment agnostic pipeline functions. All environment-specific configurations can be moved to launching stage.

Before:
```python
# Defining pipeline
def my_pipeline():
    # portable pipeline code

    dsl.get_pipeline_conf().add_op_transformer(gcp.use_gcp_secret('user-gcp-sa'))

# Launching pipeline
kfp.Clinet().create_run_from_pipeline_func(my_pipeline, arguments={})
```

After:
```python
# Defining pipeline
def my_pipeline():
    # portable pipeline code

# Launching pipeline
pipeline_conf = dsl.PipelineConf()
pipeline_conf.add_op_transformer(gcp.use_gcp_secret('user-gcp-sa'))
kfp.Clinet().create_run_from_pipeline_func(my_pipeline, arguments={}, pipeline_conf=pipeline_conf)
```

After 2 *(launching same portable pipeline using different launchers):
```python
# Loading portable pipeline
from portable_pipeline import my_pipeline

# Launching pipeline on Kubeflow
pipeline_conf = dsl.PipelineConf()
pipeline_conf.add_op_transformer(gcp.use_gcp_secret('user-gcp-sa'))
kfp.Clinet().create_run_from_pipeline_func(my_pipeline, arguments={}, pipeline_conf=pipeline_conf)

# Launching pipeline on locally (not implemented yet)
kfp.run_pipeline_func_locally(my_pipeline, arguments={})
```

* Added parameter docstring
2019-10-02 20:58:08 -07:00
Alexey Volkov ecd93a5056 SDK - Added kfp.run_pipeline_func_on_cluster function (#1521)
This commit adds alias to the kfp.Client.create_run_from_pipeline_func method as the root kfp.run_pipeline_func_on_cluster function.
In future more runners can be added (local, etc).
2019-09-03 12:26:58 -07:00