578d8de91d
* SDK - Reduce python component limitations - no import errors for custom type annotations
By default, create_component_from_func copies the source code of the function and creates a component using that source code. No global imports are captured. This is problematic for the function definition, since any annotation, that uses a type that needs to be imported, will cause error. There were some special provisions for
NamedTuple, InputPath and OutputPath, but even they were brittle (for example, "typing.NamedTuple" or "components.InputPath" annotations still caused failures at runtime).
This commit fixes the issue by stripping the type annotations from function declarations.
Fixes cases that were failing before:
```python
import typing
import collections
MyFuncOutputs = typing.NamedTuple('Outputs', [('sum', int), ('product', int)])
@create_component_from_func
def my_func(
param1: CustomType, # This caused failure previously
param2: collections.OrderedDict, # This caused failure previously
) -> MyFuncOutputs: # This caused failure previously
pass
```
* Fixed the compiler tests
* Fixed crashes on print function
Code `print(line, end="")` was causing error: "lib2to3.pgen2.parse.ParseError: bad input: type=22, value='=', context=('', (2, 15))"
* Using the strip_hints library to strip the annotations
* Updating test workflow yamls
* Workaround for bug in untokenize
* Switched to the new strip_string_to_string method
* Fixed typo.
Co-Authored-By: Jiaxiao Zheng <jxzheng@google.com>
Co-authored-by: Jiaxiao Zheng <jxzheng@google.com>
|
||
|---|---|---|
| .github/ISSUE_TEMPLATE | ||
| backend | ||
| components | ||
| contrib | ||
| docs | ||
| frontend | ||
| manifests | ||
| proxy | ||
| release | ||
| samples | ||
| sdk | ||
| test | ||
| third_party | ||
| tools | ||
| .cloudbuild.yaml | ||
| .dockerignore | ||
| .gitattributes | ||
| .gitignore | ||
| .pylintrc | ||
| .release.cloudbuild.yaml | ||
| .style.yapf | ||
| .travis.yml | ||
| BUILD.bazel | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| Makefile | ||
| OWNERS | ||
| README.md | ||
| ROADMAP.md | ||
| WORKSPACE | ||
| developer_guide.md | ||
| go.mod | ||
| go.sum | ||
README.md
SDK:
Overview of the Kubeflow pipelines service
Kubeflow is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on Kubernetes simple, portable, and scalable.
Kubeflow pipelines are reusable end-to-end ML workflows built using the Kubeflow Pipelines SDK.
The Kubeflow pipelines service has the following goals:
- End to end orchestration: enabling and simplifying the orchestration of end to end machine learning pipelines
- Easy experimentation: making it easy for you to try numerous ideas and techniques, and manage your various trials/experiments.
- Easy re-use: enabling you to re-use components and pipelines to quickly cobble together end to end solutions, without having to re-build each time.
Documentation
Get started with your first pipeline and read further information in the Kubeflow Pipelines overview.
See the various ways you can use the Kubeflow Pipelines SDK.
See the Kubeflow Pipelines API doc for API specification.
Consult the Python SDK reference docs when writing pipelines using the Python SDK.
Kubeflow Pipelines Community Meeting
The meeting is happening every other Wed 10-11AM (PST) Calendar Invite or Join Meeting Directly
Kubeflow Pipelines Slack Channel
Blog posts
- Getting started with Kubeflow Pipelines (By Amy Unruh)
- How to create and deploy a Kubeflow Machine Learning Pipeline (By Lak Lakshmanan)
Acknowledgments
Kubeflow pipelines uses Argo under the hood to orchestrate Kubernetes resources. The Argo community has been very supportive and we are very grateful.