Commit Graph

30 Commits

Author SHA1 Message Date
Chen Sun 2f19a26ffd
chore(sdk): Format all Python files under SDK folder. (#6501)
* Reformat sdk only using the new yapf config.

* Reformat docstrings using docformatter.

* update golden files to resolve diff caused by whitespaces

* fix some tests

* format .py files under sdk/python/tests using yapf

* additional docformatter

* fix some tests
2021-09-03 11:25:11 -07:00
Yaqi Ji 65174cdd93
feat(sdk): Add BaseModel to component_spec data classes (#6372)
* pydantic setup

* switch to union types

* Fix base_component tests

* switch to pydantic for python3.6 compat and fix tests

* Fix tests

* use str for type

* fix tests

* Consistent quote sign

* Address comments

* Update type
2021-08-25 14:00:02 -07:00
Alexey Volkov cc83e1089b
Assigned copyright to the project authors (#5587) 2021-05-05 13:53:22 +08:00
Jiaxiao Zheng d3a7fbf1ca
feat(sdk): Refactor v2 component building (#5163)
* add executor input placeholder and remove unused entrypoint function

* add executor input generator

* add tests and fixes

* add comments for v2 component args

* refactor output metadata path placeholder

* fix test and change yaml dumping

* fix tests
2021-02-23 15:45:14 -08:00
Jiaxiao Zheng a36a62a700
feat(sdk): Artifact metadata related placeholder for components. (#5003)
* resolve comments.

* fix tests

* wip: add structures and skeleton for component resolution logic

* add generator

* fix the problem

* cleanup

* add a test

* fix tests
2021-01-19 08:57:45 -08:00
Chen Sun 88dbfda02b
chore(sdk): Add inputUri and outputUri placeholders in v1 (#4913)
* SDK - Components - Added support for inputUri and outputUri placeholders

* remove InputUriSpec and OutputUriSpec

* Address review comments

Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com>
2020-12-18 08:28:25 -08:00
Alexey Volkov 39975e3cde
SDK - Components - Temporary removed Kubernetes options from ComponentSpec (#4230)
* SDK - Components - Replaced Kubernetes options with generic launcher options

This reduces the schema size and makes the task launcher options more flexible.

* Removed the launcherOptions for now
2020-07-16 19:59:00 -07:00
Alexey Volkov afd816161d
feat(sdk): SDK - Components - Added annotations to TaskSpec (#4153) 2020-07-12 22:22:32 -07:00
Alexey Volkov 9b1580039f
feat(sdk): SDK - Components - Added annotations to InputSpec and OutputSpec (#4154) 2020-07-12 21:34:33 -07:00
Alexey Volkov e41ee9cdf7
SDK - Components - Task objects now have the .output attribute when component has only one output (#3622) 2020-04-26 18:47:28 -07:00
Alexey Volkov 734b43e3db
SDK - Added support for maxCacheStaleness (#3318)
* SDK - Added support for maxCacheStaleness

* Added the vendor prefix to the annotation
2020-03-20 13:38:09 -07:00
Alexey Volkov 681d873fc7 SDK - Components - Added type to graph input references (#2451)
This makes the graph input references consistent with task output references.
This is a breaking change, but the graph components are not exposed in the documentation or samples yet.
2019-10-23 17:03:05 -07:00
Alexey Volkov b2f1d0071f SDK - Components - Added the ComponentSpec.save method (#2264)
* SDK - Components - Added the ComponentSpec.save method

* Fixed write call
2019-10-03 15:25:55 -07:00
Alexey Volkov 052a6ac0ce SDK - Components - Reorganized TaskSpec execution options (#2270)
This part of the spec was unused, so this is not a breaking change.
Consolidating Kubernetes-related options under a single attribute: `TaskSpec.execution_options.kubernetes_options`.
`TaskSpec.k8s_container_options` -> `TaskSpec.execution_options.kubernetes_options.main_container`
`TaskSpec.k8s_pod_options.spec` -> `TaskSpec.execution_options.kubernetes_options.pod_spec`
Added `TaskSpec.execution_options.retry_strategy.max_tetries` attribute.
2019-10-02 18:44:08 -07:00
Alexey Volkov e54fe67543 SDK - Components - Added type to TaskOutputReference (#1995)
* SDK - Components - Added type to TaskOutputReference
Now the task output references taken from TaskSpec instances can be
type-checked when passed to components.

* Renamed TypeType to TypeSpecType
2019-08-30 16:33:50 -07:00
Alexey Volkov d43de167df SDK - Components - Added output references to TaskSpec (#1991)
Also added TaskSpec.task and ComponentReference.spec attributes
2019-08-29 15:28:58 -07:00
Alexey Volkov f6cf9c5f55 SDK - Lightweight - Added support for "None" default values (#1626)
* SDK - Lightweight - Added support for "None" default values
Previously it was impossible to pass None to components since it was being converted to the string "None".

* is_required = not input.optional for now
As asked by @gaoning777
2019-07-25 18:49:59 -07:00
Alexey Volkov f40a22a3f4 SDK - Made ComponentSpec.implementation field optional (#1188)
* SDK - Made ComponentSpec.implementation field optional
Improved the error message when trying to convert tasks to ContainerOp.

* Switched from attribute checking to type checking
2019-04-24 12:54:46 -07:00
Alexey Volkov 6920aceeba SDK - Removed SourceSpec structure (#1119)
It has never been used and ComponentSpec.metadata.annotations['source'] is a better place for such metadata.
2019-04-24 12:06:26 -07:00
Alexey Volkov 9b804688d3 Added the metadata property to ComponentSpec (#1023)
The `metadata` section contains the `annotations` and `labels` dictionaries.
2019-03-23 16:27:05 -07:00
Alexey Volkov 8c4f5de1f7 SDK/Components - Command line args can only be strings or placeholders (#711)
Ultimately, command line is an array of strings. Component yaml files should have the arguments as strings instead of Python SDK doing conversion sometimes.
2019-01-24 19:13:50 -08:00
Alexey Volkov b8f5348fb3 SDK/Components - Renamed serialized command-line argument tokens (#659) 2019-01-10 00:49:37 -08:00
Alexey Volkov 83e9ffe5bc SDK/Components - Reworked the component model structures. (#642)
* Reworked the Component structures.
Rewrote parsing, type checking and serialization code.
Improved the graph component structures.
Added most of the needed k8s structures.
Added model validation (input/output existence etc).
Added task cycle detection and topological sorting to GraphSpec.
All container component tests now work.
Added some graph component tests.

* Fixed incompatibilities with python <3.7

* Added __init__.py to make the Travis tests work.

* Adding kubernetes structures to setup.py

* Addressed PR feedback: Renamed _original_names to _serialized_names

* Addressed PR feedback: Reduced indentation.

* Added descriptions for all component structures.

* Fixed a bug in ComponentSpec._post_init()

* Added documentation for ModelBase class and functions.

* Added __eq__/__ne__ and improved __repr__

* Added ModelBase tests
2019-01-09 15:51:34 -08:00
Ning 85c6413a2e Refactor Python SDK (#568)
* add some comments

* remove unused import; add license to dsl_bridge

* move_convert_k8s_obj_to_dic from compiler to k8s_helper

* move unit test
2018-12-20 09:51:09 -08:00
Alexey Volkov fa1886c39d SDK/Components - Allow dict type spec (#490) 2018-12-08 02:29:37 -08:00
Alexey Volkov e06dc88316 SDK/Components - Renamed container.arguments to container.args (#437)
This aligns us with Kubernetes spec
2018-12-03 11:02:15 -08:00
Alexey Volkov 9110296e57 SDK/Components - Support for optional inputs (#214)
* Renamed "required" to "optional"

* Added support for optional inputs

* Added tests for optional inputs. "If then *" tests now also work.
2018-11-30 13:30:09 -08:00
Alexey Volkov dd0bd45aa3 SDK/Components - Renamed DockerContainer spec to to Container (#323) 2018-11-20 12:47:49 -08:00
Alexey Volkov 09e4146e8c SDK/Components - Switched the generated inputs/outputs structure from dict to list (#173) 2018-11-10 15:17:26 -08:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00