Commit Graph

69 Commits

Author SHA1 Message Date
Alexey Volkov b935836c30 SDK/Lightweight - Enable cloudpickle installation from non-root users (#1511) 2019-06-17 18:56:15 -07:00
Alexey Volkov e90085ecb3 SDK - Refactored _func_to_component_spec to split code generation from signature analysis (#1334)
* SDK - Refactored _func_to_component_spec to split out the function signature analyzer

* Renamed function to _extract_component_interface
2019-06-17 18:02:16 -07:00
Alexey Volkov aee1b5e2e5 SDK - Improving python component logs by making stdout and stderr unbuffered (#1510)
Without this the output and error lines can be printed in wrong order and sometimes not printed at all.
2019-06-14 00:20:20 -07:00
Krassimir Valev 8938669d7d Base64 encode the pickled code (#1476)
Due to its nature, Argo will replace any strings it encounters
that are enclosed in double curly braces, which will make the code
non-executable. To workaround this, the code is encoded in the Argo
yaml template and decoded on the fly, before the execution.
2019-06-13 23:30:25 -07:00
Alexey Volkov d724a4b68d SDK - Controlling which modules are captured with Lightweight components (#1435)
* SDK - Controlling which modules are captured with Lightweight components

All func_to_* functions now accept the modules_to_capture parameter: List of module names that will be captured (instead of just referencing) during the dependency scan. By default the func.__module__ is captured.

* Described the behavior more in depth.

* Added a test to check that only dependencies are captured
2019-06-07 18:47:06 -07:00
Alexey Volkov ab97d5708d SDK - Only install cloudpickle if it's not available (#1434)
This makes unit tests much faster.
Also:
Pined the version to 1.1.1.
Made the installation quiet.
2019-06-04 17:57:53 -07:00
Alexey Volkov 16213ba62d SDK - Dynamically installing cloudpickle module (#1429)
Fixes https://github.com/kubeflow/pipelines/issues/1426
2019-06-03 16:45:53 -07:00
Alexey Volkov 9a1d47a185 SDK - Capturing function dependencies when creating lightweight components (#1372)
* Transitively capturing code dependencies
Using cloudpickle.

* Got rid of func_type_declarations_code variable

* Extracted the function code extraction functions

* Improved support for capturing module-level dependencies

* Added test for capturing module-level dependencies

* Removed the _capture_function_code_using_source_copy function
As requested by Ning
2019-05-28 18:18:18 -07:00
Alexey Volkov b61bef04a3 SDK - Renamed ModelBase.from_struct/to_struct to from_dict/to_dict (#1290) 2019-05-07 14:06:35 -07:00
Alexey Volkov 5ab368ac10 Added support for default values to Lightweight python components (#890) 2019-03-01 14:51:18 -08:00
Alexey Volkov 32475bfafb SDK/Components/Python - Improved Python2 compatibility (#718)
Improved Python2 compatibility in Lightweight python components
2019-01-24 14:42:03 -08:00
Alexey Volkov 9b4088626c SDK/Components/Python - Made the typing.NamedTuple import optional (#717)
Now it's only imported if the return type is NamedTuple.
2019-01-23 16:31:13 -08:00
Alexey Volkov 83e9ffe5bc SDK/Components - Reworked the component model structures. (#642)
* Reworked the Component structures.
Rewrote parsing, type checking and serialization code.
Improved the graph component structures.
Added most of the needed k8s structures.
Added model validation (input/output existence etc).
Added task cycle detection and topological sorting to GraphSpec.
All container component tests now work.
Added some graph component tests.

* Fixed incompatibilities with python <3.7

* Added __init__.py to make the Travis tests work.

* Adding kubernetes structures to setup.py

* Addressed PR feedback: Renamed _original_names to _serialized_names

* Addressed PR feedback: Reduced indentation.

* Added descriptions for all component structures.

* Fixed a bug in ComponentSpec._post_init()

* Added documentation for ModelBase class and functions.

* Added __eq__/__ne__ and improved __repr__

* Added ModelBase tests
2019-01-09 15:51:34 -08:00
Alexey Volkov e06dc88316 SDK/Components - Renamed container.arguments to container.args (#437)
This aligns us with Kubernetes spec
2018-12-03 11:02:15 -08:00
Alexey Volkov dd0bd45aa3 SDK/Components - Renamed DockerContainer spec to to Container (#323) 2018-11-20 12:47:49 -08:00
Alexey Volkov 6a37388a79 SDK/Components/Python - Removed python_op in favor of python_component (#85)
* SDK/Components/Python - Removed python_op in favor of python_component

* Added test for python_component + func_to_container_op.
* dsl.PythonComponent class is removed, because the metadata is now properly stored in the function object itself.
* build_python_component now optionally accepts base_image and no longer requires the decorator (but can use decorator's metadata).
* Made staging_gcs_path optional since it's only needed when build_image == True
* Added more validation
* Added description and parameter help to python_component

* Fixed the pipeline path string after merge.

* Addressed the PR comments.
Added more detailed explanation of base_image selection to the docstring.

* Removed extra empty lines
2018-11-16 19:32:08 -08:00
Alexey Volkov 06f8821abf Components - Python - Stopped using the Fire library (#40) 2018-11-05 17:58:11 -08:00
Alexey Volkov 5409792183 SDK/Components - Switching to map-based syntax for the arguments. (#29)
* Switching to map-based syntax for the arguments.

The list-based syntax is going to be deprecated.

* Switched to map-style arguments in _func_to_component_spec

* Updated testdata
2018-11-05 14:59:23 -08:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00