Commit Graph

35 Commits

Author SHA1 Message Date
Alexey Volkov 151c5349f1 Simplified the build_docker_image function (#1887)
* Simplified the build_docker_image function

* Extracted 'Dockerfile' into a variable
Addressed PR feedback

* Fixed conflict with updated master.

* Addressed the feedback
2019-08-21 12:56:33 -07:00
Alexey Volkov 1d997048ba SDK - Fixed string comparisons (#1756) 2019-08-16 18:24:09 -07:00
Ning 613086e502
Container builder default gcr (#1806)
* container builder default gcr
* get notebook id
* output explicit message when not in GCP clusters.
2019-08-14 16:21:48 -07:00
Alexey Volkov 980b4c72c8 SDK - Removed the broken build_image parameter from build_python_component function (#1657)
This parameter has never worked and has no useful purpose.
2019-08-13 04:19:41 -07:00
Alexey Volkov 17e0efe51d SDK - Containers - Returning image name with digest (#1768)
* SDK - Containers - Returning image name with digest

Image building functions now return image name with digest: image_repo@sha256:digest

Fixes https://github.com/kubeflow/pipelines/issues/1715

* Added comments
2019-08-09 17:49:13 -07:00
Zane Durante f18b7fd121 Adding multiple outputs into sdk with sample (#1667)
* Added support for mulitple outputs

* Added test for multiple output

* Adding sample for multiple outputs

* func_signature now shorter form

* Added parameters tag

* Fixed func_signature mistake
2019-08-01 13:42:14 -07:00
Ning 5387e24d87 Separate codegen from containerbuild 2 (#1680)
* refactor component build code

* remove unnecessary import

* minor changes

* fix unit tests

* separate the container build from the component build; add support for directories in the containerbuilder

* minor fixes

* fix unit test

* fix tarball error

* revert changes

* unit test fix

* minor fix

* addressing comments

* removing the check_gcs_path function

* move namespace to the contructor of containerbuilder

* fix bugs
2019-07-30 14:43:50 -07:00
Ning 0283dddbee Separate codegen from containerbuild (#1679)
* refactor component build code

* remove unnecessary import

* minor changes

* fix unit tests

* fix sample test bug

* revert the change of the dependency orders

* add -u to disable python stdout buffering

* address the comments

* separate lines to look clean

* fix unit tests

* fix
2019-07-26 16:09:58 -07:00
Ning 46aafbd062
move gcshelper out of component_builder (#1658)
* move gcshelper out of component_builder
2019-07-24 14:24:08 -07:00
Ning 17e8c89719
update kaniko executor version to speed up image build (#1652)
* update kaniko executor version to speed up image build
2019-07-23 14:28:29 -07:00
IronPan 02313e4e5e Cleanup Kaniko code (#1394)
/assign @gaoning777 @Ark-kun
2019-05-28 16:46:17 -07:00
Alexey Volkov b61bef04a3 SDK - Renamed ModelBase.from_struct/to_struct to from_dict/to_dict (#1290) 2019-05-07 14:06:35 -07:00
Ning d6a9f60ddb display kaniko log if failed (#1247) 2019-04-29 13:03:35 -07:00
Ning a6763b9599 component build support for both python2 and python3 (#730)
* component build support for both python2 and python3

* add sample test

* remove the annotations for python2 component build

* add pathlib for python2 component build

* fix component build unit test

* fix bug in the dockerfile generator

* remove exist_ok in path.mkdir to make python2 compatible

* adjust unit test

* remove pathlib dependency for python2 component build

* remove the pathlib codes in python3 component build, but use python2 code instead; add a todo to create a new sample
2019-02-25 12:56:19 -08:00
Alexey Volkov 83e9ffe5bc SDK/Components - Reworked the component model structures. (#642)
* Reworked the Component structures.
Rewrote parsing, type checking and serialization code.
Improved the graph component structures.
Added most of the needed k8s structures.
Added model validation (input/output existence etc).
Added task cycle detection and topological sorting to GraphSpec.
All container component tests now work.
Added some graph component tests.

* Fixed incompatibilities with python <3.7

* Added __init__.py to make the Travis tests work.

* Adding kubernetes structures to setup.py

* Addressed PR feedback: Renamed _original_names to _serialized_names

* Addressed PR feedback: Reduced indentation.

* Added descriptions for all component structures.

* Fixed a bug in ComponentSpec._post_init()

* Added documentation for ModelBase class and functions.

* Added __eq__/__ne__ and improved __repr__

* Added ModelBase tests
2019-01-09 15:51:34 -08:00
Ning 85c6413a2e Refactor Python SDK (#568)
* add some comments

* remove unused import; add license to dsl_bridge

* move_convert_k8s_obj_to_dic from compiler to k8s_helper

* move unit test
2018-12-20 09:51:09 -08:00
Alexey Volkov d9c133f7f7 SDK/Components/PythionContainerOp - Make the local output path configurableThis is part of a bigger effort to make all output locations manageable in preparation for storage system. (#424) 2018-12-04 19:56:56 -08:00
nealgao 632370c862 import bug (#463) 2018-12-04 12:30:59 -08:00
Alexey Volkov e0a2cf66d2 SDK/PythonContainer - Compiling pipelines without needing kubernetes (#442) 2018-12-03 13:14:08 -08:00
Alexey Volkov e06dc88316 SDK/Components - Renamed container.arguments to container.args (#437)
This aligns us with Kubernetes spec
2018-12-03 11:02:15 -08:00
Alexey Volkov 8a9e205d05 SDK/Components/PythonContainerOp - Simplified GCSHelper by extracting duplicate code (#210)
* Simplified GCSHelper by extracting duplicate code

* Revert to using get_bucket for now
As per  @gaoning777's request
2018-12-02 11:59:15 -08:00
Alexey Volkov 98cc6ee42d SDK/Components/PythonContainerOp - Switch from dict to ComponentSpec (#396) 2018-11-30 14:14:20 -08:00
nealgao cc38ca063a remove fire dependency in the component image build (#384) 2018-11-26 16:31:28 -08:00
google inc 33f191f308 mount gcp credentials for kaniko worker (#343)
* mount gcp credentials for kaniko worker

* update test
2018-11-20 23:33:43 -08:00
nealgao 6937e780c3 refactor component build codes (#260)
* refactor codes

* change name handler to handle
2018-11-20 20:03:34 -08:00
Alexey Volkov 325a44661c SDK/Components/PythonContainerOp - Fixed string escaping warning (#208)
W1401:Anomalous backslash in string: '\w'. String constant might be missing an r prefix.
2018-11-20 16:28:51 -08:00
nealgao 5908496852 fix bug for the component build with requirements (#320)
* fix bug for the component build with requirements

* fix unit test failure
2018-11-20 15:43:28 -08:00
Alexey Volkov dd0bd45aa3 SDK/Components - Renamed DockerContainer spec to to Container (#323) 2018-11-20 12:47:49 -08:00
nealgao 7d811bff0a add support for dependencies in the component image building (#219)
* add requirement to component image build

* add unit test for DependencyVersion

* add unit tests, fix bugs

* revert some unit test change

* include the package name inside the VersionedDependency, enable the unit test for the component builder

* disable the component image build, this is done in another PR

* update the version name to dependency

* delete unuseful tests and add more useful unit tests

* add unit tests

* fix some unit tests

* fix bug in DependencyHelper
2018-11-16 21:49:14 -08:00
Alexey Volkov 6a37388a79 SDK/Components/Python - Removed python_op in favor of python_component (#85)
* SDK/Components/Python - Removed python_op in favor of python_component

* Added test for python_component + func_to_container_op.
* dsl.PythonComponent class is removed, because the metadata is now properly stored in the function object itself.
* build_python_component now optionally accepts base_image and no longer requires the decorator (but can use decorator's metadata).
* Made staging_gcs_path optional since it's only needed when build_image == True
* Added more validation
* Added description and parameter help to python_component

* Fixed the pipeline path string after merge.

* Addressed the PR comments.
Added more detailed explanation of base_image selection to the docstring.

* Removed extra empty lines
2018-11-16 19:32:08 -08:00
nealgao 6a5639295f configure logger (#292) 2018-11-15 16:16:53 -08:00
nealgao 263812b070 enable component build unit test (#228)
* enable component build unit test

* minor fix
2018-11-15 15:20:51 -08:00
Alexey Volkov 199a962e42 SDK - Relative imports (#156)
Made all SDK import relative so that they files always refer to the sibling files instead of the installed package. This makes debugging and development easier since you can be sure the correct files are used.
2018-11-10 13:56:12 -08:00
Alexey Volkov 5409792183 SDK/Components - Switching to map-based syntax for the arguments. (#29)
* Switching to map-based syntax for the arguments.

The list-based syntax is going to be deprecated.

* Switched to map-style arguments in _func_to_component_spec

* Updated testdata
2018-11-05 14:59:23 -08:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00