Commit Graph

163 Commits

Author SHA1 Message Date
qimingj 19e7ff2995 Support Kaniko job in a outside-cluster jupyter. (#535)
* Make kaniko job working in a outside-cluster jupyter.

* Also support in-cluster config.
2018-12-13 00:47:51 -08:00
qimingj 6b315d4693 Change "namespace" param to "host" in kfp client so we can use Jupyter outside cluster. (#529)
* Change "namespace" param to "host" in kfp client so we can use Jupyter outside cluster.

* Fix tests.
2018-12-12 23:36:18 -08:00
Alexey Volkov 17582a8fab SDK/Components - Made component search locations configurable (#519)
* SDK/Components - Made component search locations configurable

* Raise proper error on existing, but malformed components instead of saying the component was not found.
Component name must be non-empty.
Addressed PR feedback.
2018-12-12 17:08:51 -08:00
Ning 886e113f78 Improve get_experiment and list_runs in the python sdk (#508)
* add get_experiment_id and list_runs_by_experiment

* offer only one get_experiment function

* return experiment body instead of id

* simply codes

* simply code 2

* remove experiment_id check in the while loop

* minor bug
2018-12-11 22:02:24 -08:00
Alexey Volkov e64a76656b SDK/Components - Do not crash on non-hashable objectsApparently Python's `dict.get` throws exception when it thinks that the object is not suitable for key. (#511) 2018-12-11 00:05:51 -08:00
Alexey Volkov fa1886c39d SDK/Components - Allow dict type spec (#490) 2018-12-08 02:29:37 -08:00
nealgao ad1950b8e6 Add notebook sample test: tfx sample (#470)
* add notebook sample tests for tfx

* parameterize component image tag

* parameterize base and target image tags

* install tensorflow package for the notebook tfx sample test

* bug fixes

* start debug mode

* fix bugs

* add namespace arg to check_notebook_results, copy test results to gcs, fix minor bugs
add CMLE model deletion

* install the correct KFP version in the notebook; parameterize deployer model name and version

* fix CMLE model name bug

* add notebook sample test in v2

* add gcp sa in notebook tfx sample and shutdown debug mode

* import kfp.gcp
2018-12-07 10:49:30 -08:00
hongye-sun 55e18269c3 support tpu settings in dsl (#491)
* support tpu settings in dsl

* fix issues from review comment
2018-12-06 21:55:36 -08:00
Alexey Volkov d9c133f7f7 SDK/Components/PythionContainerOp - Make the local output path configurableThis is part of a bigger effort to make all output locations manageable in preparation for storage system. (#424) 2018-12-04 19:56:56 -08:00
qimingj 1f8846cf20 Sanitize inputs from SDK when submitting run. (#466) 2018-12-04 16:25:21 -08:00
nealgao 632370c862 import bug (#463) 2018-12-04 12:30:59 -08:00
Alexey Volkov 800bd0249b SDK/GCP - Replaced default_gcp_op with task.apply(use_gcp_secret) (#430) 2018-12-03 18:02:30 -08:00
Alexey Volkov 96ec194260 SDK/Components - Removed outputs from task factory function signature (#388)
This realizes the outputs handling roadmap and solves problems with input and output name clashes.
2018-12-03 14:52:32 -08:00
Alexey Volkov e0a2cf66d2 SDK/PythonContainer - Compiling pipelines without needing kubernetes (#442) 2018-12-03 13:14:08 -08:00
Alexey Volkov b0461f51ff SDK/DSL - Added support for 5 more conditional operations (#309) 2018-12-03 12:33:24 -08:00
Alexey Volkov e06dc88316 SDK/Components - Renamed container.arguments to container.args (#437)
This aligns us with Kubernetes spec
2018-12-03 11:02:15 -08:00
Alexey Volkov 8a9e205d05 SDK/Components/PythonContainerOp - Simplified GCSHelper by extracting duplicate code (#210)
* Simplified GCSHelper by extracting duplicate code

* Revert to using get_bucket for now
As per  @gaoning777's request
2018-12-02 11:59:15 -08:00
IronPan 9d38f717b1 DSL - return self for .after() (#435) 2018-11-30 18:40:44 -08:00
Alexey Volkov 4f74ae3c0c SDK/Client - Removed import six (#425) 2018-11-30 15:03:59 -08:00
Alexey Volkov 98cc6ee42d SDK/Components/PythonContainerOp - Switch from dict to ComponentSpec (#396) 2018-11-30 14:14:20 -08:00
Alexey Volkov 9110296e57 SDK/Components - Support for optional inputs (#214)
* Renamed "required" to "optional"

* Added support for optional inputs

* Added tests for optional inputs. "If then *" tests now also work.
2018-11-30 13:30:09 -08:00
Chris Van Pelt 26fd724ec5 Fix for k8s dict parsing (#411)
* Add support for minio artifacts

* Add new tests for parity

* Fix for sdk env bug

* improved test
2018-11-30 11:44:38 -08:00
IronPan 537357ac8f Propagate secret to kaniko (#423)
* propagate secret to kaniko spec

* update name
2018-11-29 22:39:32 -08:00
qimingj 0b7120c322 Now pipeline function takes direct default values rather than dsp.PipelineParam. (#110)
* Now pipeline function takes direct default values rather than dsp.PipelineParam. It simplifies the sample code a lot.

* Remove extraneous parenthesis.

* Follow up CR comments.

* Change Dockerfile (not done).

* Fix dockerfile.

* Fix Dockerfile again.

* Remove unneeded installation of packages in Dockerfile.
2018-11-26 17:13:55 -08:00
nealgao cc38ca063a remove fire dependency in the component image build (#384) 2018-11-26 16:31:28 -08:00
IronPan 6003a63a60 move default gcp op to dsl/ (#376) 2018-11-26 12:50:18 -08:00
hongye-sun 486d43ddfb Add support for nvidia gpu limit (#346)
* Add support for nvidia gpu limit

* Expose resource limits, requests and nodeSelector to ContainerOp

* Fix test data

* Add explicit set_gpu_limit function

* Fix logical bug
2018-11-21 17:49:42 -08:00
google inc 33f191f308 mount gcp credentials for kaniko worker (#343)
* mount gcp credentials for kaniko worker

* update test
2018-11-20 23:33:43 -08:00
nealgao 6937e780c3 refactor component build codes (#260)
* refactor codes

* change name handler to handle
2018-11-20 20:03:34 -08:00
Alexey Volkov 325a44661c SDK/Components/PythonContainerOp - Fixed string escaping warning (#208)
W1401:Anomalous backslash in string: '\w'. String constant might be missing an r prefix.
2018-11-20 16:28:51 -08:00
nealgao 5908496852 fix bug for the component build with requirements (#320)
* fix bug for the component build with requirements

* fix unit test failure
2018-11-20 15:43:28 -08:00
Alexey Volkov dd0bd45aa3 SDK/Components - Renamed DockerContainer spec to to Container (#323) 2018-11-20 12:47:49 -08:00
Yang Pan 150ff9c085 Add default_gcp_op (#314)
* switch set() to add() for better inheritance.

* return self for container op initialization methods to allow chaining the initialization

* add default gcp op

* update

* Update sdk/python/kfp/dsl/components/default_gcp_op.py

Co-Authored-By: IronPan <yangpa@google.com>

* Update default_gcp_op.py
2018-11-19 18:33:39 -08:00
Yang Pan f28a9891d2 return self for container op initialization methods to allow chaining the construction (#313)
* switch set() to add() for better inheritance.

* return self for container op initialization methods to allow chaining the initialization

* comment
2018-11-19 16:46:55 -08:00
Yang Pan dfcf92bea8 switch set() to add() for better inheritance. (#312) 2018-11-19 15:18:56 -08:00
Alexey Volkov 906ad680ed SDK/DSL/Compiler - Improved compilation of dsl.Conditional - UX support done (#177)
* Fixed compilation of dsl.Conditional
The compiler no longer produced intermediate steps.

* Got rid of _create_new_groups

* Changed the sub_group.type check

* Update frontend handling of graphs (#293)

* Updates the frontend to correctly parse the new format of conditional pipelines

* WIP - Assume tasks and templates don't share names

* Greatly simplifies graphing of conditional and non-conditional pipelines

* Adds/updates StaticParser tests

* Give nodes unique names
2018-11-19 14:01:10 -08:00
Yang Pan 9225ac6945 remove duplication (#233) 2018-11-19 12:26:49 -08:00
nealgao b39435765d configurable namespaces in sample test and python SDK (#306)
* configurable namespaces in sample test and python SDK

* update run_test.sh

* fix a typo
2018-11-17 00:31:05 -08:00
Yasser Elsayed 0f2e9a3ca8 Job link -> run link (#268) 2018-11-16 22:25:32 -08:00
nealgao 7d811bff0a add support for dependencies in the component image building (#219)
* add requirement to component image build

* add unit test for DependencyVersion

* add unit tests, fix bugs

* revert some unit test change

* include the package name inside the VersionedDependency, enable the unit test for the component builder

* disable the component image build, this is done in another PR

* update the version name to dependency

* delete unuseful tests and add more useful unit tests

* add unit tests

* fix some unit tests

* fix bug in DependencyHelper
2018-11-16 21:49:14 -08:00
Alexey Volkov 6a37388a79 SDK/Components/Python - Removed python_op in favor of python_component (#85)
* SDK/Components/Python - Removed python_op in favor of python_component

* Added test for python_component + func_to_container_op.
* dsl.PythonComponent class is removed, because the metadata is now properly stored in the function object itself.
* build_python_component now optionally accepts base_image and no longer requires the decorator (but can use decorator's metadata).
* Made staging_gcs_path optional since it's only needed when build_image == True
* Added more validation
* Added description and parameter help to python_component

* Fixed the pipeline path string after merge.

* Addressed the PR comments.
Added more detailed explanation of base_image selection to the docstring.

* Removed extra empty lines
2018-11-16 19:32:08 -08:00
Yang Pan a47eb10558 Add volume, volumemount and env to container op (#300)
* [WIP] change deployment platform to gcp

* debug

* revert test

* add volume

* update test

* to list

* fix

* to list

* to list

* to list

* to list

* stage

* update

* update

* Undid style changes

* address comments

* update comments
2018-11-16 18:36:38 -08:00
nealgao 6a5639295f configure logger (#292) 2018-11-15 16:16:53 -08:00
nealgao 263812b070 enable component build unit test (#228)
* enable component build unit test

* minor fix
2018-11-15 15:20:51 -08:00
Alexey Volkov 0c6fef8870 SDK/Components - Fixes and more tests (#213)
* Fixed string boolean handling in if condition

* Fixed bug in isPresent

* Fixed list expansion when an item expands to a list

* Renamed two tests

* Fixed resolving primitive types (yaml supports and decodes them)

* Added test that checks handling arguments of all yaml types

* Added tests for handling true and false booleand and string literals in conditional expressions
2018-11-15 14:26:15 -08:00
Yang Pan 7e34b12e8d Add gcp secret parameter to container op (#261)
* add secret

* add secret to contianer op

* update comments

* address comments

* update logic

* fix
2018-11-15 10:06:14 -08:00
Alexey Volkov 09e4146e8c SDK/Components - Switched the generated inputs/outputs structure from dict to list (#173) 2018-11-10 15:17:26 -08:00
Alexey Volkov 2a7aeee184 SDK/Components - Removed the old argument syntax (#168) 2018-11-10 14:42:56 -08:00
Alexey Volkov 199a962e42 SDK - Relative imports (#156)
Made all SDK import relative so that they files always refer to the sibling files instead of the installed package. This makes debugging and development easier since you can be sure the correct files are used.
2018-11-10 13:56:12 -08:00
Alexey Volkov 43b2381d3b SDK/DSL-compiler - Compile without temporary files (#172)
This also avoids writing, closing and then reading again the same temporary file which is not always supported.
2018-11-10 12:51:29 -08:00
nealgao e639a361de add todos for the image tag update for new releases (#138) 2018-11-08 17:34:48 -08:00
Bradley Jiang d110eb8fdb Make kubernetes a dependency of kfp package. 2018-11-07 09:43:46 -08:00
qimingj bb62b321a5 Fix an issue that %%docker doesn't work. (#119) 2018-11-06 20:06:58 -08:00
nealgao f25d32b801 image tag update for release (#114)
* update image tag for new releases

* add more

* delete the accidentaly added sample
2018-11-06 18:12:13 -08:00
Alexey Volkov 7e4569324b SDK/DSL/Compiler - Reverted fix of dsl.Condition until the UI is ready. (#94) 2018-11-06 12:33:22 -08:00
qimingj 458c817d83 Add %%docker magic to jupyter kernel. It helps submitting a docker build job more easily with one cell. (#72) 2018-11-06 10:55:56 -08:00
Alexey Volkov e6a4c9ff73 mlp -> kfp.dsl (#88) 2018-11-06 09:40:07 -08:00
Alexey Volkov 98e4d2f881 SDK/DSL/Compiler - Fixed compilation of dsl.Condition (#28)
* Fixed compilation of dsl.Conditional
The compiler no longer produced intermediate steps.

* Got rid of _create_new_groups

* Changed the sub_group.type check

* Fix tfx name bug in the tfma sample test (#67)

* fix tfx name bug

* update release build for the data publish
2018-11-06 09:26:41 -08:00
Alexey Volkov 06f8821abf Components - Python - Stopped using the Fire library (#40) 2018-11-05 17:58:11 -08:00
Alexey Volkov 5409792183 SDK/Components - Switching to map-based syntax for the arguments. (#29)
* Switching to map-based syntax for the arguments.

The list-based syntax is going to be deprecated.

* Switched to map-style arguments in _func_to_component_spec

* Updated testdata
2018-11-05 14:59:23 -08:00
Alexey Volkov 6f4386884c Components - Removed debug print 2018-11-04 23:40:50 -08:00
Pascal Vicaire 67b011f228 Upgrading the container versions to 0.0.42, the version of the first release of kubeflow/pipelines. 2018-11-02 16:57:37 -07:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00