Commit Graph

282 Commits

Author SHA1 Message Date
Jiaxiao Zheng 56dfff52b1
Fix lint related issue (#1922)
* Add flake8 ignore
2019-08-21 21:45:53 -07:00
Eterna2 ad307db5b9 [Bug Fix] Delete ResourceOp should not have output parameters (#1822)
* Fix bug where delete resource op should not have success_condition, failure_condition, and output parameters

* remove unnecessary whitespace

* compiler test for delete resource ops should retrieve templates from spec instead of root
2019-08-21 17:52:32 -07:00
Alexey Volkov 593f25a5aa Collecting coverage when running python tests (#898)
* Collecting coiverage when running python tests

* Added coveralls to python unit tests

* Try removing the PATH modification

* Specifying coverage run --source

* Using the installed package

* Try getting the correct coverage paths
2019-08-21 17:16:33 -07:00
Alexey Volkov 553885ffb1
SDK - Components - Fixed ModelBase comparison bug (#1874) 2019-08-21 16:38:12 -07:00
Alexey Volkov 203307dbaf
SDK - Lightweight - Fixed custom types in multi-output case (#1875)
The type was mistakenly serialized as `_ForwardRef('CustomType')`.
The input parameter types and single-output types were not affected.
2019-08-21 16:37:21 -07:00
Alexey Volkov 9adf16301d
SDK - Airflow - Fixed bug in airflow op creation (#1911)
This PR fixes a bug in AirFlow op creation.
The `_run_airflow_op` helper function was not captured along with the `_run_airflow_op_closure` function, because they belong to different modules (`_run_airflow_op_closure` was module-less).
This was not discovered during the notebook testing of the code since in that environment the `_run_airflow_op` was also module-less as it was defined in a notebook (not in .py file).
2019-08-21 16:29:54 -07:00
hongye-sun 7ec56978b6 Release 151c5349f1 (#1916)
* Updated component images to version 151c5349f1

* Updated components to version a97f1d0ad0

* Update setup.py

* Update setup.py
2019-08-21 16:06:31 -07:00
Christian Clauss 8e1e823139 Lint Python code for undefined names (#1721)
* Lint Python code for undefined names

* Lint Python code for undefined names

* Exclude tfdv.py to workaround an overzealous pytest

* Fixup for tfdv.py

* Fixup for tfdv.py

* Fixup for tfdv.py
2019-08-21 15:04:31 -07:00
Alexey Volkov 151c5349f1 Simplified the build_docker_image function (#1887)
* Simplified the build_docker_image function

* Extracted 'Dockerfile' into a variable
Addressed PR feedback

* Fixed conflict with updated master.

* Addressed the feedback
2019-08-21 12:56:33 -07:00
Ning 79c7bdabaf fix unit tests and address some comments (#1892) 2019-08-20 09:24:56 -07:00
Aakash Bajaj 6a7b28f35d gcp cred bug fix for multiple credentials in single pipeline (#1384)
* gcp cred bug fix for multiple credentials in single pipeline

* squash to remove conflict
2019-08-20 01:37:33 -07:00
Alexey Volkov 2b246bc356 SDK - Tests - Improved the "ContainerOp.set_retry" test (#1843)
Properly testing the feature isntead of just comparing with golden data.
2019-08-16 19:54:07 -07:00
Alexey Volkov 1d997048ba SDK - Fixed string comparisons (#1756) 2019-08-16 18:24:09 -07:00
Ajay Gopinathan 5bcc665928
Update Python SDK versions for release. (#1860) 2019-08-16 13:58:23 -07:00
Alexey Volkov 54ff3e6614 SDK - Cleanup - Serialized PipelineParamTuple does not need value or type (#1469)
* SDK - Refactoring - Serialized PipelineParam does not need type
Only the types in non-serialized PipelineParams are ever used.

* SDK - Refactoring - Serialized PipelineParam does not need value
Default values are only relevant when PipelineParam is used in the pipeline function signature and even in this case compiler captures them explicitly from the pipelineParam objects in the signature.
There is no other uses for them.
2019-08-16 01:22:31 -07:00
Alexey Volkov d8eaeaad95
SDK - Preserving the pipeline input information in the compiled Workflow (#1381)
* SDK - Preserving the pipeline metadata in the compiled Workflow

* Stabilizing the DSL compiler tests
2019-08-15 17:25:59 -07:00
Ning 613086e502
Container builder default gcr (#1806)
* container builder default gcr
* get notebook id
* output explicit message when not in GCP clusters.
2019-08-14 16:21:48 -07:00
Alexey Volkov 980b4c72c8 SDK - Removed the broken build_image parameter from build_python_component function (#1657)
This parameter has never worked and has no useful purpose.
2019-08-13 04:19:41 -07:00
IronPan 4e43750c9d
add reference name to resource reference API proto (#1781)
* add resource reference

* fix tests

* fix tests

* fix e2e test

* fix e2e

* fix test

* update api requirement

* fix tests

* Update job_api_test.go

* Update run_api_test.go

* Update setup.py

* Update deploy-kubeflow.sh

* fix tests

* Update deploy-kubeflow.sh
2019-08-13 03:15:11 -07:00
Alexey Volkov 7917ea475e SDK - Lightweight - Added support for complex default values (#1696) 2019-08-12 02:35:13 -07:00
Ning 243b88dbac ContainerBuilder loading kube config (#1795)
* avoid istio injector in the container builder

* find the correct namespace

* configure default ns to kubeflow if out of cluster; fix unit tests

* container build default gcs bucket

* resolve comments

* code refactor; add create_bucket_if_not_exist in containerbuilder

* support load kube config and output error, good for ai platform notebooks/local notebooks

* remove create_bucket_if_not_exist param
2019-08-09 23:59:14 -07:00
Alexey Volkov 17e0efe51d SDK - Containers - Returning image name with digest (#1768)
* SDK - Containers - Returning image name with digest

Image building functions now return image name with digest: image_repo@sha256:digest

Fixes https://github.com/kubeflow/pipelines/issues/1715

* Added comments
2019-08-09 17:49:13 -07:00
Ning 9c79163287
Container builder (#1774)
* avoid istio injector in the container builder
* find the correct namespace
* configure default ns to kubeflow if out of cluster; fix unit tests
2019-08-09 13:57:51 -07:00
Jiaxiao Zheng 90b9ad7657 Move imagepullsecrets sample to samples/core (#1767)
* Remove redundant import.

* Simplify sample_test.yaml by using withItem syntax.

* Simplify sample_test.yaml by using withItem syntax.

* Change dict to str in withItems.

* Add image pull secret sample.

* Move imagepullsecret sample from test dir to sample dir. Waiting on corresponding unit test infra refactoring.

* Update the location of imagepullsecrets so that it can serve as an example.

* Add minimal comments documenting usage.
2019-08-08 17:35:26 -07:00
Jiaxiao Zheng 9a047fb016
Add preemtptible gpu sample (#1749)
* Remove redundant import.

* Simplify sample_test.yaml by using withItem syntax.

* Simplify sample_test.yaml by using withItem syntax.

* Change dict to str in withItems.

* Add preemptible gpu tpu sample and unittest

* Update a test utility function.

* Seperate the location of sample and gold .yaml for testing purpose.
2019-08-08 13:58:46 -07:00
Alexey Volkov dd59bc2597 SDK - Lightweight - Fixed regression for components without outputs (#1726) 2019-08-05 21:47:53 -07:00
Alexey Volkov 4fb329eaf8 SDK - ContainerOp.set_display_name should return self (#1718)
/assign @gaoning777
2019-08-05 20:47:52 -07:00
Alexey Volkov a7635f1cd4 SDK - Using Airflow ops in Pipelines (#1483)
* SDK - Using Airflow ops in Pipelines

* Documented the create_component_from_airflow_op function

* Need to set use_code_pickling=True now

* Using the original operator name as the component name

* Filtering out `*args` and `**kwargs` parameters that some operators have

* Fixed the function call

* Changed the default airflow base image
Airflow has removed most of the old images and tags.
See https://issues.apache.org/jira/browse/AIRFLOW-5093 and  https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+CI+image#AIP-10Multi-layeredandmulti-stageofficialAirflowCIimage-ProposedsetupoftheDockerHubandTravisCI .
2019-08-02 19:53:52 -07:00
Alexey Volkov 94969d6264 SDK/Lightweight - Updated default image to tensorflow:1.13.2-py3 (#1671) 2019-08-02 18:31:53 -07:00
Alexey Volkov ca93033d47 SDK/Client - Added the create_run_from_pipeline_package method (#1523)
I does not add much value over run_pipeline, buy @hongyes proposed adding it for parity.
2019-08-02 14:13:51 -07:00
Zane Durante f18b7fd121 Adding multiple outputs into sdk with sample (#1667)
* Added support for mulitple outputs

* Added test for multiple output

* Adding sample for multiple outputs

* func_signature now shorter form

* Added parameters tag

* Fixed func_signature mistake
2019-08-01 13:42:14 -07:00
Ning c79addf665
Component build fix (#1703)
* simplify the tarfile buildfiles
2019-07-31 16:31:36 -07:00
Ning 5387e24d87 Separate codegen from containerbuild 2 (#1680)
* refactor component build code

* remove unnecessary import

* minor changes

* fix unit tests

* separate the container build from the component build; add support for directories in the containerbuilder

* minor fixes

* fix unit test

* fix tarball error

* revert changes

* unit test fix

* minor fix

* addressing comments

* removing the check_gcs_path function

* move namespace to the contructor of containerbuilder

* fix bugs
2019-07-30 14:43:50 -07:00
Ning 0283dddbee Separate codegen from containerbuild (#1679)
* refactor component build code

* remove unnecessary import

* minor changes

* fix unit tests

* fix sample test bug

* revert the change of the dependency orders

* add -u to disable python stdout buffering

* address the comments

* separate lines to look clean

* fix unit tests

* fix
2019-07-26 16:09:58 -07:00
jingzhang36 80d16406f6 Increase KFP version in (1) sdk/python/setup.py (2) component_sdk/python/setup.py (3) manifests/base/kustomization.yaml (#1674)
* Increase KFP version in sdk/python/setup.py component_sdk/python/setup.py

* Increase kfp version in manifests/base/kustomization.yaml
2019-07-25 20:03:58 -07:00
Alexey Volkov f6cf9c5f55 SDK - Lightweight - Added support for "None" default values (#1626)
* SDK - Lightweight - Added support for "None" default values
Previously it was impossible to pass None to components since it was being converted to the string "None".

* is_required = not input.optional for now
As asked by @gaoning777
2019-07-25 18:49:59 -07:00
hongye-sun 68cf384ad2 Adding kfp.cil to kfp setup.py. (#1666) 2019-07-24 17:31:51 -07:00
Ning 46aafbd062
move gcshelper out of component_builder (#1658)
* move gcshelper out of component_builder
2019-07-24 14:24:08 -07:00
Eterna2 08ff76f5f1 [Feature] Set ttlSecondsAfterFinished in argo workflow with PipelineConf (#1594)
* Add PipelineConf method to set ttlSecondsAfterFinished in argo workflow spec

* remove unnecessary compile test for ttl. add unit test for ttl instead.
2019-07-24 09:26:15 -07:00
Alexey Volkov 02808c50b9 Samples - Removed the immediate_value sample (#1630)
Users can already pass constant values to components. There is no need to use PipelineParam for that.
2019-07-23 17:40:14 -07:00
Ning 17e8c89719
update kaniko executor version to speed up image build (#1652)
* update kaniko executor version to speed up image build
2019-07-23 14:28:29 -07:00
Alexey Volkov 3da6e90253 Samples - Cleaned up unnecessary usage of PipelineParam (#1631) 2019-07-23 14:16:15 -07:00
IronPan 8bc464409b add init container for container op (#1650)
* add init container

* update test

* update tests

* address comments
2019-07-22 20:40:54 -07:00
Ilias Katsakioris 56e923f8d1 SDK/DSL: Fix bug when specifying custom resource for VolumeOp (#1595)
Closes #1593.

Signed-off-by: Ilias Katsakioris <elikatsis@arrikto.com>
2019-07-17 11:30:13 -07:00
Yaron Haviv 6e5171452b add support for flexible config (via env var) for the pipline service and UI, fix broken links (pointed to API vs UI service) (#1293)
* add support for flexible config (via env var) for the pipline service and UI, fix broken links (pointed to API vs UI service)

* support https prefix

* change to LF

* init _uihost

* set _uihost from host param if specified no env var

* change config.host to host
2019-07-16 14:26:05 -07:00
Ning cce52f4dfb update manifest and sdk version (#1617) 2019-07-12 14:39:04 -07:00
Ning 28b871a2be fix dependency bug in the recursion support (#1616) 2019-07-12 12:59:05 -07:00
Kevin Bache ac833a084b List pipelines (#1588)
* added list_pipelines

* minor update
2019-07-09 18:28:24 -07:00
hongye-sun 97ccfd0c9a add_pod_env op handler (#1540)
* Configure gcp connectors in dsl

* Make configure_gcp_connector more extensible

* Add add_pod_env op handler.

* Only apply add_pod_env on ContainerOp

* Update license header
2019-07-08 11:42:35 -07:00
Ning f23b619880 fix recursion bug (#1583)
* fix recursion bug

* propagate inputs to out layers of opsgroup; adjust unit tests
2019-07-01 16:55:08 -07:00