Commit Graph

185 Commits

Author SHA1 Message Date
Ajay Gopinathan a882999c42 Use Remote Build Execution for Bazel builds. (#1031)
* [WIP] Use Remote Build Execution for Bazel builds.

* Add build_api_server.sh build script for API server.

* update build_image.yaml

* update build_image.yaml

* update build_image.yaml

* update build_image.yaml

* update build_image.yaml

* update build_image.yaml

* escape newlines

* Try using bazelbuilder's service account credentials.

* Clean up

* Add README

* Update backend README

* Update OWNERS
2019-09-30 10:41:38 -07:00
Alexey Volkov 342abae27a SDK - Moved the _container_builder from kfp.compiler to kfp.containers (#2192)
* SDK - Moved the _container_builder from kfp.compiler to kfp.containers
This only moves the files. The imports remain the same for now.

* Simplified the imports.
2019-09-25 18:27:06 -07:00
Kevin Bache b52e32b1a5 Add samples for loops (#2218)
* added loop samples

* moved samples; added samples to tests

* year
2019-09-24 14:59:58 -07:00
Alexey Volkov 7dab30085e Testing - Output Argo workflow information when the workflow times out (#2176)
* Testing - Output Argo workflow information when the workflow times out

* Further improved argo logging in case of failure or timeout

* Fixed bug and addressed the feedback

* Fixed failure and timeout detection

* Making the script quieter

* check-argo-status no longer exits the main script on success
2019-09-23 16:07:24 -07:00
Ning 86163ecd38
Remove dataflow components (#2161)
* remove dataflow components
2019-09-23 11:12:27 -07:00
IronPan 0711566754 Build inverse proxy image as part of the presubmit test (#2187)
* small fixes

* add

* Delete Makefile
2019-09-21 16:53:23 -07:00
Alexey Volkov 372efe56c5 Tests - Samples - Added the component_build sample test (#2147)
* Tests - Samples - Added the component_build sample test

* Fixed the papermill issue

* Renamed EXPERIMENT_NAME to experiment_name
2019-09-20 14:31:27 -07:00
IronPan ca4683d9e9
Add Bobgy to test infra owner (#2166)
/assign @Bobgy
2019-09-19 15:25:22 +08:00
Ning a855c5655b remove tfx-taxi sample in favor of the tfx oss sample (#2160) 2019-09-18 22:59:06 -07:00
Ning a71d5b0f28
add kubeflow serving sample test (#2163)
* add kubeflow serving sample test
2019-09-18 15:46:20 -07:00
Yuan (Bob) Gong 2572013434 Fix presubmit test failure -- build job in QUEUED state (#2152) 2019-09-17 22:43:29 -07:00
Ning 551b0dc56c add preemptible vm and volume snapshot op sample tests (#2149)
* add preemptible vm and volume snapshot op sample tests

* add config
2019-09-17 20:51:29 -07:00
Ning d72411cf35 add retry tests (#2141)
* add retry tests

* add arguments

* fix sample test infra error
2019-09-17 18:21:29 -07:00
Jiaxiao Zheng d6d3785506 Add a convention to make sure experiment_name is parameterized in notebook sample. (#2112)
* add a new convention

* Fix config experiment_name

* fix light weight example

* Fix experiment name

* Update guidelin.

* fix lightweight_component.config.yaml

* Remove config yaml

* Fix guideline.md

* Reword.
2019-09-17 15:37:14 -07:00
Ning 759d2f6576
add artifact location tests (#2142) 2019-09-17 13:34:28 -07:00
Ning 4f88a820f5
Add sidecar sample test (#2106)
* add sidecar sample test
* add more sleep sec for the sidecar to star
2019-09-17 09:32:46 -07:00
Ning 52fc2c824a
add dataflow sample with the new GCP component (#2096)
* add dataflow sample with tests
2019-09-16 14:19:35 -07:00
Jiaxiao Zheng b86f64b81c fix name. (#2132) 2019-09-16 12:32:39 -07:00
Jiaxiao Zheng dc3c7a4d37 add exec order (#2114) 2019-09-16 10:46:38 -07:00
Jiaxiao Zheng 216c80f43f add ips sample test (#2113) 2019-09-13 17:16:40 -07:00
Ning 646aa5ff64 feature parity between notebook sample and normal sample (#2095)
* feature parity between notebook sample and normal sample

* typo
2019-09-12 18:26:31 -07:00
Jiaxiao Zheng a4fa1edb42 Use GCSHelper to upload test result (#2078)
* Add GCSHelper in sample_test_launcher

* Remove redundant stuff
2019-09-10 09:06:55 -07:00
Jiaxiao Zheng 30b97ddffa Fix. (#2077) 2019-09-09 19:30:53 -07:00
Alexey Volkov 570141d7c9 Make wget quieter (#2069) 2019-09-09 14:32:54 -07:00
Jiaxiao Zheng 7b1c720a47 Improve notebook check automation (#2040)
* Add logic to detect extension name.

* Rename notebook samples

* Change to use config yaml for papermill preprocess.

* Remove ad hoc logic

* Remove duplicated logic

* Refactor

* Add run_pipeline flag in config yaml

* Add run pipeline flag for .py sample as well.

* Fix extension name

* Fix

* Fix problems in docstring.

* refactor run_sample_test.py into two functions

* Refactor the procedure into 3 steps

* Fix bug in exit code format

* Remove two redundant functions.
2019-09-09 13:40:55 -07:00
Ning 60018e3149
update sample readme for the new structure (#2058)
* update sample readme for the new structure
* sidecar seems flaky, disable for now and will re-enable later
2019-09-06 14:24:45 -07:00
Ning fe0bc678a8 add more core samples in the sample test (#2030)
* add more core samples in the sample test

* remove retry test for now

* update pipeline transformer sample

* add default value to volume_ops and fix error

* remove volume snapshot and resource op sample tests, which require platform support

* remove volume_ops since it is already included in the e2e

* add timeout
2019-09-06 11:46:15 -07:00
Ilias Katsakioris 0cc9f69b04 Extend E2E tests: Add VolumeOp test (#2019)
* Extend E2E tests: Add VolumeOp test

* Rename: `samples/core/volume_ops/volumeop.py` ->
          `samples/core/volume_ops/volume_ops.py`
  for uniformity regarding test launching
* Change VolumeOp modes from RWM to RWO

* Add volume_ops.config.yaml
2019-09-03 21:24:58 -07:00
Yuan (Bob) Gong 3d2b0ae417 Print cloud build logs when the task fails (#1989)
* Print cloud build logs when the task fails

* Add comment about why cloud build is made async
2019-09-03 18:12:58 -07:00
Kirin Patel 41b394b045 Add e2e visualization tests (#1981)
* Created visualization_api_test.go

* Updated BUILD.bazel files

* Removed clean_up from e2e test

* Revert "Removed clean_up from e2e test"

This reverts commit 82fd4f5a00.

* Update e2e tests to build visualizationserver and viewer-crd

* Fix bug where wrong image is set

* Fixed incorrect image names

* Fixed additional instance of incorrect image names
2019-08-30 13:54:10 -07:00
Alexey Volkov 3962032898 Testing - Fixed the comparison bug in presubmit test script (#1990)
* Testing - Fixed the comparison bug in presubmit test script

Bash was treating the single quotes literally.

* Addressed the feedback
2019-08-30 12:48:04 -07:00
Jiaxiao Zheng 5ac9b0449a
Add schema validation for sample test config yaml (#2000)
* Add schema for yaml files, and remove native style checks.

* Add schema check in run_sample_test.py

* fix schema
2019-08-30 11:33:56 -07:00
Jiaxiao Zheng 4918d1919a
Change sample test to read test arguments from yaml config file (#1987)
* Clean unused import

* nit

* nit: improve docstring.

* Refactor sample test into digesting params from config yaml files.

* Fix argument assignment.

* Fix path.

* Fix output params. Not every test is using it.

* Add output placeholder in yaml config.

* Fix yaml config.

* Minor fix.

* Minor fix.

* Move timeout info to config.yaml, too

* Fix import in check_notebook_results.py

* Add type hints in config.yaml

* Remove redundant close.
2019-08-29 16:29:55 -07:00
IronPan bbfe5e09cc move pipeline runner service account to backend (#1988)
* move pipeline runner service account to backend

* revert swf change

* revert swf change

* update tests

* update tests

* Update compiler.py

revert python change for backward compatibility

* Update compiler.py

revert python change for backward compatibility
2019-08-29 16:03:14 -07:00
Jiaxiao Zheng 63483b66f5
Refactor tedious redefinition using withItem. (#1985) 2019-08-28 17:34:55 -07:00
Jiaxiao Zheng 2f7d55b98c
Add travis CI for new unit test. (#1960)
* Relocate unit test for sample test infra.

* Relocate utils tests and add travis ci check.

* Revert accidental change.

* Lint

* Fix travis build

* Move dependencies to install section.

* Fix travis build

* Fix travis build

* Fix travis build
2019-08-27 23:08:32 -07:00
Ning 03143cf9e0
reduce taxi and xgboost test data to speed up sample test (#1917) 2019-08-27 18:32:35 -07:00
dushyanthsc a3a174b91e test/project-cleanup - fix (#1964)
This change fixes the missing packages issue in project-cleanup job.

Issue: project cleanup job runs in Prow and uses kubekins-e2e image, which
in turn uses go-1.12 version and enlists the source in $GOPATH/src
directory, the packages specified in go.mod file are not discovered.
The fix is to install the packages using go-get, and has been tested
using the kubekins-e2e image.

The change also has some minor logging fixes to the project-cleanup
tool.
2019-08-27 16:47:10 -07:00
Jiaxiao Zheng 77210fba7e Refactor run_sample_test.py script into python class (#1963)
* Refactor run_sample_test.py into python class.

* Introduce PySampleChecker in sample_test_launcher
2019-08-27 16:07:11 -07:00
Alexey Volkov 856949ed30
Test infra - Fixed the batch Prow tests (#1958)
Fixes https://github.com/kubeflow/pipelines/issues/1957
2019-08-27 14:16:19 -07:00
Jiaxiao Zheng e58ba48eac
Refactor check_notebook_results.py into python module. (#1947)
* Refactor check_notebook_results.py into python module.

* Fix indentation.

* Fix indentation.

* Fix indentation.

* Fix indentation.
2019-08-26 18:05:04 -07:00
Jiaxiao Zheng 3a76e4bc67
Add unit test for file_injection. (#1952)
Also fix a bug in file_injection.
2019-08-26 13:37:18 -07:00
Jiaxiao Zheng 27d67420db Fix the missing argument of re.sub (#1948) 2019-08-26 08:30:23 -07:00
Jiaxiao Zheng 555356653b
Sample test improvement - using python fire to launch sample test (#1897)
* Remove redundant import.

* Simplify sample_test.yaml by using withItem syntax.

* Simplify sample_test.yaml by using withItem syntax.

* Change dict to str in withItems.

* Add back coveralls.

* Squashed commit of the following:

commit 7dbca1afce
Author: Ning <ngao@google.com>
Date:   Fri Aug 16 12:06:37 2019 -0700

    update gcloud ml-engine to ai-platform (#1863)

commit e849f22a28
Author: Ryan Dawson <ryandawson@cantab.net>
Date:   Fri Aug 16 19:21:23 2019 +0100

    Seldon examples (#1405)

commit a6f3f40e5e
Author: sina chavoshi <sina.chavoshi@gmail.com>
Date:   Fri Aug 16 09:56:09 2019 -0700

    Adding a sample for serving component (#1830)

    * Adding a sample for serving component

    * removed typo / updated based on PR feedback

    * fixing the jupyter rendering issue

    * adding pip3 for tensorflow

    * Fixed spelling error in VERSION

    * fix indentation based on review feedback

commit 54ff3e6614
Author: Alexey Volkov <avolkov@google.com>
Date:   Fri Aug 16 01:22:31 2019 -0700

     SDK - Cleanup - Serialized PipelineParamTuple does not need value or type (#1469)

    * SDK - Refactoring - Serialized PipelineParam does not need type
    Only the types in non-serialized PipelineParams are ever used.

    * SDK - Refactoring - Serialized PipelineParam does not need value
    Default values are only relevant when PipelineParam is used in the pipeline function signature and even in this case compiler captures them explicitly from the pipelineParam objects in the signature.
    There is no other uses for them.

commit d2e94e4e04
Author: Riley Bauer <34456002+rileyjbauer@users.noreply.github.com>
Date:   Thu Aug 15 19:52:34 2019 -0700

    Fix run duration bug (#1827)

    * Allows durations >=24h and renames 'showLink' in RunList

    * Update, fix tests

commit d8eaeaad95
Author: Alexey Volkov <avolkov@google.com>
Date:   Thu Aug 15 17:25:59 2019 -0700

    SDK - Preserving the pipeline input information in the compiled Workflow (#1381)

    * SDK - Preserving the pipeline metadata in the compiled Workflow

    * Stabilizing the DSL compiler tests

commit afe8a694f6
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Thu Aug 15 17:00:28 2019 -0700

    Reduce API usage by utilizing reference name in reference resource API (#1824)

    * Regenerated run api for frontend

    * Added support for reference name to resource reference API in frontend

    * Revert "Regenerated run api for frontend"

    * Addressed PR comments

    * Removed extra if statement by setting default value of parameter

    * Removed the whole comment

    * Addressed PR feedback

    * Addressed PR feedback

    * Simplified logic after offline discussion

commit ea67c998b6
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Thu Aug 15 16:18:35 2019 -0700

    Change how Variables are Provided to Visualizations (#1754)

    * Changed way visualization variables are passed from request to NotebookNode

    Visualization variables are now saved to a json file and loaded by a NotebookNode upon execution.

    * Updated roc_curve visualization to reflect changes made to dependency injection

    * Fixed bug where checking if is_generated is provided to roc_curve visualization would crash visualizaiton

    Also changed ' -> "

    * Changed text_exporter to always sort variables by key for testing

    * Addressed PR suggestions

commit d238bef1fe
Author: Jiaxiao Zheng <jxzheng@google.com>
Date:   Thu Aug 15 12:55:29 2019 -0700

    Add back coveralls. (#1849)

    * Remove redundant import.

    * Simplify sample_test.yaml by using withItem syntax.

    * Simplify sample_test.yaml by using withItem syntax.

    * Change dict to str in withItems.

    * Add back coveralls.

commit 0517114dc2
Author: Riley Bauer <34456002+rileyjbauer@users.noreply.github.com>
Date:   Thu Aug 15 12:28:35 2019 -0700

    Reduce getPipeline calls in RunList (#1852)

    * Skips calling getPipeline in RunList if the pipeline name is in the pipeline_spec

    * Update fixed data to include pipeline names in pipeline specs

    * Remove redundant getRuns call

commit 39e5840f2f
Author: IronPan <yangpa@google.com>
Date:   Thu Aug 15 11:04:34 2019 -0700

    Add retry button in Pipeline UI (#1782)

    * add retry button

    * add retry button

    * add retry button

    * address comments

    * fix tests

    * fix tests

    * update image

    * Update StatusUtils.test.tsx

    * Update RunDetails.test.tsx

    * Update Buttons.ts

    * update test

    * update frontend

    * update

    * update

    * addrerss comments

    * update test

* WIP: rewrite sample test infra
step 1, presubmit check triggered by fire.

* WIP: rewrite sample test infra
step 2, add ComponentTest class

* WIP: rewrite sample test infra
Rewrite dockerfile

* Fixing

* Refactor the logic of run_test.sh into sample_test_launcher.py

* Fix exit_code

* Fix imports and add todo

* Squashed commit of the following:

commit 0ed5819ae9
Author: Yuan (Bob) Gong <gongyuan94@gmail.com>
Date:   Tue Aug 20 08:13:31 2019 +0800

    Refactor presubmit-tests-with-pipeline-deployment.sh to run in other projects (#1732)

    * Refactor presubmit-tests-with-pipeline-deployment.sh so that it can be run from a different project

    * Simplify getting service account from cluster.

    * Copy image builder image instead of granting permission

    * Add missed yes command

    * fix stuff

    * Let other usages of image-builder image become configurable

    * let test workflow use image builder image

* Clean up.

* Fix launcher flag.

* switch to sys.execxutable.

* Fix bugs

* Add copy blob util function
fix issues,
and
Squashed commit of the following:

commit 0ed5819ae9
Author: Yuan (Bob) Gong <gongyuan94@gmail.com>
Date:   Tue Aug 20 08:13:31 2019 +0800

    Refactor presubmit-tests-with-pipeline-deployment.sh to run in other projects (#1732)

    * Refactor presubmit-tests-with-pipeline-deployment.sh so that it can be run from a different project

    * Simplify getting service account from cluster.

    * Copy image builder image instead of granting permission

    * Add missed yes command

    * fix stuff

    * Let other usages of image-builder image become configurable

    * let test workflow use image builder image

commit 0369a4c382
Author: Ajay Gopinathan <ajaygopinathan@google.com>
Date:   Fri Aug 16 21:02:07 2019 -0700

    Update manifests to point to 0.26 release. (#1870)

commit 2b246bc356
Author: Alexey Volkov <avolkov@google.com>
Date:   Fri Aug 16 19:54:07 2019 -0700

    SDK - Tests - Improved the "ContainerOp.set_retry" test (#1843)

    Properly testing the feature isntead of just comparing with golden data.

commit d66508d486
Author: Ajay Gopinathan <ajaygopinathan@google.com>
Date:   Fri Aug 16 18:42:48 2019 -0700

    Update changelog for 0.1.26 (#1872)

    * Update changelog for 0.1.26

    * Update changelog.

commit 1d997048ba
Author: Alexey Volkov <avolkov@google.com>
Date:   Fri Aug 16 18:24:09 2019 -0700

    SDK - Fixed string comparisons (#1756)

commit 6d5ffa28dc
Author: Ajay Gopinathan <ajaygopinathan@google.com>
Date:   Fri Aug 16 16:57:33 2019 -0700

    Remove copying of tfx data for cloudbuild release steps. (#1871)

commit de538aa3e3
Author: Ning <ngao@google.com>
Date:   Fri Aug 16 15:53:07 2019 -0700

    add compile step in the samples to generate zip files (#1866)

    * add compile step in the samples to generate zip files

commit 5bcc665928
Author: Ajay Gopinathan <ajaygopinathan@google.com>
Date:   Fri Aug 16 13:58:23 2019 -0700

    Update Python SDK versions for release. (#1860)

commit 0d898cb40f
Author: Ajay Gopinathan <ajaygopinathan@google.com>
Date:   Fri Aug 16 13:16:11 2019 -0700

    Release 0517114dc2 (#1859)

    * Updated component images to version 0517114dc2

    * Updated components to version 48dd338c8a

commit 7dbca1afce
Author: Ning <ngao@google.com>
Date:   Fri Aug 16 12:06:37 2019 -0700

    update gcloud ml-engine to ai-platform (#1863)

commit e849f22a28
Author: Ryan Dawson <ryandawson@cantab.net>
Date:   Fri Aug 16 19:21:23 2019 +0100

    Seldon examples (#1405)

commit a6f3f40e5e
Author: sina chavoshi <sina.chavoshi@gmail.com>
Date:   Fri Aug 16 09:56:09 2019 -0700

    Adding a sample for serving component (#1830)

    * Adding a sample for serving component

    * removed typo / updated based on PR feedback

    * fixing the jupyter rendering issue

    * adding pip3 for tensorflow

    * Fixed spelling error in VERSION

    * fix indentation based on review feedback

commit 54ff3e6614
Author: Alexey Volkov <avolkov@google.com>
Date:   Fri Aug 16 01:22:31 2019 -0700

     SDK - Cleanup - Serialized PipelineParamTuple does not need value or type (#1469)

    * SDK - Refactoring - Serialized PipelineParam does not need type
    Only the types in non-serialized PipelineParams are ever used.

    * SDK - Refactoring - Serialized PipelineParam does not need value
    Default values are only relevant when PipelineParam is used in the pipeline function signature and even in this case compiler captures them explicitly from the pipelineParam objects in the signature.
    There is no other uses for them.

commit d2e94e4e04
Author: Riley Bauer <34456002+rileyjbauer@users.noreply.github.com>
Date:   Thu Aug 15 19:52:34 2019 -0700

    Fix run duration bug (#1827)

    * Allows durations >=24h and renames 'showLink' in RunList

    * Update, fix tests

commit d8eaeaad95
Author: Alexey Volkov <avolkov@google.com>
Date:   Thu Aug 15 17:25:59 2019 -0700

    SDK - Preserving the pipeline input information in the compiled Workflow (#1381)

    * SDK - Preserving the pipeline metadata in the compiled Workflow

    * Stabilizing the DSL compiler tests

commit afe8a694f6
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Thu Aug 15 17:00:28 2019 -0700

    Reduce API usage by utilizing reference name in reference resource API (#1824)

    * Regenerated run api for frontend

    * Added support for reference name to resource reference API in frontend

    * Revert "Regenerated run api for frontend"

    * Addressed PR comments

    * Removed extra if statement by setting default value of parameter

    * Removed the whole comment

    * Addressed PR feedback

    * Addressed PR feedback

    * Simplified logic after offline discussion

commit ea67c998b6
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Thu Aug 15 16:18:35 2019 -0700

    Change how Variables are Provided to Visualizations (#1754)

    * Changed way visualization variables are passed from request to NotebookNode

    Visualization variables are now saved to a json file and loaded by a NotebookNode upon execution.

    * Updated roc_curve visualization to reflect changes made to dependency injection

    * Fixed bug where checking if is_generated is provided to roc_curve visualization would crash visualizaiton

    Also changed ' -> "

    * Changed text_exporter to always sort variables by key for testing

    * Addressed PR suggestions

commit d238bef1fe
Author: Jiaxiao Zheng <jxzheng@google.com>
Date:   Thu Aug 15 12:55:29 2019 -0700

    Add back coveralls. (#1849)

    * Remove redundant import.

    * Simplify sample_test.yaml by using withItem syntax.

    * Simplify sample_test.yaml by using withItem syntax.

    * Change dict to str in withItems.

    * Add back coveralls.

commit 0517114dc2
Author: Riley Bauer <34456002+rileyjbauer@users.noreply.github.com>
Date:   Thu Aug 15 12:28:35 2019 -0700

    Reduce getPipeline calls in RunList (#1852)

    * Skips calling getPipeline in RunList if the pipeline name is in the pipeline_spec

    * Update fixed data to include pipeline names in pipeline specs

    * Remove redundant getRuns call

commit 39e5840f2f
Author: IronPan <yangpa@google.com>
Date:   Thu Aug 15 11:04:34 2019 -0700

    Add retry button in Pipeline UI (#1782)

    * add retry button

    * add retry button

    * add retry button

    * address comments

    * fix tests

    * fix tests

    * update image

    * Update StatusUtils.test.tsx

    * Update RunDetails.test.tsx

    * Update Buttons.ts

    * update test

    * update frontend

    * update

    * update

    * addrerss comments

    * update test

* Fix default image prefix.

* Fix dependencies.

* Fix gs op

* Lint

* Fix papermill exec

* Init for componentTest

* Debugging.

* Try to fix the issue.
Possible cause: https://github.com/nteract/papermill/issues/410

* Fix exit code

* Another fix.

* Squashed commit of the following:

commit 2592307cce
Author: Christian Clauss <cclauss@me.com>
Date:   Wed Aug 21 04:15:19 2019 +0200

    Undefined name 'e' in openvino (#1876)

    Discovered in #1721
    ```
    ./contrib/components/openvino/ovms-deployer/containers/evaluate.py:62:16: F821 undefined name 'e'
            except e:
                   ^
    ./contrib/components/openvino/ovms-deployer/containers/evaluate.py:63:50: F821 undefined name 'e'
                print("Can not read the image file", e)
                                                     ^
    ```
    Your review please @Ark-kun

commit 7b442f4a54
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Tue Aug 20 18:27:19 2019 -0700

    Created extensible code editor based on react-ace (#1855)

    * Created extensible code editor based on react-ace

    * Installed dependencies

    * Updated unit tests for Editor.tsx to test placeholder and value in simplified manner

    * Updated Editor unit tests to use snapshot testing where applicable

commit d11fae78d8
Author: Yuan (Bob) Gong <gongyuan94@gmail.com>
Date:   Wed Aug 21 08:25:20 2019 +0800

    Use KFP lite deployment for presubmit tests (#1808)

    * Refactor presubmit-tests-with-pipeline-deployment.sh so that it can be run from a different project

    * Simplify getting service account from cluster.

    * Migrate presubmit-tests-with-pipeline-deployment.sh to use kfp
    lightweight deployment.

    * Add option to cache built images to make debugging faster.

    * Fix cluster set up

    * Copy image builder image instead of granting permission

    * Add missed yes command

    * fix stuff

    * Let other usages of image-builder image become configurable

    * let test workflow use image builder image

    * Fix permission issue

    * Hide irrelevant error logs

    * Use shared service account key instead

    * Move test manifest to test folder

    * Move build-images.sh to a different script file

    * Update README.md

    * add cluster info dump

    * Use the same cluster resources as kubeflow deployment

    * Remove cluster info dump

    * Add timing to test log

    * cleaned up code

    * fix tests

    * address cr comments

    * Address cr comments

    * Enable image caching to improve retest speed

commit 0864fafbbc
Author: IronPan <yangpa@google.com>
Date:   Tue Aug 20 14:09:19 2019 -0700

    Use single part as default (#1893)

    The data stored in artifact storage are usually small. Using multi-part is not strictly a requirement.
    Change the default to true to better support more platform out of box.

commit 6284dc10c6
Author: Christian Clauss <cclauss@me.com>
Date:   Tue Aug 20 22:01:18 2019 +0200

    IBM Watson samples: from six.moves import xrange (#1877)

    Discovered in #1721

    __xrange()__ was removed in Python 3 in favor of an improved version of __range()__.  This PR ensures equivalent functionality in both Python 2 and Python 3.

    ```
    ./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py💯40: F821 undefined name 'xrange'
                fake_image = [1.0 for _ in xrange(784)]
                                           ^
    ./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py:102:41: F821 undefined name 'xrange'
                return [fake_image for _ in xrange(batch_size)], [
                                            ^
    ./samples/contrib/ibm-samples/watson/source/model-source-code/tf-model/input_data.py:103:37: F821 undefined name 'xrange'
                    fake_label for _ in xrange(batch_size)]
                                        ^
    ```

    @gaoning777 @Ark-kun Your reviews please.

commit 79c7bdabaf
Author: Ning <ngao@google.com>
Date:   Tue Aug 20 09:24:56 2019 -0700

    fix unit tests and address some comments (#1892)

commit 6a7b28f35d
Author: Aakash Bajaj <aakashbajaj5589@gmail.com>
Date:   Tue Aug 20 14:07:33 2019 +0530

    gcp cred bug fix for multiple credentials in single pipeline (#1384)

    * gcp cred bug fix for multiple credentials in single pipeline

    * squash to remove conflict

commit 60fd70c093
Author: Andy Wei <jwwandy@gmail.com>
Date:   Tue Aug 20 15:33:33 2019 +0800

    Let backend apiserver mysql dbname configurable (#1714)

    * let mysql dbname configurable

    * solve conflict

    * move declaration out to fix scope

commit 101a346c0e
Author: olegchorny <oleg.chorny@gmail.com>
Date:   Tue Aug 20 08:33:47 2019 +0300

    'core' folder included to parameters related On-Premise cluster (#1751)

    * 'core' folder included to parameters related On-Premise cluster

    Update is required because this sample was migrated to the samples/core folder

    * Update README.md

commit 4c5d34fe71
Author: dushyanthsc <43390008+dushyanthsc@users.noreply.github.com>
Date:   Mon Aug 19 18:57:33 2019 -0700

    test/project-cleanup: Support to cleanup gke-clusters in test project (#1857)

    Change to add base framework for cleaning up resources in a GCP project.
    The resource specification is specified declaratively using a YAML file.

    As per current requirements this change only adds cleaning up of GKE
    clusters.

* Fix working dir

* Fix exit code format

* Remove kfp notebook sample using tfx:oss components.

* Refactor to reduce potential dup code.

* Clear unused const.

* Fix redundant check

* Add image injection for component test.

* Fix unused import

* Squash and merge from master

* Squashed commit of the following:

commit 41d39c17a2
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Wed Aug 21 19:38:31 2019 -0700

    Add run with json data as input within fixed-data.ts for UI testing and development (#1895)

    * Added run with json data as input

    * Changed run and uid to not be duplicates of hello-world-runtime

commit 851e7c80d2
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Wed Aug 21 19:04:31 2019 -0700

    Replace codemirror editor react component with react-ace editor component (#1890)

    * Replaced CodeMirror with Editor in PipelineDetails.tsx

    * Replaced codemirror in DetailsTable with react-ace

    * Removed codemirror

    * Updated unit tests for Editor.tsx to test placeholder and value in simplified manner

    * Updated DetailsTable.test.tsx.snap to reflect changes made in DetailsTable.tsx

    * Updated PipelineDetails test snapshot

    * Changed width of Editor in DetailsTable to be 100% instead of 300px

    * Revert "Updated unit tests for Editor.tsx to test placeholder and value in simplified manner"

    This reverts commit 40103f2e01.

commit 8c3d6fe121
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Wed Aug 21 18:30:33 2019 -0700

    Add visualization-server service to lightweight deployment (#1844)

    * Add visualization-server service to lightweight deployment

    * Addressed PR suggestions

    * Added field to determine if visualization service is active and fixed unit tests for visualization_server.go

    * Additional small fixes

    * port change from 88888 -> 8888
    * version change from 0.1.15 -> 0.1.26
    * removed visualization-server from base/kustomization.yaml

    * Fixed visualization_server_test.go to reflect new changes

    * Changed implementation to be fail fast

    * Changed host name to be constant provided by environment

    * Added retry and extracted isVisualizationServiceAlive logic to function

    * Fixed deployment.yaml file

    * Fixed serviceURL configuration issuse

    serviceURL is now properly obtained from the environment, the service ip address and port are used rather than service name and namespace

    * Added log message to indicate when visualization service is unreachable

    * Addressed PR comments

    * Removed _HTTP

commit ad307db5b9
Author: Eterna2 <eterna2@hotmail.com>
Date:   Thu Aug 22 08:52:32 2019 +0800

    [Bug Fix] Delete ResourceOp should not have output parameters (#1822)

    * Fix bug where delete resource op should not have success_condition, failure_condition, and output parameters

    * remove unnecessary whitespace

    * compiler test for delete resource ops should retrieve templates from spec instead of root

commit 593f25a5aa
Author: Alexey Volkov <avolkov@google.com>
Date:   Wed Aug 21 17:16:33 2019 -0700

    Collecting coverage when running python tests (#898)

    * Collecting coiverage when running python tests

    * Added coveralls to python unit tests

    * Try removing the PATH modification

    * Specifying coverage run --source

    * Using the installed package

    * Try getting the correct coverage paths

commit 553885ffb1
Author: Alexey Volkov <avolkov@google.com>
Date:   Wed Aug 21 16:38:12 2019 -0700

    SDK - Components - Fixed ModelBase comparison bug (#1874)

commit 2622c674ea
Author: IronPan <yangpa@google.com>
Date:   Wed Aug 21 16:37:53 2019 -0700

    cleanup test dir (#1914)

commit 203307dbaf
Author: Alexey Volkov <avolkov@google.com>
Date:   Wed Aug 21 16:37:21 2019 -0700

    SDK - Lightweight - Fixed custom types in multi-output case (#1875)

    The type was mistakenly serialized as `_ForwardRef('CustomType')`.
    The input parameter types and single-output types were not affected.

commit 2e7f2d4816
Author: IronPan <yangpa@google.com>
Date:   Wed Aug 21 16:36:35 2019 -0700

    Add cloud sql and gcs connection for pipeline-lite deployment (#1910)

    * restructure

    * working example

    * working example

    * move mysql

    * moving minio and mysql out

    * add gcp

    * add files

    * fix test

commit 9adf16301d
Author: Alexey Volkov <avolkov@google.com>
Date:   Wed Aug 21 16:29:54 2019 -0700

    SDK - Airflow - Fixed bug in airflow op creation (#1911)

    This PR fixes a bug in AirFlow op creation.
    The `_run_airflow_op` helper function was not captured along with the `_run_airflow_op_closure` function, because they belong to different modules (`_run_airflow_op_closure` was module-less).
    This was not discovered during the notebook testing of the code since in that environment the `_run_airflow_op` was also module-less as it was defined in a notebook (not in .py file).

commit 7ec56978b6
Author: hongye-sun <43763191+hongye-sun@users.noreply.github.com>
Date:   Wed Aug 21 16:06:31 2019 -0700

    Release 151c5349f1 (#1916)

    * Updated component images to version 151c5349f1

    * Updated components to version a97f1d0ad0

    * Update setup.py

    * Update setup.py

commit 7e062cea77
Author: IronPan <yangpa@google.com>
Date:   Wed Aug 21 15:14:31 2019 -0700

    Update README.md

commit 8e1e823139
Author: Christian Clauss <cclauss@me.com>
Date:   Thu Aug 22 00:04:31 2019 +0200

    Lint Python code for undefined names (#1721)

    * Lint Python code for undefined names

    * Lint Python code for undefined names

    * Exclude tfdv.py to workaround an overzealous pytest

    * Fixup for tfdv.py

    * Fixup for tfdv.py

    * Fixup for tfdv.py

* # Conflicts:
#	sdk/python/tests/components/test_python_op.py

* Fix version for papermill

* Squashed commit of the following:

commit af456bcc61
Author: Eterna2 <eterna2@hotmail.com>
Date:   Fri Aug 23 02:08:33 2019 +0800

    [front-end-server] Allow viewer:tensorboard podTemplateSpec to be customizable (#1906)

    * Allow front-end server to provide custom viewer podTemplateSpec via path/configmap

    * Fix JSON.parse input to string

commit 06b7ad659f
Author: IronPan <yangpa@google.com>
Date:   Thu Aug 22 10:03:28 2019 -0700

    Cleanup pipeline-lite deployment (#1921)

    * restructure

    * working example

    * working example

    * move mysql

    * moving minio and mysql out

    * add gcp

    * add files

    * fix test

    * extract parameters to single place

    * update

    * update readme

    * update readme

    * address pr comment

commit 56160f1c83
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Thu Aug 22 09:26:33 2019 -0700

    Added support for environment specified kernel timeout (#1920)

* Fix google-cloud-storage and google fire version info.

* Delete run_test.sh

* Merge from kfp master

* Squashed commit of the following:

commit 7b6825d7d5
Author: hongye-sun <43763191+hongye-sun@users.noreply.github.com>
Date:   Thu Aug 22 18:23:18 2019 -0700

    Update changelog for 0.1.27 (#1935)

    * Update changelog for 0.1.27

    * Revert unexpected changes.

commit 336760cfd8
Author: IronPan <yangpa@google.com>
Date:   Thu Aug 22 17:47:18 2019 -0700

    sync namespaced install file (#1932)

commit 55d62fe9fd
Author: Hamed <hamedhsn@gmail.com>
Date:   Fri Aug 23 01:09:18 2019 +0100

    Support Affinity for ContainerOps (#1886)

commit d15697bdb1
Author: Animesh Singh <singhan@us.ibm.com>
Date:   Thu Aug 22 16:29:17 2019 -0700

    Initial kfserving pipeline component (#1838)

    * initial kfserving pipeline component

    * changing some descriptions

    * adding action as an input, and pinning versions

    * moving Dockerfile in correct location

    * adding namespace to delete

commit 119405deaf
Author: hongye-sun <43763191+hongye-sun@users.noreply.github.com>
Date:   Thu Aug 22 15:49:29 2019 -0700

    upgrade backend image versions (#1918)

commit c01315a89d
Author: Alexey Volkov <avolkov@google.com>
Date:   Thu Aug 22 15:31:24 2019 -0700

    SDK - Refactoring - Replaced the TypeMeta class (#1930)

    * SDK - Refactoring - Replaced the TypeMeta class
    The PipelineParam no longer exposes the private TypeMeta class
    Fixes #1420

    The refactoring PR is part of a series of PR which unifies the metadata and specification types.

commit 736c7a5750
Author: IronPan <yangpa@google.com>
Date:   Thu Aug 22 15:29:19 2019 -0700

    clean up owner file (#1928)

commit 5abba6695b
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Thu Aug 22 14:45:18 2019 -0700

    Add pipeline id to pipeline summary card (#1927)

    * Added pipeline id to pipeline summary card

    * Updated PipelineDetails.test.tsx.snap

commit 3c8952e6bf
Author: Kirin Patel <kirinpatel@gmail.com>
Date:   Thu Aug 22 13:57:20 2019 -0700

    Add TFDV, TFMA, and Table visualization support for Python based visualizations (#1898)

    * Added table and tfdv visualization

    Also fixed issue surrounding ApiVisualizationType enum

    * Fixed table visualization

    * Removed byte limit
    * Fixed issue where headers would not properly be applied
    * Fixed issue where table would not be intractable

    * Updated table visualizaiton to reflect changes made to dependency injection

    * Fixed bug where checking if headers is provided to table visualizations could crash visualization

    * Added TFMA visualization

    * Updated new visualizations to match syntax of #1878

    * Updated test snapshots to account for TFMA visualization

    * Small if statement synax changes

    * Add flake8 noqa comments to table.py and tfma.py

commit db6c9b4fd0
Author: dushyanthsc <43390008+dushyanthsc@users.noreply.github.com>
Date:   Thu Aug 22 13:11:18 2019 -0700

    pipeline-lite: Introduce metadata component to pipeline-lite (#1840)

    This change introduces the metadata component to pipeline-lite
    installation. This installation:

      1. Does not include metadata-ui
      2. mysql installation is used instead of metadata-db
      3. Replica count has been reduced 1 instead of 3

* Fix check_notebook_results args.

* Fix.

* Fix work dir path

* Fix global const naming convention

* Remove redundant check.

* Extract project name from result gcs dir.

* Add error catching

* Name change.

* Improve error catching.

* Improve error catching.

* naming correction.
2019-08-23 15:44:24 -07:00
Yuan (Bob) Gong 8e53eb43ad Move postsubmit tests to lite deployment (#1939)
* Move postsubmit tests to lite deployment

* Reduce verbose logs by wget

* Add ignored files

* add test temporary file to gitignore
2019-08-23 14:34:26 -07:00
Yuan (Bob) Gong fe8d96ffb5 Use cloud build to build images instead (#1923)
* Use cloud build to build images instead

* Batch 3 image buld tasks

* Fix check cloud build status script
2019-08-23 01:43:18 -07:00
IronPan 736c7a5750 clean up owner file (#1928) 2019-08-22 15:29:19 -07:00
IronPan 2622c674ea
cleanup test dir (#1914) 2019-08-21 16:37:53 -07:00
IronPan 2e7f2d4816
Add cloud sql and gcs connection for pipeline-lite deployment (#1910)
* restructure

* working example

* working example

* move mysql

* moving minio and mysql out

* add gcp

* add files

* fix test
2019-08-21 16:36:35 -07:00
Ning 94aa0788d5 remove tfx notebook sample in favor of the TFX OSS sample (#1908) 2019-08-21 10:22:31 -07:00