Commit Graph

185 Commits

Author SHA1 Message Date
Ning fa3ebefe09 add sigint sigterm (#863) 2019-02-26 17:35:49 -08:00
Ning 508210d40b
Add postsubmit component test (#613)
* add postsubmit script and yaml
* remove old sample tests component file
* extract deploy-pipeline.sh, deploy-kubeflow.sh and test-prep.sh from presubmit and postsubmit scripts
2019-02-25 13:29:04 -08:00
Ning a6763b9599 component build support for both python2 and python3 (#730)
* component build support for both python2 and python3

* add sample test

* remove the annotations for python2 component build

* add pathlib for python2 component build

* fix component build unit test

* fix bug in the dockerfile generator

* remove exist_ok in path.mkdir to make python2 compatible

* adjust unit test

* remove pathlib dependency for python2 component build

* remove the pathlib codes in python3 component build, but use python2 code instead; add a todo to create a new sample
2019-02-25 12:56:19 -08:00
hongye-sun ad370933c7 Move e2e tests to us-east1 (#847)
* move to us-east1

* switch to us-east1-b
2019-02-22 10:51:42 -08:00
hongye-sun 749d0aab9f Update swagger codegen version (#839) 2019-02-21 12:21:38 -08:00
Riley Bauer 600acbdd23
Updates lodash to version 4.17.11 (#803)
* Pin lodash version

* Commit updated package-locks
2019-02-13 10:40:21 -08:00
IronPan cc257f29a2 switch test to us-west1 (#808)
* switch test to east1

* Update presubmit-tests-with-pipeline-deployment.sh
2019-02-11 16:10:00 -08:00
IronPan e9bd7c6a4d
merge build image to test suit (#799)
* merge build image to test suit

* update image

* Update presubmit-tests-with-pipeline-deployment.sh

* add permission to access to gcr

* add service account

* test

* fix

* not exit

* speed boost
2019-02-09 00:23:06 -08:00
hongye-sun 969bb4ed2c Revert "Add gpu pool to test deployment and enable gpu in sample test (#696)" (#778)
This reverts commit 72a7de9d47.
2019-02-04 21:59:19 -08:00
IronPan 6ad2601ec7 Remove pipeline bootstrapper (#739)
* remove bootstrapper

* more cleanups

* more cleanups
2019-01-31 17:52:03 -08:00
Yasser Elsayed be19cbc259 Refactor UI buttons to lib file (#737)
* refactor buttons to lib file

* Add license header

* fix e2e test
2019-01-28 11:27:45 -08:00
hongye-sun 72a7de9d47 Add gpu pool to test deployment and enable gpu in sample test (#696)
* add gpu pool to test deployment and enable gpu in sample test

* enable clean up
2019-01-25 09:44:08 -08:00
IronPan 4c551bac60 bump ks version (#693) 2019-01-23 15:08:25 -08:00
qimingj 4a043c1823 Add CMLE sample test script. (#724)
The test is not added to the list to run automatically yet since it takes about 25 min.
2019-01-22 20:44:34 -08:00
Riley Bauer d9665549ce Use "create" rather than "start" except when initiating a run (#650)
* Uses 'Create' for all actions that lead to creation flow, or result in a static object (experiment). 'Start' is used solely for initiating runs

* Update integration test
2019-01-09 11:56:45 -08:00
Ning f86fb2a677 output argo log in case of exception throw (#635) 2019-01-07 10:43:30 -08:00
Ning ea72316ac4 fix deploy model name conflict in case of concurrent notebook sample test (#636)
* fix deploy model name conflict in case of concurrent notebook sample test

* minor fix
2019-01-05 09:15:18 -08:00
qimingj 410f9b979f Update sample notebook to clean up deployed models. (#622)
* Update sample notebook to clean up deployed models.

Update SDK client to return correct links in local Jupyter with user's own proxy connection.

* Fix sample tests.
2019-01-04 13:07:30 -08:00
Ning 5abc1a4f59 Add sample test without image build (#578)
* add another sample test to test the current sample codes instead of using newly built component images

* rename sample test yamls

* use the v2 name

* bash bug

* tf-training bug fix

* output argo log in case of exceptions for tf-training sample

* disable gpu
2019-01-03 15:17:51 -08:00
Yasser Elsayed 3ef16462b6 Move backend unit tests to Travis (#589)
* try go version in travis

* add back old travis tests with backend tests

* remove backend unit tests prow config

* remove unit_test_gke

* test backend/src directory

* update comment to call out unit test
2019-01-02 15:00:09 -08:00
Alexey Volkov d135be9c87 Deployment - Passing the platform parameter (#588) 2019-01-02 11:40:08 -08:00
IronPan e7af13263a switch from go dep to go module (#581)
* add vendor to gitignore

* switch to go module

* switch to go module

* switch to go module

* prune go mod

* prune go mod

* turn on go mod for test

* enable go module in docker image

* enable go module in docker image

* fix images

* debug

* debug

* debug

* update image
2018-12-21 16:28:21 -08:00
Alexey Volkov 169b92e87c Tests - Updated image-builder Makefile (#500)
Nothing should change
2018-12-21 00:13:12 -08:00
Ning 2118baf752 Fix sample test failure (#548)
* enable debug mode

* add gcp secret to tfdv; add tensorflow installation in lightweight sample test

* disable debug mode

* update the args in check_notebook_results

* change tfdv output schema

* typo fix

* update roc image in tfx sample test

* update confusionmatrix image in the tfx sample test
2018-12-19 08:55:29 -08:00
Ning f1fb2df73b Restructure dataproc components (#542)
* restructure the dataproc component folder

* more changes in cloudbuild and sample test and readme

* remove src in cloudbuild
2018-12-17 16:57:58 -08:00
Yasser Elsayed ba261f368e Skip backend integration tests when cli flag isn't passed (#527)
* skip integration tests when unit test flag is set to true

* use cli arg to run integration tests

* use runIntegrationTests flag instead
2018-12-17 10:47:37 -08:00
Ning 8e58ca9e1f add lightweight sample test (#499) 2018-12-13 14:59:06 -08:00
qimingj 6b315d4693 Change "namespace" param to "host" in kfp client so we can use Jupyter outside cluster. (#529)
* Change "namespace" param to "host" in kfp client so we can use Jupyter outside cluster.

* Fix tests.
2018-12-12 23:36:18 -08:00
Ning 886e113f78 Improve get_experiment and list_runs in the python sdk (#508)
* add get_experiment_id and list_runs_by_experiment

* offer only one get_experiment function

* return experiment body instead of id

* simply codes

* simply code 2

* remove experiment_id check in the while loop

* minor bug
2018-12-11 22:02:24 -08:00
Riley Bauer 114c99d808 Add experiment selector to NewRun (#486)
* Adds an experiment selector to the new run page. Needs tests

* Adds an experiment selector to the new run page. Needs tests

* Adds tests for the new experiment selector in NewRun

* Rename PipelineSelector -> ResourceSelector since it handles experiments as well

* Makes ResourceSelector more abstract. No longer coupled to experiments and pipelines

* PR comments, NewRun clean-up

* Moves resourceToRow function into ResourceSelector

* Fix e2e test
2018-12-07 13:50:01 -08:00
nealgao ad1950b8e6 Add notebook sample test: tfx sample (#470)
* add notebook sample tests for tfx

* parameterize component image tag

* parameterize base and target image tags

* install tensorflow package for the notebook tfx sample test

* bug fixes

* start debug mode

* fix bugs

* add namespace arg to check_notebook_results, copy test results to gcs, fix minor bugs
add CMLE model deletion

* install the correct KFP version in the notebook; parameterize deployer model name and version

* fix CMLE model name bug

* add notebook sample test in v2

* add gcp sa in notebook tfx sample and shutdown debug mode

* import kfp.gcp
2018-12-07 10:49:30 -08:00
IronPan a9d0689965 clean up test - remove unused workflow (#483)
* remove

* remove

* remove
2018-12-06 16:37:41 -08:00
Yasser Elsayed 5c9010cdf8
Refresh and wait for run to start (#472) 2018-12-04 19:14:12 -08:00
Yasser Elsayed 401f738ec2
Wait until logs appear right (#459) 2018-12-04 15:40:08 -08:00
Alexey Volkov 6be0399938 Tests - Try fix GCS file being inaccessible right after upload (#433)
https://github.com/kubeflow/pipelines/issues/403
2018-12-03 15:52:36 -08:00
nealgao 548a2d6fe5 Failed the sample tests when it should (#417)
* failed the sample tests when it should

* print log before failure
2018-11-29 23:53:33 -08:00
IronPan 9a62c319c6 Clean up the resource after test finish (#410) 2018-11-29 13:03:30 -08:00
IronPan 9b77d4a8a6 Switching test to kubeflow deployment (#351)
* test

* fix

* fix

* fix

* fix

* fix

* update

* cleanup

* fix

* coopy test

* chmod

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* update

* fix

* fix

* fix

* fix

* fix

* fix

* fix sample test

* fix

* fix

* merge

* update image builder image

* update script

* mount permission
2018-11-28 21:36:12 -08:00
Yasser Elsayed a2e533da49 increase timeout waiting for test run to start (#392) 2018-11-27 16:58:10 -08:00
Alexey Volkov 81f2df62b3 Tests - Improve Minikube source code copy process (#367) 2018-11-27 10:56:49 -08:00
Alexey Volkov bcbb2c51b9 Tests - Fixed many test issues (external PRs, code de-sync) by getting rid of git clone (#315)
* Tests - Getting rid of git clone in */run_test.sh
run_test.sh scripts no longer pull the repo code, because the code is now correctly baked in during the image build. This saves ~11 pulls per commit
Backend unit test image is now build as part of the test suite

* Added target-image-prefix parameter to simplify test configuration

* Build all images from source code prepared by Prow. Got rid of git pulls
All images are now built from archived version of code the source code prepared by Prow.
This saves 25 more pulls and improves test reliability.
The archived source code location is passed through image-build-context-gcs-uri parameter.

* Addressed the PR feedback.
2018-11-26 13:56:49 -08:00
nealgao c9d8ded51e restructure local components directory (#357)
* restructure local components directory

* typo fix
2018-11-26 11:29:18 -08:00
Alexey Volkov b298fdc9ce Tests - Stop repeating the logs three times (#366) 2018-11-22 01:21:34 -08:00
nealgao da6ff9cdd5 restructure dataflow component structure (#338)
* restructure dataflow components

* add . directory before the script name
2018-11-21 10:55:15 -08:00
Yang Pan b73a0bf952
Update presubmit-tests.sh 2018-11-20 22:39:55 -08:00
IronPan 2c2f839246 Merge branch 'master' of https://github.com/kubeflow/pipelines 2018-11-20 22:36:14 -08:00
Alexey Volkov f6a8719e69 Tests - Fixed the argo submit failure not failing the script (#324)
Removed pipe from argo submit.
Only the last pipe command affects the exit pipeline code unless `set -o pipefail` is used.
2018-11-20 20:53:58 -08:00
Yang Pan fb1bff0705 rename (#340) 2018-11-20 17:33:07 -08:00
IronPan 2938c8d0f1 Merge branch 'master' of https://github.com/kubeflow/pipelines 2018-11-19 22:39:01 -08:00
nealgao 2c6a123595 reenable basic sample tests, serially after frontend integ tests(triggering 5 times, successful always) (#298)
* reenable basic sample tests, serially after frontend integ tests

* fix namespace bug
2018-11-19 13:18:29 -08:00
nealgao 0fad57e023 add ngao to test owner (#299) 2018-11-19 10:54:25 -08:00
nealgao b39435765d configurable namespaces in sample test and python SDK (#306)
* configurable namespaces in sample test and python SDK

* update run_test.sh

* fix a typo
2018-11-17 00:31:05 -08:00
Alexey Volkov e6b4107056 Components - Reorganized components/kubeflow (#232)
* kubeflow/deployer

* kubeflow/dnntrainer

* kubeflow/launcher
2018-11-16 12:30:41 -08:00
Yang Pan 98014b65cd Revert sample test to mitigate test flakiness (#277)
* [WIP] change deployment platform to gcp

* debug

* revert test

* Update presubmit-tests.sh

* Update deploy.sh
2018-11-15 12:30:45 -08:00
IronPan 38b6cdc4ac revert test 2018-11-14 20:38:05 -08:00
IronPan f33cb44b89 Merge branch 'master' of https://github.com/kubeflow/pipelines 2018-11-14 20:36:22 -08:00
Riley Bauer 4550513ed0 Adds NewRun tests (#242)
* Adds tests for the NewRun page

* Add more tests

* Adds many more tests, cleans-up PipelineSelector a bit, and fixes bug in Utils.ts

* Update create run button ID in e2e test

* Test clean-up and PR comments
2018-11-14 16:31:21 -08:00
nealgao 790cb36f73 Move basic sample tests to e2e tests (#235)
* move basic sample tests to e2e tests

* remove pipeline deletion to avoid conflicts with concurrent basic e2e tests.
2018-11-14 13:51:54 -08:00
Alexey Volkov 1eb0adf616 Tests - Simplified test/sample-test/run_tests.sh a bit (#230) 2018-11-14 11:39:50 -08:00
IronPan d7d35bf79b debug 2018-11-13 17:45:23 -08:00
Alexey Volkov a104f1ac69 Tests/Minikube - Fix Google credentials (#194)
* Tests/Minikube - Downgraded Docker to 18.06.1

* Passing the GOOGLE_APPLICATION_CREDENTIALS env variable

* Copying the credentials file

* Using the home directory for the credentials file.
2018-11-12 13:26:35 -08:00
Alexey Volkov f0c189f3a5 Marked scripts as executable (#188)
chmod +x scripts
2018-11-09 19:50:30 -08:00
Alexey Volkov d68ea51eb9 Tests - Upgrade Minikube and Kubernetes for new Docker release (#183)
* Tests - Upgrade minikube due for new Docker release

* Upgrading Kubernetes
2018-11-09 16:32:30 -08:00
Alexey Volkov b190e729d2 Tests - Testing master+branch instead of just branch (#165)
* Tests - Testing master+branch instead of just branch

* Updated the image-builder image

* Fixed "fatal: unable to auto-detect email address"

* Updated the image builder images

* Updated the backend-unit-test image
2018-11-09 14:22:41 -08:00
nealgao e61735c3af sample tests using the dataset in the corresponding test project (#152) 2018-11-08 19:28:44 -08:00
Yang Pan a855ffb68f activate public prow service account (#153)
* activate service account

* Update presubmit-tests-gce-minikube.sh

* Update presubmit-tests.gke.sh

* Update presubmit-tests.sh

* update git clone

* update image

* update image

* Removed the SSH key copying

* update git clone

* update image

* update
2018-11-08 16:00:30 -08:00
nealgao 81341d3aa6 sample test fix: software version error (#146)
* typo

* start debug mode

* package version failure fix

* disable debug mode
2018-11-07 18:10:42 -08:00
nealgao a540cf77c6 Fix flaky basic sample test (#134)
* fix typo

* typo fix

* start debug mode

* fix typos

* make the timeout configurable

* disable debug mode

* update variable name

* update timeout to timeout_seconds
2018-11-07 14:50:14 -08:00
nealgao 8427e30c7f Add basic sample tests (#79)
* add sequential sample test

* add condition basic sample

* reuse script

* add all the other basic tests

* update sample test dockerfile to add run_basic_test file

* write test output

* typo bug
2018-11-06 22:26:28 -08:00
nealgao 3ce66aa226 add xgboost: migrate from the old repo (#46)
* migrate from the old repo

* fix bug: accidentally override tfma test

* add tfma test back

* add tfma back

* typo fix

* fix small typo

* if job fails, exit after logs are output
2018-11-06 17:15:49 -08:00
Alexey Volkov c7bb1ff1d2 Add Ning and Alexey to OWNERS for components, samples and sample-test (#102) 2018-11-06 14:59:05 -08:00
Alexey Volkov a0e55deea4 Fixed the Minikube tests after moving to the new repo (#98) 2018-11-06 13:40:07 -08:00
nealgao 077a060024 debug tfma failure (#91)
* debug tfma failure

* tft version bug

* minor fix

* comment the test validation
2018-11-06 12:52:56 -08:00
nealgao 3b602a57b0 Fix tfx name bug in the tfma sample test (#67)
* fix tfx name bug

* update release build for the data publish
2018-11-05 18:06:35 -08:00
Yang Pan 28b860db32
Update presubmit-tests.sh
The change is merged by accident. Revert the change to avoid resource leak.
2018-11-05 14:39:11 -08:00
qimingj 414a5afa13
Merge pull request #27 from kubeflow/ngao/fix-tfma-deploy-bug
TFMA deployer bug fix
2018-11-05 14:28:54 -08:00
Pascal Vicaire 3df28a9700 Updating OWNERS files. Adding per-subdirectory OWNER files. 2018-11-05 14:03:33 -08:00
Ning Gao 87f8840294 fix tfx name change 2018-11-05 13:23:12 -08:00
Ning Gao 9886ab10c0 fix tfx name change 2018-11-05 13:20:04 -08:00
Ning Gao 9dcedf094f fix name bug 2018-11-05 13:11:24 -08:00
Ning Gao 59172542d0 debug mode 2018-11-05 09:46:44 -08:00
Pascal Vicaire 34a030b199 Updating the backend unit test container. 2018-11-02 16:14:12 -07:00
Pascal Vicaire 7d3a089676 Updating the container image used to build the images needed by the e2e tests. 2018-11-02 15:34:19 -07:00
Pascal Vicaire d18f37785d Updating references to the project repository to kubeflow/pipelines. 2018-11-02 15:06:54 -07:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00