Commit Graph

93 Commits

Author SHA1 Message Date
jingzhang36 6e56960e6a
Refactor the legacy way of using pipeline id to create run in KFP backend (#3437)
* For legacy interface, we switch to the new presentation underhood

* when create run, if user specify a pipeline, we subsitute it with the pipeline's default version

* Add a case where a version and a pipeline are both specified

* comment; get ready pipeline

* comments

* fix upgrade integration test

* comments of todo; expected run/job now has resource references

* fix upgrade test expected value according to the new response

* fix a typo

* a quick hack for upgrade test

* surface err from conversion
2020-04-08 00:08:49 +08:00
jingzhang36 d9bb1be25b
Add pipeline version to job/run integration test so that job/run is c… (#3270)
* Add pipeline version to job/run integration test so that job/run is created from pipeline version

* Use set comparison to check array equation in test

* address comments to use elements match
2020-03-16 16:45:40 +08:00
Yuan (Bob) Gong 5391e88fbc
[Testing] KFP standalone test infra for upgradability (#1971)
* Implement upgrade test

* mark upgrade-tests.sh as executable

* Fix comments

* Base upgrade_test_setup.yaml

* e2e integration of upgrade test

* Fix entrypoint argument

* Fix e2e workflow yaml

* Fix run_test.sh argument processing

* Fix no closing backtick

* Restrucutre upgrade_test.go to focus the test on upgrade verification

* clean up code

* Clean up after upgrade test when it is run in integration tests.

* Include pipeline tests in upgrade test

* Reorder tests

* Add upgrade test coverage for run api resources

* Add job api resource coverage in upgrade test & refactored upgrade test

* Fix add missing step in upgrade test

* Fix BUILD.bazel

* Fix upgrade_test.go

* Try to fix upgrade test failure

* Fix hard coded namespace

* Sync upgrade-tests.sh with new changes in presubmit-tests-with-pipeline-deployment.sh

* Update upgrade test

* Remove redundant code

* Fix integration test exit code

* Fix trigger interval second mismatch
2020-03-09 16:53:37 -07:00
jingzhang36 ee587101d3
Add integration tests for version related methods (#3174)
* create a new version upload integration test file

* checkpoint

* checkpoint

* test

* checkpoint

* checkpoint

* version api integration test

* Fix a typo

* typo in comment

* error message
2020-02-26 21:29:52 +08:00
Yuan (Bob) Gong 8c0bc219ed
[Backend] swf catchup option integration tests (#3139)
* Make local testing easier

* Move cleanup to test setup stage

* Add readme for how to run integration tests

* Add warning about data loss

* Add warning also in the script

* Change flag to isDevMode and cleanup resources if not in dev mode

* Pass through arguments in the bash script

* Fix unit tests

* integration tests for swf NoCatchup option

* Also add cron catchup option tests

* Restructure test

* Adjust format

* Fix BUILD.bazel

* fix typo
2020-02-21 02:28:31 -08:00
Yuan (Bob) Gong b90040a00f
[Backend] Refactor integration tests, facilitate local testing (#3138)
* Make local testing easier

* Move cleanup to test setup stage

* Add readme for how to run integration tests

* Add warning about data loss

* Add warning also in the script

* Change flag to isDevMode and cleanup resources if not in dev mode

* Pass through arguments in the bash script

* Fix unit tests
2020-02-20 23:02:32 -08:00
jingzhang36 d1b72df5fb
Upload local file as new pipeline version Step 1 (#3001)
* add upload pipeline version to upload_pipeline_server and http main

* add apiPipelineVersion to pipeline upload swagger json

* add apiResourceReference to pipeline upload swagger json

* Add yet more types to pipeline upload swagger json

* Unit tests
2020-02-07 16:13:36 +08:00
jingzhang36 af090bffa2 Regenerate api since https://github.com/kubeflow/pipelines/pull/2445 changed api proto (#2506)
* re-generate api since #2445 changed api proto

* Fix integration test for pipeline client

* use generated constants
2019-10-29 23:54:52 -07:00
Kirin Patel 41b394b045 Add e2e visualization tests (#1981)
* Created visualization_api_test.go

* Updated BUILD.bazel files

* Removed clean_up from e2e test

* Revert "Removed clean_up from e2e test"

This reverts commit 82fd4f5a00.

* Update e2e tests to build visualizationserver and viewer-crd

* Fix bug where wrong image is set

* Fixed incorrect image names

* Fixed additional instance of incorrect image names
2019-08-30 13:54:10 -07:00
IronPan f1c9594f7d Propagate pipeline name in pipeline spec (#1842)
* update

* add pipeline name

* propagate name

* propagate name

* fix

* update test
2019-08-14 23:30:32 -07:00
IronPan 4e43750c9d
add reference name to resource reference API proto (#1781)
* add resource reference

* fix tests

* fix tests

* fix e2e test

* fix e2e

* fix test

* update api requirement

* fix tests

* Update job_api_test.go

* Update run_api_test.go

* Update setup.py

* Update deploy-kubeflow.sh

* fix tests

* Update deploy-kubeflow.sh
2019-08-13 03:15:11 -07:00
Yuan (Bob) Gong db7f978afd Disable flaky tests temporarily (#1809)
Temporarily addresses https://github.com/kubeflow/pipelines/issues/1762
2019-08-12 09:38:14 -07:00
IronPan 3c9b23216c
fix tests (#1769) 2019-08-08 11:37:58 -07:00
IronPan 44712b3c4a
Delete go CLI (#1592)
* delete go cli

* Delete cli_test.go

* Update BUILD.bazel

* Update .travis.yml

* Update BUILD.bazel
2019-07-19 01:11:25 -07:00
Ajay Gopinathan 5f1b41171f Fix API package names and regenerate checked-in proto files. (#1404)
* Fix API package names and regenerate checked-in proto files. Also bump version of GRPC gateway used.

* Fix BUILD.bazel file for api as well.

* Update Bazel version
2019-05-30 11:26:28 -07:00
Riley Bauer fbae0f855e Creates a default experiment at API server set up time (#1089)
* WIP - Create default experiment upon API server initialization

* Default experiment initialization caused crashes if API server pod was restarted without clearing DB

* Adding new table to store default experiment ID

* Add default experiment type model definition

* Minor fixes, everything seems to work now

* Clean up. Renamed to default_experiment_store

* Adds tests for the default_experiment_store

* Add integration test for verifying initial cluster state. Currently only covers existence of default experiment

* Don't run initialization tests except as integration tests

* Fixes comments

* PR comments and cleanup

* Extract code in resource_manager to helper func
2019-04-15 15:55:04 -07:00
IronPan 086d4763d9 Surface workflow finished time in list run API (#1122)
* add finished time for list runs

* add finished time for list runs

* fix tests

* add finished time for list runs

* Update run.proto

* address comments

* make query more robust

* fix e2e test

* fix e2e test
2019-04-10 23:02:10 -07:00
Alexey Volkov 2c2445df83 Enable pipeline packages with multiple files (#939)
* Enable pipeline packages with multiple files

* Added tests

* Initialize the variables to nil

* Trying to read the archive file entry immediately

* Fixed the pipeline packages used by the `TestPipelineAPI` test.
Also added a failing test case. Will disable it in next commit.

* Disabling the test for the UploadFile bug I've discovered

* Fixed the pipeline name.

* Removed the disabled extra test.

* Addressed the feedback.

* Removed the "header == nil" check (feedback).

* Fixed typo

* Addressed the PR feedback
Added space before comment.
Checking for the error again.
2019-03-28 15:09:58 -07:00
Alexey Volkov f98ec68488 Added the ability to terminate a run (#528)
* Added the terminate run command to backend and CLI.
No generated files for now.

* Added the generated files.

* Moved the code to run_store.go
Now the call chain is run_client->run_server->resource_manager->run_store

* Using the backoff package for retries.

* Trying to update run status in the DB to "Terminating" before patching the workflow.

* Stopped using the Argo errors module.

* Fixed the compilation errors due to recent backend changes.

* RILEY - WIP - Implementation of workflow_fake.go and first test

Added successful test in resource_manager_test.go and completed, barring nits and conventions, the implementation of Patch() and isTerimated() within workflow_fake.go

Additional tests and lots of clean-up are still necessary

* Adds a few more tests to resource_manager_test and cleans up a bit

* Further clean up. Stopped using squirrel for UPDATE query. Added run_store_tests

* Adds terminate run integration test

* Undo changes to go.sum

* Fixes path to long-running.yaml in integration test

* Allow runs with no Conditions to be terminated
2019-03-21 18:00:18 -07:00
IronPan f8c1dde3a8 move integration test to sub dir (#888)
* move integration test to sub dir

* revert
2019-03-01 15:32:56 -08:00
IronPan 1bde25552c support .zip pipeline package (#874)
* support zip

* update integration test

* debug

* comments

* fix

* Update .cloudbuild.yaml

* Update .release.cloudbuild.yaml

* Update .travis.yml
2019-02-28 18:50:36 -08:00
Yasser Elsayed ec4d7e8bd3 Return resource count from ListXXX calls (#595)
* add count to protos and libs

* close db rows before second query

* count -> total_size

* int32 -> int

* move scan count row to util

* add comments

* add logs when transactions fail

* dedup from and where clauses

* simplify job count query

* job count queries

* run count queries

* add job_store total size test

* added tests for list util

* pr comments

* list_utils -> list

* fix clients and fake clients to support TotalSize

* added TotalSize checks in api integration tests
2019-01-31 11:15:54 -08:00
Ajay Gopinathan b97969f882 Fix retrying logic which was causing persistenceagent to crash loop. (#633) 2019-01-05 12:00:27 -08:00
Ajay Gopinathan 163545b370 Use Bazel to build the entire backend and perform API code generation (#609)
* Use Bazel to build the entire backend.

This also uses Bazel to generate code from the API definition in the
proto files.

The Makefile is replaced with a script that uses Bazel to first generate
the code, and then copy them back into the source tree.

Most of the BUILD files were generated automatically using Gazelle.

* Fix indentation in generate_api.sh

* Clean up WORKSPACE

* Add README for building/testing backend.

Also fix the missing licenses in the generated proto files.

* Add license to files under go_http_client
2019-01-04 17:17:20 -08:00
Yasser Elsayed 549a366c39 Support archiving/unarchiving runs on the backend (#552)
* skip integration tests when unit test flag is set to true

* wip

* add StorageState enum to proto

* add StorageState to model

* archive proto/model changes

* wip archive endpoint

* wip adding tests

* archive test

* unarchive proto and implementation

* cleanup

* make storage state required, with a default value

* remove unspecified value from storage state enum

* pr comments

* pr comments

* fix archive/unarchive endpoints, add api integration test

* typo
2018-12-19 14:01:06 -08:00
Yasser Elsayed ba261f368e Skip backend integration tests when cli flag isn't passed (#527)
* skip integration tests when unit test flag is set to true

* use cli arg to run integration tests

* use runIntegrationTests flag instead
2018-12-17 10:47:37 -08:00
IronPan ddab2fdd36 update test to specify name when create pipeline (#543) 2018-12-14 19:25:12 -08:00
Ajay Gopinathan 0c0d8a2b91 Add filtering ability for all backend API ListXXX requests (#537)
* WIP: Add filter package with tests.

* Add tests for IN predicate.

* Add listing functions

* Try updating list experiments

* Cleanup and finalize list API.

Add tests for list package, and let ExperimentStore use this new API.
Update tests for the latter as well.

* Add comments. BuildSQL -> AddToSelect for flexibility

* Run dep ensure

* Add filter proto to all other resources

* Add filtering for pipeline server

* Add filtering for job server

* Add filtering for run server

* Try to fix integration tests
2018-12-14 00:18:31 -08:00
IronPan 9eee2bbb47 move name (#498) 2018-12-07 16:41:15 -08:00
IronPan 9b77d4a8a6 Switching test to kubeflow deployment (#351)
* test

* fix

* fix

* fix

* fix

* fix

* update

* cleanup

* fix

* coopy test

* chmod

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* update

* fix

* fix

* fix

* fix

* fix

* fix

* fix sample test

* fix

* fix

* merge

* update image builder image

* update script

* mount permission
2018-11-28 21:36:12 -08:00
Yang Pan b494e3daa0 Add integration tests for API servers (#112)
* add pipeline e2e test back

* delete samples

* more changes to pipeline test

* add delete run/experiment

* update test name

* stage

* add api

* fii

* add tensorboard routing rule (#143)

* add tensorboard routing rule

* rename tb routing rule

* address comments

* remove debugger

* fix url prefix

* more fixes

* update tests

* revert debugging

* update

* update test

* fix

* update test

* fix

* fix pipeline tests

* exp

* update run

* update jobs
2018-11-09 20:39:15 -08:00
Pascal Vicaire a44b4dd81a Saving progress. 2018-11-08 18:31:14 -08:00
Pascal Vicaire 26e96e2836 Saving progress. 2018-11-08 18:14:46 -08:00
Pascal Vicaire c03add4f28 Fixing imported json library in the CLI integration tests. 2018-11-07 14:12:26 -08:00
Pascal Vicaire b04883fbd5 Addressing code review comments. 2018-11-07 12:22:23 -08:00
Pascal Vicaire 7a18430ec5 Fix typo 2018-11-06 22:10:24 -08:00
Pascal Vicaire 425403ae83 Adding integrations test for the CLI commands related to pipelines. 2018-11-06 22:03:33 -08:00
Pascal Vicaire 81cd221f5d Changing the namespace to Kubeflow. 2018-11-06 16:05:35 -08:00
Pascal Vicaire 2331ad8079 Adding the --debug flag to help with debugging. 2018-11-06 15:13:21 -08:00
Pascal Vicaire aeb32c8d42 Fixing an issue with an undefined variable 2018-11-06 12:34:03 -08:00
Pascal Vicaire 204ba0d57b First integration test for the ML Pipeline CLI (Pipeline List). 2018-11-05 19:16:41 -08:00
Pascal Vicaire 1bab424abf Fixing the GO import paths to reference the kubeflow/pipelines repository. 2018-11-02 14:53:42 -07:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00