Commit Graph

24 Commits

Author SHA1 Message Date
Yuan (Bob) Gong d3ef4bf39f
[Backend] Use service account passed from api object (#3650)
* Add service account field to run and job api objects

* Update description

* Fix field casing

* Use service account from api object

* Fix bug and add unit test

* Save patched workflow spec into db instead

* Save service account to db model

* Fix unit tests

* Fix integration tests

* Fix upgrade test

* Update upgrade_test.go
2020-05-05 23:23:08 -07:00
Chen Sun f55147d9c0
Sort job resource references before equality check (#3561) 2020-04-29 02:32:24 -07:00
Chen Sun 87ddf202ea
Sort resource references before checking for run struct equality (#3547) 2020-04-20 01:53:38 -07:00
Chen Sun 0d961ce5c1
use reflect.DeepEqual to compare run structs (#3546) 2020-04-17 15:43:14 -07:00
jingzhang36 56d519e9f7
Add archive experiment feature in backend (#3359)
* add new field in db schema and api schema

* auto genereted types for experiment storage state

* add archive and unarchive methods to backend for experiments.

* auto generated archive/unarchive methods for epxeriments

* add archive and unarchive to client

* set proper storage state when creating experiment

* retrieve storage state when we get/list epxeriment(s)

* change expection in test to have storage state

* add storage state in resource manager test

* revise experiemnt server test

* revise api converter test

* integration test of experiment archive

* archive/unarchive experiment affect the storage state of runs in it

* test all the runs in archive/unarchive experiment

* test all runs are archived/unarchived with their experiment in experiment server

* integration test

* integration test: value type mismatch in assertion

* unused import; default value for storage state

* autogen code for frontend

* reorder the fields in api experiment schema

* switch the position of the two enum to verify a hypothesis

* Put a place hodler to prevent any valid item to take the value 0

* Get rid of the place holder since the cause of issue related to value 0 is found and fixed.

* The returned api experiment now has storage state field

* create experiment return doesn't contain storege state

* Cleanup needs to clean runs and pipeliens now

* a missing client

* use resource reference as fileter instead of experiment uuid

* use same namespace in archive unit test

* Leave archive/unarchive experiment integration test to a separate PR

* also need to update jobs when experiments are archived

* Change of unarchiving logic. When experiment is unarchived, jobs/runs in
it stay archived

* add unit test for the job status in archived/unarchived experiment

* change archive state to 3 value enum; add experiment integration test

* make archive state 3 value enum to avoid 0 value mapped to available; add integration test

* run swagger autogen

* fix an expected value

* fix experiment server test

* add job check in experiment server test

* update job crds

* fix a typo

* remove accidentally included irrelevant changes
2020-04-16 07:31:09 +08:00
jingzhang36 6e56960e6a
Refactor the legacy way of using pipeline id to create run in KFP backend (#3437)
* For legacy interface, we switch to the new presentation underhood

* when create run, if user specify a pipeline, we subsitute it with the pipeline's default version

* Add a case where a version and a pipeline are both specified

* comment; get ready pipeline

* comments

* fix upgrade integration test

* comments of todo; expected run/job now has resource references

* fix upgrade test expected value according to the new response

* fix a typo

* a quick hack for upgrade test

* surface err from conversion
2020-04-08 00:08:49 +08:00
jingzhang36 d9bb1be25b
Add pipeline version to job/run integration test so that job/run is c… (#3270)
* Add pipeline version to job/run integration test so that job/run is created from pipeline version

* Use set comparison to check array equation in test

* address comments to use elements match
2020-03-16 16:45:40 +08:00
Yuan (Bob) Gong 5391e88fbc
[Testing] KFP standalone test infra for upgradability (#1971)
* Implement upgrade test

* mark upgrade-tests.sh as executable

* Fix comments

* Base upgrade_test_setup.yaml

* e2e integration of upgrade test

* Fix entrypoint argument

* Fix e2e workflow yaml

* Fix run_test.sh argument processing

* Fix no closing backtick

* Restrucutre upgrade_test.go to focus the test on upgrade verification

* clean up code

* Clean up after upgrade test when it is run in integration tests.

* Include pipeline tests in upgrade test

* Reorder tests

* Add upgrade test coverage for run api resources

* Add job api resource coverage in upgrade test & refactored upgrade test

* Fix add missing step in upgrade test

* Fix BUILD.bazel

* Fix upgrade_test.go

* Try to fix upgrade test failure

* Fix hard coded namespace

* Sync upgrade-tests.sh with new changes in presubmit-tests-with-pipeline-deployment.sh

* Update upgrade test

* Remove redundant code

* Fix integration test exit code

* Fix trigger interval second mismatch
2020-03-09 16:53:37 -07:00
jingzhang36 ee587101d3
Add integration tests for version related methods (#3174)
* create a new version upload integration test file

* checkpoint

* checkpoint

* test

* checkpoint

* checkpoint

* version api integration test

* Fix a typo

* typo in comment

* error message
2020-02-26 21:29:52 +08:00
Yuan (Bob) Gong 8c0bc219ed
[Backend] swf catchup option integration tests (#3139)
* Make local testing easier

* Move cleanup to test setup stage

* Add readme for how to run integration tests

* Add warning about data loss

* Add warning also in the script

* Change flag to isDevMode and cleanup resources if not in dev mode

* Pass through arguments in the bash script

* Fix unit tests

* integration tests for swf NoCatchup option

* Also add cron catchup option tests

* Restructure test

* Adjust format

* Fix BUILD.bazel

* fix typo
2020-02-21 02:28:31 -08:00
Yuan (Bob) Gong b90040a00f
[Backend] Refactor integration tests, facilitate local testing (#3138)
* Make local testing easier

* Move cleanup to test setup stage

* Add readme for how to run integration tests

* Add warning about data loss

* Add warning also in the script

* Change flag to isDevMode and cleanup resources if not in dev mode

* Pass through arguments in the bash script

* Fix unit tests
2020-02-20 23:02:32 -08:00
jingzhang36 d1b72df5fb
Upload local file as new pipeline version Step 1 (#3001)
* add upload pipeline version to upload_pipeline_server and http main

* add apiPipelineVersion to pipeline upload swagger json

* add apiResourceReference to pipeline upload swagger json

* Add yet more types to pipeline upload swagger json

* Unit tests
2020-02-07 16:13:36 +08:00
jingzhang36 af090bffa2 Regenerate api since https://github.com/kubeflow/pipelines/pull/2445 changed api proto (#2506)
* re-generate api since #2445 changed api proto

* Fix integration test for pipeline client

* use generated constants
2019-10-29 23:54:52 -07:00
Kirin Patel 41b394b045 Add e2e visualization tests (#1981)
* Created visualization_api_test.go

* Updated BUILD.bazel files

* Removed clean_up from e2e test

* Revert "Removed clean_up from e2e test"

This reverts commit 82fd4f5a00.

* Update e2e tests to build visualizationserver and viewer-crd

* Fix bug where wrong image is set

* Fixed incorrect image names

* Fixed additional instance of incorrect image names
2019-08-30 13:54:10 -07:00
IronPan f1c9594f7d Propagate pipeline name in pipeline spec (#1842)
* update

* add pipeline name

* propagate name

* propagate name

* fix

* update test
2019-08-14 23:30:32 -07:00
IronPan 4e43750c9d
add reference name to resource reference API proto (#1781)
* add resource reference

* fix tests

* fix tests

* fix e2e test

* fix e2e

* fix test

* update api requirement

* fix tests

* Update job_api_test.go

* Update run_api_test.go

* Update setup.py

* Update deploy-kubeflow.sh

* fix tests

* Update deploy-kubeflow.sh
2019-08-13 03:15:11 -07:00
Yuan (Bob) Gong db7f978afd Disable flaky tests temporarily (#1809)
Temporarily addresses https://github.com/kubeflow/pipelines/issues/1762
2019-08-12 09:38:14 -07:00
IronPan 3c9b23216c
fix tests (#1769) 2019-08-08 11:37:58 -07:00
IronPan 44712b3c4a
Delete go CLI (#1592)
* delete go cli

* Delete cli_test.go

* Update BUILD.bazel

* Update .travis.yml

* Update BUILD.bazel
2019-07-19 01:11:25 -07:00
Riley Bauer fbae0f855e Creates a default experiment at API server set up time (#1089)
* WIP - Create default experiment upon API server initialization

* Default experiment initialization caused crashes if API server pod was restarted without clearing DB

* Adding new table to store default experiment ID

* Add default experiment type model definition

* Minor fixes, everything seems to work now

* Clean up. Renamed to default_experiment_store

* Adds tests for the default_experiment_store

* Add integration test for verifying initial cluster state. Currently only covers existence of default experiment

* Don't run initialization tests except as integration tests

* Fixes comments

* PR comments and cleanup

* Extract code in resource_manager to helper func
2019-04-15 15:55:04 -07:00
IronPan 086d4763d9 Surface workflow finished time in list run API (#1122)
* add finished time for list runs

* add finished time for list runs

* fix tests

* add finished time for list runs

* Update run.proto

* address comments

* make query more robust

* fix e2e test

* fix e2e test
2019-04-10 23:02:10 -07:00
Alexey Volkov 2c2445df83 Enable pipeline packages with multiple files (#939)
* Enable pipeline packages with multiple files

* Added tests

* Initialize the variables to nil

* Trying to read the archive file entry immediately

* Fixed the pipeline packages used by the `TestPipelineAPI` test.
Also added a failing test case. Will disable it in next commit.

* Disabling the test for the UploadFile bug I've discovered

* Fixed the pipeline name.

* Removed the disabled extra test.

* Addressed the feedback.

* Removed the "header == nil" check (feedback).

* Fixed typo

* Addressed the PR feedback
Added space before comment.
Checking for the error again.
2019-03-28 15:09:58 -07:00
Alexey Volkov f98ec68488 Added the ability to terminate a run (#528)
* Added the terminate run command to backend and CLI.
No generated files for now.

* Added the generated files.

* Moved the code to run_store.go
Now the call chain is run_client->run_server->resource_manager->run_store

* Using the backoff package for retries.

* Trying to update run status in the DB to "Terminating" before patching the workflow.

* Stopped using the Argo errors module.

* Fixed the compilation errors due to recent backend changes.

* RILEY - WIP - Implementation of workflow_fake.go and first test

Added successful test in resource_manager_test.go and completed, barring nits and conventions, the implementation of Patch() and isTerimated() within workflow_fake.go

Additional tests and lots of clean-up are still necessary

* Adds a few more tests to resource_manager_test and cleans up a bit

* Further clean up. Stopped using squirrel for UPDATE query. Added run_store_tests

* Adds terminate run integration test

* Undo changes to go.sum

* Fixes path to long-running.yaml in integration test

* Allow runs with no Conditions to be terminated
2019-03-21 18:00:18 -07:00
IronPan f8c1dde3a8 move integration test to sub dir (#888)
* move integration test to sub dir

* revert
2019-03-01 15:32:56 -08:00