Commit Graph

25 Commits

Author SHA1 Message Date
Chen Sun 8b2fba3f39
test: Add v2 API integration tests (#9185)
* Add v2 API integration tests

* move v1 client
2023-04-27 20:20:39 +00:00
gkcalat 135dc64854
feat(backend): Refactor apiserver, add Pipeline and other v2beta services (#8633)
* Staging WIP

* Fix recurring run, switch to pipeline_version_id

* Regenerate python client

* Refactor packages for pipeline service v2

* Stage progress

* Add support for pipeline service v2beta1

* Go mod tidy

* Update licenses

* Set default namespace to be empty

* Separate deletion of pipeline versions

* Change report ReportRunMetrics to obey AIP

* Use pipeline version id in run proto

* Update run.proto

* Upgrade storage

* Remove dependencies on api

* Stage refactoring progress

* Refactor apiserver

* Update v1beta1 unit test

* Update dependencies

* Fix integration test

* Revert visualization server changes (workaround)

* Address comments

* Basic lint the api server

* Update licenses file

* Adjust intergtaion tests

* Fix typos in the tests

* Update tests

* Update test

* Add support v1 delepe pipeline

* Remove gorm.Model to split into a different PR

* Lint api server

* Fix default experiment in upgrade test

* Update storage states

* Testing upgrade test run

* Add copying pod logs to the GCS bucket

* Fix upgrade test

* Adjust backend test for list

* Fix the tests

* Fix workflow manifest discrepancy at create

* Add a few unit-tests

* Add misc service and fix persistence agent

* Remove default values from TEXT columns

* Address comments. Fixes #8702.

* Address comment, reorder resource, server, storage

* Fix front end integration test

* Fix FE integration test

* Fix tabs, add pauses, make more explicit FE test

* Revert FE test changes

* Enable default experiment and disable its deletion

* Replace empty namespace with POD_NAMESPACE

* Default experiment is now the oldest

* Emplace empty with POD_NAMESPACE for argo only

* Fix api integration test

* Revert changes to default experiment's description

* Improve FE test flakiness

* Force refresh in the FE test

* Fix FE integration test

* Add delay in FE test

* Add local debug to FE test

* Update tests

* Fix typo

* Address comments

* Enforce template version verification

* Fix workflow manifest discrepancy with v1

* Fix FE test

* Revert changes in persistence agent

* Limit to one resource owner

* Change default run state to unknown

* Revert changes to FE integration test

* Make states consistent

* Remove links from error messages

* Address comments

* Address comments

* Change the order for teardown

---------

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2023-02-03 20:17:01 +00:00
Lingqing Gan 6ee767769d
feat(backend): add support for "finished_at" and others in filter. Fixes #8654 (#8662)
* add support for "finished_at" and other in filter

* update job map

* add UTs

* add integration test for filter

* update integration test

* fix integration test

* add more info

* fixed integration test

* fix small bug

* fix flakiness
2023-01-15 19:00:58 +00:00
Lingqing Gan 3f2e20d14f
chore(backend): regenerate v1beta1 api clients (#8403)
* add v2 experiment.proto

* regenerate v1beta1 client

* remove api version from combined swagger file

* Revert "regenerate v1beta1 client"

This reverts commit 02769d779b.

* regenerate v1beta1 clients

* regenerate python packages without api version

* remove regenerated v1 python packages&change proto

* refactor v1 exp apis and functions

* remove v2 api package and refactor v1 proto

* regenerate v1 go clients

* regenerate v1 python package

* refactor and update the human maintained swagger

* renew swagger file

* regenerate v1 packages

* regenerate v1 python package

* refactor

* regenerate v1 go clients

* regenerate v1 python package

* recover dockerfile and add s in python

* fix func name

* use commit for sdk execution test

* change sdk execution test

* update test script

* add v1 to methods

* proto update v1 method names

* regenerate v1 go packages

* regenerate python client

* update server methods

* refactor

* refactor

* refactor

* refactor python client

* fix get_healthz

* update sample-test.sh in v2 backend

* sample test script

* change v2 sample test

* change package install order

* Revert "change package install order"

This reverts commit c24fcfbed9.

* use KFP server api from commit for v2 sample tests

* small fix
2022-11-10 19:18:00 +00:00
Lingqing Gan 6c6b1e3a94
chore(backend): move v1 api to v1beta1 package and folder (#8376)
* move v1 api to v1beta1 package and folder

* fix test

* update README.md

* update README.md

* update readme
2022-10-26 07:11:30 +00:00
Lingqing Gan f25127fe76
feat(backend): Implement RuntimeConfig in backend for jobs(RecurringRun) (#8152)
* update server folder

* update storage folder

* update run_api_test.go

* update job_api_test.go

* update upgrade_test.go

* model converter and tests

* fix upgrade_test.go

* fix api tests

* update error messages
2022-08-23 00:01:25 +00:00
Lingqing Gan 2b556ec855
feat(backend): Implement RuntimeConfig in backend (#8085)
* add runtime config model

* add run store and tests

* add runtime_config to model converter and tests

* add runtime_config to api converter and its tests

* change api server and related tests

* remove v2 runtime_config test

* add runtimeconfig to upgrade test

* fix test values

* upgrade test debug

* tests

* add more info for debug

* use NullString instead of String, remove debug

* fix type conversion

* change function and add unit tests

* run go fmt

* Add comments for model

* marshal params using v2 structpb values

* fix small bug

* Revert "run go fmt"

This reverts commit 251c3a9535.

* No longer sort keys

* test values and explain comparison using .String()

* func toApiRuntimeConfig

* tests updates

* add api converter tests

* change test

* fix format

* change test

* simplify marshalling parameters
2022-08-11 23:21:52 +00:00
James Wu bcbdbd050f
feat(conformance): adapt remaining tests to KF env (#7670)
- Adapted job, rn, upgrade and visualization tests
- Undid earlier change where I had to pass in empty string to indicate
  that resource reference should not be overridden because experiment id
  is specified
- Updated visualization client to support SA token projection auth
2022-05-05 23:56:50 +00:00
James Wu 8b1155158e
feat(conformance): adapt run api tests to KF env (#7658)
- Undid earlier refactoring in test_util.go. I think having the util
  function to take a parameter structure is much more readable than
  constructing api parameters in the util function.
2022-05-05 18:27:13 +00:00
James Wu 1265113dff
feat(conformance): Enable experiment API tests in KF mode. (#7596)
* Conformance: Enable experiment API tests in KF mode.

Added 3 flags:
- isDebugMode is enables HTTP request/response logging
- IsKubeflowMode enables the tests in full Kubeflow mode
- resourceNamespace: the namespace/profile under which the test
  resources are created

Added a new HTTP client that uses SA token volume projection auth. The
test pods will be set up to project SA token.

Plumbed everything through for experiment API tests. The other tests
will be enabled in subsequent PRs.

* Updated change to address comments.
2022-04-29 00:45:56 +00:00
Niklas Hansson 129f46a943
chore: add missing assert in pipeline_version_api_test.go, run_api_test.go and upgrade_test.go (#6605) 2021-09-28 16:02:54 -07:00
Yuan (Bob) Gong d3ef4bf39f
[Backend] Use service account passed from api object (#3650)
* Add service account field to run and job api objects

* Update description

* Fix field casing

* Use service account from api object

* Fix bug and add unit test

* Save patched workflow spec into db instead

* Save service account to db model

* Fix unit tests

* Fix integration tests

* Fix upgrade test

* Update upgrade_test.go
2020-05-05 23:23:08 -07:00
Chen Sun f55147d9c0
Sort job resource references before equality check (#3561) 2020-04-29 02:32:24 -07:00
Chen Sun 87ddf202ea
Sort resource references before checking for run struct equality (#3547) 2020-04-20 01:53:38 -07:00
Chen Sun 0d961ce5c1
use reflect.DeepEqual to compare run structs (#3546) 2020-04-17 15:43:14 -07:00
jingzhang36 d9bb1be25b
Add pipeline version to job/run integration test so that job/run is c… (#3270)
* Add pipeline version to job/run integration test so that job/run is created from pipeline version

* Use set comparison to check array equation in test

* address comments to use elements match
2020-03-16 16:45:40 +08:00
Yuan (Bob) Gong b90040a00f
[Backend] Refactor integration tests, facilitate local testing (#3138)
* Make local testing easier

* Move cleanup to test setup stage

* Add readme for how to run integration tests

* Add warning about data loss

* Add warning also in the script

* Change flag to isDevMode and cleanup resources if not in dev mode

* Pass through arguments in the bash script

* Fix unit tests
2020-02-20 23:02:32 -08:00
jingzhang36 d1b72df5fb
Upload local file as new pipeline version Step 1 (#3001)
* add upload pipeline version to upload_pipeline_server and http main

* add apiPipelineVersion to pipeline upload swagger json

* add apiResourceReference to pipeline upload swagger json

* Add yet more types to pipeline upload swagger json

* Unit tests
2020-02-07 16:13:36 +08:00
IronPan f1c9594f7d Propagate pipeline name in pipeline spec (#1842)
* update

* add pipeline name

* propagate name

* propagate name

* fix

* update test
2019-08-14 23:30:32 -07:00
IronPan 4e43750c9d
add reference name to resource reference API proto (#1781)
* add resource reference

* fix tests

* fix tests

* fix e2e test

* fix e2e

* fix test

* update api requirement

* fix tests

* Update job_api_test.go

* Update run_api_test.go

* Update setup.py

* Update deploy-kubeflow.sh

* fix tests

* Update deploy-kubeflow.sh
2019-08-13 03:15:11 -07:00
Yuan (Bob) Gong db7f978afd Disable flaky tests temporarily (#1809)
Temporarily addresses https://github.com/kubeflow/pipelines/issues/1762
2019-08-12 09:38:14 -07:00
IronPan 3c9b23216c
fix tests (#1769) 2019-08-08 11:37:58 -07:00
IronPan 086d4763d9 Surface workflow finished time in list run API (#1122)
* add finished time for list runs

* add finished time for list runs

* fix tests

* add finished time for list runs

* Update run.proto

* address comments

* make query more robust

* fix e2e test

* fix e2e test
2019-04-10 23:02:10 -07:00
Alexey Volkov f98ec68488 Added the ability to terminate a run (#528)
* Added the terminate run command to backend and CLI.
No generated files for now.

* Added the generated files.

* Moved the code to run_store.go
Now the call chain is run_client->run_server->resource_manager->run_store

* Using the backoff package for retries.

* Trying to update run status in the DB to "Terminating" before patching the workflow.

* Stopped using the Argo errors module.

* Fixed the compilation errors due to recent backend changes.

* RILEY - WIP - Implementation of workflow_fake.go and first test

Added successful test in resource_manager_test.go and completed, barring nits and conventions, the implementation of Patch() and isTerimated() within workflow_fake.go

Additional tests and lots of clean-up are still necessary

* Adds a few more tests to resource_manager_test and cleans up a bit

* Further clean up. Stopped using squirrel for UPDATE query. Added run_store_tests

* Adds terminate run integration test

* Undo changes to go.sum

* Fixes path to long-running.yaml in integration test

* Allow runs with no Conditions to be terminated
2019-03-21 18:00:18 -07:00
IronPan f8c1dde3a8 move integration test to sub dir (#888)
* move integration test to sub dir

* revert
2019-03-01 15:32:56 -08:00