Commit Graph

26 Commits

Author SHA1 Message Date
Chen Sun 8b2fba3f39
test: Add v2 API integration tests (#9185)
* Add v2 API integration tests

* move v1 client
2023-04-27 20:20:39 +00:00
gkcalat 135dc64854
feat(backend): Refactor apiserver, add Pipeline and other v2beta services (#8633)
* Staging WIP

* Fix recurring run, switch to pipeline_version_id

* Regenerate python client

* Refactor packages for pipeline service v2

* Stage progress

* Add support for pipeline service v2beta1

* Go mod tidy

* Update licenses

* Set default namespace to be empty

* Separate deletion of pipeline versions

* Change report ReportRunMetrics to obey AIP

* Use pipeline version id in run proto

* Update run.proto

* Upgrade storage

* Remove dependencies on api

* Stage refactoring progress

* Refactor apiserver

* Update v1beta1 unit test

* Update dependencies

* Fix integration test

* Revert visualization server changes (workaround)

* Address comments

* Basic lint the api server

* Update licenses file

* Adjust intergtaion tests

* Fix typos in the tests

* Update tests

* Update test

* Add support v1 delepe pipeline

* Remove gorm.Model to split into a different PR

* Lint api server

* Fix default experiment in upgrade test

* Update storage states

* Testing upgrade test run

* Add copying pod logs to the GCS bucket

* Fix upgrade test

* Adjust backend test for list

* Fix the tests

* Fix workflow manifest discrepancy at create

* Add a few unit-tests

* Add misc service and fix persistence agent

* Remove default values from TEXT columns

* Address comments. Fixes #8702.

* Address comment, reorder resource, server, storage

* Fix front end integration test

* Fix FE integration test

* Fix tabs, add pauses, make more explicit FE test

* Revert FE test changes

* Enable default experiment and disable its deletion

* Replace empty namespace with POD_NAMESPACE

* Default experiment is now the oldest

* Emplace empty with POD_NAMESPACE for argo only

* Fix api integration test

* Revert changes to default experiment's description

* Improve FE test flakiness

* Force refresh in the FE test

* Fix FE integration test

* Add delay in FE test

* Add local debug to FE test

* Update tests

* Fix typo

* Address comments

* Enforce template version verification

* Fix workflow manifest discrepancy with v1

* Fix FE test

* Revert changes in persistence agent

* Limit to one resource owner

* Change default run state to unknown

* Revert changes to FE integration test

* Make states consistent

* Remove links from error messages

* Address comments

* Address comments

* Change the order for teardown

---------

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2023-02-03 20:17:01 +00:00
Lingqing Gan 6ee767769d
feat(backend): add support for "finished_at" and others in filter. Fixes #8654 (#8662)
* add support for "finished_at" and other in filter

* update job map

* add UTs

* add integration test for filter

* update integration test

* fix integration test

* add more info

* fixed integration test

* fix small bug

* fix flakiness
2023-01-15 19:00:58 +00:00
Lingqing Gan 7e7e1afcb0
feat(backend): Add support for converting v2 recurring run to scheduled workflow in template (#8521)
* add template support for v2 api

* copyright

* add v2 tests

* add proto files and generate clients

* update dockerfile

* update script

* refactor rec_run_id

* copyright info

* move string definition

* small fix to proto

* small refactor

* template takes model instead of api

* template changes

* convert to model in resource manager

* resolve conflicts

* fix integration test typos

* test

* test

* test

* testing

* temp change for debug

* temp change for debug 2

* temp change for debug 3

* temp change for debug 4

* temp change for debug 5

* temp change for debug 6

* revert resource manager change

* add print

* bring back integration tests

* folder name

* delete a file

* remove package debug

* fix bug

* print more variables

* add log

* add log

* small fix

* add log

* more logs

* add test

* add  logs

* test

* add another test

* use old function for Trigger convertion

* change modeltoCRDTrigger function

* removed V1 at the end of function name

* add comments

* removed unnecessary variable

* typo

* small change

* test

* remove logs for debugging

* small bug

* remove logs for debug

* nit and removed unused functions

* remove unused functions  and move var declararion
2022-12-21 19:43:13 +00:00
Lingqing Gan 3f2e20d14f
chore(backend): regenerate v1beta1 api clients (#8403)
* add v2 experiment.proto

* regenerate v1beta1 client

* remove api version from combined swagger file

* Revert "regenerate v1beta1 client"

This reverts commit 02769d779b.

* regenerate v1beta1 clients

* regenerate python packages without api version

* remove regenerated v1 python packages&change proto

* refactor v1 exp apis and functions

* remove v2 api package and refactor v1 proto

* regenerate v1 go clients

* regenerate v1 python package

* refactor and update the human maintained swagger

* renew swagger file

* regenerate v1 packages

* regenerate v1 python package

* refactor

* regenerate v1 go clients

* regenerate v1 python package

* recover dockerfile and add s in python

* fix func name

* use commit for sdk execution test

* change sdk execution test

* update test script

* add v1 to methods

* proto update v1 method names

* regenerate v1 go packages

* regenerate python client

* update server methods

* refactor

* refactor

* refactor

* refactor python client

* fix get_healthz

* update sample-test.sh in v2 backend

* sample test script

* change v2 sample test

* change package install order

* Revert "change package install order"

This reverts commit c24fcfbed9.

* use KFP server api from commit for v2 sample tests

* small fix
2022-11-10 19:18:00 +00:00
Lingqing Gan 6c6b1e3a94
chore(backend): move v1 api to v1beta1 package and folder (#8376)
* move v1 api to v1beta1 package and folder

* fix test

* update README.md

* update README.md

* update readme
2022-10-26 07:11:30 +00:00
James Wu bcbdbd050f
feat(conformance): adapt remaining tests to KF env (#7670)
- Adapted job, rn, upgrade and visualization tests
- Undid earlier change where I had to pass in empty string to indicate
  that resource reference should not be overridden because experiment id
  is specified
- Updated visualization client to support SA token projection auth
2022-05-05 23:56:50 +00:00
James Wu 1265113dff
feat(conformance): Enable experiment API tests in KF mode. (#7596)
* Conformance: Enable experiment API tests in KF mode.

Added 3 flags:
- isDebugMode is enables HTTP request/response logging
- IsKubeflowMode enables the tests in full Kubeflow mode
- resourceNamespace: the namespace/profile under which the test
  resources are created

Added a new HTTP client that uses SA token volume projection auth. The
test pods will be set up to project SA token.

Plumbed everything through for experiment API tests. The other tests
will be enabled in subsequent PRs.

* Updated change to address comments.
2022-04-29 00:45:56 +00:00
Niklas Hansson 484c61931a
chore: remove sleep from api_job_test.go and add retry instead. (#6584)
* Add retry logic instead of sleep

* retry instead of sleep

* to do comment

* Updated to 8 times and 5 seconds

* missed to update on two places
2021-10-19 02:38:33 -07:00
Niklas Hansson d5223b396e
chore: add condition check and missing error asserts in run_api_test.go (#6604)
* Fix missing asserts and add missing test case

* Update the expected value of the list
2021-09-28 17:02:46 -07:00
Niklas Hansson 00a0ffbdb7
feat(backend): upgrade argo go module to V3. Part of #5718 (#5792)
* updating argo

* updated fix deps and code

* clean up
2021-06-08 05:04:45 -07:00
Haofeng Xie 15b0f50c54
feat(backend): Add qps & burst options in clientset. Fix #5095 (#5231)
* feat(backend): Add qps & burst options in clientset. Fix #5095

* Some minor changes:
* Use comment in client-go as usage string of QPS & burst.
* Use `common.GetXXX()` to retrieve configurations in API server.
* Use a struct to include configurations in cache server.
* Update job_api_test.go to support modified `NewScheduledWorkflowClientOrFatal()`.

* Some minor fixes:
* Create `util.ClientParameters` for configurations needed to create a client.
* Add comments linking to the default value of clientQPS & clientBurst defined in k8s.io/client-go.
2021-03-16 11:14:15 -07:00
Niklas Hansson eeb7f8f04a
fix(backend): make the scheduleworkflowontroller timezone aware. Fixes #2653 (#4641)
* intial work'

* small fixes

* updated tests and how parameter are set

* try to fix test

* check with out adding missing test

* fixed small typo

* test changes

* updated config

* typo

* updated after feedback

* fixed pointer error

* test to add paramter

* moved to init so removed not needed code

* updated further

* updated tests to also check endtime

* clean up test

* fixed failing test

* fixed the expected test results

* added timezone examples

* further clean up

* fixed time format

* Update params.env

* moved location to cronjobscheduler

* clean up

* set env variable to empty

* reverted back

* updated to make magic nbr to constant

* updated the tests with comment

* added comments on cron expressions

* update naming and return types

* updated to UTC as default

* updated with an alpha notice
2021-01-10 00:59:05 -08:00
Yuan (Bob) Gong 4df8925b05
fix(backend): job api -- deletion/disabling should succeed when swf not found. Fixes #4871 (#4884)
* fix(backend): job api -- deletion should succeed when swf not found

* bug reproducing unit test

* fix the bug and pass reproducing unit test

* reproducing integration test

* fix integration test

* clarify error message

* disable job should also succeed, unify term to CR instead of CRD

* fix unit test error

* fix error message

* improve logging
2020-12-09 18:12:52 -08:00
Niklas Hansson a6c79c2e2c
test(backend): added missing assert statement in job_api_test . (#4705) 2020-11-01 16:36:52 -08:00
Yuan (Bob) Gong 7840d30274
test(backend): fix TestJobApis test is flaky. Fixes #4419 (#4451) 2020-09-01 22:13:06 -07:00
Yuan (Bob) Gong d3ef4bf39f
[Backend] Use service account passed from api object (#3650)
* Add service account field to run and job api objects

* Update description

* Fix field casing

* Use service account from api object

* Fix bug and add unit test

* Save patched workflow spec into db instead

* Save service account to db model

* Fix unit tests

* Fix integration tests

* Fix upgrade test

* Update upgrade_test.go
2020-05-05 23:23:08 -07:00
Chen Sun f55147d9c0
Sort job resource references before equality check (#3561) 2020-04-29 02:32:24 -07:00
jingzhang36 d9bb1be25b
Add pipeline version to job/run integration test so that job/run is c… (#3270)
* Add pipeline version to job/run integration test so that job/run is created from pipeline version

* Use set comparison to check array equation in test

* address comments to use elements match
2020-03-16 16:45:40 +08:00
Yuan (Bob) Gong 8c0bc219ed
[Backend] swf catchup option integration tests (#3139)
* Make local testing easier

* Move cleanup to test setup stage

* Add readme for how to run integration tests

* Add warning about data loss

* Add warning also in the script

* Change flag to isDevMode and cleanup resources if not in dev mode

* Pass through arguments in the bash script

* Fix unit tests

* integration tests for swf NoCatchup option

* Also add cron catchup option tests

* Restructure test

* Adjust format

* Fix BUILD.bazel

* fix typo
2020-02-21 02:28:31 -08:00
Yuan (Bob) Gong b90040a00f
[Backend] Refactor integration tests, facilitate local testing (#3138)
* Make local testing easier

* Move cleanup to test setup stage

* Add readme for how to run integration tests

* Add warning about data loss

* Add warning also in the script

* Change flag to isDevMode and cleanup resources if not in dev mode

* Pass through arguments in the bash script

* Fix unit tests
2020-02-20 23:02:32 -08:00
jingzhang36 d1b72df5fb
Upload local file as new pipeline version Step 1 (#3001)
* add upload pipeline version to upload_pipeline_server and http main

* add apiPipelineVersion to pipeline upload swagger json

* add apiResourceReference to pipeline upload swagger json

* Add yet more types to pipeline upload swagger json

* Unit tests
2020-02-07 16:13:36 +08:00
IronPan f1c9594f7d Propagate pipeline name in pipeline spec (#1842)
* update

* add pipeline name

* propagate name

* propagate name

* fix

* update test
2019-08-14 23:30:32 -07:00
IronPan 4e43750c9d
add reference name to resource reference API proto (#1781)
* add resource reference

* fix tests

* fix tests

* fix e2e test

* fix e2e

* fix test

* update api requirement

* fix tests

* Update job_api_test.go

* Update run_api_test.go

* Update setup.py

* Update deploy-kubeflow.sh

* fix tests

* Update deploy-kubeflow.sh
2019-08-13 03:15:11 -07:00
IronPan 3c9b23216c
fix tests (#1769) 2019-08-08 11:37:58 -07:00
IronPan f8c1dde3a8 move integration test to sub dir (#888)
* move integration test to sub dir

* revert
2019-03-01 15:32:56 -08:00