Commit Graph

741 Commits

Author SHA1 Message Date
Linchin 923b67e38f chore(release): bumped version to 2.0.0-rc.2 2023-06-13 18:43:09 +00:00
Lingqing Gan dbebbde200
fix(backend): let components add default values (#9591)
* let components add default values

* address comments and add unit tests

* address comments

* backend test

* backend test 2

* backend test 3

* backend test 4

* backend test 5

* backend test 6

* do not use python component in unit tests

* shell command

* shell command

* shell command

* does not delete tmp folder

* change folder permission

* update launcher image
2023-06-13 18:25:03 +00:00
Chen Sun ee2102a9b5
chore(samples/backend/frontend): Update preload control-flow and data-passing pipelines to v2 (#9563)
* Update preload samples to v2 pipelines

* address review comments

* fix frontend test

* format frontend code

* update comment
2023-06-07 22:06:57 +00:00
Chen Sun 0d53de7d1c
fix(backend): fix column size of `StateHistory` in `task` table. Fixes: #9553 (#9554) 2023-06-02 10:49:06 +00:00
Lingqing Gan eadeb3096b
fix(backend): enable cache for create/delete PVC (#9504)
* enable cache for create/delete pvc

* remove comments

* createCache() func

* update driver image
2023-06-02 01:12:06 +00:00
Lingqing Gan 0ccdb09a35
fix(backend): publish launcher execution even if it fails (#9513)
* publish launcher execution

* update launcher image
2023-05-31 01:16:04 +00:00
ryansteakley 8a3215ec56
feat(backend): Implement aws-go-sdk crendentials to support IRSA for s3 in V2 (#9502)
* support v2 irsa

* remove white space
2023-05-26 23:49:00 +00:00
Chen Sun 4810b7aac7
fix(backend): upgrade to `gopkg.in/yaml.v3` to fix boolean support. Fixes #9451 (#9473) 2023-05-24 08:57:19 +00:00
gkcalat 3e9f04815e
fix(backend): Fix url encoding issue. Closes #9448 (#9449) 2023-05-23 04:00:35 +00:00
gkcalat 79d8d83ce4
fix(backend): Add creation of pipeline and pipeline version atomically (#9416)
* Add create pipeline and version

* Add CreatePipelineAndVersion

* Add unit tests

* Address comments and add counters

* Add comments to pipeline.proto

* Add e2e test case for CreatePipelineAndVersion
2023-05-18 23:38:46 +00:00
Lingqing Gan 08494a2db6
feat(backend): add schema version check (#9447) 2023-05-18 21:02:10 +00:00
Lingqing Gan 724445064b
chore: change sample PVC access modes to ReadWriteOnce (#9425)
* change sample access modes to ReadWriteOnce

* change test data

* change samples to use ReadWriteOnce
2023-05-18 01:22:06 +00:00
gkcalat c2bc434c11
chore(backend): Add golangci-lint and pre-commit config (#9428)
* Add golangci-lint and pre-commit config

* Adopt golanci-lint config from k8s
2023-05-17 18:23:06 +00:00
Lingqing Gan f9409a42d6
test: remove test dependency on driver/launcher images (#9422) 2023-05-16 00:19:59 +00:00
Joe Li cb93fcc5ca chore(release): bumped version to 2.0.0-rc.1 2023-05-12 21:36:20 +00:00
Lingqing Gan 73f5f1b3ca
fix(backend): prepublish execution in the driver for create/delete pvc (#9397)
* publish execution for create/delete pvc

* update driver launcher image

* combine functions

* update driver launcher image
2023-05-12 14:05:20 -07:00
Chen Sun f35d50c8dd
chore: Remove unused binaries in cache-deployer, add curl license (#9400)
* Update Dockerfile

* Update Dockerfile

* add curl license text to third_party/license.txt

* change the location where we store third_party license.txt

* checkout license.txt from upstream master

* add https://github.com/curl/curl/blob/master/COPYING to third_party/license.txt
2023-05-12 06:28:31 +00:00
gkcalat 09c0b92771
fix(backend): Fix authorization logic and remove redundant DB calls (#9373)
* Refactor authorization

* Upgrade api version

* Revent API version bump

* Fix string comparison

* Revert test change

* Change error type

* Fix typo

* Fix namespace logic
2023-05-12 03:07:31 +00:00
gkcalat 9ec4dc7f30
fix(backend): Add pipeline spec name validation and relax display name validation (#9374)
* Add pipeline name check at creation time

* Revert changes for pipeline version

* Validate pipeline spec name and relax display name

* Fix e2e test
2023-05-11 23:57:30 +00:00
gkcalat 960eb6604e
chore(backend): Remove deprecated experiment id from run server's endpoints (#9392)
* Remove unused getRunMetrics

* Remove deprecated experiemnt id
2023-05-11 22:28:39 +00:00
gkcalat d4c43f77bc
chore(backend): Remove unused getRunMetrics (#9391) 2023-05-11 22:28:31 +00:00
Chen Sun feb1f79b3e
chore: Create and use license-compliance driver and launcher images (#9388)
* Create and use license-compliance driver and launcher

* fix launcher cmd
2023-05-11 18:44:02 +00:00
Lingqing Gan 4f4f2cbb55
fix(backend): change format of some backend error message (#9304)
* fix error message

* address comments
2023-05-10 01:19:42 +00:00
b4sus c17b6815cd
fix(backend): added rbac group to commands to create pipeline/run. Fixes #8854 (#9365) 2023-05-09 17:37:33 +00:00
Chen Sun 270257512a chore(release): bumped version to 2.0.0-beta.2 2023-05-08 09:21:48 +00:00
Chen Sun 931a68096e
chore: upgrade to Argo v3.3.10 (#9336)
* pull argo v3.3.10 upstream

* upgrade to Argo v3.3.10

* fix validateworkflow

* upgrade golang

* update golang

* fix go dep

* update go licenses file

* fix build

* fix go dep

* downgrade k8s.io/client-go to v0.24.3

* update licenses

* fix go dep

* update licenses
2023-05-08 08:38:55 +00:00
Chen Sun b6b8f94da2
chore: recompile/update v2 samples to avoid possible runtime breakage (#9333)
* recompile/update v2 samples

* fix test
2023-05-05 17:28:24 +00:00
gkcalat fbb8b399b3
fix(backend): Allow empty namespace and remove default namespace. Fixes #8945. Fixes #8897. Fixes #8854. (#9286)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.

* stage progress

* Fix timestamps in tasks and add unit test

* Add upsert to sql dialects. Closes #8851.

* Remove dead code

* Fix unit test

* Switch to upsert

* Add unit tests

* Allow empty namespace

* Cleanup

* Fix tests

* Fix inconsistencies

* Fix old tests

* Differentiate list requests in multi-user mode

* Refactor create run and create job paths

* Revert default experiment change
2023-05-04 23:53:23 +00:00
gkcalat 3c9dbee225
chore(backend): Add 503 error to the backend (#9317)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.

* Add service unavialbe error to the BE

* Fix a unit test
2023-05-04 21:37:22 +00:00
James Wu af04fda622
chore(conformance): add import alias to avoid naming conflict. (#9324)
Addressed PR comment https://github.com/kubeflow/pipelines/pull/9297#discussion_r1184185507
2023-05-04 19:24:45 +00:00
Yihong Wang 6c72d95b3f
feat(backend): avoid downcast and use ExecutionSpec if possible (#9232)
Limit the direct Workflow data struct access to common.util
and use ExecutionSpec and ExecutionStatus in other places.
Create a data struct to represent the node status information
in ExecutionStatus and add an API to access this information.
2023-05-04 17:25:45 +00:00
James Wu 2cfc4283d2
fix(conformance): enable healthz test in full Kubeflow. (#9297)
* Enable healthz test in full Kubeflow.

This change will also be ported to release-2.0-alpha branch to support
conformance tests for KF 1.7, which is using alpha.7 version of KFP.

* Fix build break due to recent refactoring.
2023-05-04 02:02:44 +00:00
Chen Sun e115e0c01f
[WIP] test: Add v2 upgrade test (#9287)
* Add v2 upgrade test

* demo recurring run missing runtime config

* Revert "demo recurring run missing runtime config"

This reverts commit 58154c5548.

* add v2 entries in upgrade  test script

* address review comments
2023-05-03 21:56:45 +00:00
Lingqing Gan 38bf3681d5
fix(backend): fix launcher importer error (#9290) 2023-05-03 00:58:13 +00:00
gkcalat a975e99d2a
fix(backend): Fix timestamp in task store, add disabled unit test (#9246)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.

* Fix timestamps in tasks and add unit test

* Add upsert to sql dialects. Closes #8851.

* Remove dead code

* Fix unit test

* Add unit tests

* Remove dead code
2023-05-02 19:34:44 +00:00
Lingqing Gan 52c4c6f70d
feat(backend): add parameter verification before submitting the workflow (#9228)
* add param verification

* address comments

* error message

* address comments and add extra param verification

* error message
2023-04-28 02:35:39 +00:00
gkcalat 2b92f9bb61
fix(backend): Add upsert for SQLite and MySQL dialects (#9245)
* Add upsert to sql dialects. Closes #8851.

* Remove unit-test

* Remove dead code

* Add unit tests
2023-04-27 20:21:01 +00:00
Chen Sun 8b2fba3f39
test: Add v2 API integration tests (#9185)
* Add v2 API integration tests

* move v1 client
2023-04-27 20:20:39 +00:00
gkcalat f5514c7662
fix(backend): Cleanup pipeline creation and upload logic (#9189)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.

* Add delete create pipeline and improve errors

* Add single-transaction pipeline upload

* Revert previous changes

* Resolve inconsistency

* Add a unit test
2023-04-25 19:20:45 +00:00
Chen Sun afe34941d6
chore: Update api-server base image to Debian buster. Fixes #9220 (#9219) 2023-04-24 20:24:44 +00:00
Chen Sun 8645cac9a6
fix(release): Update release image with the latest api-generator (#9198)
* Update release image with the latest api-generator

* Add note to keep api-generator image in sync between dev and release
2023-04-21 16:34:41 +00:00
Chen Sun 7b70cb5e37
fix(api): Regenerate v2beta1 API (#9197) 2023-04-21 14:21:41 +00:00
Lingqing Gan 9b5db59170
chore(backend): change optional parameter enforce rule (#9184) 2023-04-20 01:31:40 +00:00
Lingqing Gan b8caf811c2
feat(backend): support secrets (#9178)
* support secret in driver

* add sample tests

* fix comments
2023-04-20 00:33:39 +00:00
Lingqing Gan ca63f20049
fix(backend): fix volume bug (#9150)
* fix volume bug

* fix indentation

* update driver image

* revert launcher image sha
2023-04-18 23:53:54 +00:00
gkcalat 276f922a68
feat(backend): Add nodeSelector support. Closes #8959. (#9132)
* Stage progress

* support pvc creation, deletion and mount

* replace image address for unit tests

* go mod tidy

* address comments

* Update go.mod

nit: fix format

* go mod tidy

* fix merge error

* update driver image and nit

* Rebase

* Add node selector to driver

* Fix input parameter's name

* Add a comment on container order

* Update driver and launcher images

* Update tests with hard-coded v2 images

* Address comments

* Fix test

* Update error message

* Update error message

---------

Co-authored-by: Linchin <lingqing.gan@gmail.com>
2023-04-18 20:08:00 +00:00
Linchin e0529fe26b chore(release): bumped version to 2.0.0-beta.1 2023-04-11 19:14:29 +00:00
Lingqing Gan 761a592b89
feat(backend): support pvc creation, deletion and mount; fixes #8708 (#9106)
* support pvc creation, deletion and mount

* replace image address for unit tests

* go mod tidy

* address comments

* Update go.mod

nit: fix format

* go mod tidy

* fix merge error

* update driver image and nit
2023-04-10 23:34:13 -07:00
gkcalat ce096dd3bc
fix(backend): Fix missing parameters in jobs and recurring runs. Closes #9012. (#9118)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Fix params in recurring run

* Add support for v1 and v2 parameters in runs and jobs

* Fix convertion issues

* Update a unit-test

* Revert change of the order in sql

* Address comments
2023-04-10 23:35:54 +00:00
gkcalat b5dae77f03
fix(backend): Set status when converting API recurring run to internal representation (#9128)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.
2023-04-10 11:06:53 +00:00