Commit Graph

485 Commits

Author SHA1 Message Date
Connor McCarthy 2009dfb2f1
test(sdk): add `kfp-kubernetes` execution tests (#10304) 2023-12-15 22:31:54 +00:00
Connor McCarthy 2edfb8965d
install kfp-pipeline-spec from source for kfp tests (#10300) 2023-12-12 18:05:48 -05:00
Connor McCarthy 8d00d0eb9a
feat(sdk): support a Pythonic artifact authoring style (#9932)
* chore(sdk): test observability, refactorings, and cleanup

* implement unified i/o syntax

* clarify container component error message

* address review feedback

* add dill to packages_to_install

* update get_uri logic for KFP and tests

* update execution tests & cleanup
2023-10-19 18:49:58 +00:00
Lingqing Gan d3a15d9230
chore: Update OWNERS (#10064)
* Update OWNERS

Remove Linchin from approvers and reviewers

* remove Linchin in OWNERS files

---------

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2023-10-05 18:23:50 -07:00
Chen Sun a13e143038
test: fix sdk execution test (#9988)
* fix sdk execution test

* fix missing key

* fix state

* fix state

* fix capitalization
2023-09-14 20:02:30 +00:00
Connor McCarthy cf0e0cf87c
feat(sdk): enable dependency-free runtime install of kfp (#9886)
* handle problematic imports

* separate runtime and compile-time symbols

* wrap kfp/__init__.py imports unavailable at runtime

* update component factory + tests

* add runtime tests

* add --no-deps flag to component factory

* update release notes

* clean up

* handle containerized python components

* update golden snapshots

* update component_factory unit tests

* respond to review feedback

* fix runtime test and compilation logic

* update tests
2023-08-24 18:31:16 +00:00
James Liu eaec515812
chore(mlmd): Upgrade ML Metadata to 1.14.0. (#9856)
* chore(mlmd): Upgrade ML Metadata to 1.14.0.

* fix requirements, skip visualization.

* go mod tidy

* mlmd new enum in frontend

* mlmd license

* mlmd license again, using the failure test diff

* again
2023-08-22 17:18:09 +00:00
Chen Sun cd7e316db6
test: upgrade e2e sample tests to using sdk v2 (#9885)
* install kfp sdk 2.0 in sample test

* remove reference to GCSHelper from deprecated module

* exclude obsolete samples

* update test script

* update client usage

* revert recent changes to exit_handler.py
2023-08-17 09:33:38 +00:00
Connor McCarthy c9e54798cd
chore(sdk): undo creation of kfp-dsl namespace package (#9874)
* revert to c8204d0

* reapply #9742

* modify #9791

* reapply #9800

* reapply #9827

* revert parts of #9738

* reapply parts of #9785

* remove duplicated dsl-test code

* reapply parts of #9791

* correct version
2023-08-15 16:24:52 -07:00
Diana Atanasova 7b4ddf658c
test: enabe back archiving experiment action in kfp_functional_test (#9748)
* re-enabe archiving experiment action in kfp_functional_test

Signed-off-by: diana <difince@gmail.com>

* Use kfp.v2 instead of kfp.depricated

Signed-off-by: diana <difince@gmail.com>

* Update periodic kfp_functional_test to work with kfp=2.0.1

Signed-off-by: diana <difince@gmail.com>

* Updata dependencies and clean up instructions

---------

Signed-off-by: diana <difince@gmail.com>
Co-authored-by: gkcalat <gkcalat@ucla.edu>
2023-08-15 10:07:16 -07:00
Lingqing Gan 2f32b23c1e
chore(test): change hard-coded GKE version to 1.25 (#9841)
* update gke version

* try stable channel

* use 1.25.10-gke.2100

* use 1.25
2023-08-09 23:48:03 +00:00
Joe Li 443a67438a
test(frontend): Improve v1 frontend-integration-test (#9811)
* test: access specific field via "id".

* test: access specific field via "id".

* update snapshots.

* Add test to validata description changes.

* Change id def for input tag.

* Skip unit tests

* Add unit tests back and change id

* Add a helper for clear default value in input field.
Remove unnecessary 'tab' command

* Add comment to explain the parameter name for tests
Remove default value for runName before typing new value
2023-08-08 21:16:33 +00:00
Connor McCarthy 37b907f2a4
test(sdk): add KFP SDK upgrade presubmit test (#9827) 2023-08-08 02:46:55 +00:00
Connor McCarthy 540294aedb
fix(sdk): fix GCPC break in KFP SDK (#9791) 2023-07-27 23:02:32 +00:00
Connor McCarthy 8ad97167ff
fix(sdk): various kfp-dsl fixes (#9785)
* specify kfp-dsl version directly

* make presubmit test executable

* add yaml requirement

* assorted test and import fixes
2023-07-26 22:41:51 +00:00
Connor McCarthy 525ff90684
chore(sdk): extract DSL into `kfp-dsl` package (#9738)
* move dsl dir

* make kfp-dsl a package

* make kfp-dsl a package

* additional changes

* address review feedback
2023-07-25 23:34:02 +00:00
gkcalat d9a3eca05e
chose(test): Set image-type to cos_containerd and hard-code GKE version to 1.25 in tests (#9743)
* Set image-type to cos_containerd in tests

Reverting the previous change #9705 which removed the `--image-type`.

/hold

Check if this resolves [e2e test failures](https://oss.gprow.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/9730/kubeflow-pipeline-e2e-test/1680988618279096320#1:build-log.txt%3A2) in #9730

* Update deploy-cluster.sh

* Set cluster-version to 1.25.10-gke.1200
2023-07-18 01:42:21 +00:00
Chen Sun 7e89d05b19
test: Use GKE stable release channel to unblock presubmit tests. Fixes #9704. Part of #9706. (#9705)
* Use GKE stable release channel to unblock presubmit tests

* update TODO comment
2023-07-05 22:28:51 +00:00
Chen Sun f3ce0151f0
test: pin an older golang image in building presubmit test images. Fixes #9650 (#9648)
* use older base image

* add comments
2023-06-17 07:10:37 +00:00
Chen Sun 90ed23d183
chore: exclude `sdk-` tags when preparing base deployment for upgrade test (#9477)
* exclude `sdk-` tags when preparing base deployment for upgrade test

* Update deploy-pipeline-lite.sh

* Update deploy-pipeline-lite.sh

* Update upgrade-tests.sh
2023-05-23 00:08:35 +00:00
Chen Sun 931a68096e
chore: upgrade to Argo v3.3.10 (#9336)
* pull argo v3.3.10 upstream

* upgrade to Argo v3.3.10

* fix validateworkflow

* upgrade golang

* update golang

* fix go dep

* update go licenses file

* fix build

* fix go dep

* downgrade k8s.io/client-go to v0.24.3

* update licenses

* fix go dep

* update licenses
2023-05-08 08:38:55 +00:00
Joe Li 59db2835fa
fix(frontend): Parameter field is missing when creating run from run list (#9335)
* Redirect to v2 new run after selecting v2 pipeline when creating run

* Add a tab in FE-integration test

* Add tab to the failed tests

* Add query keys back to NewRunSwitcher
2023-05-05 21:40:52 +00:00
Chen Sun e115e0c01f
[WIP] test: Add v2 upgrade test (#9287)
* Add v2 upgrade test

* demo recurring run missing runtime config

* Revert "demo recurring run missing runtime config"

This reverts commit 58154c5548.

* add v2 entries in upgrade  test script

* address review comments
2023-05-03 21:56:45 +00:00
Chen Sun 8b2fba3f39
test: Add v2 API integration tests (#9185)
* Add v2 API integration tests

* move v1 client
2023-04-27 20:20:39 +00:00
Chen Sun 8645cac9a6
fix(release): Update release image with the latest api-generator (#9198)
* Update release image with the latest api-generator

* Add note to keep api-generator image in sync between dev and release
2023-04-21 16:34:41 +00:00
gkcalat f626629f79
chore(testing): Enable samples in e2e tests with v1 kfp SDK (#9169)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.

* Enable sample tests
2023-04-19 04:55:53 +00:00
gkcalat 5aa2a58d5a
chore(tests): Disable samples in e2e testing (#9164)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.

* Replace pipeline job id with workflow.uid

* Disable samples in e2e test

* Fix typo
2023-04-18 19:07:53 +00:00
Joe Li ab2499dd27
chore(frontend): Integrate pipeline / pipeline version functions with KFP v2 API (#9123)
* Separate pipelineServiceApiV1 and pipelineServiceApiV2.
Add uploadPipelineVersionV2 helper.

* Ingegrate new pipeline functionality with v2 API.
Create a new pipeline dialog for v2 pipeline because the error is
not string type.

* Integrate getPipelineVersion() in run/recurring run details router
with v2 API.

* Integrate listPipelines() / listPipelineVersions() in PipelineList and
PipelineVersionList with v2 API.

* Update private and shared pipelines unit tests.

* Separate v1 and v2 logics (based on API return value) in pipeline
details (router).
Noted: add additional listPipelines() to get latest version because v2
don't default version field.

* Change pipeline props in PipelineDetailsV2 and PipelineVersionCard
Remove default version in PipelineVersionCard test.
Append version id to URL for unspecified versionId scenario.

* Create new ResourceSelectorV2 and make corresponding changes (back to
v2beta1Pipeline) in PipelineDialogV2 and NewPipelineVersion

* Create resource converter to make various API returned objects fit in
base resource.
Change the behavior of "selectionChanged" to achive more general usage.

* Integrate getPipeline() / getPipelineVersion() in "create / clone run"
with v2 API.

* Integrate get started page with v2 pipeline API.

* Integrate getPipelineVersion() in run list with v2 API.

* Integrate delete pipeline / pipeline version with v2 API.

* Remove unused imports. Change relative path to absolute path.

* Add unit test for PipelineDialogV2 and ResourceSelectorV2.

* Fix incorrect Version Description in PipelineVersionCard and rename the
constant in unit test.
Add size limitation in the listPipelineVersions() called by
getSelectedVersion.
Add missing props in NewRunV2 test.
Fix FE-integration test (trial)

* Simplify the logic of deletePipelineVersion. Remove
dialoghandlerMultiId.
2023-04-09 00:04:52 +00:00
Chen Sun 7079164135
test: Update frontend-integration-test image (#9028) 2023-03-23 00:29:20 +00:00
Connor McCarthy 540342af41
chore: add kfp-kubernetes docs and update process infrastructure (#8976)
* add docs and release infrastructure; update test script

* implement review feedback
2023-03-15 18:23:12 +00:00
Chen Sun dabc02724f
test: Adding v2 initialization tests (#8978)
* Adding v2 initialization tests

* remove unused (yet) test utils

* small cleanup

* remove used code

* fix build

* address review comments
2023-03-15 00:24:09 +00:00
Connor McCarthy 40cb409e1c
chore: rename storage_class to storage_class_name (#8973) 2023-03-11 06:14:21 +00:00
Connor McCarthy 0c11cce06d
feat: implement `kfp-kubernetes` library (#8950)
* support compiling platform-specific features

* add kubernetes functions/components
2023-03-10 22:54:20 +00:00
Chen Sun 5fe67919db
chore(components): clean up deprecated GCP components (#8685)
* delete deprecated GCP components

* remove build and test scripts, references to code/folder from master
2023-02-14 22:42:33 +00:00
Chen Sun 8552226c41
chore(test): revert api changes to deprecated client (#8841)
* revert api changes to deprecated client

* list experiments

* fix

* fix sample v2 test
2023-02-14 11:01:58 -08:00
Chen Sun fd3f694d9a
test: install compatible `kfp-server-api` and `kfp` packages (#8829)
* don't install kfp-server-api and kfp both from HEAD

* install compatible kfp-server-api and kfp
2023-02-09 20:30:01 +00:00
Chen Sun eb5702ab39
chore(test): create test cluster in us-west1-b (#8821) 2023-02-07 11:53:56 -08:00
gkcalat e5aa67c61f
fix(release): Add v1 and v2 proto generation (#8819) 2023-02-07 11:52:55 -08:00
Chen Sun fe71e17a4d
test(frontend): upgrade frontend integration test dependencies and base image. (#8806)
* upgrade frontend integration test dependencies and base image.

* remote unused libraries

* fix junit config

* fix junit config
2023-02-07 01:24:39 +00:00
gkcalat 135dc64854
feat(backend): Refactor apiserver, add Pipeline and other v2beta services (#8633)
* Staging WIP

* Fix recurring run, switch to pipeline_version_id

* Regenerate python client

* Refactor packages for pipeline service v2

* Stage progress

* Add support for pipeline service v2beta1

* Go mod tidy

* Update licenses

* Set default namespace to be empty

* Separate deletion of pipeline versions

* Change report ReportRunMetrics to obey AIP

* Use pipeline version id in run proto

* Update run.proto

* Upgrade storage

* Remove dependencies on api

* Stage refactoring progress

* Refactor apiserver

* Update v1beta1 unit test

* Update dependencies

* Fix integration test

* Revert visualization server changes (workaround)

* Address comments

* Basic lint the api server

* Update licenses file

* Adjust intergtaion tests

* Fix typos in the tests

* Update tests

* Update test

* Add support v1 delepe pipeline

* Remove gorm.Model to split into a different PR

* Lint api server

* Fix default experiment in upgrade test

* Update storage states

* Testing upgrade test run

* Add copying pod logs to the GCS bucket

* Fix upgrade test

* Adjust backend test for list

* Fix the tests

* Fix workflow manifest discrepancy at create

* Add a few unit-tests

* Add misc service and fix persistence agent

* Remove default values from TEXT columns

* Address comments. Fixes #8702.

* Address comment, reorder resource, server, storage

* Fix front end integration test

* Fix FE integration test

* Fix tabs, add pauses, make more explicit FE test

* Revert FE test changes

* Enable default experiment and disable its deletion

* Replace empty namespace with POD_NAMESPACE

* Default experiment is now the oldest

* Emplace empty with POD_NAMESPACE for argo only

* Fix api integration test

* Revert changes to default experiment's description

* Improve FE test flakiness

* Force refresh in the FE test

* Fix FE integration test

* Add delay in FE test

* Add local debug to FE test

* Update tests

* Fix typo

* Address comments

* Enforce template version verification

* Fix workflow manifest discrepancy with v1

* Fix FE test

* Revert changes in persistence agent

* Limit to one resource owner

* Change default run state to unknown

* Revert changes to FE integration test

* Make states consistent

* Remove links from error messages

* Address comments

* Address comments

* Change the order for teardown

---------

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2023-02-03 20:17:01 +00:00
gkcalat c79d35f53b
fix(test): Add frontend integration test manual run (#8767)
* Fix bug in the front-end test

* Add local debugging to the FE test

* Save testing artifacts to GCS

* Address comments
2023-01-31 06:49:30 +00:00
Connor McCarthy 78d31dc8a3
chore(sdk): remove v1 dependencies from SDK v2 (#8668)
* split out deprecated requirements

* pip install deprecated requirements for sample tests

* remove superfluous comments in requirements.in

* update release notes

* ignore deprecated directory in unit tests

* remove unused argo compiler

* ignore deprecated path

* update run sample yaml

* install deprecated requirements for e2e tests

* add todos
2023-01-19 12:53:27 -08:00
dependabot[bot] 3fa2ac5f4e
chore(deps): bump jupyter-core from 4.9.1 to 4.11.2 in /test/sample-test (#8442)
Bumps [jupyter-core](https://github.com/jupyter/jupyter_core) from 4.9.1 to 4.11.2.
- [Release notes](https://github.com/jupyter/jupyter_core/releases)
- [Changelog](https://github.com/jupyter/jupyter_core/blob/main/CHANGELOG.md)
- [Commits](https://github.com/jupyter/jupyter_core/compare/4.9.1...4.11.2)

---
updated-dependencies:
- dependency-name: jupyter-core
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-09 14:07:10 -08:00
Chen Sun cf3175ef4e
chore: address kubectl dry-run deprecation (#8643) 2023-01-03 23:08:24 +00:00
dependabot[bot] 3c3bb1fdcd
chore(deps): bump qs from 6.5.2 to 6.5.3 in /test/frontend-integration-test (#8618)
Bumps [qs](https://github.com/ljharb/qs) from 6.5.2 to 6.5.3.
- [Release notes](https://github.com/ljharb/qs/releases)
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/qs/compare/v6.5.2...v6.5.3)

---
updated-dependencies:
- dependency-name: qs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-30 11:25:27 +00:00
dependabot[bot] d31026d10d
chore(deps): bump decode-uri-component from 0.2.0 to 0.2.2 in /test/frontend-integration-test (#8617)
Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases)
- [Commits](https://github.com/SamVerschueren/decode-uri-component/compare/v0.2.0...v0.2.2)

---
updated-dependencies:
- dependency-name: decode-uri-component
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-30 11:25:20 +00:00
dependabot[bot] 11e4d094a9
chore(deps): bump wheel from 0.36.1 to 0.38.1 in /test/kfp-functional-test (#8635)
Bumps [wheel](https://github.com/pypa/wheel) from 0.36.1 to 0.38.1.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.36.1...0.38.1)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-30 00:35:20 +00:00
JOCSTAA 49db63c916
feat(sdk): Add comments to IR YAML file (#8467)
* base

* add tests

* fix bug

* adress comments

* address comments 2

* sort comments

* sort signatures

* add indempotent test

* add indempotent test2

* support multiline docstring

* review

* docformatter presubmit exclude

* docformatter presubmit exclude

* docformatter

* docformatter

* merge 1

* update readme

* nit .items()

* remove reduntant test
2022-12-05 14:43:14 -08:00
Connor McCarthy 834bfbe90a
fix(sdk): revert premature client changes (#8507)
* revert _v1 client changes

* do not install kfp_server_api from source
2022-11-29 20:39:23 -08:00
Lingqing Gan 3f2e20d14f
chore(backend): regenerate v1beta1 api clients (#8403)
* add v2 experiment.proto

* regenerate v1beta1 client

* remove api version from combined swagger file

* Revert "regenerate v1beta1 client"

This reverts commit 02769d779b.

* regenerate v1beta1 clients

* regenerate python packages without api version

* remove regenerated v1 python packages&change proto

* refactor v1 exp apis and functions

* remove v2 api package and refactor v1 proto

* regenerate v1 go clients

* regenerate v1 python package

* refactor and update the human maintained swagger

* renew swagger file

* regenerate v1 packages

* regenerate v1 python package

* refactor

* regenerate v1 go clients

* regenerate v1 python package

* recover dockerfile and add s in python

* fix func name

* use commit for sdk execution test

* change sdk execution test

* update test script

* add v1 to methods

* proto update v1 method names

* regenerate v1 go packages

* regenerate python client

* update server methods

* refactor

* refactor

* refactor

* refactor python client

* fix get_healthz

* update sample-test.sh in v2 backend

* sample test script

* change v2 sample test

* change package install order

* Revert "change package install order"

This reverts commit c24fcfbed9.

* use KFP server api from commit for v2 sample tests

* small fix
2022-11-10 19:18:00 +00:00