Commit Graph

23 Commits

Author SHA1 Message Date
Lingqing Gan 89bdac9340
feat(backend): support optional and default parameters. Fixes #8716 (#8765)
* support optional and default parameters

* add sample test

* change assert value

* update test

* update test

* fix sample test

* fix sample test

* fix sample test

* test fix

* change parameter logic

* test

* test

* change logic handling not provided parameter

* remove unnecessary code and add more sample case

* fix sample test bug

* fix sample test

* update logic

* change parameter logic

* update logic

* update logic handling when parameter not provided

* fix sample test

* add error message

* fix samples test

* samples test

* remove unused code
2023-03-02 17:47:04 +00:00
Yaqi Ji 04123280a4
fix(sdk): add default value for inputs (#7405)
* fix(sdk): add default value for inputs

* merge conflict

* release

* fix sample
2022-03-23 18:59:20 +00:00
Yaqi Ji b693909a08
feat(sdk)!: move v1 to deprecated folder (#7291)
* chore(sdk): move v1 to deprecated

* fix testsg

* fix testsg

* fix setup.py

* fix test

* s'

* fix tests

* fix test

* retore v2 test changes

* fix py3.6 test

* fix py3.6 test

* fix py3.6 import fallback error

* remove deprecated

* fix samples test

* sample test

* fix samples

* add readme

* restroe test

* python require

* remove path

* fix tests

* inteegration tests

* remove tfx tests for dependency with kfp v1

* fix e2e

* fix e2e

* fix integration tests

* fix sampe

* move client down

* change to kfp

* add import alias

* fix

* runid

* fix dsl

* only use kfp for function

* revert train_until_good

* tfx test

* kfp

* try import

* onprem
2022-03-04 00:52:59 +00:00
Yuan (Bob) Gong c7448c71df
test(v2): refactor sample test utils into a py package (#7105)
* test(v2): refactor sample test utils into a py package

* fix error
2021-12-23 11:10:01 +08:00
Yuan (Bob) Gong 713558cfb8
feat(v2): iterator and sub-DAG basic support (#6985)
* chore(samples): add loop_output_v2 sample

* feat(v2): iterator support

* refactor v2 backend compiler tests to golden files

* fix unit tests

* test(v2): verify loop_output_test MLMD data

* clean up

* fix

* fix2

* fix
2021-12-09 14:23:55 +00:00
Chen Sun 5bbb384cc7
test: clean up v2 compatible mode sample tests (#6943) 2021-11-23 02:08:06 +00:00
Chen Sun 2910d10cc3
fix(sdk.v2): Fix bug on executor input with default value. (#6937)
* fix bug on executor input

* release note

* fix test on empty message

* fix part of the test
2021-11-22 21:09:06 +00:00
Chen Sun 73804f8928
test(sample): reenable samples tests run with V2_ENGINE mode (#6932)
* reenable v2 engine

* fix

* fix

* fix

* log v2 engine test

* don't fill in default value at test util

* fix

* pass to job instead of spec
2021-11-20 18:18:35 +00:00
Ajay Gopinathan 0be57c3880
feat(sdk): Use google.protobuf.Value in v2 for passing parameters. (#6804)
* Use google.protobuf.Value in v2 for passing parameters.

* retest samples.

* Fix tests.

* Update release, more cleanup.

* Use github.com/kubeflow/pipelines/api from same repo.

* Run go mod tidy

* chore: go mod tidy

* fix v2 compile error and clean up unused code

* pr comments.

* update goldens

* Fix metadata recording.

* Update kfp mlmd client.

* fix test again

* another try.

* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings

* fix v2 launcher + clean up

* fix a compile error

* fix a few more tests

* fix number parsing

* clean up

* disable cache_v2 test.

Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
2021-10-28 14:25:51 -07:00
Ajay Gopinathan f3f383c2ff
chore(sdk): Refactor and move all v2 related code to under the v2 namespace. (#6358)
* Refactor and move all v2 related code to under the v2 namespace.

Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
  were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.

Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.

* Update setup.py.

* update tests.

* revert accidental change of gcpc

* Fix component entrypoint.

* Update goldens.

* fix tests.

* fix merge conflict.

* revert gcpc change.

* fix tests.

* fix tests.

* Add type aliases for moved files.

* merge and update goldens.
2021-08-17 19:25:37 -07:00
Yuan (Bob) Gong 674756c76f
chore(v2): support output metadata file (#6361)
* chore(v2): support output metadata file

* add one more sample
2021-08-17 03:56:42 -07:00
Yuan (Bob) Gong afd28c605d
test(v2): add more samples. Part of #6150 (#6353)
* test(v2): add all samples using parameter + artifact passing features to v2 engine

* fix some samples

* fix output file

* comment failing samples

* improve comments
2021-08-16 21:38:41 -07:00
Yuan (Bob) Gong ee663d9593
chore(v2): standardize MLMD data model. Fixes #5669 (#6054)
* chore(v2): standardize MLMD data model

* change context type to system namespace

* update sdk snapshots

* fix go v2 tests

* update

* update v2 compat snapshots

* fix all samples

* fix must specify pipeline root

* add artifact display name

* add UI rendering of new fields

* fix sample tests

* let ui read artifact and execution names consistently

* fix samples

* fix frontend tests

* fix sample test

* fix last sample

* address feedback
2021-07-19 22:26:15 -07:00
capri-xiyue 3f4bf8fc48
test(v2): added v2 compatible cache e2e test (#6004)
* added v2 compatible cache e2e test

* fixed typo

* fixed e2e test

* resued methods

* combine cache_v2 and two_step pipeline

* fixed sample v2 test

* fixed cache e2e test

* fixed e2e test

* enabled cache e2e test

* fixed typo

* fixed typo

* added enable caching info

* added debug logging

* fixed launcher bug

* removed debug logging
2021-07-15 00:25:48 -07:00
Yuan (Bob) Gong f3b369e153
fix: v2compat - handle parameter values with special characters stably. Fixes #5830 (#5973)
* fix(launcher): handle parameter values with special characters stably

* include new test case

* add tensorboard minio test case

* fix go unit tests

* update test golden

* address feedback

* fix tests
2021-07-08 22:02:23 -07:00
Chen Sun 3f45f78a3d
fix(sdk): Fix default value of component input not picked up. Fixes #5880. (#5903)
* Fix default value of component input not picked up

* fix v2 compat
2021-06-23 13:25:18 -07:00
Yuan (Bob) Gong 834a5cc80f
test(sdk/v2compat): reproduce v2 py component default value does not work (#5879)
* test(sdk/v2compat): reproduce v2 py component default value does not work

* fix

* fix
2021-06-18 08:43:15 -07:00
Yuan (Bob) Gong f1f99c01e6
chore(launcher): revert meadata to custom properties & artifact name -> event path (#5842)
* chore(launcher): revert meadataa to custom properties & artifact name -> event path

* fix

* fix2

* fix

* fix2

* fix test

* fix
2021-06-12 19:28:35 -07:00
Yuan (Bob) Gong 77c1de8152
fix(launcher): duplicate input artifacts logged (#5836)
* fix(launcher: input artifact logging when more than one

* fix

* fix

* disable presubmit v2 go test
2021-06-11 21:39:52 +08:00
Alexey Volkov cc83e1089b
Assigned copyright to the project authors (#5587) 2021-05-05 13:53:22 +08:00
Yuan (Bob) Gong 4abbdec8c9
test(v2/samples): verify sample mlmd state (#5486)
* test(v2/samples): verify sample mlmd state

* finish artifact verification and fix v2 test integration

* fix unrecognized argument error

* fix tests

* fix test failure

* fix test
2021-04-20 03:06:38 -07:00
Ajay Gopinathan 5651bd1efd
feat(v2): Support v2 components in KFP v2-compatible mode. (#5458)
* Support v2 components in KFP v2-compatible mode.

Adds support in v2-compatible mode for the following
* v2 lightweight function components
* support for passing ExecutorInput to component
* support for parsing ExecutorOutput from component

Also renamed several variables in RuntimeInfo as follow ups to previous
PR.

* Remove newly irrelevant variables in runtimeInfo

* update goldens.

* Renaming in runtimeInfo.

* Add v2 component pipeline tests and update goldens.

* create output directory for parameter files.

* Fix output schemas.

* Support concat placeholder.

* Use Path and MetadataPath

* update launcher.go too with new names.

* move functions around.

* add some TODOs, replace cmd placeholder.

* fix tests
2021-04-15 17:44:13 -07:00
Yuan (Bob) Gong 02dbcfa062
test: more samples in v2 sample test (#5431)
* test: set up sample test for many samples

* test: rm loop_* tests from v1 sample test, they are already covered in v2 sample test

* fix condition pipeline for basic e2e test

* remove condition from e2e test
2021-04-09 08:34:03 -07:00