6dfaeebd92
fix(sdk.v2)!: Block task dependency referencing tasks inside a sibling condition or loop group. ( #7050 )
2021-12-13 19:08:37 +00:00
c1b67154a2
fix(sdk.v2): fix regression on nested loops ( #6990 )
...
* fix regression on nested loops
* release note
2021-12-01 22:34:05 +00:00
cd20373310
feat(sdk): add set_env_variable for Pipeline task ( #6919 )
...
* feat(sdk): add set_env_variable
* release notes
* enable test
* fix test
* fix test
* address comments
2021-11-23 21:46:47 +00:00
c4d8dfd8c6
feat(sdk): ImporterSpec v2 ( #6917 )
...
* importer
* compiler
* check diff
* fix format
* remove v1 node test
* remove importer v2 compat test
* release note
* address comments
2021-11-18 23:11:32 +00:00
7921946b07
feat(sdk.v2)!: Merge v2 experimental change back to v2 namespace ( #6890 )
...
* merge v2 experimental change back to v2 namespace
* release note
* fix tests
* fix cli
* fix py3.6 test
* skip v2 compatible mode tests
* fix tests
* skip v2 engine tests
2021-11-11 00:25:42 +00:00
334307d6c3
chore(sdk): add deprecation warnings for v2 ( #6851 )
2021-11-02 16:26:06 -07:00
0be57c3880
feat(sdk): Use google.protobuf.Value in v2 for passing parameters. ( #6804 )
...
* Use google.protobuf.Value in v2 for passing parameters.
* retest samples.
* Fix tests.
* Update release, more cleanup.
* Use github.com/kubeflow/pipelines/api from same repo.
* Run go mod tidy
* chore: go mod tidy
* fix v2 compile error and clean up unused code
* pr comments.
* update goldens
* Fix metadata recording.
* Update kfp mlmd client.
* fix test again
* another try.
* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings
* fix v2 launcher + clean up
* fix a compile error
* fix a few more tests
* fix number parsing
* clean up
* disable cache_v2 test.
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
2021-10-28 14:25:51 -07:00
b466f59e2f
fix(sdk.v2): Fix a couple of ParallelFor related bugs. Fixes #6383 , fixes #6628 ( #6643 )
...
* fix a couple of loop related bugs
* add release note
2021-09-29 11:27:09 -07:00
5d82fa7dc8
chore(sdk): v2 compiler to throw no task defined error ( #6545 )
...
* chore(sdk): v2 compiler to throw no task defined error
* update and clean up release notes
* Update RELEASE.md
* Update RELEASE.md
2021-09-14 01:13:32 -07:00
0fba85cfab
fix(sdk.v2): Support dict, list, bool typed input parameters from constant values and pipeline inputs. ( #6523 )
...
* fix dict list typed inputs support
* Support passing dict, list, bool typed parameter via client
* update release note
* fix test
2021-09-07 19:53:07 -07:00
2f19a26ffd
chore(sdk): Format all Python files under SDK folder. ( #6501 )
...
* Reformat sdk only using the new yapf config.
* Reformat docstrings using docformatter.
* update golden files to resolve diff caused by whitespaces
* fix some tests
* format .py files under sdk/python/tests using yapf
* additional docformatter
* fix some tests
2021-09-03 11:25:11 -07:00
cfefc6d397
feat(sdk.v2): Support `set_display_name` in v2. ( #6471 )
...
* Support `set_display_name` in v2.
* update comment in proto
* fix test
2021-08-30 12:07:25 -07:00
f3f383c2ff
chore(sdk): Refactor and move all v2 related code to under the v2 namespace. ( #6358 )
...
* Refactor and move all v2 related code to under the v2 namespace.
Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.
Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.
* Update setup.py.
* update tests.
* revert accidental change of gcpc
* Fix component entrypoint.
* Update goldens.
* fix tests.
* fix merge conflict.
* revert gcpc change.
* fix tests.
* fix tests.
* Add type aliases for moved files.
* merge and update goldens.
2021-08-17 19:25:37 -07:00
c6cb8acf7a
feat(sdk): Add interface for enable_caching at task level. ( #6007 )
...
* feat(sdk): Add pipeline level caching options
* Update golden
* fixing caching options
* fix tests
* add a caching disabled case
Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-07-13 17:33:58 -07:00
a1740b4dfa
fix(sdk compiler):Fix v2 compile for exit handler Fixes #5854 ( #5899 )
...
* punch the hole to exit handler
* add pipeline parameter to test data
2021-06-22 02:21:54 -07:00
6b87155a33
feat(sdk.v2): Support Exit handler in v2 compiler. ( #5784 )
2021-06-03 12:58:36 -07:00
ee77fd86ec
fix(sdk.v2): fix metrics outputs populating to unblock pipeline run ( #5770 )
2021-06-02 13:18:35 -07:00
2991602cfc
feat(sdk): Better error message when compiling for v2. Fixes #5688 , Fixes #5727 ( #5759 )
...
* Better error message when compiling for v2
* update samples
2021-05-31 13:39:03 -07:00
cc83e1089b
Assigned copyright to the project authors ( #5587 )
2021-05-05 13:53:22 +08:00
d5a1853857
feat(sdk.v2): Raise error on artifacts vs. parameters mismatch in v2. ( #5557 )
...
* error out on passing artifact to parameter or vice versa.
* error out on missing type annotation for pipeline arguments.
* address test failures
2021-04-29 03:17:29 -07:00
5c08cf91e2
chore(sdk.v2): block recursive subgroup in v2. ( #5551 )
2021-04-28 18:51:29 -07:00
3988706b7d
chore(sdk.v2): Sort compilation output json to make it deterministic. ( #5538 )
2021-04-24 19:50:44 -07:00
a2d3f234ea
chore(sdk.v2): Remove warning on missing pipeline root at compile time ( #5527 )
2021-04-22 16:48:43 -07:00
74950d620f
fix(sdk.v2): Fix loop subvar support and remove redundant loop iterator component. ( #5516 )
...
* fix loop subvar
* remove loop intermediate component
* fix subvar
* improve condition operand type handling
2021-04-22 15:13:42 -07:00
5dafda1270
feat(sdk.v2): Support explicit importer ( #5502 )
2021-04-19 13:14:09 -07:00
5f97a4502d
feat(sdk.v2): remove pipeline_root from compiler interface ( #5492 )
2021-04-16 08:49:13 -07:00
85cb99173d
feat(sdk.v2): fix loop implementation and add subvar support ( #5461 )
...
* fix loop dag and add subvar support
* Bump kfp-pipeline-spec min version to 0.1.7
2021-04-12 16:59:03 -07:00
f32915e88f
feat(sdk.v2): Support setting custom job spec in v2. ( #5454 )
...
* Support setting custom job spec.
* auto fill custom job jobSpec using container spec.
* fix test
* address comments
2021-04-10 09:20:04 -07:00
a80421191d
feat(sdk.v2): surface metrics output artifacts to pipeline outputs ( #5445 )
...
* populates metrics outputs in root
* avoid kfp import in io_types.py
* populates outputs following the right path
2021-04-09 16:37:04 -07:00
0e921c3083
feat(sdk.v2): use consistent argument name in v2 compiler ( #5430 )
2021-04-06 09:05:01 -07:00
eb558423ec
fix(sdk.v2): fix bug for component input parameter/artifact referencing ( #5419 )
2021-04-01 22:10:20 -07:00
eb3a092b64
feat(sdk.v2): Support optional component inputs in v2. ( #5383 )
...
* support optional inputs
* fix typo
* update chicago taxi dataset component
* fix executor
2021-03-31 01:16:18 -07:00
eabc10f364
chore(sdk.v2): fix runtime config wrong parameter type. ( #5389 )
2021-03-30 15:10:17 -07:00
4c51b576f5
chore(sdk.v2): fix a couple of issues related to cross-DAG i/o referencing and more. ( #5382 )
...
* fix a couple of issues related to cross-DAG i/o
* address review comments
2021-03-29 19:12:17 -07:00
06f0ee51d0
chore(sdk.v2): Do not inject component input for loop arguments from component output ( #5331 )
2021-03-18 14:36:16 -07:00
d7eb168347
chore(sdk): merge v2 dsl code back to v1. ( #5227 )
...
* Merge v2 dsl back to v1 dsl
* fix tests
* fix tests and always attach container_spec to container_op
* address review comments
* fix a bad indentation
2021-03-04 13:48:48 -08:00
280cae3718
feat(sdk.v2): Support dsl.ParallelFor in v2. ( #5162 )
2021-02-25 05:12:15 -08:00
ec7201db5a
feat(sdk.v2): Implement AI Platform (Unified) Custom job component in KFP v2 SDK ( #5146 )
...
* code skeleton
* wip
* add param scaffold
* add param scaffold
* wip: add new DSL class for AiPlatformCustomJob
* wip: play with container op i/o
* wip
* wip
* enhance get_parameter_type
* add tests
* add get_ir_type and tests
* add get_ir_type and tests
* wip: implementing get_custom_job, without IO
* add op IO
* clean up
* fix issues in aiplatform module
* cmd/arg resolution
* fix
* fix nonetyped cmds
* revert _structures
* revert component_tests
* revert component_bridge
* add basic tests
* add an e2e test
* fix tests
* resolve comments.
* fix test
* fix test
2021-02-24 17:18:15 -08:00
a2e2e6ca6f
feat(sdk.v2): support dsl.Condition in v2. ( #5127 )
...
* support condition in sdk.v2
* address review comments
2021-02-12 15:05:57 -08:00
051a022937
feat(sdk.v2): Allow set pipeline_root via @dsl.pipeline decorator. Make pipeline_root optional. ( #5107 )
...
* Allow set pipeline_root via @dsl.pipeline decorator.
* test covering pipeline_root not set
2021-02-07 02:06:32 -08:00
7c50708aad
feat(sdk.v2): Update v2 compiler to support v2 component. ( #5104 )
...
* update compiler
* lint
* fix test
2021-02-05 17:40:37 -08:00
91c5a93c8e
chore(sdk.v2): Migrate to the new IR with subdag support ( #4981 )
...
* migrate to new ir
* address review comments
* fix bugs
* fix pipeline parameters and tests
* fix components import
* fix typo
2021-02-03 04:11:37 -08:00
ecb14f40bb
chore(sdk): Remove v2 components fork, use v1 instead. ( #5042 )
...
* Remove v2.components fork
* fix setup.py
2021-01-28 18:20:07 -08:00
0c600a2cb8
fix(sdk.v2): Fix the bug where pipeline_parameters is not used. ( #5002 )
...
* fix bug on pipeline parameters not being used.
* address review comments
2021-01-19 07:59:45 -08:00
03904ebacd
chore(sdk.v2): Error out on invalid pipeline name ( #4851 )
2020-12-01 14:36:51 -08:00
6b8bd4c7ab
chore(sdk.v2): Support explicit task dependency ( #4824 )
2020-11-24 12:10:40 -08:00
7cf620323f
chore(sdk): Use pipeline_spec proto from kfp-pipeline-spec package ( #4745 )
...
* Use pipeline_spec proto from kfp-pipeline-spec package
* update imports
* use pre-released kfp-pipeline-spec (temporarily)
* Revert "use pre-released kfp-pipeline-spec (temporarily)"
This reverts commit 77f2e9a39c
2020-11-10 16:14:41 -08:00
a51380b065
chore(sdk): Add InputUri and, OutputUris placeholders in v2 ( #4728 )
...
* Fix bug where we missed injecting importer node
* moved files
* address review comments
* Add InputUriPlaceholder and OutputUriPlaceholder
* support uri placholder in v2
* lint
* test
* Preserve a test case with inputPath and outputPath usage.
* fix ut
* fix import and setup
* address comments
2020-11-09 11:37:58 -08:00
82b1ed4d17
chore(sdk): v2 compiler compiles to pipeline job spec ( #4729 )
...
* fix proto
* compile to pipeline job spec
* make pipeline_root required
* address comments
2020-11-06 15:05:38 -08:00
5d15b19b47
chore(sdk): Ensure all pipeline parameters present in runtime_parameters mapping. ( #4722 )
...
* Fix bug that pipline parameter missing from runtime parameter list
* fix test data
* refactor and tests
2020-11-06 01:56:42 -08:00
Chen Sun
Yaqi Ji
Ajay Gopinathan
Trs
Alexey Volkov
Jiaxiao Zheng