Commit Graph

26 Commits

Author SHA1 Message Date
Connor McCarthy 78902274f1
feat(sdk): support compiling components with `dsl.PipelineTaskFinalStatus` type (#9082) 2023-05-12 22:14:28 +00:00
Connor McCarthy c01288d673
feat(sdk): support writing `TASK_FINAL_STATUS` parameter type (#9080)
* write new parameter type

* update golden snapshots

* add compiler tests
2023-05-12 17:13:32 +00:00
Chen Sun b6b8f94da2
chore: recompile/update v2 samples to avoid possible runtime breakage (#9333)
* recompile/update v2 samples

* fix test
2023-05-05 17:28:24 +00:00
Connor McCarthy 946c51bafe
feat(sdk): support extracting input/output descriptions from component/pipeline docstrings (#9156)
* feat(sdk): support extracting input/output descriptions from component/pipeline docstrings

* remove numpydoc tests
2023-04-24 17:16:44 +00:00
Connor McCarthy 91abbeaf2f
feat(sdk): support `display_name` and `description` in `@dsl.pipeline` decorator (#9153)
* feat(sdk): support display_name and description in @dsl.pipeline decorator

* add release note

* test additional case
2023-04-17 14:15:39 -07:00
Connor McCarthy 80bd8f17b1
chore(sdk): bump kfp version in test data to 2.0.0-beta.14 (#9145) 2023-04-12 16:27:09 -07:00
Chen Sun a6ef77b784
feat(sdk): Support setting cpu/memory requests. (#9121)
* Support setting cpu/memory requests.

* address review comments

* address review comments
2023-04-10 23:35:16 -07:00
Connor McCarthy e96754cb53
feat(sdk): deprecate .add_node_selector_constraint in favor of .set_accelerator_type (#8980)
* rename method

* update docstring
2023-03-14 09:22:35 -07:00
Chen Sun 8d3e0dd805
fix(sdk): Fix environment variable set in component yaml lost during compilation. Fixes #8884 (#8885)
* Fix env set in component yaml not captured

* release note

* clean up unnecessary code in test
2023-02-23 14:28:26 -08:00
Connor McCarthy 31298c479b
fix(sdk): fix bug when passing data between tasks using f-strings (#8879) 2023-02-23 08:36:53 -08:00
Connor McCarthy c046879dda
support fanning-in artifacts (#8808) 2023-02-22 10:39:00 -08:00
JOCSTAA 1e1a02066a
fix(sdk): Add outputDefinitions of root pipeline to compiled components IR YAML file (#8848)
* base

* add to readme

* adress comments
2023-02-15 09:28:33 -08:00
JOCSTAA 5684bfccff
feat(sdk): Ignore upstream failure (#8838)
* main

* adress comments

* adress comments

* adress comments 2

* address comments 3

* address comments 4
2023-02-13 11:40:42 -08:00
Connor McCarthy e7ad073f1d
feat(sdk): deprecate .set_gpu_limit in favor of .set_accelerator_limit (#8836) 2023-02-09 15:46:10 -08:00
Connor McCarthy b575950a75
feat(sdk): support for fan-in from `dsl.ParallelFor` (#8631)
* implement parallelfor fan-in

* temporarily block fan-in of artifacts

* clean up

* unify and clarify error messages

* remove comment

* remove unnecessary compiler condition

* make test cases more realistic

* use better ParallelFor variable name

* add and rename test cases

* add ParallelFor fan-in test cases

* fix return collected bug

* clean up compilation tests

* update docstrings; update error message

* remove fan-in artifacts read/write tests

* implement review feedback

* change use of additional_input_name_for_pipeline_channel

* move additional_input_name_for_pipeline_channel

* provide additional pipeline topology validation and tests; refactor existing validation

* add release note for parameter fan-in
2023-02-04 20:50:51 -08:00
Connor McCarthy 151e8a3c6e
feat(sdk): support optional artifact inputs (#8623)
* add compiler tests

* support optional artifact inputs

* update executor test

* update existing golden snapshot

* add new compiler test/snapshot

* add release note

* bump kfp-pipeline-spec dependency
2022-12-28 13:40:12 -05:00
Connor McCarthy 5491f6f3db
fix(sdk): support writing/reading isOptional for input parameters (#8612)
* add tests

* support writing/reading isOptional for parameters

* update golden snapshots

* temporarily maintain workaround for v1 optional artifact inputs

* add release note
2022-12-21 14:46:59 -08:00
Connor McCarthy 47eb1cad92
chore(sdk): update golden snapshots with pipeline interface comments (#8575)
* update path in readme

* update golden snapshots
2022-12-15 01:32:21 +00:00
Connor McCarthy b829e4a5da
feat(sdk): enable use of primitive placeholders in f-string (#8494)
* support primitive placeholders in fstring

* add read/write test component
2022-11-28 14:45:48 -08:00
Connor McCarthy 9e9e1081c6
fix(sdk): fix boolean default value serialization bug (#8444)
* add test cases

* fix bug

* add v1 component yaml back compat changes

* update golden snapshot
2022-11-11 15:42:33 -08:00
Connor McCarthy 24e17f7352
chore(sdk): quiet noisy SDK unit test output (#8434) 2022-11-09 17:47:39 +00:00
Connor McCarthy c353245baf
feat(sdk): enable dynamic importer metadata (#7660)
* support placeholder in metadata dict

* add unit tests

* add compilation test

* fix tests

* update compiler test

* use 'metadata' as input key

* update compiler test and golden snapshot

* add int input to compiler test

* change constant name

* use util function

* support f-strings in dynamic import metadata; add and update tests

* update method name after rebase

* make compile consistent across python versions
2022-09-19 21:58:31 +00:00
Connor McCarthy 166d6bb917
feat(sdk): add runtime logic for custom artifact types (support for custom artifact types pt. 3) (#8233)
* add runtime artifact instance creation logic

* refactor executor

* add executor tests

* add custom artifact type import handling and tests

* fix artifact class construction

* fix custom artifact type in tests

* add typing extensions dependency for all python versions

* use mock google namespace artifact for tests

* remove print statement

* update google artifact golden snapshot

* resolve some review feedback

* remove handling for OutputPath and InputPath custom artifact types; update function names and tests

* clarify named tuple tests

* update executor tests

* add artifact return and named tuple support; refactor; clean tests

* implement review feedback; clean up artifact names

* move test method
2022-09-15 01:00:40 +00:00
Scott_Xu 5dff29a071
fix(sdk): keeping placeholder strings in command and args even if default value is set (#8259)
* return string representation of placeholder in pipeline_task

* return string representation of placeholder in pipeline_task
2022-09-13 23:44:31 +00:00
Connor McCarthy e371fa3795
feat(sdk): add compilation logic for google artifact types (support custom artifact types pt. 2) (#8232)
* remove dependency of deprecated on v2

* use new artifact identification logic

* add custom artifact type compiler test

* compile schema_version from non-kfp artifacts

* update tests

* add importer to compiler test

* update golden snapshots

* fix type checking code

* undo executor_test diff

* remove gcpc types compiler test

* add compilation test comment

* update google artifact handling

* update compiler test

* only permit system. and google. artifact types; add tests

* update compiler test

* simplify type checking logic

* use bundled artifact types in PipelineArtifactChannel

* raise error on unknown v1 input or output

* add type utils and move functions

* add type utils tests

* remove schema_version from inputspec and outputspec

* update artifact proto creation and tests

* propogate bundled artifact type string changes throughout sdk

* update golden snapshot

* remove unused v1 compat and tests

* update type checking and tests to validate schema_version using semantic versioning

* update importer sample

* update test data config

* update test_data config

* clean up sample

* clean up sample
2022-09-12 14:00:02 -06:00
Connor McCarthy 16c7d6658e
test(sdk): create SDK execution test suite and restructure read/write tests (#8245)
* update client for graph components

* use yaml for config

* simplify read write tests

* add test script

* restructure yaml and read/write tests

* add execution test file

* move test_data; update paths; update readme

* update kfp endpoint

* update pipeline with env

* update logging

* update config

* update kfp endpoint and timeout

* add kfp as requirement

* mock kfp_package_path for lightweight component execution

* revert pipeline name changes

* update config with name

* update read write tests to use explicit name

* update execution tests to use explicit name
2022-09-12 18:33:06 +00:00