Commit Graph

57 Commits

Author SHA1 Message Date
Connor McCarthy 21c5ffebb0
fix(sdk): fix bug where `dsl.OneOf` with multiple consumers cannot be compiled (#10452) 2024-02-08 23:51:37 +00:00
rickyxie0929 fe04a5a842
fix(sdk): fixes type issues for ParallelFor. Fixes #9366 (#10436)
* fix type issues

* fix type issue

* fix format

* fix failed test

* fix format

* delete comments

* resolve comments

* resolve comments

* resolve format

* resolve import

* move unnecessary file

* resolve compiler_test failures

* resolve comments

* remove unnecessary imports

* fix format sort

* fix nits

* add new compiled yaml file

* solve merge conflicts

* solve conflicts

* format

* sort

* resolve comments

* resolve comments
2024-02-08 18:34:37 +00:00
Connor McCarthy 11f60d813a
feat(sdk): support `.after()` referencing task in `ParallelFor` group (#10257) 2023-11-29 10:22:14 -05:00
Connor McCarthy c0adea9946
chore(sdk): add test for key error bug resolved in #10067 (#10128) 2023-10-19 23:49:03 +00:00
Connor McCarthy 8d00d0eb9a
feat(sdk): support a Pythonic artifact authoring style (#9932)
* chore(sdk): test observability, refactorings, and cleanup

* implement unified i/o syntax

* clarify container component error message

* address review feedback

* add dill to packages_to_install

* update get_uri logic for KFP and tests

* update execution tests & cleanup
2023-10-19 18:49:58 +00:00
Connor McCarthy 2d3171cbfe
feat(sdk): support collecting outputs from conditional branches using `dsl.OneOf` (#10067)
* support dsl.OneOf

* address review feedback

* address review feedback
2023-10-18 02:34:55 +00:00
Connor McCarthy 55f174e5bb
chore(sdk): clean up compiler_test.py (#10070) 2023-10-09 20:50:18 +00:00
Connor McCarthy fcdff294a6
fix(sdk): fix incorrect sub-DAG output type when using `dsl.Collected` (#10069) 2023-10-09 17:04:18 +00:00
Connor McCarthy 87005cc015
chore(sdk): wrap conditional branch groups in outer pipeline (#10034) 2023-10-05 02:45:15 +00:00
Connor McCarthy e99f2704fc
feat(sdk): support `PipelineTaskFinalStatus` in tasks that use `.ignore_upstream_failure()` (#10010)
* support taskfinalstatus in tasks that ignore upstream failure

* address review feedback
2023-09-21 01:46:37 +00:00
Connor McCarthy c6b236d1a0
feat(sdk): support dsl.If, dsl.Elif, and dsl.Else (#9894)
* support if/elif/else

* deprecate dsl.Condition

* alter rebase

* update release notes

* address review feedback

* change BinaryOperation to ConditionOperation
2023-09-11 20:19:35 +00:00
Connor McCarthy c9e54798cd
chore(sdk): undo creation of kfp-dsl namespace package (#9874)
* revert to c8204d0

* reapply #9742

* modify #9791

* reapply #9800

* reapply #9827

* revert parts of #9738

* reapply parts of #9785

* remove duplicated dsl-test code

* reapply parts of #9791

* correct version
2023-08-15 16:24:52 -07:00
Connor McCarthy 309c5b9f0a
chore(sdk): release KFP SDK 2.1.0 (#9782) 2023-07-26 17:09:02 +00:00
Connor McCarthy c8204d0285
chore(sdk): partition KFP SDK source code into runtime and non-runtime code (#9710) 2023-07-07 23:18:52 +00:00
Chen Sun a727783abd
chore: remove 'dummy_root' in test samples (#9640)
* remove 'dummy_root' in test samples

* more cleanup
2023-06-21 23:51:41 +00:00
Connor McCarthy 48ecc7d879
fix(sdk): fix bug when iterating over output from upstream task in nested parallelfor loops (#9580)
* refactor for clarity

* fix iter over upstream output in nested parallelfor

* update release notes

* support empty `List` generic annotation

* fix typos

* add comment about refactoring opportunity

* fix annotation

* update comment again
2023-06-09 08:58:28 -07:00
Connor McCarthy 96947e6fb9
fix(sdk): fix compilation of boolean constant passed to component (#9390) 2023-05-12 17:13:39 +00:00
Connor McCarthy c01288d673
feat(sdk): support writing `TASK_FINAL_STATUS` parameter type (#9080)
* write new parameter type

* update golden snapshots

* add compiler tests
2023-05-12 17:13:32 +00:00
Connor McCarthy 946c51bafe
feat(sdk): support extracting input/output descriptions from component/pipeline docstrings (#9156)
* feat(sdk): support extracting input/output descriptions from component/pipeline docstrings

* remove numpydoc tests
2023-04-24 17:16:44 +00:00
Connor McCarthy 91abbeaf2f
feat(sdk): support `display_name` and `description` in `@dsl.pipeline` decorator (#9153)
* feat(sdk): support display_name and description in @dsl.pipeline decorator

* add release note

* test additional case
2023-04-17 14:15:39 -07:00
Chen Sun a6ef77b784
feat(sdk): Support setting cpu/memory requests. (#9121)
* Support setting cpu/memory requests.

* address review comments

* address review comments
2023-04-10 23:35:16 -07:00
Connor McCarthy 0cbcebcf88
fix(sdk): support accessing `.dsl` attribute on `kfp` module object (#9048) 2023-03-27 15:12:17 -07:00
Ze Mao 0fd1e69c74
cpu and memory resource can be optional (#8992)
Co-authored-by: Ze Mao <zemao@google.com>
2023-03-16 09:23:55 -07:00
Connor McCarthy a38d7479d7
feat(sdk): support compiling platform specific features (#8940)
* support compiling platform-specific features

* fix map size change during iteration
2023-03-10 11:18:25 -08:00
Connor McCarthy c046879dda
support fanning-in artifacts (#8808) 2023-02-22 10:39:00 -08:00
Connor McCarthy 41752608bc
support list of artifact input placeholders (#8484) 2023-02-16 09:26:53 -08:00
JOCSTAA 1e1a02066a
fix(sdk): Add outputDefinitions of root pipeline to compiled components IR YAML file (#8848)
* base

* add to readme

* adress comments
2023-02-15 09:28:33 -08:00
JOCSTAA 5684bfccff
feat(sdk): Ignore upstream failure (#8838)
* main

* adress comments

* adress comments

* adress comments 2

* address comments 3

* address comments 4
2023-02-13 11:40:42 -08:00
Connor McCarthy b575950a75
feat(sdk): support for fan-in from `dsl.ParallelFor` (#8631)
* implement parallelfor fan-in

* temporarily block fan-in of artifacts

* clean up

* unify and clarify error messages

* remove comment

* remove unnecessary compiler condition

* make test cases more realistic

* use better ParallelFor variable name

* add and rename test cases

* add ParallelFor fan-in test cases

* fix return collected bug

* clean up compilation tests

* update docstrings; update error message

* remove fan-in artifacts read/write tests

* implement review feedback

* change use of additional_input_name_for_pipeline_channel

* move additional_input_name_for_pipeline_channel

* provide additional pipeline topology validation and tests; refactor existing validation

* add release note for parameter fan-in
2023-02-04 20:50:51 -08:00
Connor McCarthy 151e8a3c6e
feat(sdk): support optional artifact inputs (#8623)
* add compiler tests

* support optional artifact inputs

* update executor test

* update existing golden snapshot

* add new compiler test/snapshot

* add release note

* bump kfp-pipeline-spec dependency
2022-12-28 13:40:12 -05:00
Connor McCarthy 5491f6f3db
fix(sdk): support writing/reading isOptional for input parameters (#8612)
* add tests

* support writing/reading isOptional for parameters

* update golden snapshots

* temporarily maintain workaround for v1 optional artifact inputs

* add release note
2022-12-21 14:46:59 -08:00
JOCSTAA 49db63c916
feat(sdk): Add comments to IR YAML file (#8467)
* base

* add tests

* fix bug

* adress comments

* address comments 2

* sort comments

* sort signatures

* add indempotent test

* add indempotent test2

* support multiline docstring

* review

* docformatter presubmit exclude

* docformatter presubmit exclude

* docformatter

* docformatter

* merge 1

* update readme

* nit .items()

* remove reduntant test
2022-12-05 14:43:14 -08:00
JOCSTAA 614c231ae6
fix(sdk): unblock valid topology. (#8416)
* unblock valid topology

* add more tests

* handle git fork

* sample_test_cases

* main

* restore to master

* resolve comments on PR

* resolve conflicts

* resolve conflict 2

* revert conflict fix

* fix changes

* address comments

* review

* docformatter presubmit

* revert docformatter

* update release.md
2022-12-02 00:15:15 +00:00
Connor McCarthy 9e9e1081c6
fix(sdk): fix boolean default value serialization bug (#8444)
* add test cases

* fix bug

* add v1 component yaml back compat changes

* update golden snapshot
2022-11-11 15:42:33 -08:00
Connor McCarthy 24e17f7352
chore(sdk): quiet noisy SDK unit test output (#8434) 2022-11-09 17:47:39 +00:00
Connor McCarthy e371fa3795
feat(sdk): add compilation logic for google artifact types (support custom artifact types pt. 2) (#8232)
* remove dependency of deprecated on v2

* use new artifact identification logic

* add custom artifact type compiler test

* compile schema_version from non-kfp artifacts

* update tests

* add importer to compiler test

* update golden snapshots

* fix type checking code

* undo executor_test diff

* remove gcpc types compiler test

* add compilation test comment

* update google artifact handling

* update compiler test

* only permit system. and google. artifact types; add tests

* update compiler test

* simplify type checking logic

* use bundled artifact types in PipelineArtifactChannel

* raise error on unknown v1 input or output

* add type utils and move functions

* add type utils tests

* remove schema_version from inputspec and outputspec

* update artifact proto creation and tests

* propogate bundled artifact type string changes throughout sdk

* update golden snapshot

* remove unused v1 compat and tests

* update type checking and tests to validate schema_version using semantic versioning

* update importer sample

* update test data config

* update test_data config

* clean up sample

* clean up sample
2022-09-12 14:00:02 -06:00
Connor McCarthy 16c7d6658e
test(sdk): create SDK execution test suite and restructure read/write tests (#8245)
* update client for graph components

* use yaml for config

* simplify read write tests

* add test script

* restructure yaml and read/write tests

* add execution test file

* move test_data; update paths; update readme

* update kfp endpoint

* update pipeline with env

* update logging

* update config

* update kfp endpoint and timeout

* add kfp as requirement

* mock kfp_package_path for lightweight component execution

* revert pipeline name changes

* update config with name

* update read write tests to use explicit name

* update execution tests to use explicit name
2022-09-12 18:33:06 +00:00
Chen Sun 48574dc648
feat(sdk): Support pipeline outputs (#8204)
* Support pipeline outputs

* release note
2022-08-29 20:26:53 +00:00
Chen Sun 5ce4954d6b
feat(sdk)!: Implement Graph Component (#8179)
* Implement Graph Component

* isort imports

* release note

* address review comments

* address review comments
2022-08-26 09:28:49 +00:00
Bo 51bea09833
feat(sdk): Support Parallelism in ParallelFor in KFP SDK V2 (#8146)
* feat(sdk): add support for ParallelFor parallelism setting

* add parallelism value check

* Add unit tests

* Adding a compiler test

* used None as default parallelism input signature, other minor fixes on format and tests

* fix import statements

* add release.md message

* update ParallelFor docstring

* fixed docstring comments

* removed 'optional' in docstrings
2022-08-17 21:06:01 +00:00
Connor McCarthy bdff332ac6
feat(sdk): support more than one exit handler per pipeline (#8088)
* add compiler test pipeline with multiple exit handlers

* remove blocker of multiple exit handlers

* move exit handler builder logic to pipeline_spec_builder

* build all exit handlers per pipeline

* add compiler test with IR inspection

* prevent usage of cross-pipeline after

* test cross-pipeline after is prevented

* update existing task dependency logic and tests

* add v2 sample test

* remove cross-pipeline .after

* prevent cross-dag data dependency for dsl features

* add compiler test pipeline with nested exit handlers

* add support for nested exit handlers

* clean up pipeline with nested exit handlers

* remove sample with multiple exit handlers

* remove compiler test with nested exit handlers

* add compilation guard against nested exit handlers in subdag

* update release notes
2022-08-10 16:56:21 +00:00
Scott_Xu 04c827fc3a
feat(sdk): supporting bring your own container for arbitrary input and outputs (#8066)
* support container_component decorator for function with no inputs

* resolve review comments

* add sample tests for milestone 1

* modify compiler test data

* resolve reviews

* resolve reviews

* WIP

* implementation of function of no inputs

* fixed sample test

* re-fix sample test

* fix rebase merge conflict

* resolve formatting

* resolve isort error for test data

* resolve comments

* fix nit

* resolve nit

* add implementation for placeholders i/o, sample and compiler tests

* resolve comments and merge logic for constructing container component

* resolve comments

* resolve comments

* fix assertion messages

* add error handling for accessing artifact by itself

* add test for raising error for accessing artifact by itself
2022-08-08 21:00:53 +00:00
Chen Sun 1dd06cd8e8
feat(sdk): Allow artifact inputs in pipeline definition. (#8044)
* Allow artifact inputs in pipeline definition.

* release note

* isort fix

* remove unused imports

* Delete flycheck_compiler.py

* Delete flycheck_pipeline_spec_builder.py

* Delete flycheck_structures.py

* address cr comment
2022-07-28 00:17:53 +00:00
Connor McCarthy 13736b85b6
chore(sdk): clean up kfp code style (#8027)
* remove unused imports

* use google as isort profile

* sort imports

* format with yapf

* clean end of file new line, trailing whitespace, double quoted strings
2022-07-15 00:06:34 +00:00
Connor McCarthy 850a750496
feat(sdk): add retry policy support to kfp v2 (#7867)
* depend on kfp-pipeline-spec>=0.1.16

* implement RetryPolicy structure

* implement RetryPolicy structure tests

* implement .set_retry method on PipelineTask

* implement retry policy compilation logic

* implement compiler tests for retry

* implement pipeline with retry compilation test

* add release note

* clean up
2022-07-11 16:44:52 +00:00
Connor McCarthy 281151c051
chore(sdk): improve IR read/write testing (#7920)
* add more load component test cases

* add read write equality tests

* add copyright

* remove duplicative structures tests

* add if __name__ == '__main__' block
2022-06-24 09:45:15 -06:00
Connor McCarthy c6125ffc44
feat(sdk): enable component compilation via cli (#7649)
* enable component compilation via cli

* reorganize compiler/test_data dir

* clean up pipeline samples

* add component compilation test cases

* update compiler test_data README

* add compiler tests

* test component deserialization is lossless
2022-06-17 05:49:10 +00:00
Chen Sun 0ada48b55a
chore(sdk): move pipeline test samples closer to compiler unit tests. (#7849)
* chore(sdk): move pipeline test samples closer to compiler unit tests.

* explicitly list test files

* remove dead code
2022-06-08 18:35:17 +00:00
Connor McCarthy a6ec449273
feat(sdk): only use IR when saving component (#7690)
* move function into compiler module

* remove v2 readable yaml write path

* remove v2 readable yaml test data; remove duplicate tests
2022-05-24 22:23:33 +00:00
Connor McCarthy 62972eccf9
feat(sdk): enable compilation of primitive components (#7580)
* clean up relevant files

* use context manager for temporary directory cleanup

* add component compilation methods

* add component compilation tests

* add more complete pipeline_func argument handling

* use context manager for handling type check mode

* improve checks on inputs for component compilation

* correct pipeline name for primitive component IR

* simplify component compilation logic

* extract compiler methods

* move compiler helpers to helpers file to avoid circular imports

* remove underscore prefix from functions in hidden module

* move helpers tests to helper_test module

* put to_pipeline_spec on ComponentSpec

* fix inputs reference

* use same naming regex for pipelines and components

* permit compilation of all BaseComponent subclasses

* clean up docstrings

* merge compiler.helpers into pipeline_spec_builder

* modify docstrings; refactor

* modify docstring
2022-05-13 20:12:34 +00:00