Commit Graph

11 Commits

Author SHA1 Message Date
Connor McCarthy a38d7479d7
feat(sdk): support compiling platform specific features (#8940)
* support compiling platform-specific features

* fix map size change during iteration
2023-03-10 11:18:25 -08:00
Connor McCarthy b575950a75
feat(sdk): support for fan-in from `dsl.ParallelFor` (#8631)
* implement parallelfor fan-in

* temporarily block fan-in of artifacts

* clean up

* unify and clarify error messages

* remove comment

* remove unnecessary compiler condition

* make test cases more realistic

* use better ParallelFor variable name

* add and rename test cases

* add ParallelFor fan-in test cases

* fix return collected bug

* clean up compilation tests

* update docstrings; update error message

* remove fan-in artifacts read/write tests

* implement review feedback

* change use of additional_input_name_for_pipeline_channel

* move additional_input_name_for_pipeline_channel

* provide additional pipeline topology validation and tests; refactor existing validation

* add release note for parameter fan-in
2023-02-04 20:50:51 -08:00
JOCSTAA 49db63c916
feat(sdk): Add comments to IR YAML file (#8467)
* base

* add tests

* fix bug

* adress comments

* address comments 2

* sort comments

* sort signatures

* add indempotent test

* add indempotent test2

* support multiline docstring

* review

* docformatter presubmit exclude

* docformatter presubmit exclude

* docformatter

* docformatter

* merge 1

* update readme

* nit .items()

* remove reduntant test
2022-12-05 14:43:14 -08:00
Connor McCarthy e371fa3795
feat(sdk): add compilation logic for google artifact types (support custom artifact types pt. 2) (#8232)
* remove dependency of deprecated on v2

* use new artifact identification logic

* add custom artifact type compiler test

* compile schema_version from non-kfp artifacts

* update tests

* add importer to compiler test

* update golden snapshots

* fix type checking code

* undo executor_test diff

* remove gcpc types compiler test

* add compilation test comment

* update google artifact handling

* update compiler test

* only permit system. and google. artifact types; add tests

* update compiler test

* simplify type checking logic

* use bundled artifact types in PipelineArtifactChannel

* raise error on unknown v1 input or output

* add type utils and move functions

* add type utils tests

* remove schema_version from inputspec and outputspec

* update artifact proto creation and tests

* propogate bundled artifact type string changes throughout sdk

* update golden snapshot

* remove unused v1 compat and tests

* update type checking and tests to validate schema_version using semantic versioning

* update importer sample

* update test data config

* update test_data config

* clean up sample

* clean up sample
2022-09-12 14:00:02 -06:00
Chen Sun d88358201d
feat(sdk): Support using pipeline in exit handlers (#8220)
* Support using pipeline in exit handlers

* release note

* remove dead code

* fix component spec merging flakiness

* address comments in PR#8209

* address review comments

* fix bad merge in release note
2022-09-01 01:45:54 +00:00
Chen Sun 5ce4954d6b
feat(sdk)!: Implement Graph Component (#8179)
* Implement Graph Component

* isort imports

* release note

* address review comments

* address review comments
2022-08-26 09:28:49 +00:00
Connor McCarthy 13736b85b6
chore(sdk): clean up kfp code style (#8027)
* remove unused imports

* use google as isort profile

* sort imports

* format with yapf

* clean end of file new line, trailing whitespace, double quoted strings
2022-07-15 00:06:34 +00:00
Connor McCarthy de0b824be1
feat(sdk): add support for IfPresentPlaceholder and ConcatPlaceholder strings (#7795)
* move placeholders to placeholders module; implement if-present and concat placeholders

* update module name and method names throughout codebase

* rename placeholder abc

* add back compat support for CEL-style concat string

* add copyright

* add docstrings; clean up

* remove duplicate code
2022-06-09 07:42:00 +00:00
Connor McCarthy 62972eccf9
feat(sdk): enable compilation of primitive components (#7580)
* clean up relevant files

* use context manager for temporary directory cleanup

* add component compilation methods

* add component compilation tests

* add more complete pipeline_func argument handling

* use context manager for handling type check mode

* improve checks on inputs for component compilation

* correct pipeline name for primitive component IR

* simplify component compilation logic

* extract compiler methods

* move compiler helpers to helpers file to avoid circular imports

* remove underscore prefix from functions in hidden module

* move helpers tests to helper_test module

* put to_pipeline_spec on ComponentSpec

* fix inputs reference

* use same naming regex for pipelines and components

* permit compilation of all BaseComponent subclasses

* clean up docstrings

* merge compiler.helpers into pipeline_spec_builder

* modify docstrings; refactor

* modify docstring
2022-05-13 20:12:34 +00:00
Connor McCarthy 5ca4cb78d3
chore(sdk): remove unused imports and variables (#7393) 2022-03-09 21:42:11 +00:00
Yaqi Ji 8bb0d0e025
feat(sdk)!: move v2 to main namespace (#7376)
* feat(sdk)!: move v2 to main namespace

* edit samples
2022-03-04 21:56:51 +00:00