Commit Graph

33 Commits

Author SHA1 Message Date
Chen Sun a6ef77b784
feat(sdk): Support setting cpu/memory requests. (#9121)
* Support setting cpu/memory requests.

* address review comments

* address review comments
2023-04-10 23:35:16 -07:00
Connor McCarthy e96754cb53
feat(sdk): deprecate .add_node_selector_constraint in favor of .set_accelerator_type (#8980)
* rename method

* update docstring
2023-03-14 09:22:35 -07:00
Connor McCarthy a38d7479d7
feat(sdk): support compiling platform specific features (#8940)
* support compiling platform-specific features

* fix map size change during iteration
2023-03-10 11:18:25 -08:00
Connor McCarthy c046879dda
support fanning-in artifacts (#8808) 2023-02-22 10:39:00 -08:00
JOCSTAA 5684bfccff
feat(sdk): Ignore upstream failure (#8838)
* main

* adress comments

* adress comments

* adress comments 2

* address comments 3

* address comments 4
2023-02-13 11:40:42 -08:00
Connor McCarthy e7ad073f1d
feat(sdk): deprecate .set_gpu_limit in favor of .set_accelerator_limit (#8836) 2023-02-09 15:46:10 -08:00
Connor McCarthy f4588f3112
chore(sdk): idiomatic Python pure refactorings (#8537)
* use f-strings

* inline immediately returned variables

* simplify comprehension

* use list literals

* lift return statements into control flow blocks

* other pure refactorings

* remove passthrough function

* use a ternary when it improves readability

* use __getitem__ for regex match groups; introduced in Python 3.6

* fix changed comments

* revert one change
2022-12-06 23:44:36 +00:00
Connor McCarthy 00c6b01040
fix(sdk): fix bug when referencing component.pipeline_spec in a pipeline body (#8514)
* add tests

* implement fix

* add TODO
2022-12-02 09:35:16 +00:00
JOCSTAA 614c231ae6
fix(sdk): unblock valid topology. (#8416)
* unblock valid topology

* add more tests

* handle git fork

* sample_test_cases

* main

* restore to master

* resolve comments on PR

* resolve conflicts

* resolve conflict 2

* revert conflict fix

* fix changes

* address comments

* review

* docformatter presubmit

* revert docformatter

* update release.md
2022-12-02 00:15:15 +00:00
Connor McCarthy be766dc352
docs(sdk): update pipeline task .set_gpu_limit reference docs (#8477) 2022-11-18 17:58:22 +00:00
Connor McCarthy d33f359797
fix(sdk): fix nested placeholders and block illegal IfPresent form in Concat (#8414)
* support single element IfPresentPlaceholders that contain a primitive placeholder

* block list then and else_ inside concat

* support single element IfPresent in v1

* clean up typing

* skip loading batch_predict_job yaml
2022-11-04 18:20:16 +00:00
Connor McCarthy 4c6abe5afc
refactor(sdk): simplify placeholders and logic (#8402)
* simplify placeholders

* update placeholders_test.py

* update structures to use placeholders correctly

* update use of placeholders elsewhere in codebase

* clean up

* move placeholder type checking logic to PipelineTask to prevent v1 load breaks

* clean up

* update docstrings

* make to_dict and to_string private methods

* revert unnecessary change to tests

* improve type safety of code

* prefix module-level constant with underscore
2022-11-02 00:31:45 +00:00
Connor McCarthy ca6ad27020
chore(sdk): remove unused linter comments (#8405) 2022-11-01 18:31:46 +00:00
Scott_Xu 6443fc17dc
chore(sdk): hide private methods in placeholders for simpler doc (#8271)
* hide private methods in ConcatPlaceholders and IfPresentPlaceholder by renaming

* rename transform and validate methods
2022-09-16 16:16:30 -06:00
Scott_Xu 5dff29a071
fix(sdk): keeping placeholder strings in command and args even if default value is set (#8259)
* return string representation of placeholder in pipeline_task

* return string representation of placeholder in pipeline_task
2022-09-13 23:44:31 +00:00
Connor McCarthy e371fa3795
feat(sdk): add compilation logic for google artifact types (support custom artifact types pt. 2) (#8232)
* remove dependency of deprecated on v2

* use new artifact identification logic

* add custom artifact type compiler test

* compile schema_version from non-kfp artifacts

* update tests

* add importer to compiler test

* update golden snapshots

* fix type checking code

* undo executor_test diff

* remove gcpc types compiler test

* add compilation test comment

* update google artifact handling

* update compiler test

* only permit system. and google. artifact types; add tests

* update compiler test

* simplify type checking logic

* use bundled artifact types in PipelineArtifactChannel

* raise error on unknown v1 input or output

* add type utils and move functions

* add type utils tests

* remove schema_version from inputspec and outputspec

* update artifact proto creation and tests

* propogate bundled artifact type string changes throughout sdk

* update golden snapshot

* remove unused v1 compat and tests

* update type checking and tests to validate schema_version using semantic versioning

* update importer sample

* update test data config

* update test_data config

* clean up sample

* clean up sample
2022-09-12 14:00:02 -06:00
Chen Sun 48574dc648
feat(sdk): Support pipeline outputs (#8204)
* Support pipeline outputs

* release note
2022-08-29 20:26:53 +00:00
Chen Sun 5ce4954d6b
feat(sdk)!: Implement Graph Component (#8179)
* Implement Graph Component

* isort imports

* release note

* address review comments

* address review comments
2022-08-26 09:28:49 +00:00
Scott_Xu 35fccb1fae
feat(sdk): expose new ContainerSpec API (#8144)
* rename ContainerSpec to ContainerSpecImplementation for internal usage

* fix nit
2022-08-15 20:33:34 +00:00
Connor McCarthy bdff332ac6
feat(sdk): support more than one exit handler per pipeline (#8088)
* add compiler test pipeline with multiple exit handlers

* remove blocker of multiple exit handlers

* move exit handler builder logic to pipeline_spec_builder

* build all exit handlers per pipeline

* add compiler test with IR inspection

* prevent usage of cross-pipeline after

* test cross-pipeline after is prevented

* update existing task dependency logic and tests

* add v2 sample test

* remove cross-pipeline .after

* prevent cross-dag data dependency for dsl features

* add compiler test pipeline with nested exit handlers

* add support for nested exit handlers

* clean up pipeline with nested exit handlers

* remove sample with multiple exit handlers

* remove compiler test with nested exit handlers

* add compilation guard against nested exit handlers in subdag

* update release notes
2022-08-10 16:56:21 +00:00
Connor McCarthy c6a5e387dd
docs(sdk): refresh SDK public API docstrings (#8057)
* add module-level docstrings

* update compiler docstrings

* update registry module docstrings

* add BaseComponent and children to public api, but discourage use

* update artifact docstrings and type annotations

* update dsl docstrings

* update client docstrings

* clean up kfp.__init__

* add dsl placeholder docstrings

* many more docstring updates

* document type aliases in dsl module
2022-07-21 19:51:18 +00:00
Connor McCarthy 850a750496
feat(sdk): add retry policy support to kfp v2 (#7867)
* depend on kfp-pipeline-spec>=0.1.16

* implement RetryPolicy structure

* implement RetryPolicy structure tests

* implement .set_retry method on PipelineTask

* implement retry policy compilation logic

* implement compiler tests for retry

* implement pipeline with retry compilation test

* add release note

* clean up
2022-07-11 16:44:52 +00:00
Connor McCarthy f877a1ffc8
fix(sdk): fix ifpresent and concat placeholder compile (#7930)
* fix ifpresent and concat placeholder expansion

* update golden snapshots

* uncomment v1 ifpresent component test case
2022-06-27 16:59:34 -06:00
Connor McCarthy ad458d535c
fix(sdk): fix placeholder read/write logic (#7928)
* fix ExecutorInputPlaceholder() usage

* fix OutputPath and OutputParameter placeholder task creation logic

* fix v1 component yaml deserialization logic

* fix typo in test

* ignore description when comparing ComponentSpec

* uncomment fixed test cases
2022-06-27 15:06:38 -06:00
Connor McCarthy de0b824be1
feat(sdk): add support for IfPresentPlaceholder and ConcatPlaceholder strings (#7795)
* move placeholders to placeholders module; implement if-present and concat placeholders

* update module name and method names throughout codebase

* rename placeholder abc

* add back compat support for CEL-style concat string

* add copyright

* add docstrings; clean up

* remove duplicate code
2022-06-09 07:42:00 +00:00
Connor McCarthy cbc2ac5266
feat(sdk): enable read in component using IR (#7689)
* enable read in primitive components

* enable read in via YAML component

* convert OutputValuePlaceholder to OutputParameterPlaceholder

* add docstring
2022-05-24 18:58:57 +00:00
Connor McCarthy 6698801fc0
refactor(sdk): move placeholder logic to structures (#7687)
* move placeholder formatting logic to structures

* add placeholder formatting logic tests

* update placeholder references throughout codebase

* remove ABC meta inheritance
2022-05-19 07:45:56 +00:00
Connor McCarthy 62972eccf9
feat(sdk): enable compilation of primitive components (#7580)
* clean up relevant files

* use context manager for temporary directory cleanup

* add component compilation methods

* add component compilation tests

* add more complete pipeline_func argument handling

* use context manager for handling type check mode

* improve checks on inputs for component compilation

* correct pipeline name for primitive component IR

* simplify component compilation logic

* extract compiler methods

* move compiler helpers to helpers file to avoid circular imports

* remove underscore prefix from functions in hidden module

* move helpers tests to helper_test module

* put to_pipeline_spec on ComponentSpec

* fix inputs reference

* use same naming regex for pipelines and components

* permit compilation of all BaseComponent subclasses

* clean up docstrings

* merge compiler.helpers into pipeline_spec_builder

* modify docstrings; refactor

* modify docstring
2022-05-13 20:12:34 +00:00
Connor McCarthy 5da3826bb5
feat(sdk): use custom basemodel and remove pydantic (#7639)
* fix discovered bug

* update tests

* implement custom BaseModel

* use basemodel for structures

* remove pydantic dependency

* assorted code cleanup
2022-05-04 12:56:32 -06:00
Connor McCarthy 4621aec146
chore(sdk): apply yapf formatting (#7414) 2022-03-16 20:34:36 +00:00
Connor McCarthy 5dcc6d9910
chore(sdk): rename PipelineTask arguments parameter to args (#7392) 2022-03-10 21:42:54 +00:00
Connor McCarthy 82d17ff616
feat(sdk)!: rename commands, arguments parameters to command, args (#7391) 2022-03-09 05:39:11 +00:00
Yaqi Ji 8bb0d0e025
feat(sdk)!: move v2 to main namespace (#7376)
* feat(sdk)!: move v2 to main namespace

* edit samples
2022-03-04 21:56:51 +00:00