Commit Graph

158 Commits

Author SHA1 Message Date
Connor McCarthy c8204d0285
chore(sdk): partition KFP SDK source code into runtime and non-runtime code (#9710) 2023-07-07 23:18:52 +00:00
Connor McCarthy 5a769e9080
Revert "update compiler type annotations (#9090)" (#9095)
This reverts commit ad6f55e8f3.
2023-04-03 16:01:26 -07:00
Connor McCarthy ad6f55e8f3
update compiler type annotations (#9090) 2023-04-03 09:20:17 -07:00
Connor McCarthy a38d7479d7
feat(sdk): support compiling platform specific features (#8940)
* support compiling platform-specific features

* fix map size change during iteration
2023-03-10 11:18:25 -08:00
JOCSTAA 49db63c916
feat(sdk): Add comments to IR YAML file (#8467)
* base

* add tests

* fix bug

* adress comments

* address comments 2

* sort comments

* sort signatures

* add indempotent test

* add indempotent test2

* support multiline docstring

* review

* docformatter presubmit exclude

* docformatter presubmit exclude

* docformatter

* docformatter

* merge 1

* update readme

* nit .items()

* remove reduntant test
2022-12-05 14:43:14 -08:00
Chen Sun 410e99c604
feat(sdk): Support loading pipeline from yaml (#8209)
* support loading pipeline from yaml

* release note

* cleanup

* maintain read/write consistency with component/pipeline compilation.
2022-08-31 12:05:55 +00:00
Chen Sun 48574dc648
feat(sdk): Support pipeline outputs (#8204)
* Support pipeline outputs

* release note
2022-08-29 20:26:53 +00:00
Chen Sun 5ce4954d6b
feat(sdk)!: Implement Graph Component (#8179)
* Implement Graph Component

* isort imports

* release note

* address review comments

* address review comments
2022-08-26 09:28:49 +00:00
Connor McCarthy bdff332ac6
feat(sdk): support more than one exit handler per pipeline (#8088)
* add compiler test pipeline with multiple exit handlers

* remove blocker of multiple exit handlers

* move exit handler builder logic to pipeline_spec_builder

* build all exit handlers per pipeline

* add compiler test with IR inspection

* prevent usage of cross-pipeline after

* test cross-pipeline after is prevented

* update existing task dependency logic and tests

* add v2 sample test

* remove cross-pipeline .after

* prevent cross-dag data dependency for dsl features

* add compiler test pipeline with nested exit handlers

* add support for nested exit handlers

* clean up pipeline with nested exit handlers

* remove sample with multiple exit handlers

* remove compiler test with nested exit handlers

* add compilation guard against nested exit handlers in subdag

* update release notes
2022-08-10 16:56:21 +00:00
Connor McCarthy e728d0871b
chore(sdk): add pre-commit hook to remove unused imports (#8123)
* add pycln pre-commit hook to remove unused imports

* remove existing unused imports
2022-08-10 08:17:21 +00:00
Deepyaman Datta 257f8dbe0a
chore(sdk): change "PipelienSpec" to "PipelineSpec" (#8102) 2022-08-04 16:55:06 +00:00
Chen Sun 1dd06cd8e8
feat(sdk): Allow artifact inputs in pipeline definition. (#8044)
* Allow artifact inputs in pipeline definition.

* release note

* isort fix

* remove unused imports

* Delete flycheck_compiler.py

* Delete flycheck_pipeline_spec_builder.py

* Delete flycheck_structures.py

* address cr comment
2022-07-28 00:17:53 +00:00
Connor McCarthy c6a5e387dd
docs(sdk): refresh SDK public API docstrings (#8057)
* add module-level docstrings

* update compiler docstrings

* update registry module docstrings

* add BaseComponent and children to public api, but discourage use

* update artifact docstrings and type annotations

* update dsl docstrings

* update client docstrings

* clean up kfp.__init__

* add dsl placeholder docstrings

* many more docstring updates

* document type aliases in dsl module
2022-07-21 19:51:18 +00:00
Connor McCarthy a6ec449273
feat(sdk): only use IR when saving component (#7690)
* move function into compiler module

* remove v2 readable yaml write path

* remove v2 readable yaml test data; remove duplicate tests
2022-05-24 22:23:33 +00:00
Connor McCarthy 62972eccf9
feat(sdk): enable compilation of primitive components (#7580)
* clean up relevant files

* use context manager for temporary directory cleanup

* add component compilation methods

* add component compilation tests

* add more complete pipeline_func argument handling

* use context manager for handling type check mode

* improve checks on inputs for component compilation

* correct pipeline name for primitive component IR

* simplify component compilation logic

* extract compiler methods

* move compiler helpers to helpers file to avoid circular imports

* remove underscore prefix from functions in hidden module

* move helpers tests to helper_test module

* put to_pipeline_spec on ComponentSpec

* fix inputs reference

* use same naming regex for pipelines and components

* permit compilation of all BaseComponent subclasses

* clean up docstrings

* merge compiler.helpers into pipeline_spec_builder

* modify docstrings; refactor

* modify docstring
2022-05-13 20:12:34 +00:00
Connor McCarthy f70413ad7c
chore(sdk): stop using pydantic for serialization (#7536)
* add ir_utils

* add ir_utils tests

* use ir_utils to write ir to file

* cast posix path to string in `kfp components build` command

* add copyright

* remove unused import

* update setup.py packages
2022-04-12 17:03:55 +00:00
Connor McCarthy c98f14abb4
chore(sdk): make field order in compile deterministic (#7520)
* apply docformatter

* use sort_keys=True in compiler

* update golden snapshots

* set other sort_keys=True

* update structures_test yamls
2022-04-06 21:46:13 +00:00
Yaqi Ji 04123280a4
fix(sdk): add default value for inputs (#7405)
* fix(sdk): add default value for inputs

* merge conflict

* release

* fix sample
2022-03-23 18:59:20 +00:00
Connor McCarthy c7301c449c
feat(sdk)!: use yaml for IR package (#7431)
* change serialization format from json to yaml

* refactor: use guard clause for clarity

* convert test data from json to yaml

* add and update tests

* permit deprecated compile to json

* re-add json test data

* update compiler tests

* refactor Compiler._write_pipeline_spec_file

* update golden snapshots

* update sample output extensions

* test file extension exceptions and warnings

* update release notes

* remove json golden snapshots
2022-03-21 21:45:31 +00:00
Connor McCarthy 2c0b38162f
chore(sdk): make kfp v2 hermetic (#7428)
* move v1 files to v2

* format with yapf

* remove unused imports

* clean up

* update copyright

* move imports to module top level

* apply some pylint changes

* loosen pylintrc

* remove v2_yaml_utils.py
2022-03-18 03:00:40 +00:00
Chen Sun c03701b82a
fix(sdk): fix wrong kfp import causes wrong sdk_version being set in pipeline_spec. (#7433)
* fix wrong kfp import in compiler

* release note
2022-03-18 00:05:39 +00:00
Connor McCarthy 5ca4cb78d3
chore(sdk): remove unused imports and variables (#7393) 2022-03-09 21:42:11 +00:00
Yaqi Ji 8bb0d0e025
feat(sdk)!: move v2 to main namespace (#7376)
* feat(sdk)!: move v2 to main namespace

* edit samples
2022-03-04 21:56:51 +00:00
Yaqi Ji b693909a08
feat(sdk)!: move v1 to deprecated folder (#7291)
* chore(sdk): move v1 to deprecated

* fix testsg

* fix testsg

* fix setup.py

* fix test

* s'

* fix tests

* fix test

* retore v2 test changes

* fix py3.6 test

* fix py3.6 test

* fix py3.6 import fallback error

* remove deprecated

* fix samples test

* sample test

* fix samples

* add readme

* restroe test

* python require

* remove path

* fix tests

* inteegration tests

* remove tfx tests for dependency with kfp v1

* fix e2e

* fix e2e

* fix integration tests

* fix sampe

* move client down

* change to kfp

* add import alias

* fix

* runid

* fix dsl

* only use kfp for function

* revert train_until_good

* tfx test

* kfp

* try import

* onprem
2022-03-04 00:52:59 +00:00
Seb 577902ca46
fix(sdk): Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest (#7141)
* Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest

* Update test for TTL

* Declare fix in release note of SDK

* update syntaxe

* Update RELEASE.md

Add Braking change due to incompatibility with  KFP pre 1.7 due to Argo 2.X
2022-01-20 00:43:52 +00:00
Yuan (Bob) Gong 2460aaa653
test: migrate sample integration test to v2 sample test infra (#7115)
* test: migrate sample integration test to v2 sample test infra

* test: configure v2 integration test timeout
2021-12-26 04:17:44 +00:00
Chen Sun 8aee62142a
feat(sdk)!: Deprecate V2 compatible mode in v1 compiler (#6958)
* Deprecate V2 compatible mode in v1 compiler

* release note

* fix tests
2021-11-24 22:11:34 +00:00
Chen Sun 439d8c87c1
chore(sdk): remove dead code on importer check in v1. (#6508)
* remove dead code

* Update RELEASE.md

* Update compiler.py

* Update v2_compatible_compiler_test.py

* Update compiler_tests.py

* Update RELEASE.md
2021-09-07 15:01:06 -07:00
Chen Sun 2f19a26ffd
chore(sdk): Format all Python files under SDK folder. (#6501)
* Reformat sdk only using the new yapf config.

* Reformat docstrings using docformatter.

* update golden files to resolve diff caused by whitespaces

* fix some tests

* format .py files under sdk/python/tests using yapf

* additional docformatter

* fix some tests
2021-09-03 11:25:11 -07:00
Chen Sun 434e5c3489
fix(sdk): block dsl.importer usage in KFP OSS. Fixes: #6323 (#6330)
* block dsl.importer in KFP OSS

* address cr comments
2021-08-16 01:02:06 -07:00
Yuan (Bob) Gong befd89c6ec
fix(sdk): pipeline name is required in v2 compatible mode. Fixes #6197 (#6332)
* fix(sdk): pipeline name is required in v2 compatible mode

* Update xgboost_sample.py
2021-08-13 14:00:04 -07:00
Niklas Hansson ed2ba1a6b3
chore(sdk): Argo lint requires '--kinds=workflows' argument for offline linting. (#6135) 2021-07-26 00:40:41 -07:00
Chen Sun b200e1bc7d
fix(sdk): Fix URI placeholder in v2 compatible mode. (#6040)
* fix uri placeholder in v2 compatible mode

* fix tests

* fix path generation

* fix tests

* fix test

* cleanup

* clean up

* fix test

* fix test

* fix test
2021-07-21 08:32:50 -07:00
Niklas Hansson 354e4251e8
chore: add offline linting (#5981)
* Added the offline linting

* update after feedback
2021-07-20 08:39:15 -07:00
Yuan (Bob) Gong fff62e1521
chore(v2): unify pipeline name between MLMD and KFP DB. Fixes #5978 (#6038)
* chore(sdk): unify pipeline name between MLMD and KFP DB

* backend changes pipeline-name during upload

* update sdk snapshots

* add backend tests

* fix launcher

* fix tests

* address feedback
2021-07-14 20:53:51 -07:00
Yuan (Bob) Gong 8a256db1bf
feat(sdk/dsl/compiler): dsl-compile --mode flag to turn on V2_COMPATIBLE, defaults to KF_PIPELINES_COMPILER_MODE env var. Fixes #5840 (#5952)
* feat(sdk/dsl/compiler): support --mode flag which can turn on v2 compatible mode

* override compiler default mode using KF_PIPELINES_COMPILER_MODE env var

* update V1_LEGACY to V1

* add unit tests

* address feedback

* clean up

* cleanup again

* use absl.testing.parameterized for table driven tests

* update
2021-07-03 02:22:49 -07:00
Yaqi Ji e7c281bd65
chore: Change python/kfp/compiler and cli folders from relative to absolute path (#5891)
* Update deprecated download_as_string to download_as_bytes

* Add more usage

* chore: Change python/kfp/cli from relative to absolute path

* chore: Change python/kfp/compiler from relative to absolute path
2021-06-22 11:28:31 -07:00
Chen Sun 2991602cfc
feat(sdk): Better error message when compiling for v2. Fixes #5688, Fixes #5727 (#5759)
* Better error message when compiling for v2

* update samples
2021-05-31 13:39:03 -07:00
Chen Sun fa5ac25678
fix(sdk): Avoid inserting duplicate input parameters in v2 compatible mode. (#5740) 2021-05-28 15:06:00 -07:00
Yuan (Bob) Gong 7629e10130
chore(sdk): warn v2 compatible mode is Beta quality (#5658) 2021-05-17 21:55:40 -07:00
capri-xiyue 6717434978
[SDK] Add pod labels for telemetry purpose (#5582)
* Add pod labels for telemetry purpose

* fixed test

* added sdk label in pods

* added sdk type label

* fixed test

* added UT back

* updated UT
2021-05-05 10:43:27 -07:00
Alexey Volkov cc83e1089b
Assigned copyright to the project authors (#5587) 2021-05-05 13:53:22 +08:00
capri-xiyue 0dba00518c
chore(sdk.v2):add v2 compact metrics label (#5566)
* add-v2-compact-metrics-label

* add telemetry pod label
2021-04-29 16:06:30 -07:00
Chen Sun 5dafda1270
feat(sdk.v2): Support explicit importer (#5502) 2021-04-19 13:14:09 -07:00
Chen Sun 5f97a4502d
feat(sdk.v2): remove pipeline_root from compiler interface (#5492) 2021-04-16 08:49:13 -07:00
Chen Sun f32915e88f
feat(sdk.v2): Support setting custom job spec in v2. (#5454)
* Support setting custom job spec.

* auto fill custom job jobSpec using container spec.

* fix test

* address comments
2021-04-10 09:20:04 -07:00
Ajay Gopinathan a1d453af21
chore(sdk): Allow the launcher image to be configurable. (#5360)
* Allow the launcher image to be configurable. This is especially useful
for tests.

* Address PR comments.
2021-03-23 08:27:44 -07:00
Niklas Hansson 8a32632edf
fix(sdk): make workflow validation compatible with argo CLI v2.12+. Fixes #4775, Fixes #5296 (#4804)
* updated the validation default test

* updated the version

* Update compiler.py

* Update compiler.py

* Update compiler.py

Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
2021-03-18 21:10:16 -07:00
Ajay Gopinathan 83eded130c
feat(sdk): Introduce experimental v2-compatibility in KFP SDK (#5218)
* WIP: Enable v2 compatibility in KFP SDK compiler.

* First pass clean up

* Clean up and introduce enum instead of boolean for execution mode.

* More cleanup

* Clean up and add comments.

* Undo formatting changes.

* Undo formatting changes.

* Add method to unconditionally add kfp pod env.

* minor formatting change.

* Update docstrings.

* Undo formatting changes.

* fix imports.

* fix pod_env tests

* rebased.

* undo format changes.

* Undo compiler changes.:

* format _default_transformers.py for consistency

* Fix various rebasing issues.

* fix bug referring to pipeline_name/pipeline_root in v1 pipelines.

* revert output dir name.

* allow both types of attributes for pipeline root.

* fix pod env yaml golden.

* fix for input/output uri tests.

* Add v2 compatible compiler test.

* Use ordereddict to fix flaky golden file tests.

* Address PR comments.

* Address PR comments.

* Address PR comments.
2021-03-08 15:40:23 -08:00
S N Munendra 37d6554e5f
fix(sdk): update default value for type_check in compiler docstrings (#5169) 2021-03-01 02:16:16 -08:00