c9e54798cd
chore(sdk): undo creation of kfp-dsl namespace package ( #9874 )
...
* revert to c8204d0#9742
* modify #9791
* reapply #9800
* reapply #9827
* revert parts of #9738
* reapply parts of #9785
* remove duplicated dsl-test code
* reapply parts of #9791
* correct version
2023-08-15 16:24:52 -07:00
540294aedb
fix(sdk): fix GCPC break in KFP SDK ( #9791 )
2023-07-27 23:02:32 +00:00
525ff90684
chore(sdk): extract DSL into `kfp-dsl` package ( #9738 )
...
* move dsl dir
* make kfp-dsl a package
* make kfp-dsl a package
* additional changes
* address review feedback
2023-07-25 23:34:02 +00:00
c8204d0285
chore(sdk): partition KFP SDK source code into runtime and non-runtime code ( #9710 )
2023-07-07 23:18:52 +00:00
48ecc7d879
fix(sdk): fix bug when iterating over output from upstream task in nested parallelfor loops ( #9580 )
...
* refactor for clarity
* fix iter over upstream output in nested parallelfor
* update release notes
* support empty `List` generic annotation
* fix typos
* add comment about refactoring opportunity
* fix annotation
* update comment again
2023-06-09 08:58:28 -07:00
296120290b
fix(sdk): relax executor type checking ( #9481 )
...
* relax executor type checking
* address review comments
* add release note
2023-05-23 19:05:35 +00:00
78902274f1
feat(sdk): support compiling components with `dsl.PipelineTaskFinalStatus` type ( #9082 )
2023-05-12 22:14:28 +00:00
946c51bafe
feat(sdk): support extracting input/output descriptions from component/pipeline docstrings ( #9156 )
...
* feat(sdk): support extracting input/output descriptions from component/pipeline docstrings
* remove numpydoc tests
2023-04-24 17:16:44 +00:00
41c0d78f5f
fix(sdk): fix handling of PipelineTaskFinalStatus in ir_type_ attribute hack ( #9186 )
2023-04-20 13:53:31 -07:00
91abbeaf2f
feat(sdk): support `display_name` and `description` in `@dsl.pipeline` decorator ( #9153 )
...
* feat(sdk): support display_name and description in @dsl.pipeline decorator
* add release note
* test additional case
2023-04-17 14:15:39 -07:00
3164dfb01a
fix(sdk): fixes module not found error for containerized python components. Fixes #8385 ( #9157 )
...
* Add module directory to sys.path
* Add nested module imports unit test
* Add release note to release.md
2023-04-17 11:46:02 -07:00
a6ef77b784
feat(sdk): Support setting cpu/memory requests. ( #9121 )
...
* Support setting cpu/memory requests.
* address review comments
* address review comments
2023-04-10 23:35:16 -07:00
e907b6343e
set _ir_type attribute ( #9105 )
2023-04-10 12:01:36 -07:00
5828632c2f
feat(sdk): support indexing into top-level artifact metadata in Container Components ( #9131 )
2023-04-10 11:46:01 -07:00
3026e452ff
fix(sdk): fix loading non-canonical type strings from v1 component YAML ( #9041 )
2023-03-27 10:25:23 -07:00
2bbfd5e89f
fix: support setting task dependencies via kfp.kubernetes.mount_pvc ( #8999 )
...
* enable accessing .task on pipeline channel
* set task dependencies in mount_pvc
* update tests
2023-03-16 21:10:54 -07:00
e96754cb53
feat(sdk): deprecate .add_node_selector_constraint in favor of .set_accelerator_type ( #8980 )
...
* rename method
* update docstring
2023-03-14 09:22:35 -07:00
a38d7479d7
feat(sdk): support compiling platform specific features ( #8940 )
...
* support compiling platform-specific features
* fix map size change during iteration
2023-03-10 11:18:25 -08:00
6777e8f445
fix(sdk): Support python 3.11 ( #8907 )
...
* new branch
* add to release.md
* edit setup.py
2023-03-03 13:36:53 -08:00
ed66ba327f
feat(sdk): using component's pip_index_urls for Dockerfile generation. Fixes #8816 ( #8871 )
...
* feat(sdk): using component's pip_index_urls for Dockerfile generation
* Update sdk/python/kfp/cli/component.py
Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
* feat(sdk): PR changes: removed additional space from index url options string; duplicate removal not necessary
* feat(sdk): feature mentioned in RELEASE.md
* deduplication back - more components in a file can specify same pypi url - we need it only once for docker image
* feature note moved to current development section
---------
Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
2023-03-01 09:32:56 -08:00
87062dbf22
fix(sdk): fix attribute error for Containerized Python Components ( #8887 )
2023-02-27 16:42:50 -08:00
8d3e0dd805
fix(sdk): Fix environment variable set in component yaml lost during compilation. Fixes #8884 ( #8885 )
...
* Fix env set in component yaml not captured
* release note
* clean up unnecessary code in test
2023-02-23 14:28:26 -08:00
386ff88236
fix(sdk): fix unknown artifact string in dsl.OutputPath/dsl.InputPath ( #8880 )
2023-02-23 08:37:04 -08:00
c046879dda
support fanning-in artifacts ( #8808 )
2023-02-22 10:39:00 -08:00
41752608bc
support list of artifact input placeholders ( #8484 )
2023-02-16 09:26:53 -08:00
1e1a02066a
fix(sdk): Add outputDefinitions of root pipeline to compiled components IR YAML file ( #8848 )
...
* base
* add to readme
* adress comments
2023-02-15 09:28:33 -08:00
5684bfccff
feat(sdk): Ignore upstream failure ( #8838 )
...
* main
* adress comments
* adress comments
* adress comments 2
* address comments 3
* address comments 4
2023-02-13 11:40:42 -08:00
e7ad073f1d
feat(sdk): deprecate .set_gpu_limit in favor of .set_accelerator_limit ( #8836 )
2023-02-09 15:46:10 -08:00
b575950a75
feat(sdk): support for fan-in from `dsl.ParallelFor` ( #8631 )
...
* implement parallelfor fan-in
* temporarily block fan-in of artifacts
* clean up
* unify and clarify error messages
* remove comment
* remove unnecessary compiler condition
* make test cases more realistic
* use better ParallelFor variable name
* add and rename test cases
* add ParallelFor fan-in test cases
* fix return collected bug
* clean up compilation tests
* update docstrings; update error message
* remove fan-in artifacts read/write tests
* implement review feedback
* change use of additional_input_name_for_pipeline_channel
* move additional_input_name_for_pipeline_channel
* provide additional pipeline topology validation and tests; refactor existing validation
* add release note for parameter fan-in
2023-02-04 20:50:51 -08:00
84241d6460
fix(sdk): fix deserializing v1 component yaml defaults ( #8639 )
...
* fix deserializing v1 defaults
* update test cases
* update type annotation
* add unit tests
2022-12-29 17:48:58 -05:00
151e8a3c6e
feat(sdk): support optional artifact inputs ( #8623 )
...
* add compiler tests
* support optional artifact inputs
* update executor test
* update existing golden snapshot
* add new compiler test/snapshot
* add release note
* bump kfp-pipeline-spec dependency
2022-12-28 13:40:12 -05:00
5491f6f3db
fix(sdk): support writing/reading isOptional for input parameters ( #8612 )
...
* add tests
* support writing/reading isOptional for parameters
* update golden snapshots
* temporarily maintain workaround for v1 optional artifact inputs
* add release note
2022-12-21 14:46:59 -08:00
f64348f29e
fix(sdk): make executor output writer more permissive [sdk v2] ( #8607 )
2022-12-21 00:23:13 +00:00
2eef97cfc7
fix(sdk): add special handling for PipelineTaskFinalStatus to get_parameter_type ( #8587 )
2022-12-19 18:48:46 +00:00
08b2408d3d
feat(sdk): add support for lists of artifacts in Python components [support lists of artifacts pt. 3] ( #8465 )
...
* remove unused codepath
* rename to avoid confusing variable names
* implement support in executor
* add comment
* make executor code safer
2022-12-15 21:34:21 +00:00
d4b896d3e3
fix(sdk): new fix for multiple workers writing with gcsfuse bug [KFP SDK v2] ( #8544 )
...
* fix and test
* update release notes
* extract constants
2022-12-13 18:01:29 +00:00
f4588f3112
chore(sdk): idiomatic Python pure refactorings ( #8537 )
...
* use f-strings
* inline immediately returned variables
* simplify comprehension
* use list literals
* lift return statements into control flow blocks
* other pure refactorings
* remove passthrough function
* use a ternary when it improves readability
* use __getitem__ for regex match groups; introduced in Python 3.6
* fix changed comments
* revert one change
2022-12-06 23:44:36 +00:00
49db63c916
feat(sdk): Add comments to IR YAML file ( #8467 )
...
* base
* add tests
* fix bug
* adress comments
* address comments 2
* sort comments
* sort signatures
* add indempotent test
* add indempotent test2
* support multiline docstring
* review
* docformatter presubmit exclude
* docformatter presubmit exclude
* docformatter
* docformatter
* merge 1
* update readme
* nit .items()
* remove reduntant test
2022-12-05 14:43:14 -08:00
00c6b01040
fix(sdk): fix bug when referencing component.pipeline_spec in a pipeline body ( #8514 )
...
* add tests
* implement fix
* add TODO
2022-12-02 09:35:16 +00:00
614c231ae6
fix(sdk): unblock valid topology. ( #8416 )
...
* unblock valid topology
* add more tests
* handle git fork
* sample_test_cases
* main
* restore to master
* resolve comments on PR
* resolve conflicts
* resolve conflict 2
* revert conflict fix
* fix changes
* address comments
* review
* docformatter presubmit
* revert docformatter
* update release.md
2022-12-02 00:15:15 +00:00
e7c82c0593
feat(sdk): support list of artifacts annotations [list of artifacts support pt. 1] ( #8464 )
...
* update get_io_artifact_class function
* update various type annotation tests
* support list of artifacts in component interfaces
* move helper function
* clarify function name
* respond to review feedback
2022-11-30 05:24:15 +00:00
b829e4a5da
feat(sdk): enable use of primitive placeholders in f-string ( #8494 )
...
* support primitive placeholders in fstring
* add read/write test component
2022-11-28 14:45:48 -08:00
4bb57e6723
fix(sdk): fix multiple workers writing with gcsfuse bug [KFP SDK v2] ( #8455 )
...
* fix(sdk): fix multiple workers writing with gcsfuse bug [KFP SDK v1]
* add comment for context
2022-11-23 00:38:48 +00:00
be766dc352
docs(sdk): update pipeline task .set_gpu_limit reference docs ( #8477 )
2022-11-18 17:58:22 +00:00
6b3b4df3d2
refactor(sdk): simplify InputSpec ( #8422 )
...
* simplify usage of inputspec
* remove underscore from optional
* add error message and test
* remove optional
2022-11-14 18:39:26 +00:00
d33f359797
fix(sdk): fix nested placeholders and block illegal IfPresent form in Concat ( #8414 )
...
* support single element IfPresentPlaceholders that contain a primitive placeholder
* block list then and else_ inside concat
* support single element IfPresent in v1
* clean up typing
* skip loading batch_predict_job yaml
2022-11-04 18:20:16 +00:00
08a8b1df97
refactor(sdk): remove dead BaseModel code ( #8415 )
...
* remove dead basemodel code
* remove unused deserialization logic
* remove more dead code
* remove basemodel entirely
2022-11-03 21:58:15 +00:00
4c6abe5afc
refactor(sdk): simplify placeholders and logic ( #8402 )
...
* simplify placeholders
* update placeholders_test.py
* update structures to use placeholders correctly
* update use of placeholders elsewhere in codebase
* clean up
* move placeholder type checking logic to PipelineTask to prevent v1 load breaks
* clean up
* update docstrings
* make to_dict and to_string private methods
* revert unnecessary change to tests
* improve type safety of code
* prefix module-level constant with underscore
2022-11-02 00:31:45 +00:00
ca6ad27020
chore(sdk): remove unused linter comments ( #8405 )
2022-11-01 18:31:46 +00:00
ac02a0e191
fix(sdk): Throw exception for component parameter named Output ( #8367 )
...
* fix(sdk): Throw exception for component parameter named Output
* Add unit test
* lint fix
* nit fixes
* nit
* nit
* format fixes
2022-10-17 23:40:27 +00:00
Connor McCarthy
Chen Sun
Mithil Poojary
JOCSTAA
b4sus
Sune Debel