fe04a5a842
fix(sdk): fixes type issues for ParallelFor. Fixes #9366 ( #10436 )
...
* fix type issues
* fix type issue
* fix format
* fix failed test
* fix format
* delete comments
* resolve comments
* resolve comments
* resolve format
* resolve import
* move unnecessary file
* resolve compiler_test failures
* resolve comments
* remove unnecessary imports
* fix format sort
* fix nits
* add new compiled yaml file
* solve merge conflicts
* solve conflicts
* format
* sort
* resolve comments
* resolve comments
2024-02-08 18:34:37 +00:00
b528568718
feat(sdk): Support dsl.ParallelFor over list of Artifacts ( #10441 )
...
* samples message
* update pr number
* Split LoopArgument into LoopParameterArgument and LoopArtifactArgument
* formatting
* address some comments
* resolve release notes conflict
* flatten loops in pipeline_spec_builder
* update artifact type checking logic
* simplify artifact checking logic
* re-add issubtype_of_artifact
* move name_is_loop_argument to for_loop_test.py
* simplify LoopArtifactArgument is_artifact_list=False logic
* update typeerror
* typo
* typo
* small fix
* formatting
* formatting
* remove issubtype_of_artifact()
* small changes
* assert LoopArtifactArgument channel.is_artifact_list is True
* remove whitespace
* remove newline
* Update single artifact check and error message
* formatting
* add unit test for is_artifact_list==False
* formatting
* update valueerror test.
* typo
* regex formatting
* formatting
2024-02-06 23:03:40 +00:00
7bd31d104b
support dsl.importer locally; resolve merge conflicts ( #10431 )
2024-01-29 19:17:15 -08:00
442d457057
feat(sdk): support local pipeline execution ( #10423 )
...
* local pipeline implementation
* address review feedback
2024-01-29 07:50:37 -08:00
a990446b9c
chore(sdk): local task execution refactor + cleanup ( #10420 )
2024-01-23 15:54:03 +00:00
8a5a17e910
feat(sdk): remove local execution feature flag #localexecution ( #10355 )
2024-01-05 20:20:08 +00:00
76aad8b18a
feat(sdk): add local execution output collection #localexecution ( #10325 )
...
* feat(sdk): add local execution output collection #localexecution
* update ExecutorInput processing
* address review feedback
* fix exception handling test
2023-12-16 09:53:18 -05:00
21f8e9c72b
feat(sdk): add subprocess task handler #localexecution ( #10302 )
...
* add subprocess task handler
* response to review feedback
2023-12-15 19:51:55 +00:00
d41efc3e96
update task dispatcher ( #10298 )
2023-12-12 14:23:49 -05:00
5cd708de37
feat(sdk): add local execution skeleton #localexecution ( #10292 )
2023-12-09 14:08:21 -05:00
d3b3e7a5ba
chore(sdk): support Python 3.12 ( #10272 )
...
* chore(sdk): support Python 3.12
* fix kfp-kubernetes test import error
* use assertEqual instead of assertEquals
* fix kfp-kubernetes snapshot yaml
2023-11-29 23:18:03 +00:00
d3323c06f3
feat(sdk): add executor output path and executor input message placeholders ( #10240 )
...
* add support for '{{$.outputs.output_file}}' placeholder
* dedupe executor input code
2023-11-20 20:30:10 +00:00
e3b1863791
chore(sdk): add Python 3.7 EOL notice ( #10139 )
2023-10-24 22:06:08 +00:00
8d00d0eb9a
feat(sdk): support a Pythonic artifact authoring style ( #9932 )
...
* chore(sdk): test observability, refactorings, and cleanup
* implement unified i/o syntax
* clarify container component error message
* address review feedback
* add dill to packages_to_install
* update get_uri logic for KFP and tests
* update execution tests & cleanup
2023-10-19 18:49:58 +00:00
73d51c8a23
fix(sdk): fix bug when `dsl.importer` argument is provided by loop variable ( #10116 )
2023-10-18 20:37:56 +00:00
2d3171cbfe
feat(sdk): support collecting outputs from conditional branches using `dsl.OneOf` ( #10067 )
...
* support dsl.OneOf
* address review feedback
* address review feedback
2023-10-18 02:34:55 +00:00
87005cc015
chore(sdk): wrap conditional branch groups in outer pipeline ( #10034 )
2023-10-05 02:45:15 +00:00
e99f2704fc
feat(sdk): support `PipelineTaskFinalStatus` in tasks that use `.ignore_upstream_failure()` ( #10010 )
...
* support taskfinalstatus in tasks that ignore upstream failure
* address review feedback
2023-09-21 01:46:37 +00:00
20218a9c1d
chore(sdk): test observability, refactorings, and cleanup ( #10005 )
...
* chore(sdk): test observability, refactorings, and cleanup
* address review feedback
* rename classes
2023-09-18 22:51:27 +00:00
d001b8055f
chore(sdk): refactor python component executor ( #9990 )
2023-09-15 22:00:30 +00:00
cc2cd58918
fix(sdk): fix --no-deps flag usage ( #9982 )
...
* update --no-deps flag usage
* update runtime test code
2023-09-13 15:31:02 -07:00
c6b236d1a0
feat(sdk): support dsl.If, dsl.Elif, and dsl.Else ( #9894 )
...
* support if/elif/else
* deprecate dsl.Condition
* alter rebase
* update release notes
* address review feedback
* change BinaryOperation to ConditionOperation
2023-09-11 20:19:35 +00:00
959f54efd7
chore(sdk): release kfp sdk 2.1.3 ( #9896 )
...
* release kfp sdk 2.1.3
* update tests
* update OWNERS file
2023-09-05 18:32:31 +00:00
ef0788d986
feat(sdk): add logging at end of executor execution ( #9895 )
...
* feat(sdk): add logging at end of executor execution
* Update executor.py
* Update executor.py
2023-09-05 17:25:31 +00:00
cf0e0cf87c
feat(sdk): enable dependency-free runtime install of kfp ( #9886 )
...
* handle problematic imports
* separate runtime and compile-time symbols
* wrap kfp/__init__.py imports unavailable at runtime
* update component factory + tests
* add runtime tests
* add --no-deps flag to component factory
* update release notes
* clean up
* handle containerized python components
* update golden snapshots
* update component_factory unit tests
* respond to review feedback
* fix runtime test and compilation logic
* update tests
2023-08-24 18:31:16 +00:00
c9e54798cd
chore(sdk): undo creation of kfp-dsl namespace package ( #9874 )
...
* revert to c8204d0#9742
* modify #9791
* reapply #9800
* reapply #9827
* revert parts of #9738
* reapply parts of #9785
* remove duplicated dsl-test code
* reapply parts of #9791
* correct version
2023-08-15 16:24:52 -07:00
525ff90684
chore(sdk): extract DSL into `kfp-dsl` package ( #9738 )
...
* move dsl dir
* make kfp-dsl a package
* make kfp-dsl a package
* additional changes
* address review feedback
2023-07-25 23:34:02 +00:00
c8204d0285
chore(sdk): partition KFP SDK source code into runtime and non-runtime code ( #9710 )
2023-07-07 23:18:52 +00:00
96742a648a
docs(sdk): add dsl placeholder exports ( #9423 )
2023-05-15 19:46:31 +00:00
5e98b3b6f0
feat(sdk): Support PIPELINE_ROOT_PLACEHOLDER ( #9134 )
...
* Support PIPELINE_ROOT_PLACEHOLDER
/assign @connor-mccarthy
* Update __init__.py
2023-04-11 08:56:49 -07:00
b575950a75
feat(sdk): support for fan-in from `dsl.ParallelFor` ( #8631 )
...
* implement parallelfor fan-in
* temporarily block fan-in of artifacts
* clean up
* unify and clarify error messages
* remove comment
* remove unnecessary compiler condition
* make test cases more realistic
* use better ParallelFor variable name
* add and rename test cases
* add ParallelFor fan-in test cases
* fix return collected bug
* clean up compilation tests
* update docstrings; update error message
* remove fan-in artifacts read/write tests
* implement review feedback
* change use of additional_input_name_for_pipeline_channel
* move additional_input_name_for_pipeline_channel
* provide additional pipeline topology validation and tests; refactor existing validation
* add release note for parameter fan-in
2023-02-04 20:50:51 -08:00
e97fe8e04f
feat(sdk): add pipeline job placholders for scheduled runs ( #8447 )
...
* add pipeline job placeholders
* simplify example code in docs
* add _PLACEHOLDER suffix to variable names
2022-11-23 00:38:42 +00:00
30669e46dc
chore(sdk): improve KFP SDK reference documentation ( #8337 )
...
* use autodoc default options
* sort symbol order by typical usage patterns
* include summary of submodule symbols
* expose Input[] and Output[] type generics
* include tables of contents on API reference docs pages
* expand global TOC for discoverability
* fix missing reference / circular import problem
2022-10-17 19:58:53 +00:00
c7a90a3fa8
feat(sdk): expose IfPresentPlaceholder and ConcatPlaceholder to kfp.dsl ( #8145 )
2022-08-15 23:22:06 +00:00
04c827fc3a
feat(sdk): supporting bring your own container for arbitrary input and outputs ( #8066 )
...
* support container_component decorator for function with no inputs
* resolve review comments
* add sample tests for milestone 1
* modify compiler test data
* resolve reviews
* resolve reviews
* WIP
* implementation of function of no inputs
* fixed sample test
* re-fix sample test
* fix rebase merge conflict
* resolve formatting
* resolve isort error for test data
* resolve comments
* fix nit
* resolve nit
* add implementation for placeholders i/o, sample and compiler tests
* resolve comments and merge logic for constructing container component
* resolve comments
* resolve comments
* fix assertion messages
* add error handling for accessing artifact by itself
* add test for raising error for accessing artifact by itself
2022-08-08 21:00:53 +00:00
c6a5e387dd
docs(sdk): refresh SDK public API docstrings ( #8057 )
...
* add module-level docstrings
* update compiler docstrings
* update registry module docstrings
* add BaseComponent and children to public api, but discourage use
* update artifact docstrings and type annotations
* update dsl docstrings
* update client docstrings
* clean up kfp.__init__
* add dsl placeholder docstrings
* many more docstring updates
* document type aliases in dsl module
2022-07-21 19:51:18 +00:00
d8b9439ef9
add kfp v2 api reference docs ( #8042 )
...
* use __all__ in top-level modules to record public api
* add index and source files
* add kubeflow assets to _static/
* add and pin requirements
* use block quote instead of header for readme notice
* update conf.py
* delete old files
2022-07-19 15:53:54 -06:00
13736b85b6
chore(sdk): clean up kfp code style ( #8027 )
...
* remove unused imports
* use google as isort profile
* sort imports
* format with yapf
* clean end of file new line, trailing whitespace, double quoted strings
2022-07-15 00:06:34 +00:00
8bb0d0e025
feat(sdk)!: move v2 to main namespace ( #7376 )
...
* feat(sdk)!: move v2 to main namespace
* edit samples
2022-03-04 21:56:51 +00:00
b693909a08
feat(sdk)!: move v1 to deprecated folder ( #7291 )
...
* chore(sdk): move v1 to deprecated
* fix testsg
* fix testsg
* fix setup.py
* fix test
* s'
* fix tests
* fix test
* retore v2 test changes
* fix py3.6 test
* fix py3.6 test
* fix py3.6 import fallback error
* remove deprecated
* fix samples test
* sample test
* fix samples
* add readme
* restroe test
* python require
* remove path
* fix tests
* inteegration tests
* remove tfx tests for dependency with kfp v1
* fix e2e
* fix e2e
* fix integration tests
* fix sampe
* move client down
* change to kfp
* add import alias
* fix
* runid
* fix dsl
* only use kfp for function
* revert train_until_good
* tfx test
* kfp
* try import
* onprem
2022-03-04 00:52:59 +00:00
3bbe81a463
chore(sdk): Remove redundant check in set_gpu_limit ( #6866 )
...
* remove redundant check in set_gpu_limit
* release note
2021-11-08 10:35:02 -08:00
dea0823fe8
fix(sdk): Make `Artifact` type be compatible with any sub-artifact types bidirectionally ( #6859 )
2021-11-03 16:34:21 -07:00
2f2ebf8962
fix(sdk): Fix invalid doc example of python_component ( #6841 )
2021-11-02 11:41:06 -07:00
0be57c3880
feat(sdk): Use google.protobuf.Value in v2 for passing parameters. ( #6804 )
...
* Use google.protobuf.Value in v2 for passing parameters.
* retest samples.
* Fix tests.
* Update release, more cleanup.
* Use github.com/kubeflow/pipelines/api from same repo.
* Run go mod tidy
* chore: go mod tidy
* fix v2 compile error and clean up unused code
* pr comments.
* update goldens
* Fix metadata recording.
* Update kfp mlmd client.
* fix test again
* another try.
* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings
* fix v2 launcher + clean up
* fix a compile error
* fix a few more tests
* fix number parsing
* clean up
* disable cache_v2 test.
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
2021-10-28 14:25:51 -07:00
41275b4228
fix(sdk): fixes the specified 'mlpipeline-ui-metadata','mlpipeline-metrics' path is overrided by default value ( #6796 )
...
* fix the issue:
the 'mlpipeline-ui-metadata','mlpipeline-metrics' path is replaced by the default path as '_components._generate_output_file_name(output.name)'
* reformat code
* Update _container_op.py
* Update RELEASE.md
2021-10-28 01:33:00 -07:00
ec4ab2dc4c
fix(sdk): Add missing retry policy. ( #6808 )
2021-10-27 23:41:30 -07:00
7f1e34f17d
chore(sdk.v2): Implement LoopArgument and LoopArgumentVariable v2 ( #6755 )
...
* for_loop v2
* release note
* address review comments
2021-10-19 03:09:23 -07:00
14b3d6751a
fix(sdk): Fix type_utils ( #6719 )
...
* chore(kfp): Fix type_utils
* Update RELEASE.md
2021-10-12 11:17:44 -07:00
b466f59e2f
fix(sdk.v2): Fix a couple of ParallelFor related bugs. Fixes #6383 , fixes #6628 ( #6643 )
...
* fix a couple of loop related bugs
* add release note
2021-09-29 11:27:09 -07:00
4e89973504
fix(sdk): sanitize op name. Fix #6433 ( #6600 )
...
* fix(sdk): sanitize op name. Fix #6433
* Add release note: Fix the error that kfp v1 compiler failed to provide unique name for ops of the same component.
2021-09-28 00:10:10 -07:00
rickyxie0929
KevinGrantLee
Connor McCarthy
IronPan
Scott_Xu
Yaqi Ji
Chen Sun
Toshiya Kawasaki
Ajay Gopinathan
dinimicky
Joshua Carp
chongyouquan
Lin Yiming