c03701b82a
fix(sdk): fix wrong kfp import causes wrong sdk_version being set in pipeline_spec. ( #7433 )
...
* fix wrong kfp import in compiler
* release note
2022-03-18 00:05:39 +00:00
5ca4cb78d3
chore(sdk): remove unused imports and variables ( #7393 )
2022-03-09 21:42:11 +00:00
8bb0d0e025
feat(sdk)!: move v2 to main namespace ( #7376 )
...
* feat(sdk)!: move v2 to main namespace
* edit samples
2022-03-04 21:56:51 +00:00
b693909a08
feat(sdk)!: move v1 to deprecated folder ( #7291 )
...
* chore(sdk): move v1 to deprecated
* fix testsg
* fix testsg
* fix setup.py
* fix test
* s'
* fix tests
* fix test
* retore v2 test changes
* fix py3.6 test
* fix py3.6 test
* fix py3.6 import fallback error
* remove deprecated
* fix samples test
* sample test
* fix samples
* add readme
* restroe test
* python require
* remove path
* fix tests
* inteegration tests
* remove tfx tests for dependency with kfp v1
* fix e2e
* fix e2e
* fix integration tests
* fix sampe
* move client down
* change to kfp
* add import alias
* fix
* runid
* fix dsl
* only use kfp for function
* revert train_until_good
* tfx test
* kfp
* try import
* onprem
2022-03-04 00:52:59 +00:00
577902ca46
fix(sdk): Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest ( #7141 )
...
* Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest
* Update test for TTL
* Declare fix in release note of SDK
* update syntaxe
* Update RELEASE.md
Add Braking change due to incompatibility with KFP pre 1.7 due to Argo 2.X
2022-01-20 00:43:52 +00:00
2460aaa653
test: migrate sample integration test to v2 sample test infra ( #7115 )
...
* test: migrate sample integration test to v2 sample test infra
* test: configure v2 integration test timeout
2021-12-26 04:17:44 +00:00
8aee62142a
feat(sdk)!: Deprecate V2 compatible mode in v1 compiler ( #6958 )
...
* Deprecate V2 compatible mode in v1 compiler
* release note
* fix tests
2021-11-24 22:11:34 +00:00
439d8c87c1
chore(sdk): remove dead code on importer check in v1. ( #6508 )
...
* remove dead code
* Update RELEASE.md
* Update compiler.py
* Update v2_compatible_compiler_test.py
* Update compiler_tests.py
* Update RELEASE.md
2021-09-07 15:01:06 -07:00
2f19a26ffd
chore(sdk): Format all Python files under SDK folder. ( #6501 )
...
* Reformat sdk only using the new yapf config.
* Reformat docstrings using docformatter.
* update golden files to resolve diff caused by whitespaces
* fix some tests
* format .py files under sdk/python/tests using yapf
* additional docformatter
* fix some tests
2021-09-03 11:25:11 -07:00
434e5c3489
fix(sdk): block dsl.importer usage in KFP OSS. Fixes : #6323 ( #6330 )
...
* block dsl.importer in KFP OSS
* address cr comments
2021-08-16 01:02:06 -07:00
befd89c6ec
fix(sdk): pipeline name is required in v2 compatible mode. Fixes #6197 ( #6332 )
...
* fix(sdk): pipeline name is required in v2 compatible mode
* Update xgboost_sample.py
2021-08-13 14:00:04 -07:00
ed2ba1a6b3
chore(sdk): Argo lint requires '--kinds=workflows' argument for offline linting. ( #6135 )
2021-07-26 00:40:41 -07:00
b200e1bc7d
fix(sdk): Fix URI placeholder in v2 compatible mode. ( #6040 )
...
* fix uri placeholder in v2 compatible mode
* fix tests
* fix path generation
* fix tests
* fix test
* cleanup
* clean up
* fix test
* fix test
* fix test
2021-07-21 08:32:50 -07:00
354e4251e8
chore: add offline linting ( #5981 )
...
* Added the offline linting
* update after feedback
2021-07-20 08:39:15 -07:00
fff62e1521
chore(v2): unify pipeline name between MLMD and KFP DB. Fixes #5978 ( #6038 )
...
* chore(sdk): unify pipeline name between MLMD and KFP DB
* backend changes pipeline-name during upload
* update sdk snapshots
* add backend tests
* fix launcher
* fix tests
* address feedback
2021-07-14 20:53:51 -07:00
8a256db1bf
feat(sdk/dsl/compiler): dsl-compile --mode flag to turn on V2_COMPATIBLE, defaults to KF_PIPELINES_COMPILER_MODE env var. Fixes #5840 ( #5952 )
...
* feat(sdk/dsl/compiler): support --mode flag which can turn on v2 compatible mode
* override compiler default mode using KF_PIPELINES_COMPILER_MODE env var
* update V1_LEGACY to V1
* add unit tests
* address feedback
* clean up
* cleanup again
* use absl.testing.parameterized for table driven tests
* update
2021-07-03 02:22:49 -07:00
e7c281bd65
chore: Change python/kfp/compiler and cli folders from relative to absolute path ( #5891 )
...
* Update deprecated download_as_string to download_as_bytes
* Add more usage
* chore: Change python/kfp/cli from relative to absolute path
* chore: Change python/kfp/compiler from relative to absolute path
2021-06-22 11:28:31 -07:00
2991602cfc
feat(sdk): Better error message when compiling for v2. Fixes #5688 , Fixes #5727 ( #5759 )
...
* Better error message when compiling for v2
* update samples
2021-05-31 13:39:03 -07:00
fa5ac25678
fix(sdk): Avoid inserting duplicate input parameters in v2 compatible mode. ( #5740 )
2021-05-28 15:06:00 -07:00
7629e10130
chore(sdk): warn v2 compatible mode is Beta quality ( #5658 )
2021-05-17 21:55:40 -07:00
6717434978
[SDK] Add pod labels for telemetry purpose ( #5582 )
...
* Add pod labels for telemetry purpose
* fixed test
* added sdk label in pods
* added sdk type label
* fixed test
* added UT back
* updated UT
2021-05-05 10:43:27 -07:00
cc83e1089b
Assigned copyright to the project authors ( #5587 )
2021-05-05 13:53:22 +08:00
0dba00518c
chore(sdk.v2):add v2 compact metrics label ( #5566 )
...
* add-v2-compact-metrics-label
* add telemetry pod label
2021-04-29 16:06:30 -07:00
5dafda1270
feat(sdk.v2): Support explicit importer ( #5502 )
2021-04-19 13:14:09 -07:00
5f97a4502d
feat(sdk.v2): remove pipeline_root from compiler interface ( #5492 )
2021-04-16 08:49:13 -07:00
f32915e88f
feat(sdk.v2): Support setting custom job spec in v2. ( #5454 )
...
* Support setting custom job spec.
* auto fill custom job jobSpec using container spec.
* fix test
* address comments
2021-04-10 09:20:04 -07:00
a1d453af21
chore(sdk): Allow the launcher image to be configurable. ( #5360 )
...
* Allow the launcher image to be configurable. This is especially useful
for tests.
* Address PR comments.
2021-03-23 08:27:44 -07:00
8a32632edf
fix(sdk): make workflow validation compatible with argo CLI v2.12+. Fixes #4775 , Fixes #5296 ( #4804 )
...
* updated the validation default test
* updated the version
* Update compiler.py
* Update compiler.py
* Update compiler.py
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
2021-03-18 21:10:16 -07:00
83eded130c
feat(sdk): Introduce experimental v2-compatibility in KFP SDK ( #5218 )
...
* WIP: Enable v2 compatibility in KFP SDK compiler.
* First pass clean up
* Clean up and introduce enum instead of boolean for execution mode.
* More cleanup
* Clean up and add comments.
* Undo formatting changes.
* Undo formatting changes.
* Add method to unconditionally add kfp pod env.
* minor formatting change.
* Update docstrings.
* Undo formatting changes.
* fix imports.
* fix pod_env tests
* rebased.
* undo format changes.
* Undo compiler changes.:
* format _default_transformers.py for consistency
* Fix various rebasing issues.
* fix bug referring to pipeline_name/pipeline_root in v1 pipelines.
* revert output dir name.
* allow both types of attributes for pipeline root.
* fix pod env yaml golden.
* fix for input/output uri tests.
* Add v2 compatible compiler test.
* Use ordereddict to fix flaky golden file tests.
* Address PR comments.
* Address PR comments.
* Address PR comments.
2021-03-08 15:40:23 -08:00
37d6554e5f
fix(sdk): update default value for type_check in compiler docstrings ( #5169 )
2021-03-01 02:16:16 -08:00
f27d3b81b8
chore(sdk): remove redundant method: `_pipelineparam_full_name()` ( #5174 )
2021-02-25 16:52:15 -08:00
cfe59139be
fix(sdk): generate valid pipeline from non-ascii name. Fixes #5119 ( #5120 )
2021-02-18 18:20:30 -08:00
846423a870
feat(sdk): Always add pipeline root as a pipeline parameter ( #5122 )
...
* refactor pipeline root passing
* fix test
2021-02-10 16:29:57 -08:00
86f03a90a3
chore: Improve the error message when incompatible placeholders are connected ( #5082 )
2021-02-03 00:10:36 -08:00
ce985bc287
fix(sdk): Allow keyword-only arguments in pipeline function signature ( #4544 )
...
* add test for keyword-only arguments in pipeline func
* fix: kwargs-only argument for pipeline func
* test: kwargs generate same yaml as args
* remove whole metadata
* assert -> self.assertEqual
* programmatic example --> fixed example
* same name for both
Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com>
2021-01-29 18:31:02 -08:00
5633b9abda
fix(sdk): fixes unresolved PipelineParam when static list passed to dsl.ParallelFor. Fixes #4890 ( #4891 )
...
* fix parallelfor compiling items + add tests
* remove debug print
* fix tests
* fix parallelfor_pipeline_param_in_items_resolving test
* debug test
* fix tests
* Revert "debug test"
This reverts commit 57451143bd
2021-01-14 00:09:03 -08:00
7540ba5c3b
feat(sdk): Implements artifact URI placeholder. ( #4932 )
...
* add placeholder to spec
* add output_directory to pipeline
* respect uri placeholder in file outputs
* wip: add data passing rewriting logic to respect the uri semantics
* merge input_uri and paths when instantiating ContainerOp
* fix
* fix workflow rewriting
* Add topology rewriting
* add a test case, and various fixes
* make the test case more complex
* Fix the case when working with OpsGroup
* Fix test case
* fix resolving test
* fix redundant cmd lines
* fix redundant cmd lines
* resolve comments
* fix file outputs
* resolve comments
* copy file outputs instead of modifying inplace.
2021-01-05 20:39:51 -08:00
24732b9dae
feat(compiler): add dsl operation for parallelism on sub dag level ( #4199 )
...
* Added subdag parallelism
Authored-by: NikeNano <niklas.sven.hansson@gmail.com>
Co-authored-by: guanhuichen <guanhuichen@gmail.com>
* added error handling, fixed comment and refactored
* updated with sleep and TODO
* fix imports
Co-authored-by: guanhuichen <guanhuichen@gmail.com>
2020-12-26 22:10:27 -08:00
5a4b70e37c
feat(sdk): Add settings of the dnsConfig field. Fixes #4836 ( #4837 )
...
* feat(sdk): Add settings of the dnsConfig field. Fixes #4836
* feat(sdk): Add dnsConfig example and sample.
* feat(sdk): Refactor dnsConfig param.
* feat(sdk): Refactor dnsConfig param.
2020-12-14 20:05:49 -08:00
fb15223f7e
chore: Add doc strings marking the feature stages for SDK. ( #4575 )
...
* add doc strings
* Simplify the docstring
* fix unittest
* recover cli.py
* recover cli.py
* substitute docstring in resource ops with TODOs
* revert stable labels
2020-11-24 00:19:00 -08:00
f7874d38ff
fix(sdk): Compiler - Fixed pipeline parameters with empty default values ( #4552 )
...
Fixes https://github.com/kubeflow/pipelines/issues/4549
2020-11-12 15:52:28 -08:00
9efc9e59b2
fix(sdk) - fixes missing import in KFP compiler ( #4741 )
2020-11-09 17:30:41 -08:00
c32ea232d5
feat(compiled): set pod disruption budget for pipelines. Fixes #3877 ( #4178 )
...
* Update _client.py
* Update _client.py
* added pod disruption budget
* clean up
* Update sdk/python/kfp/dsl/_pipeline.py
* fixed parameter
* updated after feedback
* removed selector
2020-09-14 13:45:26 -07:00
22b7b99a8b
fix(sdk): Fix opsgroups dependency resolution ( #4370 )
2020-08-27 09:03:53 -07:00
704c8c7660
chore: Clean up KFP SDK docstrings, make formatting a little more consistent ( #4218 )
...
* Prepare SDK docs environment so its easier to understand how to build the docs locally so theyre consistent with ReadTheDocs.
* Clean up docstrings for kfp.Client
* Add in updates to the docs for compiler and components
* Update components area to add in code references and make formatting a little more consistent.
* Clean up containers, add in custom CSS to ensure we do not overflow on inline code blocks
* Clean up containers, add in custom CSS to ensure we do not overflow on inline code blocks
* Remove unused kfp.notebook package links
* Clean up a few more errant references
* Clean up the DSL docs some more
* Update SDK docs for KFP extensions to follow Sphinx guidelines
* Clean up formatting of docstrings after Ark-Kuns comments
2020-08-04 00:33:47 +08:00
bbc9ff5ec3
SDK - Compiler - Validating Argo validator ( #3874 )
...
* SDK - Compiler - Validating Argo validator
* Added warning if argo is available, but not working
2020-07-10 19:07:21 -07:00
c6ac83f72c
feat: add parallelism for dsl.ParallelFor. Fixes #4089 ( #4149 )
...
* Added parallism at sub-dag level
* updated the parallism
* remove yaml file
* reformatting
* Update sdk/python/kfp/compiler/compiler.py
* Update sdk/python/kfp/compiler/compiler.py
* Update samples/core/loop_parallelism/loop_parallelism.py
Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com>
Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com>
2020-07-08 11:27:13 -07:00
48889a99d1
fix(sdk): Compiler - Fixed input artifact name sanitization when using raw string arguments. Fixes #4110 ( #4120 )
2020-07-08 10:43:09 -07:00
6960366846
fix(sdk): Compiler - Fixed the input argument mapping when using dsl.graph_component. Fixes #3915 (4082)
...
* SDK - Compiler - Fixed the input argument mapping when using dsl.graph_component
Fixes https://github.com/kubeflow/pipelines/issues/3915
* Stopped relying on the argument order at all
This can make the compilation less fragile.
2020-06-29 02:31:37 -07:00
b099c6f5d3
chore: Rollback telemetry related changes (4088)
...
* Revert "fix length (#3934 )"
This reverts commit 7fbb7cae#3861 )"
This reverts commit 1e2b9d4e#3578 )"
This reverts commit aa8da64b
2020-06-27 15:46:14 -07:00
Chen Sun
Connor McCarthy
Yaqi Ji
Seb
Yuan (Bob) Gong
Niklas Hansson
capri-xiyue
Alexey Volkov
Ajay Gopinathan
S N Munendra
Yuku Takahashi
Jiaxiao Zheng
Michalina Kotwica
radcheb
Kenta Onishi
Asav Patel
Niklas Hansson
Victor
Alex Latchford