For context, the commit 70aaf8a9a4 removed
support for the old fields (without a resource_ prefix). This was
added back in commit 6ebf4aae03 but done in a way
that broke any usage of pipeline input parameters but was to support the
current KFP backend which did not yet support the new fields. In commit
7c931ae201, the old fields were removed
again but added support for the new field in KFP backend.
This commit addresses the case where a user is using a new SDK but with
a KFP backend prior to 2.4.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* make component names in dag more unique
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
* tweak ordering so that renaming is handled in LIFO fashion
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
* add test that reuses a pipeline multiple times
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
---------
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
The API introduced new fields prefixed with Resource (e.g.
ResourceCpuLimit) to replace the old fields without the prefix. The
Driver hadn't been updated to honor those fields but the SDK started
using them which led to unexpected behavior.
The Driver now honors both fields but prioritizes the new fields. The
SDK now only sets the new fields.
The outcome is that resource limits/requests can now use input
parameters.
Note that pipeline_spec_builder.py was doing some validation on the
limits/requests being set, but that's already handled in the user facing
method (e.g. set_cpu_limit).
Resolves:
https://github.com/kubeflow/pipelines/issues/11500
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
This change allows component base images to be parameterized using runtime pipeline parameters. The container images can be specified within an @pipeline decorated function, and takes precedence over the @component(base_image=..) argument.
This change also adds logic to resolve these runtime parameters in the argo driver logic. It also includes resolution steps for resolving the accelerator type which functions the same way but was missing the resolution logic. The resolution logic is a generic workaround solution for any run time pod spec input parameters that cannot be resolved because they cannot be added dynamically in the argo pod spec container template.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* feat(sdk): Allow setting a default of execution caching disabled via a compiler CLI flag and env var
Co-authored-by: Greg Sheremeta <gshereme@redhat.com>
Signed-off-by: ddalvi <ddalvi@redhat.com>
* Add tests for disabling default caching var and flag
Signed-off-by: ddalvi <ddalvi@redhat.com>
---------
Signed-off-by: ddalvi <ddalvi@redhat.com>
Co-authored-by: Greg Sheremeta <gshereme@redhat.com>
* fix(sdk): Prevents dsl.ParallelFor over single paramter from compiling.
* fix(sdk): Prevents dsl.ParallelFor over single paramter from compiling.
* update PR number in release notes
* formatting
* Add compiler_test.py test for single param compile failure
* Update some docstrings and add todo
* formatting
* Update sdk/python/kfp/compiler/compiler_test.py
Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
* Update sdk/python/kfp/compiler/compiler_test.py
Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
* Update sdk/python/kfp/dsl/for_loop.py
Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
* Use print_and_return and other small changes
* typo
* typo
---------
Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
* revert to c8204d0
* reapply #9742
* modify #9791
* reapply #9800
* reapply #9827
* revert parts of #9738
* reapply parts of #9785
* remove duplicated dsl-test code
* reapply parts of #9791
* correct version
* remove dependency of deprecated on v2
* use new artifact identification logic
* add custom artifact type compiler test
* compile schema_version from non-kfp artifacts
* update tests
* add importer to compiler test
* update golden snapshots
* fix type checking code
* undo executor_test diff
* remove gcpc types compiler test
* add compilation test comment
* update google artifact handling
* update compiler test
* only permit system. and google. artifact types; add tests
* update compiler test
* simplify type checking logic
* use bundled artifact types in PipelineArtifactChannel
* raise error on unknown v1 input or output
* add type utils and move functions
* add type utils tests
* remove schema_version from inputspec and outputspec
* update artifact proto creation and tests
* propogate bundled artifact type string changes throughout sdk
* update golden snapshot
* remove unused v1 compat and tests
* update type checking and tests to validate schema_version using semantic versioning
* update importer sample
* update test data config
* update test_data config
* clean up sample
* clean up sample
Mai Nakagawa
Matt Prahl
Stijn Tratsaert
Ze Mao
Humair Khan
Ricardo Martinelli de Oliveira
Dharmit Dalvi
KevinGrantLee
Kevin James
Diego Lovison
Connor McCarthy
rickyxie0929
Chen Sun
JOCSTAA