* feat(backend/sdk): enable dsl.Collected for params & artifacts
Signed-off-by: zazulam <m.zazula@gmail.com>
* feat(backend): collect through loops & dags
Signed-off-by: zazulam <m.zazula@gmail.com>
To enable users to use loops similar to subdags, the initial collecting
implementation went only 1 layer deep of loops/subdags. This
implementation serves to handle multifacted approaches of pipelines that
users can generate.
---------
Signed-off-by: zazulam <m.zazula@gmail.com>
It is checked that not both are passed in as parameters, however in the
case of passing in the `pipeline_name` the `pipeline_id` is fetched via
the client and then both values are passed to `upload_pipeline_version`
defeating the initial check and raising an error.
Simply removing the `pipeline_name` argument from the call to
`upload_pipeline_version` fixes this as the `pipeline_id` is always set.
Signed-off-by: Christian Junker <chjdev@users.noreply.github.com>
For context, the commit 70aaf8a9a4 removed
support for the old fields (without a resource_ prefix). This was
added back in commit 6ebf4aae03 but done in a way
that broke any usage of pipeline input parameters but was to support the
current KFP backend which did not yet support the new fields. In commit
7c931ae201, the old fields were removed
again but added support for the new field in KFP backend.
This commit addresses the case where a user is using a new SDK but with
a KFP backend prior to 2.4.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
AFAIU no remote schema is currently supported for Local. This patch
consolidates all possible values under an Enum and then checks the
supplied url against all possible prefixes.
Signed-off-by: Ihar Hrachyshka <ihar.hrachyshka@gmail.com>
* make component names in dag more unique
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
* tweak ordering so that renaming is handled in LIFO fashion
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
* add test that reuses a pipeline multiple times
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
---------
Signed-off-by: Stijn Tratsaert <stijn.tratsaert.it@gmail.com>
This allows dsl.import to leverage Modelcar container images in an OCI
repository. This works by having an init container prepull the image and
then adding a sidecar container when the launcher container is running.
The Modelcar container adds a symlink to its /models directory in an
emptyDir volume that is accessible by the launcher container. Once the
launcher is done running the user code, it stops the Modelcar
containers.
This approach has the benefit of leveraging image pull secrets
configured on the Kubernetes cluster rather than require separate
credentials for importing the artifact. Additionally, no data is copied
to the emptyDir volume, so the storage cost is just pulling the Modelcar
container image on the Kubernetes worker node.
Note that once Kubernetes supports OCI images as volume mounts for
several releases, consider replacing the init container with that
approach.
This also adds a new environment variable of PIPELINE_RUN_AS_USER to
set the runAsUser on all Pods created by Argo Workflows.
Resolves:
https://github.com/kubeflow/pipelines/issues/11584
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
This adds support for the following placeholders in the KFP backend:
- dsl.PIPELINE_JOB_NAME_PLACEHOLDER
- dsl.PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER
- dsl.PIPELINE_JOB_ID_PLACEHOLDER
- dsl.PIPELINE_TASK_NAME_PLACEHOLDER
- dsl.PIPELINE_TASK_ID_PLACEHOLDER
Resolves:
https://github.com/kubeflow/pipelines/issues/10453
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* bump requests-toolbelt to 1.0.0
Signed-off-by: Simon De Ridder <simon.deridder+gitlab@silverfin.com>
* remove changes in test/kfp-functional-test
Signed-off-by: Simon De Ridder <simon.deridder+gitlab@silverfin.com>
---------
Signed-off-by: Simon De Ridder <simon.deridder+gitlab@silverfin.com>
The SDK execution tests were not run on the PR CI that fixed the
pipeline run state to be failed even when an exit handler is run after a
failed component.
This also makes the SDK execution tests run whenever the backend
compiler code is modified to help prevent a similar issue.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
The API introduced new fields prefixed with Resource (e.g.
ResourceCpuLimit) to replace the old fields without the prefix. The
Driver hadn't been updated to honor those fields but the SDK started
using them which led to unexpected behavior.
The Driver now honors both fields but prioritizes the new fields. The
SDK now only sets the new fields.
The outcome is that resource limits/requests can now use input
parameters.
Note that pipeline_spec_builder.py was doing some validation on the
limits/requests being set, but that's already handled in the user facing
method (e.g. set_cpu_limit).
Resolves:
https://github.com/kubeflow/pipelines/issues/11500
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
This change allows component base images to be parameterized using runtime pipeline parameters. The container images can be specified within an @pipeline decorated function, and takes precedence over the @component(base_image=..) argument.
This change also adds logic to resolve these runtime parameters in the argo driver logic. It also includes resolution steps for resolving the accelerator type which functions the same way but was missing the resolution logic. The resolution logic is a generic workaround solution for any run time pod spec input parameters that cannot be resolved because they cannot be added dynamically in the argo pod spec container template.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* chore(sdk): support Python 3.13
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
* chore: remove python_requires upperbound for kfp-kubernetes
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
---------
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
An api breaking change was introduced in 2.10 that removed deprecate
fields in the compilation step for the accelerator fields. This has
resulted in the driver being unable to fetch the old fields. This change
re-introduces the deprecated fields to allow for a proper timespan for
allowing driver to adjust to the new values.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* chore(sdk): remove kfp.deprecated and legacy samples and tests
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
* chore: clean up unused imports
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
---------
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
* chore(sdk): release KFP SDK 2.10.0
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
* fix: version check regex in test
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
---------
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
* feat(sdk)!: stop auto-populating metrics as dag output
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
* add release note
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
---------
Signed-off-by: Chen Sun <chensun@users.noreply.github.com>
KFP v1 supported setting pipeline-level configuration via a
`PipelineConf` class. This class was deprecated and no replacement
was added to KFP v2.
add new PipelineConfig class to support setting pipeline-level
configuration in KFP v2.
Signed-off-by: Greg Sheremeta <gshereme@redhat.com>
This commit introduces the ability to run the component inside a virtual
environment, which is particularly useful when the Python system site package
directory is read-only, causing pip to fail during the installation of kfp and
other dependencies.
To bypass this issue, a writable temporary directory is selected to create a
virtual environment that inherits the global system site packages. The entire
workflow is then executed from within this virtual environment.
Furthermore, a new field, `use_venv`, has been added to the component
decorator, with a default value of `False`.
Signed-off-by: Sébastien Han <seb@redhat.com>
* feat(sdk): Allow setting a default of execution caching disabled via a compiler CLI flag and env var
Co-authored-by: Greg Sheremeta <gshereme@redhat.com>
Signed-off-by: ddalvi <ddalvi@redhat.com>
* Add tests for disabling default caching var and flag
Signed-off-by: ddalvi <ddalvi@redhat.com>
---------
Signed-off-by: ddalvi <ddalvi@redhat.com>
Co-authored-by: Greg Sheremeta <gshereme@redhat.com>
Mai Nakagawa
Daniel Dowler
Michael
Christian
Humair Khan
Matt Prahl
Michele Dolfi
Trevor Royer
Sidney Radcliffe
Ihar Hrachyshka
Anthonios Partheniou
Stijn Tratsaert
Chen Sun
chahat sagar
Ze Mao
SimonDR-Boltzmann
Dharmit Dalvi
Ricardo Martinelli de Oliveira
Yael Fishel
Tom Coufal
Alex
Greg Sheremeta
Sébastien Han
KevinGrantLee
Kevin James
Diego Lovison
ege uçak