* Minimize the Kubernetes version range in the CI
This reduces the matrix to only include the low and high versions.
See the KFP community call notes for more context:
https://docs.google.com/document/d/1cHAdK1FoGEbuQ-Rl6adBDL5W2YpDiUbnMLIwmoXBoAU/edit?tab=t.0#heading=h.fovolzywu84d
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Reduce the testing matrix for Python versions
This reduces the Python versions being tested to the low and high
versions to reduce GitHub CI consumption.
See the KFP community call discussion for more context:
https://docs.google.com/document/d/1cHAdK1FoGEbuQ-Rl6adBDL5W2YpDiUbnMLIwmoXBoAU/edit?tab=t.0#heading=h.fovolzywu84d
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Add an option to run integration tests locally
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Stop using whalesay in the tests
The container image is over 10 years old and is in a format that is
deprecated on newer Kubernetes versions.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
---------
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
Updates V2_DRIVER_COMMAND argument and adds runtime arguments for starting a remote debug session in GoLand.
Signed-off-by: agoins <alyssacgoins@gmail.com>
Adjusted the timeout from 3 to 4 minutes and polling interval from 10 to 5 seconds in the cache integration test. This ensures more robust testing by allowing sufficient time for operations to complete while increasing polling frequency.
Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>
Prior to this commit, there was a workflow template added for every unique
retryStrategy setting. This is now consolidates to a single retryStrategy
template.
Signed-off-by: agoins <alyssacgoins@gmail.com>
* feat(backend/sdk): enable dsl.Collected for params & artifacts
Signed-off-by: zazulam <m.zazula@gmail.com>
* feat(backend): collect through loops & dags
Signed-off-by: zazulam <m.zazula@gmail.com>
To enable users to use loops similar to subdags, the initial collecting
implementation went only 1 layer deep of loops/subdags. This
implementation serves to handle multifacted approaches of pipelines that
users can generate.
---------
Signed-off-by: zazulam <m.zazula@gmail.com>
This allows the launcher command to be overridden with
the V2_LAUNCHER_COMMAND environment variable. This is useful if you need
to override the command to launch Delve for debugging or you have a
situation that requires using a different binary in the container image
based on the environment.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
Various owners' files reviewers/approvers are no longer involved with
the community. Additionaly, we have various other folks that are
involved and have shown an active interest in reviewing various portions
of the code base. This change updates all owners files to reflect this
current state of the community.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
Prior to this change, if a user requested a pipeline version that no
longer exists, it'd cause a whole stack trace to be displayed in the
logs. For benign errors, this now is a single info log.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Add the Kubernetes native pipeline store
This also improves cache update race conditions in the webhooks.
Co-authored-by: Matt Prahl <mprahl@users.noreply.github.com>
Signed-off-by: Ricardo M. Oliveira <rmartine@redhat.com>
* Use controller-runtime for the non-caching client
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Put the Kubernetes native CI manifests under the Argo manifests
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Fix the flaky Kubernetes pipeline store tests
Some tests set the viper configuration of POD_NAMESPACE while others
didn't and so the order of the tests mattered. This now sets and resets
the viper configuration for each test.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Modify the suggested pipeline version name to be a valid K8s name
This is more important for the Kubernetes pipeline store.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Fix the ml-pipeline Service ports in K8s native mode
The KFP UI automatically uses the first port listed in the ml-pipeline
Service to communicate with the KFP API. Using a JSON patch to add the
webhook port ensures it doesn't change the order.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
---------
Signed-off-by: Ricardo M. Oliveira <rmartine@redhat.com>
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
Co-authored-by: Ricardo M. Oliveira <rmartine@redhat.com>
* chore(release): bumped version to 2.5.0
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* remove unneeded ci files
these are added from the release script, until the scripts are updated,
these are manually removed.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* revert incorrect changelog changes for 2.5
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* correct 2.5.x change log (#11878)
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
---------
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* use packaged pipeline spec in k8s platform pkg
kfp-kubernetes expects pipeline spec to be present in global namespace
this change updates generate proto code to replace this import to
correctly reference the spec in the kfp.pipeline_spec namespace.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* update to protoc 3.20
upgrading to 3.20 resolves issues where some of the proto python
generated code attempts to directly create descriptors. This issue is
encountered when trying to use kfp-kubernetes packages with kfp sdk,
forcing the user to downgrade protobuf to 3.20 or lower despite the
kfp sdk and kfp-kubernetes supporting >=4.0. This change resolves
this issue.
updating protoc also requires regeneration of all go and python proto code, as well us updates to the api generate and release images.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
---------
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
This change relies on manifest yamls to specify the launcher & driver
that is pinned to a specific KFP version. The goal is to decouple having
to build launcher/driver at separate stages compared to api server.
This is accomplished by setting the hardcoded default to point to
"latest" and during release, api server is built with this hardcoding,
and the images for driver/launcher are patched into manifests post build
along with the other images.
The apiserver deployment manifest is reformatted using yq so the next
time the release.sh is ran, the user is not surprised by the entire file
reformatting unexpectedly.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
# Conflicts:
# manifests/kustomize/base/pipeline/ml-pipeline-apiserver-deployment.yaml
# Conflicts:
# .github/resources/manifests/tekton/kustomization.yaml
As per community discussion, the tekton backend is being removed due to
lack of maintainers.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
If the pipeline run is submitted without specifying an optional
parameter and there is no default, it was not handled by the driver.
The approach taken is explicitly set null for these values and let the
driver handle if the component parameter has a default that can be
used in the launcher.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Add a mutating webhook for the PipelineVersion kind
This mutating webhook adds labels to query pipeline versions by their
pipeline ID and pipeline name, and adds an owner's reference to the
pipeline it's a part of.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
* Delete the Argo Workflow object after successful SDK execution test
This will hopefully free up resources on the cluster and reduce the CI
flakes.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
---------
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
Replaced the previous workaround for checking if the DB is empty with
mlmd_store.get_context_types(), as the upstream issue (#28) is now resolved.
Signed-off-by: Abhinav Sai D <abhinavsai491@tutanota.com>
* Create a validating webhook for the PipelineVersion kind
Signed-off-by: VaniHaripriya <vmudadla@redhat.com>
* Fix the manifests for the validating webhook
This fixes deployment and local development for the Kubernetes native
API manifests.
This also addresses other feedback.
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
---------
Signed-off-by: VaniHaripriya <vmudadla@redhat.com>
Signed-off-by: mprahl <mprahl@users.noreply.github.com>
Co-authored-by: VaniHaripriya <vmudadla@redhat.com>
* add backend support for k8s platform inputs
This change adds driver support for input parameter support for the
kubernetes platform spec. Input resolution change is extracted and made
more generic so it may be re-used when building out the container spec
for the k8s config.
Also add unit tests for constant & runtime input parameters.
TaskOutput parameter support are omitted due to a lack of appropriate
mlmd mock framework.
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* add sdk implementation for k8s params inputs
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* add tests for k8s input params
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* add setup/teardown of prereqs and secret tests
and update/re-enable secret env tests
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* have kfp sample tests use local python pkgs
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
* use a better configmap k8s name input...
Add support for multiple input types for pull secrets.
Clarify toleration docstring
Remove unnecessary resolve function
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
---------
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
Alyssa Goins
Matt Prahl
Humair Khan
Anton Pechenin
Caroline DeVoto
Helber Belmiro
Vani Haripriya Mudadla
Michael
Ricardo M. Oliveira
Alex
co63oc
Giulio Frasca
Abhinav2777
Anish Asthana