[![PyPI](https://img.shields.io/pypi/v/kfp-tekton?label=PyPI)](https://pypi.org/project/kfp-tekton/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/kfp-tekton?label=Downloads)](https://pypi.org/project/kfp-tekton/#files)
[![PyPI - License](https://img.shields.io/pypi/l/kfp-tekton?label=License)](https://www.apache.org/licenses/LICENSE-2.0)


# Kubeflow Pipelines SDK for Tekton

The Kubeflow Pipelines [SDK](https://www.kubeflow.org/docs/components/pipelines/sdk/sdk-overview/)
allows data scientists to define end-to-end machine learning and data pipelines.
The output of the Kubeflow Pipelines SDK compiler is YAML for [Argo](https://github.com/argoproj/argo-workflows).

The `kfp-tekton` SDK is extending the `Compiler` and the `Client` of the Kubeflow
Pipelines SDK to generate [Tekton](https://github.com/tektoncd/pipeline) YAML
and to subsequently upload and run the pipeline with the Kubeflow Pipelines engine
backed by Tekton.

## Table of Contents

<!-- START of ToC generated by running ./tools/mdtoc.sh sdk/README.md -->

  - [SDK Packages Overview](#sdk-packages-overview)
  - [Project Prerequisites](#project-prerequisites)
  - [Installation](#installation)
  - [Compiling a Kubeflow Pipelines DSL Script](#compiling-a-kubeflow-pipelines-dsl-script)
  - [Big data passing workspace configuration](#big-data-passing-workspace-configuration)
  - [Running the Compiled Pipeline on a Tekton Cluster](#running-the-compiled-pipeline-on-a-tekton-cluster)
  - [List of Available Features](#list-of-available-features)
  - [List of Helper Functions for Python Kubernetes Client](#list-of-helper-functions-for-python-kubernetes-client)
  - [Tested Pipelines](#tested-pipelines)
  - [Troubleshooting](#troubleshooting)

<!-- END of ToC generated by running ./tools/mdtoc.sh sdk/README.md -->


## SDK Packages Overview

The `kfp-tekton` SDK is an extension to the [Kubeflow Pipelines SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/)
adding the `TektonCompiler` and the `TektonClient`:

* `kfp_tekton.compiler` includes classes and methods for compiling pipeline Python DSL into a Tekton PipelineRun YAML spec. The methods in this package
  include, but are not limited to, the following:

  * `kfp_tekton.compiler.TektonCompiler.compile` compiles your Python DSL code
    into a single static configuration (in YAML format) that the Kubeflow Pipelines service
    can process. The Kubeflow Pipelines service converts the static
    configuration into a set of Kubernetes resources for execution.

* `kfp_tekton.TektonClient` contains the Python client libraries for the [Kubeflow Pipelines API](https://www.kubeflow.org/docs/pipelines/reference/api/kubeflow-pipeline-api-spec/).
  Methods in this package include, but are not limited to, the following:

  * `kfp_tekton.TektonClient.upload_pipeline` uploads a local file to create a new pipeline in Kubeflow Pipelines.
  * `kfp_tekton.TektonClient.create_experiment` creates a pipeline
    [experiment](https://www.kubeflow.org/docs/pipelines/concepts/experiment/) and returns an
    experiment object.
  * `kfp_tekton.TektonClient.run_pipeline` runs a pipeline and returns a run object.
  * `kfp_tekton.TektonClient.create_run_from_pipeline_func` compiles a pipeline
    function and submits it for execution on Kubeflow Pipelines.
  * `kfp_tekton.TektonClient.create_run_from_pipeline_package` runs a local
    pipeline package on Kubeflow Pipelines.


## Project Prerequisites

 - Python: `3.8` or later. For Python 3.12, make sure to not have the `SETUPTOOLS_USE_DISTUTILS` flag because it's already [deprecated](https://github.com/pypa/setuptools/issues/4002).
 - Tekton: [`v0.53.2`](https://github.com/tektoncd/pipeline/releases/tag/v0.53.2) or [later](https://github.com/tektoncd/pipeline/releases/latest)
 - Tekton CLI: [`0.30.1`](https://github.com/tektoncd/cli/releases/tag/v0.30.1)
 - Kubeflow Pipelines: [KFP with Tekton backend](/guides/kfp_tekton_install.md)

Follow the instructions for [installing project prerequisites](/sdk/python/README.md#development-prerequisites)
and take note of some important caveats.


## Installation

You can install the latest release of the `kfp-tekton` compiler from
[PyPi](https://pypi.org/project/kfp-tekton/). We recommend to create a Python
virtual environment first:

    python3 -m venv .venv
    source .venv/bin/activate

    pip install kfp-tekton

Alternatively you can install the latest version of the `kfp-tekton` compiler
from the source by cloning the repository [https://github.com/kubeflow/kfp-tekton](https://github.com/kubeflow/kfp-tekton):

1. Clone the `kfp-tekton` repo:

   ```
   git clone https://github.com/kubeflow/kfp-tekton.git
   cd kfp-tekton
   ```

2. Setup Python environment with Conda or a Python virtual environment:

   ```
   python3 -m venv .venv
   source .venv/bin/activate
   ```

3. Build the compiler:

   ```
   pip install -e sdk/python
   ```

4. Run the compiler tests (optional):

   ```
   pip install pytest
   make test
   ```

## Compiling a Kubeflow Pipelines DSL Script

The `kfp-tekton` Python package comes with the `dsl-compile-tekton` command line
executable, which should be available in your terminal shell environment after
installing the `kfp-tekton` Python package.

If you cloned the `kfp-tekton` project, you can find example pipelines in the
`samples` folder or under `sdk/python/tests/compiler/testdata` folder.

    dsl-compile-tekton \
        --py sdk/python/tests/compiler/testdata/parallel_join.py \
        --output pipeline.yaml


**Note**: If the KFP DSL script contains a `__main__` method calling the
`kfp_tekton.compiler.TektonCompiler.compile()` function:

```Python
if __name__ == "__main__":
    from kfp_tekton.compiler import TektonCompiler
    TektonCompiler().compile(pipeline_func, "pipeline.yaml")
```

... then the pipeline can be compiled by running the DSL script with `python3`
executable from a command line shell, producing a Tekton YAML file `pipeline.yaml`
in the same directory:

    python3 pipeline.py


## Big data passing workspace configuration

When [big data files](/samples/big_data_passing/big_data_passing_description.ipynb)
are defined in KFP. Tekton will create a workspace to share these big data files
among tasks that run in the same pipeline. By default, the workspace is a
Read Write Many PVC with 2Gi storage using the kfp-csi-s3 storage class to push artifacts to S3.
But you can change these configuration using the environment variables below:

```shell
export DEFAULT_ACCESSMODES=ReadWriteMany
export DEFAULT_STORAGE_SIZE=2Gi
export DEFAULT_STORAGE_CLASS=kfp-csi-s3
```

To pass big data using cloud provider volumes, it's recommended to use the
[volume_based_data_passing_method](/sdk/python/tests/compiler/testdata/artifact_passing_using_volume.py)
for both Tekton and Argo runtime.

If you want to change the input and output copy artifact images, please modify the following environment variables:

```shell
export TEKTON_BASH_STEP_IMAGE=busybox  # input and output copy artifact images
export TEKTON_COPY_RESULTS_STEP_IMAGE=library/bash # output copy results images
export CONDITION_IMAGE_NAME=python:3.9.17-alpine3.18 # condition task default image name
```

## Running the Compiled Pipeline on a Tekton Cluster

After compiling the `sdk/python/tests/compiler/testdata/parallel_join.py` DSL script
in the step above, we need to deploy the generated Tekton YAML to Kubeflow Pipeline engine.

You can run the pipeline directly using a pre-compiled file and KFP-Tekton SDK. For more details, please look at the [KFP-Tekton user guide SDK documentation](/guides/kfp-user-guide#2-run-pipelines-using-the-kfp_tektontektonclient-in-python)

```python
experiment = kfp_tekton.TektonClient.create_experiment(name=EXPERIMENT_NAME, namespace=KUBEFLOW_PROFILE_NAME)
run = client.run_pipeline(experiment.id, 'parallal-join-pipeline', 'pipeline.yaml')
```

You can also deploy directly on Tekton cluster with `kubectl`. The Tekton server will automatically start a pipeline run.
We can then follow the logs using the `tkn` CLI.

    kubectl apply -f pipeline.yaml

    tkn pipelinerun logs --last --follow

Once the Tekton Pipeline is running, the logs should start streaming:

    Waiting for logs to be available...

    [gcs-download : main] With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate

    [gcs-download-2 : main] I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath

    [echo : main] Text 1: With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
    [echo : main]
    [echo : main] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
    [echo : main]


## List of Available Features

To understand how each feature is implemented and its current status, please visit
the [FEATURES](FEATURES.md) doc.


## List of Helper Functions for Python Kubernetes Client

KFP Tekton provides a list of common Kubernetes client helper functions to simplify
the process of creating certain Kubernetes resources. please visit the
[K8S_CLIENT_HELPER](K8S_CLIENT_HELPER.md) doc for more details.


## Tested Pipelines

We are [testing the compiler](/sdk/python/tests/README.md) on more than 80 pipelines
found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler
`testdata` folder, the KFP core samples and the samples contributed by third parties.

A report card of Kubeflow Pipelines samples that are currently supported by the `kfp-tekton`
compiler can be found [here](/sdk/python/tests/test_kfp_samples_report.txt).
If you work on a PR that enables another of the missing features please ensure that
your code changes are improving the number of successfully compiled KFP pipeline samples.


## Troubleshooting

- When you encounter ServiceAccount related permission issues, refer to the
  ["Service Account and RBAC" doc](sa-and-rbac.md)

- If you run into the error `bad interpreter: No such file or director` when trying
  to use Python's venv, remove the current virtual environment in the `.venv` directory
  and create a new one using `virtualenv .venv`
