* fix big data passing with multiple type of volume templates
* fix lint and license
* fix multi input step bug
* refactor update volume code
* add missing tests
* Added support to directories in step-copy-results-artifacts script
Check if the provided result (src) is a directory; if so, create a tar.gz archive and compute the ARTIFACT_SIZE on {src}.tar.gz.
Moreover, print on {dst} file the list of files in {src}.tar.gz (namely, the result of `tar -tzf {src}.tar.gz`) instead of copying directly {src} into {dst}
* Add support to directories in step-copy-artifacts script
Added check [ -d "$2" ] to allow directories to be transmitted as artifacts
* Update unit tests
* Update sdk/python/kfp_tekton/compiler/_data_passing_rewriter.py
Co-authored-by: Tommy Li <Tommy.chaoping.li@ibm.com>
* reducing script size
* generated tests
* removed whitespace identified by lint test
* fixed error (missing suffix)
* generated tests
Co-authored-by: Tommy Li <Tommy.chaoping.li@ibm.com>
when using script, it generates an init container to process the script.
we'd like to avoid the init container in pipelinerun.
remove `script` and use `command` and `arg` instead.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add labels for artifact outputs when tasks are not referred
* do not generate noninline test
* add option to also do individual artifactoutput
* add option to also do individual artifactoutput
* remove all output to artifact label
* fix sorting for test
* fix upstream conflicts
eliminate the use of `script`, replace it with
command and args. this also removes the init container that
prepares the script.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* upgrade tekton to 0.35.1
Upgrade tekton to 0.35.1, including:
- tekton manifests
- go.mod to use new pkg and update all related pkgs
- update pipeline-loops to also use new tekton pkg
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* update tekton to 0.36
update tekton, license files, and cache server
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* fix compile error on test code
for taskrun's step, it doesn't use container data struct
any more
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* update compiler
add `type` for the taskSpec.results and update all
yaml files accordingly
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add pipelinerun pod template support for security context and automount_service_account_token
* update flag to the correct type
* update flag to the correct type
* update flag to the correct type
* update usage docs
* feat(sdk): Add helper class AddOnGroup
add a helper class for users to write a custom
OpsGroup which maps to a custom task in Tekton.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* move finally under pipelineSpec
`finally` shall be under `pipelineSpec`.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Enhance params of AddOnGroup
make `params` argument of AddOnGroup class support
static and dynamic values.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Add post_param() api in AddOnGroup
Add `post_param()` api to update the params of
the task
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Handle intermediate params
Create a dedicated API to create intermediate params
which are only used by downstream Ops/OpsGroup, but not
list in the spec.params
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* fix incorrect param value
properly handle params that are created
from AddOnGroups in downstream Ops
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* handle nested AddOnGroup case
An AddOnGroup can contains another AddOnGroup
as downstream OpsGroup. The group name of an AddOnGroup
needs to follow the correct name pattern to allow
the compiler to calculate the nested case properly.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* clean up params inside the taskSpec.pipelineSpec
only keep params that are used by downstream tasks in
taskSpec.pipelineSpec.params
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* support annotations and labels
Add properties in AddOnGroup to store annotations
and labels. Then add their key/value pairs to
metadata.annotation and metadata.labels.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* rename the annotation property to annotations
amend the property name to plural
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add test for any-seq inside of loop
* fix style: newline at the end of file
* recur when handling tekton pipeline variables
* update test name
* refactor adding a param
* add type to extracted task params inside loop
* fix style: no spaces in single-elem dict
* map sanitization logic to original kfp
* update test data to reproduce sanitization bug
* fix upstream dsl bugs on undeterministic params on long names
* fix(sdk): Apply node_selector in PipelineConf
Apply the node_selector in PipelineConf to spec.podTemplate
instead of taskPodTemplate of each task.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* update Makefile under pipelineloop
when running cli, it should do update target first
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
Handle the node selector info from PipelineConf in
compiler. Node selector info in Op level could overide
PipelinConf.
settings.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* feat(SDK): extension function `Loop.enumerate()`
Add helper function: `enumerate()` in Loop to
return indices and item. It could be used to get
current iteration number and item value.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add logic to handle nested loop
Add logic to handle nested loop and test cases
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add tests for non-literal separator
* plug-in the tests
* process 'separator' like 'from'-'step'-'to'
* explicitly allow PipelineParam as a separator
* add test for custom-tasks with literals
* fix orig_params check for custom tasks
* fix: style: min 2 spaces before inline comment
* fix: style: 2 empty line before function decl
Tommy Li
Federica Filippini
Alan Chin
Yihong Wang
Maximilien de Bayser
Michalina Kotwica
Prashant Sharma
ted chang