* fix big data passing with multiple type of volume templates
* fix lint and license
* fix multi input step bug
* refactor update volume code
* add missing tests
* add labels for artifact outputs when tasks are not referred
* do not generate noninline test
* add option to also do individual artifactoutput
* add option to also do individual artifactoutput
* remove all output to artifact label
* fix sorting for test
* fix upstream conflicts
* add pipelinerun pod template support for security context and automount_service_account_token
* update flag to the correct type
* update flag to the correct type
* update flag to the correct type
* update usage docs
* feat(sdk): Add helper class AddOnGroup
add a helper class for users to write a custom
OpsGroup which maps to a custom task in Tekton.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* move finally under pipelineSpec
`finally` shall be under `pipelineSpec`.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Enhance params of AddOnGroup
make `params` argument of AddOnGroup class support
static and dynamic values.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Add post_param() api in AddOnGroup
Add `post_param()` api to update the params of
the task
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* Handle intermediate params
Create a dedicated API to create intermediate params
which are only used by downstream Ops/OpsGroup, but not
list in the spec.params
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* fix incorrect param value
properly handle params that are created
from AddOnGroups in downstream Ops
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* handle nested AddOnGroup case
An AddOnGroup can contains another AddOnGroup
as downstream OpsGroup. The group name of an AddOnGroup
needs to follow the correct name pattern to allow
the compiler to calculate the nested case properly.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* clean up params inside the taskSpec.pipelineSpec
only keep params that are used by downstream tasks in
taskSpec.pipelineSpec.params
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* support annotations and labels
Add properties in AddOnGroup to store annotations
and labels. Then add their key/value pairs to
metadata.annotation and metadata.labels.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* rename the annotation property to annotations
amend the property name to plural
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add test for any-seq inside of loop
* fix style: newline at the end of file
* recur when handling tekton pipeline variables
* update test name
* refactor adding a param
* add type to extracted task params inside loop
* fix style: no spaces in single-elem dict
* map sanitization logic to original kfp
* update test data to reproduce sanitization bug
* fix upstream dsl bugs on undeterministic params on long names
* fix(sdk): Apply node_selector in PipelineConf
Apply the node_selector in PipelineConf to spec.podTemplate
instead of taskPodTemplate of each task.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* update Makefile under pipelineloop
when running cli, it should do update target first
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
Handle the node selector info from PipelineConf in
compiler. Node selector info in Op level could overide
PipelinConf.
settings.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* feat(SDK): extension function `Loop.enumerate()`
Add helper function: `enumerate()` in Loop to
return indices and item. It could be used to get
current iteration number and item value.
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add logic to handle nested loop
Add logic to handle nested loop and test cases
Signed-off-by: Yihong Wang <yh.wang@ibm.com>
* add tests for non-literal separator
* plug-in the tests
* process 'separator' like 'from'-'step'-'to'
* explicitly allow PipelineParam as a separator
* add test for custom-tasks with literals
* fix orig_params check for custom tasks
* fix: style: min 2 spaces before inline comment
* fix: style: 2 empty line before function decl
* chain nested conditions
* add missing license
* add dependency check to remove unnecessary when expression
* minor improvement to check for the exact condition rather than just the condition task
* make results to store in multi-steps if needed
* add warning messages on oversized parameter
* regenerate tesrs
* fix copy-results format
* optimize the bin packing problem
* add docs and warning test cases
* add test for taskRef-vs-taskSpec params
* fix: taskSpec accepting params
* add license headers
* m: no spaces around "buzz"
* fix: test file custom_task_spec.yaml
* m: follow links on newlines
* fix: big_data_passing test result yamls
* fix: yamls of other tests
* factor out conditions to process params
* fix big data rewrite for regular tasks
* add auto apply custom task CRs before pipelineRun creation
* fix lint
* store loop resource in ordered dict
* Update kfp-admin-guide.md
* sorted pipeline params to avoid randomness
* sorted pipeline params to avoid randomness
* regenerate tests
* convert template string from yaml to json
* regenerate tests
* clean up unnecessary helper function
* resolve merge conflicts
* resolve merge conflicts
* remove unnecessary go mod package
* only opt-in to embed the recursion loop annotations
* only opt-in to embed the recursion loop annotations
* enable opt-in only via package variable
* add example for how to inline recursive loops
* resolve conflicts
* regenerate tests
* make auto generate and apply recursive pipelineloop in annotations as default
* move parameter sorting to the end
* add parameter sorting for recursion loops
* sort recursive task parameters
* add more details on how to opt-out auto apply custom resource
* TODO1: PipelineLoop inlined including nested loops and added tests.
* Supported recursive and parallel tasks inlining.
For recursive tasks, we simply skip them.
For Parallel tasks, relevant tests were added.
* Run all tests against inlined taskSpec and normal taskRefs
* Added CLI Flag disable/enable spec inlining.
Tommy Li
Yihong Wang
Michalina Kotwica
Prashant Sharma
huixa
Gang Pu