Commit Graph

117 Commits

Author SHA1 Message Date
Michalina Kotwica b370cbd1bc
fix(sdk): fix artifacts of ops with long names (#541)
* add test for artifacts of ops with long names

* add test on correct artifact_items

* fix: stepTemplate generated

* fix: artifact_items name collision

* refactor: mount_path to a separate variable

* add license to the new test file

* m: two free lines after imports in testcase

* m: other linting
2021-04-14 12:29:12 -07:00
Tommy Li 6adfcc64f3
feature(sdk): Optimize unnecessary condition task when using User definable CEL Custom task (#478)
* add cel custom task poc

* apply suggestions with minor fixes

* add initial version of plugable cel template

* fix lint

* update staging code for converting containerOp to custom task

* update staging code for converting containerOp to custom task

* remove duplicate cel conditions

* update flip-coin example to use custom task

* update flip-coin example to use custom task

* remove dsl dependencies;

* add POC comments

* fix lint

* fix self-defined CEL custom task mapping

* move custom task condition example to a separate folder

* fix tests

* make the new custom task spec and CEL condition feature as opt-in

* update instructions and generalize custom task images

* only let custom task result comparasion to skip the condition container to provide backward compatibility

* only let custom task result comparasion to skip the condition container to provide backward compatibility

* update example

* update tests

* fix comment
2021-04-13 09:43:11 -07:00
Tommy Li 48ac3bce5b
feature(sdk): Add custom task definition support (#526)
* add custom task template support

* fix typo

* fix lint

* update cel example to use official custom task

* add troubleshooting for example
2021-04-06 18:53:02 -07:00
Feng Li 8d5d584cb7
Fix(sdk): Fix nested loop params issue, and ensure multi nested loop works well (#525)
* fix nested loop params issue, and ensure multi nested loop works well

* address the issue of ut failed on python3.6

* Trigger CI

* address the issue of ut failed on python3.6
2021-04-06 16:10:01 -07:00
Tommy Li 991d7dcf16
fix(sdk): Fix long resource name (#521)
* fix long resource name;

* fix python 3.6 test
2021-03-31 20:21:19 -07:00
Gang Pu 504dc25c63
Support graph and recursion for tekton backend (#515)
* 1: Support graph dsl for tekton backend
2: Support recursive graph for tekton backen

* Make check license and lint pass
2021-03-30 02:13:17 -07:00
Andrew Butler 4b5230b694
Add automated labels and annotations for pipeline caching (#507)
* Add disable-cache flag and default caching for compiler

* Add updated testdata and features doc

* Update docs

* Add task level cache enabling

* Remove disable-cache flag and add warning

* Update docs

* fix typo

* Overwrite pipeline level label with task level label and add examples

* Update imports

* update format

* Update features docs
2021-03-24 11:07:45 -07:00
Gang Pu 05e6f4aba5
Enhance AnySequencer to make it an op (#499) 2021-03-16 19:24:16 -07:00
Tommy Li 6b1e02a46b
add configuration for tekton pipeline spec (#487)
* add configuration for tekton pipeline spec

* sort annotation order
2021-03-05 17:20:49 -08:00
Christian Kadner af861733e6
Update E2E tests to use kfp client (#301) (#480)
* Use kfp client to run E2E tests

* Update Tekton server and tkn client versions

* Regenerate pipelinerun logs

* Give unique names to pipelineruns resembling file names

* Delete previous pipelineruns

* Mark tests as skipped instead of not generating them

* Regenerate pipeline-loop CRDs, update copyright statements

   for f in sdk/python/tests/compiler/testdata/*_pipelineloop_cr*.yaml; do
     echo ${f/_pipelineloop_cr*.yaml/.py};
   done | sort -u | while read f; do
     echo $f; dsl-compile-tekton --py $f --output ${f/.py/.yaml};
   done;
   make unit_test GENERATE_GOLDEN_YAML=True
2021-03-05 15:02:49 -08:00
kfp-tekton-bot ede6750224
KFP 1.4.0 Rebase (#481)
* KFP 1.4.0 Rebase

Resolves #469

* Remove duplicated KFP samples

* Resolve api and backend and SDK test conflicts

* Upgrade kfp requirements to 1.4.0

* Update testdata regenerate YAML files

* Resolve CONTRIBUTING.md

* Regenerate KFP samples compile report

* Update README.md files for KFP version 1.4.0

* Resolve frontend conflicts

* Revert list-wrapped loop arguments

* Remove compiler_tests.py.rej
2021-03-01 10:21:26 -08:00
Feng Li 99e506c244
feat(backend): Support loop pipelineruns of defining dictionaries params (#460) 2021-02-08 10:37:37 -08:00
Feng Li 1f943852ae
fix(backend): Fix condition with loop together issue (#456)
* enhance the when condition UI display

* Fix condition with loop together issue
2021-02-03 18:04:28 -08:00
Feng Li 6e1f367841
fix(backend): Fix condition with multi parameter issue (#455)
* enhance the when condition UI display

* Fix condition with multi parameter issue
2021-02-03 08:10:29 -08:00
Feng Li 9b89a082bc
Fix loop issue that the loop param is from task output (#452)
* enhance the when condition UI display

* Fix loop issue that the loop param is from task output
2021-02-02 00:16:28 -08:00
Feng Li 3f750f12a0
feat(backend): Support dsl to define the pipeline loops via tekton (#440)
* enhance the when condition UI display

* Support dsl to define the pipeline loops via tekton
2021-01-29 14:25:48 -08:00
Jin Chi He fb765134d6
Any Sequencer implements (#399) 2021-01-05 22:11:52 -08:00
Tommy Li 51b9824382
fix(ui and sdk): Support display name annotations (#383)
* fix ui and sdk to support display name

* lint test file
2020-12-04 12:25:26 -08:00
Tommy Li 7cd39a7335
Feature branch for moving to Tekton 0.16 (#321)
* add tekton 0.16.3 go mod

* update dev setup

* update dev docs

* update readme

* support more argo variable mapping

* update feature doc on the list of supported argo variables translation

* add task-wise metadata support

* add beta whenexpression for condition, unblock runAfter for conditional task

* update condition docs

* update ui image for tekton-0.16

* Apply suggestions from code review

Co-authored-by: Andrew Butler <Andrew.Butler@ibm.com>

* address comments and fix conflicts

* update doc reference to tekton 0.16

* regenerate tests

* updated latest images with tekton 0.16

* regenerate tests

Co-authored-by: Andrew Butler <Andrew.Butler@ibm.com>
2020-11-13 11:33:04 -08:00
Feng Li cdd5625c7c
Fix #340: handle big data passing with multi dependencies (#360) 2020-11-10 17:09:56 -08:00
Tommy Li 1854b92ae0
fix(sdk): Check whether the dependent component is a tekton task (#313) 2020-09-22 09:26:49 -07:00
Tommy Li b1807476d1
fix(sdk): fixes outputfile key formatting (#308)
* fix outputfile formatting

* ignore python 3.5 test with unsortable characters

* Add python 3.5 error comments

* lint tailing whitespace
2020-09-17 17:32:45 -07:00
Tommy Li 7f860a9127
Refactor artifact and logging client logics to server side (#281)
* refactor artifact and logging logics to server

* remove client side logging tests

* update logging file keywords and add configurable flags

* add simple strip output logics

* update user and admin instructions

* add missing rollout command

* minor cleanup

* golang code cleanup

* clarify guide with a better UX

* add troubleshooting

* update metadata writer to also catch k8s api timeout errors

* add troubleshooting guide for redeploy

* address comments on variable names and admin guide path

* update test logs and unittest timeout interval

* Apply suggestions from code review

Co-authored-by: Animesh Singh <singhan@us.ibm.com>

* rename variables

* allow artifact image to be customizable

Co-authored-by: Animesh Singh <singhan@us.ibm.com>
2020-09-09 15:51:44 -07:00
Tommy Li 6f7eced282
fix(sdk): throw errors on deprecated volume definition (#283) 2020-08-25 16:26:02 -07:00
Tommy Li 1712b216c5
fix(sdk): Fix volumes bug on wrong dict key (#282) 2020-08-25 16:02:03 -07:00
Feng Li 819e4ed7d7
An easy solution of archiving pipeline logs to S3 (#248)
* update code to address the review comments

* change the dir name

* change the dir name

* update yaml file after rebase master

* update yaml file to trigger travic

* address comments from Tommy
2020-08-06 09:39:42 -07:00
Christian Kadner c37c600486
Upgrade E2E tests for tkn CLI version 0.11 (#246)
* Now including logs for 'finally' tasks
* Add env var for SLEEP_BETWEEN_TEST_PHASES
* Add 'retry.yaml' to ignored tests
* Make node_selector more general
* Use pipelinerun status.conditions[].reason instead of .type
* Treat status "Completed" as "Succeeded"
2020-08-04 13:26:22 -07:00
Christian Kadner 879d783ae0
Make E2E tests more robust (#235)
* Tolerate erratic log lines in E2E tests
* Add option to re-run only tests that failed previously
* Add option to exclude/include certain test_cases only
* Add volume_op.yaml to ignored tests as the created PVC
  needs to be deleted before/after test run
* Regenerate "golden" log files after recent commits
  changed the log output regarding artifacts
* Add option to override expected Tekton Pipeline and CLI
  versions, default: 0.14 and 0.10

Closes #229
2020-07-21 11:16:05 -07:00
Christian Kadner cfdc996c16
Make unit tests run with Python 3.5 (#210) 2020-07-06 09:14:53 -07:00
Tommy Li eea0c0104e
Update samples with the new PipelineRun SDK (#206)
* fix bugs, update examples, and add tests

* Address comments

* Update readme to a shared prerequisites section

* update instructions to use python commands if possible

* add details on the yaml name produced by the SDK

* Update samples/README.md

Co-authored-by: Animesh Singh <singhan@us.ibm.com>

Co-authored-by: Animesh Singh <singhan@us.ibm.com>
2020-07-02 16:56:47 -07:00
Tommy Li fcd4308297
Enable artifacts as default (#204)
* enable artifacts as default

* Apply suggestions from code review

Co-authored-by: Animesh Singh <singhan@us.ibm.com>

* Update sdk/FEATURES.md

Co-authored-by: Animesh Singh <singhan@us.ibm.com>

* address comments

* Apply suggestions from code review

Co-authored-by: Animesh Singh <singhan@us.ibm.com>

Co-authored-by: Animesh Singh <singhan@us.ibm.com>
2020-06-26 19:34:14 -07:00
Christian Kadner 196b822532
PipelineRun with embedded pipelineSpec and taskSpec (#166)
* PipelineRun with embedded pipelineSpec and taskSpec

Closes #143

* Preserve empty string default values

* Remove success_condition and failure_condition
2020-06-23 17:21:19 -07:00
Christian Kadner c1b7410029
Add end-to-end compiler tests (#174)
- Regenerate all "Golden" YAML files with PipelineRun
- Dynamically generate E2E test cases from "Golden" YAML files
- Add e2e_test target to Makefile
- Generate new "Golden" log files from Tekton pipeline run
- Fix PendingDeprecationWarning: dsl.ContainerOp....` will be removed in future releases
  Use `dsl.ContainerOp.container....` instead.
- Fix UserWarning: Missing type name was inferred as ...

Resolves #28
2020-06-11 15:10:57 -07:00
Tommy Li 07ff4c160f
Deprecate artifact location and update to kfp 0.5.1 (#175)
* deprecate artifact location and update to kfp 0.5.1

* address comments

* update python dependency map
2020-06-11 12:02:57 -07:00
Feng Li 460422b4ed
Support Exit handler (#179) 2020-06-11 10:38:57 -07:00
Feng Li 81f578a602
Change bigger to big, fix issues introduced by PR 170 (#173) 2020-06-09 23:28:04 -07:00
Tommy Li a10340d684
Add artifact annotation for metadata tracking (#170)
* add artifact annotation for metadata tracking

* update op_to_container argument, add new test case
2020-06-08 21:33:19 -07:00
Feng Li b05bc9fa30
commit code for big data passing (#169) 2020-06-08 21:31:18 -07:00
Christian Kadner 2c68de47e4
make lint (#164) 2020-05-29 09:29:16 -07:00
Christian Kadner 7a12ca4f3b
Sanitize Kubernetes Names (#158)
Resolves #151
2020-05-28 15:57:14 -07:00
Christian Kadner ac4521780e
Upgrade kfp version from 0.2.2 to 0.5.0 (#135)
Closes #133
2020-05-07 00:43:09 -07:00
Andrew Butler 2d06e98f30
Add unit tests for 'basic_no_decorator.py' and 'compose.py' (#111)
* Add unit tests

* Update condition.yaml

* Update test_kfp_samples.sh for new test_util.py

* Add results logging and refactor

* Refactor and add licenses

* Add license

* Add __main__ to introduced testdata

* Adjust new testdata for #119

* Add and edit doc string

* Fix golden yaml
2020-04-27 11:59:23 -07:00
Tommy Li 4f6a564dc0
Fix edge cases such as disable istio-inject for Tekton compiler (#119)
* update flag to disable istio-inject, also remove some pyyaml conflicts

* move license header as a variable

* move license header as global var
2020-04-24 17:43:28 -07:00
Tommy Li 8830915424
Support input artifact (#104)
* support input artifacts

* fix conflicts

* fix merging bug
2020-04-22 15:53:45 -07:00
Andrew Butler e8d973a57d
Add support for passing task outputs to conditions as parameters (#105)
* Add support for task output parameter passing to conditions
Add support for highly nested conditions

* Add minor commenting

* Minor formatting

* Update test case and sample report

* Refactor and add warning

* Add licenses and unit test

* Remove redundant import
2020-04-22 13:17:44 -07:00
Tommy Li a47ae08d51
Add volumeop and volumesnapshot support (#93)
* add volumeop support workaround

* add volumesnapshot workaround

* update resourceop integration

* add sorting to avoid randomness

* fix bug

* update sample

* minor code cleanup
2020-04-21 13:11:52 -07:00
Tommy Li 3d57d617c3
Support artifact outputs (#92)
* init commit

* update tests

* update readme

* add missing comments

* fix confilcts

* update test cases with integration test

* remove extra spaces

* fix readme command

* address comments

* fix init condition for new test methods
2020-04-21 11:37:52 -07:00
Feng Li ebe05e3a4b
Add support for dsl.ResourceOp (#89) 2020-04-11 16:27:47 -07:00
Kevin Yu cf8e99866d
Add support for pulling the image secret (#88)
* initial code

* address comments

* address comments

* generate service account name based on pipelinerun name

* re-generate the yaml
2020-04-10 11:31:47 -07:00
Tommy Li 768220989a
fix regex to only find argo specific variables (#83) 2020-04-09 13:01:47 -07:00
Tommy Li 332881895a
Add argo variable checking when compile to Tekton. (#80)
* add initial parsing

* add regex mapping for the equivalent tekton variables

* revert old changes

* fix typos

* raise error when unsupported Argo vars are used

* fix comment
2020-04-03 15:29:47 -07:00
Feng Li cf3968125d
Extend compiler to support parallelFor with loop static params (#67) 2020-04-02 19:55:46 -07:00
Tommy Li d990f2414f
Copy param output files using emptyDir volumes (#75)
* init code

* add initial workspace workaround

* added workspaces to pipelinerun

* fix typo

* only use workspace if necessary

* add extra steps for copying non configurable output file path

* update comment

* update container image

* update copy step to use busybox

* update comments

* replace workspaces with emptydir volume

* revert workspace changes

* fix volume edge case

* update comments

* update test report
2020-04-02 15:55:46 -07:00
Tommy Li 6df545ad41
add transformer test for annotations and labels (#76) 2020-03-31 17:37:27 -07:00
Tommy Li f088d2a465
Add more k8s resource support (#65)
* add more k8s resource support

* add test main functions as requested

* add description on why we need pipelinerun
2020-03-30 16:18:40 -07:00
Feng Li 0fb5ab8440
Extend compiler to support workflow level timeout (#68) 2020-03-27 14:57:53 -07:00
Tommy Li 5b52b79bd8
Add initial pipelinerun support (#62)
* add initial pipelinerun support

* update comment

* rearrange compile function
2020-03-26 14:20:23 -07:00
Feng Li 359e750b17
support retry, address the comments (#55) 2020-03-24 18:13:02 -07:00
Feng Li 10896c8036
Support timeout defining for pipeline tasks (#46) 2020-03-24 12:57:02 -07:00
Feng Li 8d6c412619
Add pipelineparams in compiler, address comments (#45) 2020-03-23 22:48:47 -07:00
Tommy Li a5992f95f7
add init_container as steps (#50) 2020-03-23 22:46:46 -07:00
Feng Li 9be5a87d69
Address review comments, remove with_param from this PR (#42) 2020-03-20 18:16:44 -07:00
Tommy Li 8e944cfc1e
Add volumemount support and fix parameter bugs (#40)
* add volumemount support and fix parameter bugs

* remove sdk/samples folder and use real kfp example

* fix formatting

* update unittest and instructions

* update license header

* update license header

* update license
2020-03-20 15:32:44 -07:00
Christian Kadner 9ff9467daf
Task output parameters as input for subsequent tasks (#27)
Closes #19
2020-03-17 17:08:14 -07:00
Tommy Li 7c843f66fe
update samples and tests to also include parallel use case (#24)
* update example and tests to also include parallel use case

* update example and tests to also include parallel use case

* add back the sequential example

* update permission

* remove sub-folders

* update readme instructions

* rename files to make it consistent

* update tests to be consistent with kfp samples
2020-03-16 14:56:30 -07:00
Christian Kadner 3167232886
Generate Tekton Pipeline with Tasks (#17)
Replace Steps with Tasks to allow parallel task execution.
Use 'runAfter' to support sequential task execution.
2020-03-11 19:46:38 -07:00
Christian Kadner faf700f222
Refactor code to extend KFP SDK compiler (#16)
This code change removes the requirement to clone and
copy the kubeflow/pipelines repository. The project
has been refactored to only contain the minimal set of
Python classes and methods that needed to be modified
to generate Tekton YAML instead of Argo YAML. The
overriden methods are dynamically linked at runtime
by monkey-patching the KFP SDK module.

Additionally adding a simple test case and setup.py
install script that creates the dsl-compile-tekton
CLI executable which works independent of the KFP
compiler executable.
2020-03-03 18:37:48 -08:00