Commit Graph

607 Commits

Author SHA1 Message Date
hsinhoyeh b40912cc5d
fix(sdk): use kfp.dsl.types to replace kfp.components.types Fixes #10282 (#10283)
* use kfp.dsl.types to replace kfp.components.types

Fixes: #10282
Signed-off-by: hsinhoyeh <yhh92u@gmail.com>

* fix import artifact_types

---------

Signed-off-by: hsinhoyeh <yhh92u@gmail.com>
2023-12-12 19:38:38 -05:00
Christian Geier 3a67f20d4e
chore(sample): fix pipeline with secrets examples (#10245) 2023-12-01 00:30:03 +00:00
Kelli Belcher 271d4ebfaf
Intel oneAPI XGBoost daal4py example pipeline (#10044)
* Intel oneAPI XGBoost daal4py sample pipeline

* Intel oneAPI XGBoost daal4py sample pipeline

* Intel oneAPI XGBoost daal4py sample pipeline
2023-10-05 22:35:15 +00:00
Chen Sun cd7e316db6
test: upgrade e2e sample tests to using sdk v2 (#9885)
* install kfp sdk 2.0 in sample test

* remove reference to GCSHelper from deprecated module

* exclude obsolete samples

* update test script

* update client usage

* revert recent changes to exit_handler.py
2023-08-17 09:33:38 +00:00
Junggil Lee 90cec167c0
fix(samples): Update volume_ops sample to v2 pipelines (#9877)
* Update volume_ops sample to v2 pipelines

* Update samples/core/kubernetes_pvc/kubernetes_pvc.py

minor change on copyright range

Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>

---------

Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
2023-08-17 00:21:38 +00:00
Junggil Lee 3e3a747b58
fix(samples): Update execution_order, loop_output samples to v2 pipelines (#9867)
* Update execution_order, loop_output samples to v2 pipelines

* Update execution_order, loop_output samples to v2 pipelines
2023-08-16 14:37:03 +00:00
Junggil Lee ff2e002157
fix(samples): Update loop_parameter, loop_static samples to v2 pipelines (#9870) 2023-08-15 17:59:04 +00:00
Ryan Jaemun Jung 1002e0cf8c
fix(samples): update samples to v2 pipelines (#9851)
* Update samples to v2 pipelines

* lint
2023-08-11 19:53:33 +00:00
Chen Sun 4a57ab69dd
chore(components): Move openvino components and samples to the right locations. (#9498) 2023-07-05 16:55:42 +00:00
Chen Sun a727783abd
chore: remove 'dummy_root' in test samples (#9640)
* remove 'dummy_root' in test samples

* more cleanup
2023-06-21 23:51:41 +00:00
Chen Sun ee2102a9b5
chore(samples/backend/frontend): Update preload control-flow and data-passing pipelines to v2 (#9563)
* Update preload samples to v2 pipelines

* address review comments

* fix frontend test

* format frontend code

* update comment
2023-06-07 22:06:57 +00:00
rd-pong d50d178936
chore(components): Add sample for Sagemaker Model Monitor component (#9405)
* Add sample pipeline for hosting and model monitor

* Update sample pipeline doc

* Address comments
2023-05-15 23:22:59 +00:00
ananth102 09d5151542
chore(components): Update Sagemaker sample to use v2 Hosting components (#9398)
* Update mnist sample with Hosting

* updated sample with new directory

* Updating sample to use output rather than after

* Address comments
2023-05-12 22:27:28 +00:00
Lingqing Gan b8caf811c2
feat(backend): support secrets (#9178)
* support secret in driver

* add sample tests

* fix comments
2023-04-20 00:33:39 +00:00
gkcalat f626629f79
chore(testing): Enable samples in e2e tests with v1 kfp SDK (#9169)
* Add pipeline_id to run and recurring run protos

* Add pipeline_id to run and recurring_run

* Set status when creating a new job. Closes #9125.

* Enable sample tests
2023-04-19 04:55:53 +00:00
Lingqing Gan ca63f20049
fix(backend): fix volume bug (#9150)
* fix volume bug

* fix indentation

* update driver image

* revert launcher image sha
2023-04-18 23:53:54 +00:00
Aditya Soni a0074c8963
test: added verify_func in TestCase (#9002) 2023-04-18 20:07:53 +00:00
gkcalat 37dac45caf
feat(backend): add support for resource requests. Closes #7047. Closes #6354. (#9088)
* Add support for resource requests

* Update pipelinespec reference

* Stash unit tests

* Fix tests

* Update license

* Update driver image

* Fix tests

* Add resource requests sample test

* Revert sample test and add TODO
2023-04-09 17:51:08 -07:00
Lingqing Gan 89bdac9340
feat(backend): support optional and default parameters. Fixes #8716 (#8765)
* support optional and default parameters

* add sample test

* change assert value

* update test

* update test

* fix sample test

* fix sample test

* fix sample test

* test fix

* change parameter logic

* test

* test

* change logic handling not provided parameter

* remove unnecessary code and add more sample case

* fix sample test bug

* fix sample test

* update logic

* change parameter logic

* update logic

* update logic handling when parameter not provided

* fix sample test

* add error message

* fix samples test

* samples test

* remove unused code
2023-03-02 17:47:04 +00:00
Lingqing Gan faca1e5aab
feat(backend): add support for environment variables. Fixes #8704 (#8886)
* support env var and add test

* sample test fix

* remove compiler test case

* fix test script

* update sample test
2023-02-27 18:02:08 +00:00
Chen Sun 5fe67919db
chore(components): clean up deprecated GCP components (#8685)
* delete deprecated GCP components

* remove build and test scripts, references to code/folder from master
2023-02-14 22:42:33 +00:00
ryansteakley 7be9693959
chore(components): Update aws owners files to reflect current owners (#8622)
* update aws owners files to reflect current owners

* fix list
2022-12-22 22:29:14 +00:00
Chen Sun 684779eb2e
fix(samples): update sklearn package name (#8439) 2022-11-10 12:35:00 +00:00
ryansteakley 4c097171c1
chore(components): Add sample for blended pipeline using v1 and v2 AWS Sagemaker components (#8284)
* blended pipeline

* add sample to samples folder

* update pinned version of TrainingJob

* update samples instructions

* update bucket and sagemaker role sections

* update broken link and fix indentation:
2022-11-01 23:55:45 +00:00
James Liu 91fe29f099
feat(samples): Full Kubeflow E2E sample using KFPv2. (#8303)
* feat(samples): Full Kubeflow E2E sample using KFPv2.

* fix output

* fix import json

* use dict

* clear output
2022-10-18 21:47:58 +00:00
Jagadeesh J 72aab0b5b8
chore(components/pytorch) - fix url paths (#8293) 2022-10-05 05:53:22 +00:00
Jagadeesh J 58d22d4ba9
chore(components/pytorch) - Samples fix for PT and PTL Upgrade (#8148)
* WIP: chore(components/pytorch) - Samples fix for PT and PTL Upgrade

* fix: bert example

 - fix minio secret
 - remove pth file upload to minio
 - add captum to pip packages

* fix: bert-dist training args

* fix: cifar10 example

* fix: cifar10 example notebook

* fix: captum example

* fix: gpu fixes for bert and cifar10 example

* fix: bert dist ptl upgrade
2022-09-22 05:29:38 +00:00
Connor McCarthy e371fa3795
feat(sdk): add compilation logic for google artifact types (support custom artifact types pt. 2) (#8232)
* remove dependency of deprecated on v2

* use new artifact identification logic

* add custom artifact type compiler test

* compile schema_version from non-kfp artifacts

* update tests

* add importer to compiler test

* update golden snapshots

* fix type checking code

* undo executor_test diff

* remove gcpc types compiler test

* add compilation test comment

* update google artifact handling

* update compiler test

* only permit system. and google. artifact types; add tests

* update compiler test

* simplify type checking logic

* use bundled artifact types in PipelineArtifactChannel

* raise error on unknown v1 input or output

* add type utils and move functions

* add type utils tests

* remove schema_version from inputspec and outputspec

* update artifact proto creation and tests

* propogate bundled artifact type string changes throughout sdk

* update golden snapshot

* remove unused v1 compat and tests

* update type checking and tests to validate schema_version using semantic versioning

* update importer sample

* update test data config

* update test_data config

* clean up sample

* clean up sample
2022-09-12 14:00:02 -06:00
Chen Sun 5ce4954d6b
feat(sdk)!: Implement Graph Component (#8179)
* Implement Graph Component

* isort imports

* release note

* address review comments

* address review comments
2022-08-26 09:28:49 +00:00
Scott_Xu 121ddcbc04
fix(sdk): fix v2 sample tests for cuj3 (#8168)
* fix v2 sample tests for cuj3

* separate sh -c

* fix two step pipeline

* fix piping error

* fix nit

* change path to uri to correct cat command

* correct bash script

* remove display name

* change directory reference in commands

* remove old files
2022-08-25 15:30:22 -07:00
Connor McCarthy 00419591c9
chore(sdk): update samples/ owners (#8178) 2022-08-22 18:08:30 +00:00
Connor McCarthy bdff332ac6
feat(sdk): support more than one exit handler per pipeline (#8088)
* add compiler test pipeline with multiple exit handlers

* remove blocker of multiple exit handlers

* move exit handler builder logic to pipeline_spec_builder

* build all exit handlers per pipeline

* add compiler test with IR inspection

* prevent usage of cross-pipeline after

* test cross-pipeline after is prevented

* update existing task dependency logic and tests

* add v2 sample test

* remove cross-pipeline .after

* prevent cross-dag data dependency for dsl features

* add compiler test pipeline with nested exit handlers

* add support for nested exit handlers

* clean up pipeline with nested exit handlers

* remove sample with multiple exit handlers

* remove compiler test with nested exit handlers

* add compilation guard against nested exit handlers in subdag

* update release notes
2022-08-10 16:56:21 +00:00
Scott_Xu 04c827fc3a
feat(sdk): supporting bring your own container for arbitrary input and outputs (#8066)
* support container_component decorator for function with no inputs

* resolve review comments

* add sample tests for milestone 1

* modify compiler test data

* resolve reviews

* resolve reviews

* WIP

* implementation of function of no inputs

* fixed sample test

* re-fix sample test

* fix rebase merge conflict

* resolve formatting

* resolve isort error for test data

* resolve comments

* fix nit

* resolve nit

* add implementation for placeholders i/o, sample and compiler tests

* resolve comments and merge logic for constructing container component

* resolve comments

* resolve comments

* fix assertion messages

* add error handling for accessing artifact by itself

* add test for raising error for accessing artifact by itself
2022-08-08 21:00:53 +00:00
Jagadeesh J 49c3587591
chore(components/pytorch):kserve migration (#7615)
* chore(components/pytorch):kserve migration

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* fix: pytorch dist training

 - enable env vars in config.properties
 - upgrade pip in dockerfile

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* Bert - KServe v2 handler changes

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* fix: bert notebook for kserve v2

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* fix: add protocol verion to bert gpu yaml

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* Adding utility to convert image to bytes - Cifar

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* Cifar10 - captum update

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

* fix: cifar10 example

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* fix: predictor component for kserve v2

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* fix: pytorch dist training for kserve v2

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* fix: cifar10 hpo example

Signed-off-by: Jagadeesh J <jagadeeshj@ideas2it.com>

* Bumping pytorch-kfp-components version

Signed-off-by: Shrinath Suresh <shrinath@ideas2it.com>

Co-authored-by: Shrinath Suresh <shrinath@ideas2it.com>
2022-07-08 08:55:52 +00:00
WhiteSource Renovate 624f197d52
fix(deps): update dependency npm to v8 [security] (#7885) 2022-07-08 05:52:51 +00:00
ryansteakley 086f0c960d
chore: Update oidc commands to include region and add brackets for trust policy (#7899) 2022-07-07 23:15:50 +00:00
ryansteakley cc8e3fdec1
chore: add missing imports for aws samples data scripts (#7877)
* update source for input data

* add imports for samples

* use aws legal approved s3 buckets

* change bucket in pipeline input
2022-07-07 22:33:51 +00:00
Connor McCarthy 3a8d2e2c3c
test(samples): skip concat placeholder test (#7988) 2022-07-07 18:17:48 +00:00
ryansteakley 77ab758a4e
chore: update source for aws samples input data (#7772) 2022-06-07 17:04:30 +00:00
Chen Sun d25bbc88b1
fix(test): Make exit handler test install KFP SDK from source (#7785)
* Update exit_handler_v2.py

* Update exit_handler_v2.py

* Update exit_handler_v2.py
2022-06-01 23:30:51 +00:00
ryansteakley a8a3b93ade
chore: fix link for installing kubeflow on AWS (#7762) 2022-05-19 18:40:56 +00:00
Lingqing Gan 98a58c7a0c
Pipeline level exit handler and v2 test (#7642)
* support IR YAML format in API

* Check the error message and return false if it is not nil

* update error message

* fixed simple loop but need cleaning up

* Deleted debug logs

* remove logs and fix some format

* fix static_loop_arguments

* change the driver image 

change the driver image back to the kfp container registry.

* change variable declaration

* remove logs

* remove log

* changed argo template

* change argocompiler dependency to my own repo

* Update go.mod

* Update VERSION

* v2 pipeline level exit handler; add test case

* Update VERSION

* remove logs and change temp dependency

* remove logs

* Update argo.go

* Update argo_test.go

* Update driver.go

* Update driver.go

* change VERSION

* Update VERSION

* Modifyb PR: remove loop fix and some changes

* update dsl compile command

* revert VERSION

* Update Makefile

* Update Makefile
2022-05-05 02:37:12 +00:00
Lingqing Gan 1180e10632
fix(backend): fix simple loop bug (#7578)
* support IR YAML format in API

* Check the error message and return false if it is not nil

* update error message

* fixed simple loop but need cleaning up

* Deleted debug logs

* remove logs and fix some format

* fix static_loop_arguments

* change the driver image 

change the driver image back to the kfp container registry.

* change variable declaration

* remove logs

* remove log

* move `ok` definition

* change test file for debug purpose

* change test for debug purpose

* update sample test for static loop

* update test file, remove code for debug
2022-05-05 01:31:12 +00:00
Connor McCarthy a5548440a3
use click for dsl-compile command (#7560) 2022-04-20 14:57:10 -06:00
Benjamin Tan Wei Hao 67c19fab67
fix: typo in VolumeOps example (#7479) 2022-03-31 19:07:31 +00:00
Yaqi Ji 04123280a4
fix(sdk): add default value for inputs (#7405)
* fix(sdk): add default value for inputs

* merge conflict

* release

* fix sample
2022-03-23 18:59:20 +00:00
Connor McCarthy c7301c449c
feat(sdk)!: use yaml for IR package (#7431)
* change serialization format from json to yaml

* refactor: use guard clause for clarity

* convert test data from json to yaml

* add and update tests

* permit deprecated compile to json

* re-add json test data

* update compiler tests

* refactor Compiler._write_pipeline_spec_file

* update golden snapshots

* update sample output extensions

* test file extension exceptions and warnings

* update release notes

* remove json golden snapshots
2022-03-21 21:45:31 +00:00
Vedant Padwal d93b390e62
feat(doc): Update kubeflow-e2e-mnist notebook to use KServe (#7355)
* Update kubeflow-e2e-mnist notebook to use KServe

* Add outputs

* Add old KFServing code

* Made changes as paul suggested

* Update multi user mode guide

* Made changes as Paul suggested

* Remove service account as The service_account issue is fixed if you use KFP 1.8.1
2022-03-07 06:54:53 +00:00
Yaqi Ji 8bb0d0e025
feat(sdk)!: move v2 to main namespace (#7376)
* feat(sdk)!: move v2 to main namespace

* edit samples
2022-03-04 21:56:51 +00:00
Yaqi Ji b693909a08
feat(sdk)!: move v1 to deprecated folder (#7291)
* chore(sdk): move v1 to deprecated

* fix testsg

* fix testsg

* fix setup.py

* fix test

* s'

* fix tests

* fix test

* retore v2 test changes

* fix py3.6 test

* fix py3.6 test

* fix py3.6 import fallback error

* remove deprecated

* fix samples test

* sample test

* fix samples

* add readme

* restroe test

* python require

* remove path

* fix tests

* inteegration tests

* remove tfx tests for dependency with kfp v1

* fix e2e

* fix e2e

* fix integration tests

* fix sampe

* move client down

* change to kfp

* add import alias

* fix

* runid

* fix dsl

* only use kfp for function

* revert train_until_good

* tfx test

* kfp

* try import

* onprem
2022-03-04 00:52:59 +00:00