Commit Graph

1393 Commits

Author SHA1 Message Date
Tara Tufano e50f40305d
fix(sdk): fix upload_pipeline when no pipeline name is provided (#8695) 2023-01-31 13:45:39 -08:00
Connor McCarthy edfb267790
bump version; update release notes; add docs (#8745) 2023-01-20 12:56:56 -08:00
Connor McCarthy 1c7ab4c5f5
pin kfp-pipeline-spec version exactly (#8737) 2023-01-20 12:49:36 -08:00
Connor McCarthy 78d31dc8a3
chore(sdk): remove v1 dependencies from SDK v2 (#8668)
* split out deprecated requirements

* pip install deprecated requirements for sample tests

* remove superfluous comments in requirements.in

* update release notes

* ignore deprecated directory in unit tests

* remove unused argo compiler

* ignore deprecated path

* update run sample yaml

* install deprecated requirements for e2e tests

* add todos
2023-01-19 12:53:27 -08:00
Owen 07615eb88a
chore(sdk): Improve CLI utility function (#8686) 2023-01-17 14:51:50 -08:00
Alex 5adc7f1b50
chore(sdk): clean up client.py (#8656) 2023-01-17 11:04:46 -08:00
Mai Nakagawa 6040334e84
feat(sdk): Accepts higher PyYAML versions (#8665)
* Accepts higher PyYAML versions

* Update the release note
2023-01-17 10:46:42 -08:00
Connor McCarthy fdf3ee7b68
chore(sdk): release KFP SDK 2.0.0b10 (#8638)
* bump version

* update release notes

* add docs version
2022-12-29 18:05:36 -05:00
Connor McCarthy 84241d6460
fix(sdk): fix deserializing v1 component yaml defaults (#8639)
* fix deserializing v1 defaults

* update test cases

* update type annotation

* add unit tests
2022-12-29 17:48:58 -05:00
Connor McCarthy 151e8a3c6e
feat(sdk): support optional artifact inputs (#8623)
* add compiler tests

* support optional artifact inputs

* update executor test

* update existing golden snapshot

* add new compiler test/snapshot

* add release note

* bump kfp-pipeline-spec dependency
2022-12-28 13:40:12 -05:00
Connor McCarthy 5491f6f3db
fix(sdk): support writing/reading isOptional for input parameters (#8612)
* add tests

* support writing/reading isOptional for parameters

* update golden snapshots

* temporarily maintain workaround for v1 optional artifact inputs

* add release note
2022-12-21 14:46:59 -08:00
Connor McCarthy ecfdc665c4
chore(sdk): release KFP SDK 2.0.0b9 (#8609) 2022-12-21 00:55:12 +00:00
Connor McCarthy f64348f29e
fix(sdk): make executor output writer more permissive [sdk v2] (#8607) 2022-12-21 00:23:13 +00:00
Sune Debel 2eef97cfc7
fix(sdk): add special handling for PipelineTaskFinalStatus to get_parameter_type (#8587) 2022-12-19 18:48:46 +00:00
Connor McCarthy 08b2408d3d
feat(sdk): add support for lists of artifacts in Python components [support lists of artifacts pt. 3] (#8465)
* remove unused codepath

* rename to avoid confusing variable names

* implement support in executor

* add comment

* make executor code safer
2022-12-15 21:34:21 +00:00
Connor McCarthy 47eb1cad92
chore(sdk): update golden snapshots with pipeline interface comments (#8575)
* update path in readme

* update golden snapshots
2022-12-15 01:32:21 +00:00
Connor McCarthy d4b896d3e3
fix(sdk): new fix for multiple workers writing with gcsfuse bug [KFP SDK v2] (#8544)
* fix and test

* update release notes

* extract constants
2022-12-13 18:01:29 +00:00
Connor McCarthy f4588f3112
chore(sdk): idiomatic Python pure refactorings (#8537)
* use f-strings

* inline immediately returned variables

* simplify comprehension

* use list literals

* lift return statements into control flow blocks

* other pure refactorings

* remove passthrough function

* use a ternary when it improves readability

* use __getitem__ for regex match groups; introduced in Python 3.6

* fix changed comments

* revert one change
2022-12-06 23:44:36 +00:00
JOCSTAA 49db63c916
feat(sdk): Add comments to IR YAML file (#8467)
* base

* add tests

* fix bug

* adress comments

* address comments 2

* sort comments

* sort signatures

* add indempotent test

* add indempotent test2

* support multiline docstring

* review

* docformatter presubmit exclude

* docformatter presubmit exclude

* docformatter

* docformatter

* merge 1

* update readme

* nit .items()

* remove reduntant test
2022-12-05 14:43:14 -08:00
Alex 1d6a52b9ff
fix(sdk): print max_attempts instead of sleep_duration (#8523) 2022-12-02 19:32:16 +00:00
Connor McCarthy 00c6b01040
fix(sdk): fix bug when referencing component.pipeline_spec in a pipeline body (#8514)
* add tests

* implement fix

* add TODO
2022-12-02 09:35:16 +00:00
JOCSTAA 614c231ae6
fix(sdk): unblock valid topology. (#8416)
* unblock valid topology

* add more tests

* handle git fork

* sample_test_cases

* main

* restore to master

* resolve comments on PR

* resolve conflicts

* resolve conflict 2

* revert conflict fix

* fix changes

* address comments

* review

* docformatter presubmit

* revert docformatter

* update release.md
2022-12-02 00:15:15 +00:00
andreafehrman b26dd100e1
test(sdk): implement small tests for Client class (#8517)
Co-authored-by: droctothorpe <mythicalsunlight@gmail.com>

Co-authored-by: droctothorpe <mythicalsunlight@gmail.com>
2022-12-01 19:39:45 +00:00
Connor McCarthy e7c82c0593
feat(sdk): support list of artifacts annotations [list of artifacts support pt. 1] (#8464)
* update get_io_artifact_class function

* update various type annotation tests

* support list of artifacts in component interfaces

* move helper function

* clarify function name

* respond to review feedback
2022-11-30 05:24:15 +00:00
Connor McCarthy 249e7af384
chore(sdk): release KFP SDK 2.0.0b8 (#8508)
* update release notes

* bump version

* add beta 8 to docs; remove beta 7
2022-11-30 05:04:15 +00:00
Connor McCarthy 834bfbe90a
fix(sdk): revert premature client changes (#8507)
* revert _v1 client changes

* do not install kfp_server_api from source
2022-11-29 20:39:23 -08:00
Connor McCarthy 1043a88842
chore(sdk): release KFP SDK 2.0.0b7 (#8503)
* update release notes

* bump version

* add docs version to dropdown
2022-11-28 22:58:42 +00:00
Connor McCarthy b829e4a5da
feat(sdk): enable use of primitive placeholders in f-string (#8494)
* support primitive placeholders in fstring

* add read/write test component
2022-11-28 14:45:48 -08:00
Connor McCarthy 4bb57e6723
fix(sdk): fix multiple workers writing with gcsfuse bug [KFP SDK v2] (#8455)
* fix(sdk): fix multiple workers writing with gcsfuse bug [KFP SDK v1]

* add comment for context
2022-11-23 00:38:48 +00:00
Connor McCarthy e97fe8e04f
feat(sdk): add pipeline job placholders for scheduled runs (#8447)
* add pipeline job placeholders

* simplify example code in docs

* add _PLACEHOLDER suffix to variable names
2022-11-23 00:38:42 +00:00
Connor McCarthy be766dc352
docs(sdk): update pipeline task .set_gpu_limit reference docs (#8477) 2022-11-18 17:58:22 +00:00
Connor McCarthy 6b3b4df3d2
refactor(sdk): simplify InputSpec (#8422)
* simplify usage of inputspec

* remove underscore from optional

* add error message and test

* remove optional
2022-11-14 18:39:26 +00:00
Connor McCarthy 9e9e1081c6
fix(sdk): fix boolean default value serialization bug (#8444)
* add test cases

* fix bug

* add v1 component yaml back compat changes

* update golden snapshot
2022-11-11 15:42:33 -08:00
Lingqing Gan 3f2e20d14f
chore(backend): regenerate v1beta1 api clients (#8403)
* add v2 experiment.proto

* regenerate v1beta1 client

* remove api version from combined swagger file

* Revert "regenerate v1beta1 client"

This reverts commit 02769d779b.

* regenerate v1beta1 clients

* regenerate python packages without api version

* remove regenerated v1 python packages&change proto

* refactor v1 exp apis and functions

* remove v2 api package and refactor v1 proto

* regenerate v1 go clients

* regenerate v1 python package

* refactor and update the human maintained swagger

* renew swagger file

* regenerate v1 packages

* regenerate v1 python package

* refactor

* regenerate v1 go clients

* regenerate v1 python package

* recover dockerfile and add s in python

* fix func name

* use commit for sdk execution test

* change sdk execution test

* update test script

* add v1 to methods

* proto update v1 method names

* regenerate v1 go packages

* regenerate python client

* update server methods

* refactor

* refactor

* refactor

* refactor python client

* fix get_healthz

* update sample-test.sh in v2 backend

* sample test script

* change v2 sample test

* change package install order

* Revert "change package install order"

This reverts commit c24fcfbed9.

* use KFP server api from commit for v2 sample tests

* small fix
2022-11-10 19:18:00 +00:00
Connor McCarthy 24e17f7352
chore(sdk): quiet noisy SDK unit test output (#8434) 2022-11-09 17:47:39 +00:00
Connor McCarthy d33f359797
fix(sdk): fix nested placeholders and block illegal IfPresent form in Concat (#8414)
* support single element IfPresentPlaceholders that contain a primitive placeholder

* block list then and else_ inside concat

* support single element IfPresent in v1

* clean up typing

* skip loading batch_predict_job yaml
2022-11-04 18:20:16 +00:00
Connor McCarthy 08a8b1df97
refactor(sdk): remove dead BaseModel code (#8415)
* remove dead basemodel code

* remove unused deserialization logic

* remove more dead code

* remove basemodel entirely
2022-11-03 21:58:15 +00:00
Connor McCarthy 4c6abe5afc
refactor(sdk): simplify placeholders and logic (#8402)
* simplify placeholders

* update placeholders_test.py

* update structures to use placeholders correctly

* update use of placeholders elsewhere in codebase

* clean up

* move placeholder type checking logic to PipelineTask to prevent v1 load breaks

* clean up

* update docstrings

* make to_dict and to_string private methods

* revert unnecessary change to tests

* improve type safety of code

* prefix module-level constant with underscore
2022-11-02 00:31:45 +00:00
Connor McCarthy ca6ad27020
chore(sdk): remove unused linter comments (#8405) 2022-11-01 18:31:46 +00:00
Jonny Browning (Datatonic) 5b75d75ad7
feat(sdk): add --build-image option to 'kfp components build' to allow users to skip docker build. Fixes #8382 for 2.0 (#8387)
* feat: added --build-image/--no-build-image flags to skip docker build

* fix: logic error for warnings/errors
2022-10-26 15:58:31 +00:00
Chen Sun 8df22db757
chore(release): release sdk 2.0.0b6 (#8373)
* release sdk 2.0.0b6

* address review comments
2022-10-18 00:30:27 +00:00
JOCSTAA ac02a0e191
fix(sdk): Throw exception for component parameter named Output (#8367)
* fix(sdk): Throw exception for component parameter named Output

* Add unit test

* lint fix

* nit fixes

* nit

* nit

* format fixes
2022-10-17 23:40:27 +00:00
Connor McCarthy 30669e46dc
chore(sdk): improve KFP SDK reference documentation (#8337)
* use autodoc default options

* sort symbol order by typical usage patterns

* include summary of submodule symbols

* expose Input[] and Output[] type generics

* include tables of contents on API reference docs pages

* expand global TOC for discoverability

* fix missing reference / circular import problem
2022-10-17 19:58:53 +00:00
Chen Sun 54922b32c6
fix(sdk): fix `kfp component build` only produces empty requirements.txt (#8372)
* fix CUJ2 always writes empty requirements.txt

* sort dependencies
2022-10-17 19:22:53 +00:00
Chen Sun 4cc0e80183
fix(sdk): Allow using cli `kfp dsl compile` to compile components (#8371)
* Allow using cli `kfp dsl compile` to compile components

* improve error message

* address review comments
2022-10-17 16:06:52 +00:00
Chen Sun 83e0f16af1
fix(sdk): Fix "No KFP components found in file" error. Fixes #8353 (#8359)
* Update component.py

* Update component.py

* Update component.py
2022-10-13 21:04:03 +00:00
JOCSTAA 764140153f
fix(sdk): Fix PyPi typo in cli/component docstring (#8361) 2022-10-13 20:28:03 +00:00
Connor McCarthy 26f1fe4dfa
fix(sdk): fix NamedTuple output with Dict/List bug (#8316) 2022-09-28 02:25:10 +00:00
Connor McCarthy 19b8709ca3
chore(release): release sdk 2.0.0b5 (#8309)
* bump reference docs

* update release notes; modify structure

* bump version

* implement review feedback
2022-09-26 21:38:35 +00:00
Connor McCarthy 8ab690f3ed
feat(sdk): add workaround for google artifact (support custom artifact types pt. 4) (#8279) 2022-09-19 22:32:38 +00:00
Connor McCarthy 0cb0d99d69
fix(sdk): fix InputPath error in executor (#8273)
* fix inputpath error

* add tests

* fix nit
2022-09-19 22:32:31 +00:00
Connor McCarthy c353245baf
feat(sdk): enable dynamic importer metadata (#7660)
* support placeholder in metadata dict

* add unit tests

* add compilation test

* fix tests

* update compiler test

* use 'metadata' as input key

* update compiler test and golden snapshot

* add int input to compiler test

* change constant name

* use util function

* support f-strings in dynamic import metadata; add and update tests

* update method name after rebase

* make compile consistent across python versions
2022-09-19 21:58:31 +00:00
Scott_Xu 6443fc17dc
chore(sdk): hide private methods in placeholders for simpler doc (#8271)
* hide private methods in ConcatPlaceholders and IfPresentPlaceholder by renaming

* rename transform and validate methods
2022-09-16 16:16:30 -06:00
gkcalat 65f2b644df
fix(sdk): Replace OOB authentication with loopback flow (#8262)
* Replace oob authentication with loopback flow. Fixes #7474. Closes #7526.

* Replace oob authentication with loopback flow. Fixes #7474. Closes #7526.

* Improve error messages

* Resolve comments in the PR

* Fix refresh/access token misconception

* Fix type hint annotation
2022-09-15 23:27:24 +00:00
Connor McCarthy f54f0b9341
fix v1 output deserialization (#8272) 2022-09-15 15:51:23 -06:00
Connor McCarthy 166d6bb917
feat(sdk): add runtime logic for custom artifact types (support for custom artifact types pt. 3) (#8233)
* add runtime artifact instance creation logic

* refactor executor

* add executor tests

* add custom artifact type import handling and tests

* fix artifact class construction

* fix custom artifact type in tests

* add typing extensions dependency for all python versions

* use mock google namespace artifact for tests

* remove print statement

* update google artifact golden snapshot

* resolve some review feedback

* remove handling for OutputPath and InputPath custom artifact types; update function names and tests

* clarify named tuple tests

* update executor tests

* add artifact return and named tuple support; refactor; clean tests

* implement review feedback; clean up artifact names

* move test method
2022-09-15 01:00:40 +00:00
Sune Debel 231c5079f0
fix(sdk): make sdk installable with poetry (#8263)
* Make kfp package installable with poetry

The poetry package manager executes `setup.py` differently than pip, which means that `__file__` is initialized to just `setup.py`, which means that `where` would be set to the empty string. The proposed change has been tested with both pip and poetry.

* apply yapf formatting
2022-09-14 21:08:08 +00:00
Scott_Xu 5dff29a071
fix(sdk): keeping placeholder strings in command and args even if default value is set (#8259)
* return string representation of placeholder in pipeline_task

* return string representation of placeholder in pipeline_task
2022-09-13 23:44:31 +00:00
Connor McCarthy e371fa3795
feat(sdk): add compilation logic for google artifact types (support custom artifact types pt. 2) (#8232)
* remove dependency of deprecated on v2

* use new artifact identification logic

* add custom artifact type compiler test

* compile schema_version from non-kfp artifacts

* update tests

* add importer to compiler test

* update golden snapshots

* fix type checking code

* undo executor_test diff

* remove gcpc types compiler test

* add compilation test comment

* update google artifact handling

* update compiler test

* only permit system. and google. artifact types; add tests

* update compiler test

* simplify type checking logic

* use bundled artifact types in PipelineArtifactChannel

* raise error on unknown v1 input or output

* add type utils and move functions

* add type utils tests

* remove schema_version from inputspec and outputspec

* update artifact proto creation and tests

* propogate bundled artifact type string changes throughout sdk

* update golden snapshot

* remove unused v1 compat and tests

* update type checking and tests to validate schema_version using semantic versioning

* update importer sample

* update test data config

* update test_data config

* clean up sample

* clean up sample
2022-09-12 14:00:02 -06:00
Connor McCarthy 16c7d6658e
test(sdk): create SDK execution test suite and restructure read/write tests (#8245)
* update client for graph components

* use yaml for config

* simplify read write tests

* add test script

* restructure yaml and read/write tests

* add execution test file

* move test_data; update paths; update readme

* update kfp endpoint

* update pipeline with env

* update logging

* update config

* update kfp endpoint and timeout

* add kfp as requirement

* mock kfp_package_path for lightweight component execution

* revert pipeline name changes

* update config with name

* update read write tests to use explicit name

* update execution tests to use explicit name
2022-09-12 18:33:06 +00:00
Alex 92a1e1fe26
feature(sdk): print links outside of ipython. Fixes #8141 (#8142) 2022-09-09 00:00:27 +00:00
Connor McCarthy e14a784327
feat(sdk): migrate v1 auth to v2 (#7789)
* initial auth migration (@ji-yaqi)

* clean up files

* update imports

* add type annotations

* update docstrings

* add tests
2022-09-06 17:54:29 -06:00
Connor McCarthy eb453d10a9
fix(sdk): fix dsl control flow docstrings (#8183)
* fix dsl control flow docstrings

* update docstring
2022-09-01 16:33:07 -06:00
Chen Sun 2a566181bf
chore(sdk): bump kfp version to 2.0.0-beta.4 (#8229) 2022-09-01 06:41:55 +00:00
Connor McCarthy 25cd45d772
chore(sdk): add script to check for unused imports (#8221)
* add dev requirement

* add unused import check to isort script
2022-08-31 22:53:51 -07:00
Chen Sun d88358201d
feat(sdk): Support using pipeline in exit handlers (#8220)
* Support using pipeline in exit handlers

* release note

* remove dead code

* fix component spec merging flakiness

* address comments in PR#8209

* address review comments

* fix bad merge in release note
2022-09-01 01:45:54 +00:00
Alex Latchford c34e5e4edd
chore(sdk): Update upper bound for kubernetes dependency (#8173)
* Update upper bound for kubernetes dependency

* Add in release note for change
2022-08-31 22:47:54 +00:00
Connor McCarthy 889637ade1
fix(sdk): fix constructor signature on dsl.Condition (#8182) 2022-08-31 21:35:54 +00:00
Chen Sun 410e99c604
feat(sdk): Support loading pipeline from yaml (#8209)
* support loading pipeline from yaml

* release note

* cleanup

* maintain read/write consistency with component/pipeline compilation.
2022-08-31 12:05:55 +00:00
Yusuke Nishioka f5277d3078
Fix the example pipeline in sdk/python/README.md (#8206) 2022-08-30 14:55:45 -06:00
Chen Sun 48574dc648
feat(sdk): Support pipeline outputs (#8204)
* Support pipeline outputs

* release note
2022-08-29 20:26:53 +00:00
Connor McCarthy 72c1d10a33
feat(sdk): rename kfp artifact attributes (support custom artifact types pt. 1) (#8191) 2022-08-27 13:20:50 +00:00
Chen Sun 5ce4954d6b
feat(sdk)!: Implement Graph Component (#8179)
* Implement Graph Component

* isort imports

* release note

* address review comments

* address review comments
2022-08-26 09:28:49 +00:00
Scott_Xu 121ddcbc04
fix(sdk): fix v2 sample tests for cuj3 (#8168)
* fix v2 sample tests for cuj3

* separate sh -c

* fix two step pipeline

* fix piping error

* fix nit

* change path to uri to correct cat command

* correct bash script

* remove display name

* change directory reference in commands

* remove old files
2022-08-25 15:30:22 -07:00
Alex Latchford 459e5358e6
chore(sdk): Bump KFP Python SDK to support Python v3.10 (#8186)
* Update upper bound for kubernetes dependency

* Bump KFP Python SDK to support Py v3.10

* Reset requirements.* changes to isolate out Python v3.10 changes

* Add in release note
2022-08-24 21:34:08 +00:00
chongyouquan 5ccf53a251
fix(sdk): fix output type of importer (#8172)
* change output type of importer to artifact_class.TYPE_NAME

* use the same logic as component_factory._annotation_to_type_struct for determining type_name

* format using yapf

* fix type_name

* format using yapf

* add yaml file

* changes with regards to PR comments

* format pipeline_with_importer_and_gcpc_types.py

* remove unused imports

* change name of test component

* add pipeline_with_importer_and_gcpc_types to config

* update yaml file
2022-08-24 20:15:07 +00:00
Scott_Xu 88a1b314c9
feat(sdk): add support for metadata placeholders (#8151) 2022-08-17 23:10:06 +00:00
Bo 51bea09833
feat(sdk): Support Parallelism in ParallelFor in KFP SDK V2 (#8146)
* feat(sdk): add support for ParallelFor parallelism setting

* add parallelism value check

* Add unit tests

* Adding a compiler test

* used None as default parallelism input signature, other minor fixes on format and tests

* fix import statements

* add release.md message

* update ParallelFor docstring

* fixed docstring comments

* removed 'optional' in docstrings
2022-08-17 21:06:01 +00:00
Connor McCarthy 41b69fd90d
chore(sdk): bump kfp version to 2.0.0-beta.3 (#8149)
* update release notes

* bump version
2022-08-17 14:21:52 +00:00
Scott_Xu c7a90a3fa8
feat(sdk): expose IfPresentPlaceholder and ConcatPlaceholder to kfp.dsl (#8145) 2022-08-15 23:22:06 +00:00
Scott_Xu 35fccb1fae
feat(sdk): expose new ContainerSpec API (#8144)
* rename ContainerSpec to ContainerSpecImplementation for internal usage

* fix nit
2022-08-15 20:33:34 +00:00
Chen Sun 6137a236a5
chore(sdk): bump kfp sdk version to 2.0.0b2 (#8135) 2022-08-11 00:08:03 +00:00
Scott_Xu d3c0537c8b
chore(sdk): Add test for if and concat placeholder and validate artifact channel usage (#8120)
* test if and concat placeholder and validate artifact channel usage

* fix nit in read write test config

* resolve comments

* resolve comments and move test cases

* resolve comments and add placeholder validation

* remove unused imports

* update release file
2022-08-10 23:35:03 +00:00
Connor McCarthy bdff332ac6
feat(sdk): support more than one exit handler per pipeline (#8088)
* add compiler test pipeline with multiple exit handlers

* remove blocker of multiple exit handlers

* move exit handler builder logic to pipeline_spec_builder

* build all exit handlers per pipeline

* add compiler test with IR inspection

* prevent usage of cross-pipeline after

* test cross-pipeline after is prevented

* update existing task dependency logic and tests

* add v2 sample test

* remove cross-pipeline .after

* prevent cross-dag data dependency for dsl features

* add compiler test pipeline with nested exit handlers

* add support for nested exit handlers

* clean up pipeline with nested exit handlers

* remove sample with multiple exit handlers

* remove compiler test with nested exit handlers

* add compilation guard against nested exit handlers in subdag

* update release notes
2022-08-10 16:56:21 +00:00
Connor McCarthy e728d0871b
chore(sdk): add pre-commit hook to remove unused imports (#8123)
* add pycln pre-commit hook to remove unused imports

* remove existing unused imports
2022-08-10 08:17:21 +00:00
Connor McCarthy e47a784035
docs(sdk): clean up kfp v2 cli reference docs (#8115)
* use expanded "Command Line Interface" for title

* remove alpha feature stage in help text

* hide deprecated argument

* use compile as command name
2022-08-09 18:40:39 +00:00
Scott_Xu 04c827fc3a
feat(sdk): supporting bring your own container for arbitrary input and outputs (#8066)
* support container_component decorator for function with no inputs

* resolve review comments

* add sample tests for milestone 1

* modify compiler test data

* resolve reviews

* resolve reviews

* WIP

* implementation of function of no inputs

* fixed sample test

* re-fix sample test

* fix rebase merge conflict

* resolve formatting

* resolve isort error for test data

* resolve comments

* fix nit

* resolve nit

* add implementation for placeholders i/o, sample and compiler tests

* resolve comments and merge logic for constructing container component

* resolve comments

* resolve comments

* fix assertion messages

* add error handling for accessing artifact by itself

* add test for raising error for accessing artifact by itself
2022-08-08 21:00:53 +00:00
Chen Sun 45f3bacc0d
fix(sdk): fix compiling components with artifact inputs (#8106) 2022-08-05 17:36:15 +00:00
Deepyaman Datta 257f8dbe0a
chore(sdk): change "PipelienSpec" to "PipelineSpec" (#8102) 2022-08-04 16:55:06 +00:00
Deepyaman Datta f221d855b1
chore(sdk): remove the unreachable validation code (#8082) 2022-08-01 21:03:47 +00:00
Chen Sun 1dd06cd8e8
feat(sdk): Allow artifact inputs in pipeline definition. (#8044)
* Allow artifact inputs in pipeline definition.

* release note

* isort fix

* remove unused imports

* Delete flycheck_compiler.py

* Delete flycheck_pipeline_spec_builder.py

* Delete flycheck_structures.py

* address cr comment
2022-07-28 00:17:53 +00:00
Connor McCarthy c6a5e387dd
docs(sdk): refresh SDK public API docstrings (#8057)
* add module-level docstrings

* update compiler docstrings

* update registry module docstrings

* add BaseComponent and children to public api, but discourage use

* update artifact docstrings and type annotations

* update dsl docstrings

* update client docstrings

* clean up kfp.__init__

* add dsl placeholder docstrings

* many more docstring updates

* document type aliases in dsl module
2022-07-21 19:51:18 +00:00
Connor McCarthy d8b9439ef9
add kfp v2 api reference docs (#8042)
* use __all__ in top-level modules to record public api

* add index and source files

* add kubeflow assets to _static/

* add and pin requirements

* use block quote instead of header for readme notice

* update conf.py

* delete old files
2022-07-19 15:53:54 -06:00
Connor McCarthy e6c9411ddf
fix(sdk): fixes kfp.components module type annotations, docstrings, and string/bytes processing (#8040)
* clean up components types, docstrings, and processing

* update tests

* use real request to get component yaml

* move docstring below copyright
2022-07-19 21:02:49 +00:00
Connor McCarthy 13736b85b6
chore(sdk): clean up kfp code style (#8027)
* remove unused imports

* use google as isort profile

* sort imports

* format with yapf

* clean end of file new line, trailing whitespace, double quoted strings
2022-07-15 00:06:34 +00:00
Connor McCarthy be5ff7215f
chore(sdk): update contributing guidelines/tests (#7802)
* add presubmit checks, contributing guidelines, and pre-commit file

* delete old unused formatting script

* modify scripts
2022-07-12 20:35:29 -06:00
Scott_Xu b95c5aba85
feat(sdk/client): implements overriding caching options at submission (#7912)
* implement override caching options for client

* use absolute path in test for overriding caching

* format code and add unit tests from pipeline

* delete typo

* add two-step pipeline test and review changes

* fix formatting

* fix typo on deprecated code

* update release file

* resolve comments on docstring and testing

* update release file
2022-07-12 16:49:30 +00:00
Connor McCarthy 850a750496
feat(sdk): add retry policy support to kfp v2 (#7867)
* depend on kfp-pipeline-spec>=0.1.16

* implement RetryPolicy structure

* implement RetryPolicy structure tests

* implement .set_retry method on PipelineTask

* implement retry policy compilation logic

* implement compiler tests for retry

* implement pipeline with retry compilation test

* add release note

* clean up
2022-07-11 16:44:52 +00:00
Connor McCarthy d3592b9bbd
chore(sdk): bump kfp version to 2.0.0b1 (#7993)
* bump version

* update release notes
2022-07-07 11:30:01 -06:00
Connor McCarthy 7253203239
add registry json to package data (#7987) 2022-07-07 10:48:40 -06:00
Christian Kadner aab7fda89b
feat(sdk): Add default registry context (#7948)
* Add default registry context

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* Address review comments

* Reinstate `_is_ar_host()` check
* Remove `view=FULL` flag for default list and get
  version URLs

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* Change outdated test name

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
2022-06-30 13:51:05 -06:00
Connor McCarthy 42600a796d
chore(sdk): bump kfp version to 2.0.0b0 (#7952) 2022-06-27 18:11:59 -06:00
Connor McCarthy f877a1ffc8
fix(sdk): fix ifpresent and concat placeholder compile (#7930)
* fix ifpresent and concat placeholder expansion

* update golden snapshots

* uncomment v1 ifpresent component test case
2022-06-27 16:59:34 -06:00
Connor McCarthy ad458d535c
fix(sdk): fix placeholder read/write logic (#7928)
* fix ExecutorInputPlaceholder() usage

* fix OutputPath and OutputParameter placeholder task creation logic

* fix v1 component yaml deserialization logic

* fix typo in test

* ignore description when comparing ComponentSpec

* uncomment fixed test cases
2022-06-27 15:06:38 -06:00
Connor McCarthy ecb55bde74
feat(sdk): throw informative exception on uncalled @pipeline decorator (#7913)
* add informative error message for uncalled @pipeline

* add test for exception
2022-06-27 12:29:48 -06:00
Connor McCarthy a020c9b01c
fix(sdk): fix extract docstring in load component (#7921)
* fix extract docstring in load component

* implement review feedback
2022-06-24 12:12:35 -06:00
Connor McCarthy 281151c051
chore(sdk): improve IR read/write testing (#7920)
* add more load component test cases

* add read write equality tests

* add copyright

* remove duplicative structures tests

* add if __name__ == '__main__' block
2022-06-24 09:45:15 -06:00
chongyouquan 58f7ab8b49
chore(sdk): Registry Client - default to read full spec of versions (#7936)
* default to read full spec of versions

* update tests

* fix formatting
2022-06-24 09:43:51 -06:00
chongyouquan 4f01b7e602
feat(kfp): implement config file support for registry client (#7908)
* implement config file support

* update tests

* update PR in response to comments
2022-06-21 15:54:09 -06:00
Connor McCarthy c6125ffc44
feat(sdk): enable component compilation via cli (#7649)
* enable component compilation via cli

* reorganize compiler/test_data dir

* clean up pipeline samples

* add component compilation test cases

* update compiler test_data README

* add compiler tests

* test component deserialization is lossless
2022-06-17 05:49:10 +00:00
Connor McCarthy 6935a47e72
feat(sdk): enable component compilation via component decorator (#7554)
* enable component compilation via decorator

* add release notes

* use decorator syntactic sugar in tests
2022-06-17 00:23:10 +00:00
Chen Sun f84b51e72d
chore(sdk): Support loading v1 yaml with {executorInput: } placeholder (#7870)
* Support {executorInput: } placeholder

* merge

* use single quotes for consistency
2022-06-14 13:14:31 -07:00
Connor McCarthy 12d5cc3284
chore(sdk): use pytest instead of unittest for test execution (#7869)
* fix compiler test data path

* condense and add development requirements

* use pytest instead of unittest

* update contributing guidelines
2022-06-10 05:49:19 +00:00
Connor McCarthy 684f477a88
chore(sdk): fix component compilation test path (#7868) 2022-06-09 15:28:00 +00:00
Connor McCarthy de0b824be1
feat(sdk): add support for IfPresentPlaceholder and ConcatPlaceholder strings (#7795)
* move placeholders to placeholders module; implement if-present and concat placeholders

* update module name and method names throughout codebase

* rename placeholder abc

* add back compat support for CEL-style concat string

* add copyright

* add docstrings; clean up

* remove duplicate code
2022-06-09 07:42:00 +00:00
Chen Sun 0ada48b55a
chore(sdk): move pipeline test samples closer to compiler unit tests. (#7849)
* chore(sdk): move pipeline test samples closer to compiler unit tests.

* explicitly list test files

* remove dead code
2022-06-08 18:35:17 +00:00
johnmacnamararseg 9ffcb2e9db
feat(sdk): Add function to sdk client for terminating run (#7835)
* Add function to sdk client for terminating run

* RELEASE notes
2022-06-08 17:08:16 +00:00
Connor McCarthy 340318625c
chore(sdk): use markdown for long description content type (#7825) 2022-06-02 23:37:25 +00:00
Connor McCarthy c491f331d3
chore(sdk): bump kfp version to 2.0.0a5 (#7814)
* bump version

* update release notes
2022-06-02 22:34:26 +00:00
Connor McCarthy 0cf817364c
fix(sdk): fix load v1 if present placeholder (#7765)
* fix load v1 if present placeholder

* add test for load v1 if present placeholder
2022-06-02 16:49:53 +00:00
Connor McCarthy 2c27475fbc
add README with project description (#7782) 2022-05-26 14:59:12 -06:00
chongyouquan 281c86d41e
chore(sdk): 2.0.0-alpha.4 release (#7771)
* add release notes

* bump version to alpha.4

* fix release notes

* fix release notes

* fix release notes
2022-05-24 16:49:12 -07:00
Connor McCarthy a6ec449273
feat(sdk): only use IR when saving component (#7690)
* move function into compiler module

* remove v2 readable yaml write path

* remove v2 readable yaml test data; remove duplicate tests
2022-05-24 22:23:33 +00:00
Connor McCarthy cbc2ac5266
feat(sdk): enable read in component using IR (#7689)
* enable read in primitive components

* enable read in via YAML component

* convert OutputValuePlaceholder to OutputParameterPlaceholder

* add docstring
2022-05-24 18:58:57 +00:00
chongyouquan eaa8ec7807
fix(sdk): Fix corner cases and implement validation (#7763)
* Handle corner case when list is empty

* add validation for tags and versions

* add tests for empty lists

* add tests for version and tag validation

* fix formatting and optional arguments

* fix type annotation
2022-05-23 17:47:02 +00:00
Connor McCarthy b0db428165
feat(sdk)!: make CLI output consistent, readable, and usable (#7739)
* fix cli upload pipeline version

* organize requirements.in

* generate requirements.txt

* add pip-tools to requirements-dev

* improve CLI output

* update release notes
2022-05-20 00:03:23 +00:00
Connor McCarthy 6698801fc0
refactor(sdk): move placeholder logic to structures (#7687)
* move placeholder formatting logic to structures

* add placeholder formatting logic tests

* update placeholder references throughout codebase

* remove ABC meta inheritance
2022-05-19 07:45:56 +00:00
Connor McCarthy 101d243f48
feat(sdk): validate pipeline resource name before submission (#7713)
* add pipeline resource name validation

* add pipeline name validation tests

* update downstream pipeline name logic in cli
2022-05-13 20:12:41 +00:00
Connor McCarthy 62972eccf9
feat(sdk): enable compilation of primitive components (#7580)
* clean up relevant files

* use context manager for temporary directory cleanup

* add component compilation methods

* add component compilation tests

* add more complete pipeline_func argument handling

* use context manager for handling type check mode

* improve checks on inputs for component compilation

* correct pipeline name for primitive component IR

* simplify component compilation logic

* extract compiler methods

* move compiler helpers to helpers file to avoid circular imports

* remove underscore prefix from functions in hidden module

* move helpers tests to helper_test module

* put to_pipeline_spec on ComponentSpec

* fix inputs reference

* use same naming regex for pipelines and components

* permit compilation of all BaseComponent subclasses

* clean up docstrings

* merge compiler.helpers into pipeline_spec_builder

* modify docstrings; refactor

* modify docstring
2022-05-13 20:12:34 +00:00
Connor McCarthy 49cdb42c12
fix cli upload pipeline version (#7722) 2022-05-13 12:13:25 -06:00
Connor McCarthy 2570922a7f
feat(sdk): compile JSON with formatting (#7712)
* clean files

* implement formatted write

* add tests
2022-05-13 17:34:34 +00:00
Connor McCarthy 6b2f015672
chore(sdk): use unittest.mock instead of mock (#7694) 2022-05-10 22:55:33 +00:00
Connor McCarthy 1ae6e11bab
feat(sdk): make client return values consistent (#7659)
* clean up client.py

* update client method return values

* update other types

* use preferred path to kfp_server_api classes

* fix incorrect module name

* other type annotation cleanup

* make docstrings consistent
2022-05-10 14:35:12 -06:00
chongyouquan 25e4c58820
feat(sdk): Implement Registry client (#7597)
* Implement registry client

* Update registry client code

* Add test skeleton

* Add some tests

* Update code

* add tests

* update tests

* update tests

* Rename Client -> RegistryClient

* Update wrt comments

* add type annotations

* fix renaming in __init__.py

* remove unused imports

* extract host variable in test

* format using yapf

* remove locals and use arg keywords

* remove json conversion

* fix header

* write bytes when downloading file

* fix create_tag; fix tests

* fix request_body for update_tag and create_tag using json.dumps

* simply return json for delete_tag

* rename files

* format files

* update return types and format double quotes

* add comments and format files

* add todos

* update credentials and change open to use context

* format using yapf

* move request into context

* Update comments

* Update release notes

* Update release notes
2022-05-10 06:53:06 +00:00
Connor McCarthy 49ccb4f61f
chore(sdk): 2.0.0-alpha.3 release (#7675)
* reorganize release notes

* finalize alpha.3 notes

* bump version to alpha.3

* add one more release note
2022-05-06 16:45:28 -06:00
Connor McCarthy 18f0842ee0
fix(sdk): some cli fixes (#7668)
* use correct type when option is path

* fix options on create recurring-run

* make output logging more consistent

* show deprecation warning for dsl-compile on invocation

* fix unarchive experiment

* fix broken f-string

* convert camelcase methods to snake case
2022-05-05 12:47:18 -06:00
Connor McCarthy 5da3826bb5
feat(sdk): use custom basemodel and remove pydantic (#7639)
* fix discovered bug

* update tests

* implement custom BaseModel

* use basemodel for structures

* remove pydantic dependency

* assorted code cleanup
2022-05-04 12:56:32 -06:00
Mai Nakagawa 1118f4859b
feat(sdk): Accept newer `typing-extensions` versions (#7632)
* Accept newer typing-extensions versions

* Add a description to the release notes.
2022-04-30 00:01:25 +00:00
Connor McCarthy 652ec3b936
feat(sdk): rename CLI methods to 'create' (#7607)
* add deprecated alias group

* add deprecated alias group tests

* implement deprecated alias group

* clean up other alias tests

* clean up docstring

* fix type annotations / mypy

* format with yapf
2022-04-29 18:52:58 +00:00
Connor McCarthy 6cecfd948e
add deprecation warning to dsl-compile (#7628) 2022-04-29 11:34:48 -06:00
Connor McCarthy ec0081372a
feat(sdk): improve cli help text (#7618)
* add docstring parsing utils

* implement docstring parsing for help text

* implement smoke test of all cli commands

* clean up use of optional in docstring

* make command descriptions more consistent

* fix disable_type_check

* update release notes
2022-04-29 17:20:57 +00:00
Connor McCarthy 027ac3274f
feat(sdk): add client methods to cli (#7606) 2022-04-27 13:21:13 -06:00
Connor McCarthy e12ac39ba4
feat(sdk): use dsl noun group for compile CLI commands (#7603)
* use kfp dsl compile for compilation commands

* create and move tests
2022-04-26 15:05:31 -06:00
Connor McCarthy fbfeadd4a4
feat(sdk): add autocomplete and version options to kfp cli (#7567)
* add helpful options to cli

* add tests
2022-04-25 14:01:02 -06:00
Alex 2636727141
feat(sdk): add archive, unarchive, and delete run methods and tidy docstrings (#7562) 2022-04-25 19:05:11 +00:00
Connor McCarthy e077d01cf9
feat(sdk): add noun aliasing to cli (#7569)
* allow both singular and plural versions of nouns

* use command table for client creation

* cleanup

* simplify docker mock

* add docker to test requirements; clean up existing requirements
2022-04-25 12:24:57 -06:00
Connor McCarthy 2db431b5f4
use click for components commands (#7559) 2022-04-20 14:59:11 -06:00
Connor McCarthy a5548440a3
use click for dsl-compile command (#7560) 2022-04-20 14:57:10 -06:00
Connor McCarthy b58a31b307
chore(sdk): clean up v2 CLI (#7558)
* remove linting comments

* move global variable to local

* simplify diagnose_me command

* use f-strings

* idiomatic python refactorings

* remove unused import

* add typestub library

* silence unresolvable mypy error

* update type information

* remove unused variable

* remove unused import

* use __main__.py for cli entrypoint

* expand imports
2022-04-20 10:40:38 -06:00
Connor McCarthy 2494447578
feat(sdk): add .list_pipeline_versions and .unarchive_experiment methods to Client (#7563)
* add client methods

* update release notes
2022-04-14 21:28:38 +00:00
Connor McCarthy d46fafe4ce
feat(sdk): port cli code from v1 to v2 (#7547)
* port cli

* fix docker mock
2022-04-13 20:19:52 +00:00
Connor McCarthy 94960cf5ab
chore(sdk): use setuptools.find_packages (#7548) 2022-04-12 21:52:51 +00:00
Connor McCarthy f70413ad7c
chore(sdk): stop using pydantic for serialization (#7536)
* add ir_utils

* add ir_utils tests

* use ir_utils to write ir to file

* cast posix path to string in `kfp components build` command

* add copyright

* remove unused import

* update setup.py packages
2022-04-12 17:03:55 +00:00
Connor McCarthy c98f14abb4
chore(sdk): make field order in compile deterministic (#7520)
* apply docformatter

* use sort_keys=True in compiler

* update golden snapshots

* set other sort_keys=True

* update structures_test yamls
2022-04-06 21:46:13 +00:00
Connor McCarthy 4eeb420905
feat(sdk)!: promote v2 compiler command and deprecate v1 command (#7508) 2022-04-05 18:07:37 +00:00
Chen Sun 9db6e74556
chore(sdk): release KFP sdk 2.0.0-alpha.2 (#7519) 2022-04-05 00:08:35 -07:00
Connor McCarthy d74d7b979b
add typestub libraries (#7451) 2022-04-04 18:04:33 -06:00
Chen Sun 9e708b9208
fix(sdk): fix yaml support `create_run_from_pipeline_func` (#7500) 2022-04-01 19:18:25 +00:00
Hao Xin 48644edf57
chore(deps): Bump google-cloud-storage to 2.2.1 (#7493) 2022-04-01 18:33:25 +00:00
Chen Sun 2019a46114
chore(sdk): Update proto link for PipelineTaskFinalStatus (#7496) 2022-03-31 20:12:30 +00:00
Connor McCarthy 1c3e2768e6
chore(sdk): modify deprecated tests (#7446) 2022-03-25 21:10:11 +00:00
Chen Sun d6e4a34c50
feat(sdk): Add pipeline_task_name to PipelineTaskFinalStatus (#7464)
* Add pipeline_task_name to PipelineTaskFinalStatus

* release note
2022-03-25 20:48:11 +00:00
Connor McCarthy 9f413706df
feat(sdk): enable pip install from custom index url (#7453)
* add pip_index_urls to component constructor

* add unit tests

* add compiler cli tests for pip_index_urls

* update golden snapshots

* add release notes

* include test file in compiler_cli_tests

* change spacing in command

* revert golden snapshots

* rename test method

* fix diff

* improve quote consistency

* extract string template from function

* fix component_op

* remove json golden

* update compiler_cli_tests
2022-03-24 20:41:47 +00:00
Chen Sun dfc85b0a0d
feat(sdk)!: Support submitting pipeline IR in yaml format via `kfp.client` (#7458)
* support yaml payload for pipeline manifest

* release note
2022-03-23 22:48:46 +00:00
Yaqi Ji 04123280a4
fix(sdk): add default value for inputs (#7405)
* fix(sdk): add default value for inputs

* merge conflict

* release

* fix sample
2022-03-23 18:59:20 +00:00
Connor McCarthy 41caca8850
chore(sdk): remove python 3.6 dead code (#7440)
* migrate modules to >= python 3.7

* update test images to greatest supported python version

* remove unused import

* update test data

* clean comment

* undo 3.7 image change

* revert 3.9 -> 3.7

* revert OrderedDict to Dict
2022-03-21 22:15:30 +00:00
Connor McCarthy c7301c449c
feat(sdk)!: use yaml for IR package (#7431)
* change serialization format from json to yaml

* refactor: use guard clause for clarity

* convert test data from json to yaml

* add and update tests

* permit deprecated compile to json

* re-add json test data

* update compiler tests

* refactor Compiler._write_pipeline_spec_file

* update golden snapshots

* update sample output extensions

* test file extension exceptions and warnings

* update release notes

* remove json golden snapshots
2022-03-21 21:45:31 +00:00
Connor McCarthy d6affa5afe
docs(sdk): update contributing guidelines (#7436)
* update SDK contributing guidelines

* add requirements-dev.txt

* update shell script instructions

* update shell script instructions

* add mypy to code style guidelines

* fix path

* Update README.md
2022-03-18 23:41:06 +00:00
Connor McCarthy bfffa83fba
chore(sdk): apply docformatter (#7438) 2022-03-18 17:02:40 +00:00
Connor McCarthy 2c0b38162f
chore(sdk): make kfp v2 hermetic (#7428)
* move v1 files to v2

* format with yapf

* remove unused imports

* clean up

* update copyright

* move imports to module top level

* apply some pylint changes

* loosen pylintrc

* remove v2_yaml_utils.py
2022-03-18 03:00:40 +00:00
Chen Sun c03701b82a
fix(sdk): fix wrong kfp import causes wrong sdk_version being set in pipeline_spec. (#7433)
* fix wrong kfp import in compiler

* release note
2022-03-18 00:05:39 +00:00
Jonny Browning (Datatonic) 02a52a4185
fix(sdk): no longer require KFP client for kfp components build (#7430)
* fix(sdk): no longer require KFP client for kfp components build

* docs(sdk): added release note for kfp components build fix
2022-03-17 22:25:39 +00:00
Hao Xin a5a8c750b5
chore: fix ref link (#7434) 2022-03-17 16:03:11 +00:00
Chen Sun 481b108319
fix(sdk): update kfp-server-api dependencies and bump sdk version to 2.0.0-alpha.1 (#7427)
* update kfp-server-api dependencies

* release note

* bump sdk version to 2.0.0-alpha.1

* fix
2022-03-16 23:30:09 +00:00
Chen Sun 40e9610711
fix(sdk): include requirements.in in sdk manifest (#7429) 2022-03-16 15:55:14 -07:00
Connor McCarthy 4621aec146
chore(sdk): apply yapf formatting (#7414) 2022-03-16 20:34:36 +00:00
Chen Sun b741926bb4
feat(sdk): client to support KFP v2 API. (#7411)
* copy _client.py and _auth.py from deprecated folder

* Make kfp.Client support v2 API (IR).

* fix test golden

* address comments

* copyright year

* chore(backend): clean up pipelinespec.Value usage (#7407)

* chore(sdk): clean, dedup, reconcile, organize requirements (#7400)

* chore(sdk): clean, dedup, reconcile, organize requirements

* apply yapf formatting

* correct copyright

* chore(sdk): fix test missing tempdir cleanup (#7403)

* chore(sdk): fix test missing tempdir cleanup

* fix test path

* fix test path

* clean up setUp method

* change string placeholder from s to test_dir

* release note

Co-authored-by: Connor McCarthy <mccarthy.connor.james@gmail.com>
2022-03-15 21:45:37 +00:00
Connor McCarthy 5fb126a715
feat(sdk): alias v2 imports and provide DeprecationWarning (#7398)
* feat(sdk): alias v2 imports and provide DeprecationWarning

* simplify tests

* enable import of v2 namespace

* assert DeprecationWarning is raised

* update copyright

* silence pylint

* move v2 alias tests

* use assertWarnsRegex instead of assertWarns

* fix tests

* remove test for deprecation warning

* update copyright
2022-03-15 03:28:39 +00:00
Connor McCarthy 247672c75c
chore(sdk): fix test missing tempdir cleanup (#7403)
* chore(sdk): fix test missing tempdir cleanup

* fix test path

* fix test path

* clean up setUp method

* change string placeholder from s to test_dir
2022-03-14 21:25:40 +00:00
Connor McCarthy 7613cf9a1f
chore(sdk): clean, dedup, reconcile, organize requirements (#7400)
* chore(sdk): clean, dedup, reconcile, organize requirements

* apply yapf formatting

* correct copyright
2022-03-14 18:27:57 +00:00
Connor McCarthy 5dcc6d9910
chore(sdk): rename PipelineTask arguments parameter to args (#7392) 2022-03-10 21:42:54 +00:00
Connor McCarthy 5ca4cb78d3
chore(sdk): remove unused imports and variables (#7393) 2022-03-09 21:42:11 +00:00
Connor McCarthy 82d17ff616
feat(sdk)!: rename commands, arguments parameters to command, args (#7391) 2022-03-09 05:39:11 +00:00
Yaqi Ji d695ec556a
feat(sdk)!: throw error for output_component_file (#7390)
* feat(sdk)!: throw error for output_component_file

* Update readme
2022-03-09 01:05:10 +00:00
Yaqi Ji 8bb0d0e025
feat(sdk)!: move v2 to main namespace (#7376)
* feat(sdk)!: move v2 to main namespace

* edit samples
2022-03-04 21:56:51 +00:00
Chen Sun 7f67130c6e
fix(sdk): require google-api-core>=1.31.5, >=2.3.2 (#7377)
* require google-api-core>=1.31.5, >=2.3.2

* add release note
2022-03-04 19:56:59 +00:00
Yaqi Ji b693909a08
feat(sdk)!: move v1 to deprecated folder (#7291)
* chore(sdk): move v1 to deprecated

* fix testsg

* fix testsg

* fix setup.py

* fix test

* s'

* fix tests

* fix test

* retore v2 test changes

* fix py3.6 test

* fix py3.6 test

* fix py3.6 import fallback error

* remove deprecated

* fix samples test

* sample test

* fix samples

* add readme

* restroe test

* python require

* remove path

* fix tests

* inteegration tests

* remove tfx tests for dependency with kfp v1

* fix e2e

* fix e2e

* fix integration tests

* fix sampe

* move client down

* change to kfp

* add import alias

* fix

* runid

* fix dsl

* only use kfp for function

* revert train_until_good

* tfx test

* kfp

* try import

* onprem
2022-03-04 00:52:59 +00:00
Chen Sun 913539728b
chore(sdk): Fix a minor bug in task final status test sample (#7314) 2022-02-17 00:27:34 +00:00
Chen Sun de2a4a45f2
chore(sdk): bump version to 2.0.0-alpha.0 (#7313)
* chore(sdk): bump version to 2.0.0-alpha.0

* relax regex to match sdk version in tests
2022-02-15 20:07:41 +00:00
Yaqi Ji 54d5789bb4
feat(sdk)!: drop support for python 3.6 (#7303)
* chore(sdk): drop support for python 3.6

* empty

* address comments
2022-02-15 10:54:46 -08:00
Chen Sun ab805f886c
feat(sdk): Support getting pipeline status in exit handler. (#7309)
* Support PipelineTaskFinalStatus in exit handler.

* release note
2022-02-15 17:58:41 +00:00
Mehdi Drissi 9be1720f55
chore(sdk): Relax Typing Extensions Constraint (#7288)
* Relax Typing Extensions Constraint

* Update requirements.txt for typing extensions

Co-authored-by: Mehdi Drissi <mdrissi@snapchat.com>
2022-02-15 10:32:40 +00:00
Alex e07643453c
feat(sdk): Add `enable_job` method to client (#7239)
* feat(sdk): implement enable_job method

* Update RELEASE.md

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2022-02-03 18:29:22 -08:00
vmallya-123 060588e1d9
fix(sdk): add verify_ssl to kfp client, fixes #4569 (#7174)
* adding verify_ssl

* adding code formatter changes

* retrigger checks

* updating release.md

* retrigger checks

* retrigger checks

* updating release.md
2022-02-03 18:28:16 -08:00
Alex Latchford 808ff5d8ed
fix(sdk): Add in filter to list_pipeline_versions SDK method (#7223)
* AIP-5661: Add in filter to list_pipeline_versions SDK method

* AIP-5661: Update type hints + docstrings to conform to standard. Include message in release notes
2022-02-03 16:52:45 -08:00
Chen Sun 1acb90fc94
fix(sdk.v2): Fix importer not using correct output artifact type (#7235)
* fix importer output type

* release note
2022-01-31 14:34:08 -08:00
Chen Sun 7f178900ae
chore(sdk): Bump SDK version to 1.8.11 (#7202) 2022-01-24 22:42:19 +00:00
Seb 577902ca46
fix(sdk): Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest (#7141)
* Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest

* Update test for TTL

* Declare fix in release note of SDK

* update syntaxe

* Update RELEASE.md

Add Braking change due to incompatibility with  KFP pre 1.7 due to Argo 2.X
2022-01-20 00:43:52 +00:00
Jiyong Jung 22a8fdfe31
fix(sdk): bumps up absl version to 1.x for compatibility with TF 2.8+. Fixes #7171 (#7172)
* Bumps up absl version to 1.x for compatibility with TF 2.8+.

* Adds a release note for absl version.
2022-01-19 22:14:52 +00:00
Chen Sun 83d54d3cb7
fix(sdk): Fix create_runtime_artifact not covering all types. (#7168)
* Fix create_runtime_artifact not covering all types.

* release note
2022-01-19 04:16:19 +00:00
Gerard Casas Saez fc2d05f90d
fix(sdk): Respect namespace if set on client (#7056)
* respect namespace if set on client

* make sure is_multi_user

* overwrite directly context_setting

* revert non-related changes

* revert overwrite_namespace

* expand text

* add release notes

* add PR link
2022-01-11 06:04:42 +00:00
Yaqi Ji 5a29d7f95b
feat(sdk): add metadata field for importer (#7112)
* feat(sdk): add metadata

* metadata

* release notes

* address comments
2022-01-06 00:41:35 +00:00
Yuan (Bob) Gong ca6e05591d
test: update README of v2 sample test (#7121)
* test: update README of v2 sample test

* address review comments by @zijianjoy

* use absolute paths, suggested by @zijianjoy

* Update README.md
2021-12-30 16:22:54 +08:00
Yuan (Bob) Gong 2460aaa653
test: migrate sample integration test to v2 sample test infra (#7115)
* test: migrate sample integration test to v2 sample test infra

* test: configure v2 integration test timeout
2021-12-26 04:17:44 +00:00
Yuan (Bob) Gong 44228c242b
test(v2): v2 sample test supports notebooks (#7109)
* test(v2): v2 sample test supports notebooks

* fix

* fix relative path problem
2021-12-24 02:37:42 +00:00
Chen Sun 6dfaeebd92
fix(sdk.v2)!: Block task dependency referencing tasks inside a sibling condition or loop group. (#7050) 2021-12-13 19:08:37 +00:00
Yaqi Ji 85d74337d6
fix(sdk): fix input/outputspec and positional arguments (#6980)
* fix(sdk): fix load_test

* concat placeholder

* release notes

* fix(sdk): fix resource spec test

* release notes

* test for ontology

* fix(sdk): fix input/outputspec

* fix reuse

* fix xgboost sample

* release

* fix tests

* address comments

* fix test

* restore tests

* address comments
2021-12-13 18:12:37 +00:00
Yaqi Ji 1a0bef7102
fix(sdk.v2): fix boolean error for inputs (#7041) 2021-12-11 17:25:35 +00:00
Chen Sun 6ca35800c7
chore: SDK release note for 1.8.10 (#7017)
* Update RELEASE.md

* Update __init__.py
2021-12-08 00:05:59 +00:00
michaelliqx a71fce0841
fix(sdk): fix bug in checking values in _param_values. Fixes #6799 (#6965) 2021-12-03 18:14:57 +00:00
Yuan (Bob) Gong bc7aad8b75
chore(sdk): fix error 'ApiResourceReference' object has no attribute 'id' (#6997)
* chore(sdk): fix error 'ApiResourceReference' object has no attribute 'id'

* fix more
2021-12-03 01:28:58 +00:00
Chen Sun c1b67154a2
fix(sdk.v2): fix regression on nested loops (#6990)
* fix regression on nested loops

* release note
2021-12-01 22:34:05 +00:00
Alex Latchford ddbfcde16f
feat(sdk): Improve CLI experience for archiving experiments, managing recurring runs and listing resources (#6934) 2021-12-01 18:07:13 +00:00
Yaqi Ji 485a07719f
fix(sdk): fix resource spec test (#6979)
* fix(sdk): fix load_test

* concat placeholder

* release notes

* fix(sdk): fix resource spec test

* release notes

* test for ontology
2021-11-30 19:25:31 +00:00
Yaqi Ji 0842934bae
fix(sdk): fix load_test (#6978)
* fix(sdk): fix load_test

* concat placeholder

* release notes
2021-11-29 23:00:30 +00:00
Chen Sun 8aee62142a
feat(sdk)!: Deprecate V2 compatible mode in v1 compiler (#6958)
* Deprecate V2 compatible mode in v1 compiler

* release note

* fix tests
2021-11-24 22:11:34 +00:00
Chen Sun 59dfffebea
fix(sdk.v2): Avoid pydantic union bug (#6957)
* avoid pydantic union bug

* release note
2021-11-24 21:48:34 +00:00
Yaqi Ji cd20373310
feat(sdk): add set_env_variable for Pipeline task (#6919)
* feat(sdk): add set_env_variable

* release notes

* enable test

* fix test

* fix test

* address comments
2021-11-23 21:46:47 +00:00
Yaqi Ji 4484ccd0b3
fix(sdk): change otherwise to else in yaml (#6952)
* fix(sdk): change otherwise to else in yaml

* release notes
2021-11-23 21:31:49 +00:00
hieuhc 6d55e262b4
fix(sdk): refresh access token only when it expires. Fixes #6883 (#6941)
* refresh access token whenerver it expires

* tight the condition when refreshing the access token
2021-11-23 19:35:07 +00:00
Bu Sun Kim 8064383cf1
chore(sdk): allow google-auth < 3 (#6939)
* Allow google-auth < 3

google-auth recently published a 2.0.0 release which removed support for Python 2.7. google-auth now requires Python >=3.6. No other breaking changes were made. You can see the full list of changes [here](googleapis/google-auth-library-python@560cf1e).

I am opening PRs to expand google-auth version ranges for packages that meet either of the following criteria:
* Package has >10,000 monthly downloads as of June 2021
* Package is owned by a Google team

**Motivation:**
google-auth is a dependency of many different libraries that interact with Google APIs. Increasing the time and number of packages with compatible pins on google-auth lowers the chance end developers who use multiple libraries will see dependency conflicts. 

If possible, please also retain compatibility with 1.x.x versions of google-auth until this issue is resolved
googleapis/google-cloud-python#10566, as that will further reduce the likelihood of diamond dependency conflicts.

Googlers, see [this doc](https://docs.google.com/document/d/1euAvUsia_4zf98lNvpwA3K0o2b4y5Xr9xAkyzCnIMzQ/edit) for more information.

* chore: also update requirements.in

* chore: update RELEASE.md
2021-11-23 18:47:06 +00:00
Chen Sun 228d37ec5c
chore(sdk.v2): Add test sample for runtime placeholders (#6946) 2021-11-23 17:23:06 +00:00
Yusuke Nishioka b6406b02f4
chore(sdk): Fix a warning message (#6911)
* Fix(sdk): Enable importing `kfp.v2.components.create_component_from_func`

* Revert "Fix(sdk): Enable importing `kfp.v2.components.create_component_from_func`"

This reverts commit b5d3f13720.

* Fix the warning message
2021-11-22 16:17:05 -08:00
Chen Sun 2910d10cc3
fix(sdk.v2): Fix bug on executor input with default value. (#6937)
* fix bug on executor input

* release note

* fix test on empty message

* fix part of the test
2021-11-22 21:09:06 +00:00
Yaqi Ji c4d8dfd8c6
feat(sdk): ImporterSpec v2 (#6917)
* importer

* compiler

* check diff

* fix format

* remove v1 node test

* remove importer v2 compat test

* release note

* address comments
2021-11-18 23:11:32 +00:00
juliusvonkohout b482ba83d8
fix(sdk): visualizations and metrics do not work with data_passing_methods (#6882)
* Update _data_passing_using_volume.py

* Update _data_passing_using_volume.py

* Update _data_passing_using_volume.py

* Update _data_passing_using_volume.py

* Update _data_passing_using_volume.py

* Update artifact_passing_using_volume.yaml

* Update artifact_passing_using_volume.py

* Update _data_passing_using_volume.py
2021-11-16 01:33:12 +00:00
Chen Sun 9ee4534aef
fix(sdk.v2): Fix regression on optional inputs (#6905)
* Fix regression on optional inputs

* add release note
2021-11-15 18:43:43 +00:00
Chen Sun 7921946b07
feat(sdk.v2)!: Merge v2 experimental change back to v2 namespace (#6890)
* merge v2 experimental change back to v2 namespace

* release note

* fix tests

* fix cli

* fix py3.6 test

* skip v2 compatible mode tests

* fix tests

* skip v2 engine tests
2021-11-11 00:25:42 +00:00
Yaqi Ji b7fd47f70b
chore(sdk): Remove sdk/python/kfp/v2/google directory for v2 (#6886)
* Delete sdk/python/kfp/v2/google directory

* Update RELEASE.md

* Update MANIFEST.in

* Update requirements-test.txt

* Update requirements.in

* fix setup

* remove test
2021-11-10 01:18:04 -08:00
Chen Sun 3bbe81a463
chore(sdk): Remove redundant check in set_gpu_limit (#6866)
* remove redundant check in set_gpu_limit

* release note
2021-11-08 10:35:02 -08:00
Chen Sun bb9db6a19d
chore(sdk): Release KFP SDK 1.8.9 off `sdk/release-1.8` branch (#6864)
* Update RELEASE.md

* Update __init__.py
2021-11-03 23:00:21 -07:00
Chen Sun dea0823fe8
fix(sdk): Make `Artifact` type be compatible with any sub-artifact types bidirectionally (#6859) 2021-11-03 16:34:21 -07:00
Yaqi Ji 334307d6c3
chore(sdk): add deprecation warnings for v2 (#6851) 2021-11-02 16:26:06 -07:00
Trs b2414f3eb8
fix(sdk): fix cloud scheduler's job name (#6844) 2021-11-02 13:46:06 -07:00
Toshiya Kawasaki 2f2ebf8962
fix(sdk): Fix invalid doc example of python_component (#6841) 2021-11-02 11:41:06 -07:00
Chen Sun 0f2cab9b39
fix(sdk.v2): Fix display name support for groups (#6832)
* fix display name for experimental tasks_group

* release note
2021-10-29 00:30:51 -07:00
Yaqi Ji f1bb852852
feat(sdk): add load_component_from_* (#6822)
* feat(sdk): add load_component_from_*

* address comments'
:

* update release notes
2021-10-28 22:30:34 -07:00
Chen Sun 8c6843fdb8
fix(sdk.v2): Fix importer ignoring `reimport` setting, and switch to Protobuf.Value for import uri. (#6827)
* fix importer

* release note

* remove error usage in test sample

* disable importer test
2021-10-28 20:40:51 -07:00
Chen Sun ea2e5be81a
feat(sdk.v2): Implement experimental `@component` component (#6825)
* Implement experimental `@component` component

* release note
2021-10-28 18:29:51 -07:00
Ajay Gopinathan 0be57c3880
feat(sdk): Use google.protobuf.Value in v2 for passing parameters. (#6804)
* Use google.protobuf.Value in v2 for passing parameters.

* retest samples.

* Fix tests.

* Update release, more cleanup.

* Use github.com/kubeflow/pipelines/api from same repo.

* Run go mod tidy

* chore: go mod tidy

* fix v2 compile error and clean up unused code

* pr comments.

* update goldens

* Fix metadata recording.

* Update kfp mlmd client.

* fix test again

* another try.

* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings

* fix v2 launcher + clean up

* fix a compile error

* fix a few more tests

* fix number parsing

* clean up

* disable cache_v2 test.

Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
2021-10-28 14:25:51 -07:00
Yaqi Ji 0c40154f6d
chore: Release KFP SDK and v2 launcher 1.8.7 (#6823)
* Update RELEASE.md

* Update __init__.py

* Update v2_compat.py
2021-10-28 10:53:26 -07:00
Omar Marzouk 3e6c776360
feat(sdk): Add load_component_from_spec. Fixes #5708 #3748 (#6690)
* add load_component_from_spec

* update release.md

* update RELEASE.md

* update RELEASE.md
2021-10-28 10:27:57 -07:00
dinimicky 41275b4228
fix(sdk): fixes the specified 'mlpipeline-ui-metadata','mlpipeline-metrics' path is overrided by default value (#6796)
* fix the issue:

the 'mlpipeline-ui-metadata','mlpipeline-metrics' path is replaced by the default path as '_components._generate_output_file_name(output.name)'

* reformat code

* Update _container_op.py

* Update RELEASE.md
2021-10-28 01:33:00 -07:00
Joshua Carp 37d3234ce5
feat(sdk): Add filters to python client. (#6748)
* Add filters to python client.

* Add filter docstrings.

h/t @chensun

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-10-28 01:15:50 -07:00
Joshua Carp ec4ab2dc4c
fix(sdk): Add missing retry policy. (#6808) 2021-10-27 23:41:30 -07:00
Chen Sun 7768f28de3
fix(sdk.v2): Fix missing experimental modules in setup.py (#6817)
* fix missing experimental modules

* fix typo

* fix test
2021-10-27 10:58:39 -07:00
Ajay Gopinathan fca4c1c2fd
Fix missing experimental type module in setup.py. (#6816) 2021-10-27 09:17:03 -07:00
Chen Sun 02b96f0a63
feat(sdk.v2): Implement v2 experimental compiler. (#6803)
* Implement v2 experimental compiler.

* add release note

* Support protobuf.Value in Condition operand
2021-10-26 14:12:37 -07:00
Gerard Casas Saez 5dfae1d8cc
feat(sdk): Add version description optionally to Py SDK. Part of #6256 (#6472)
* add description optionally

* add release notes

* add new line

* capture if description is not accepted by server

* add todo comment

* add better description api missing exception
2021-10-24 21:24:07 -07:00
Chen Sun 61fea6ba1f
fix(sdk.v2): Fix InputPathPlaceholder changed to InputValuePlaceholder (#6794)
* fix InputPathPlaceholder changed to InputValuePlaceholder

* release note
2021-10-24 17:10:07 -07:00