Commit Graph

250 Commits

Author SHA1 Message Date
Connor McCarthy 96742a648a
docs(sdk): add dsl placeholder exports (#9423) 2023-05-15 19:46:31 +00:00
IronPan 5e98b3b6f0
feat(sdk): Support PIPELINE_ROOT_PLACEHOLDER (#9134)
* Support PIPELINE_ROOT_PLACEHOLDER

/assign @connor-mccarthy

* Update __init__.py
2023-04-11 08:56:49 -07:00
Connor McCarthy b575950a75
feat(sdk): support for fan-in from `dsl.ParallelFor` (#8631)
* implement parallelfor fan-in

* temporarily block fan-in of artifacts

* clean up

* unify and clarify error messages

* remove comment

* remove unnecessary compiler condition

* make test cases more realistic

* use better ParallelFor variable name

* add and rename test cases

* add ParallelFor fan-in test cases

* fix return collected bug

* clean up compilation tests

* update docstrings; update error message

* remove fan-in artifacts read/write tests

* implement review feedback

* change use of additional_input_name_for_pipeline_channel

* move additional_input_name_for_pipeline_channel

* provide additional pipeline topology validation and tests; refactor existing validation

* add release note for parameter fan-in
2023-02-04 20:50:51 -08:00
Connor McCarthy e97fe8e04f
feat(sdk): add pipeline job placholders for scheduled runs (#8447)
* add pipeline job placeholders

* simplify example code in docs

* add _PLACEHOLDER suffix to variable names
2022-11-23 00:38:42 +00:00
Connor McCarthy 30669e46dc
chore(sdk): improve KFP SDK reference documentation (#8337)
* use autodoc default options

* sort symbol order by typical usage patterns

* include summary of submodule symbols

* expose Input[] and Output[] type generics

* include tables of contents on API reference docs pages

* expand global TOC for discoverability

* fix missing reference / circular import problem
2022-10-17 19:58:53 +00:00
Scott_Xu c7a90a3fa8
feat(sdk): expose IfPresentPlaceholder and ConcatPlaceholder to kfp.dsl (#8145) 2022-08-15 23:22:06 +00:00
Scott_Xu 04c827fc3a
feat(sdk): supporting bring your own container for arbitrary input and outputs (#8066)
* support container_component decorator for function with no inputs

* resolve review comments

* add sample tests for milestone 1

* modify compiler test data

* resolve reviews

* resolve reviews

* WIP

* implementation of function of no inputs

* fixed sample test

* re-fix sample test

* fix rebase merge conflict

* resolve formatting

* resolve isort error for test data

* resolve comments

* fix nit

* resolve nit

* add implementation for placeholders i/o, sample and compiler tests

* resolve comments and merge logic for constructing container component

* resolve comments

* resolve comments

* fix assertion messages

* add error handling for accessing artifact by itself

* add test for raising error for accessing artifact by itself
2022-08-08 21:00:53 +00:00
Connor McCarthy c6a5e387dd
docs(sdk): refresh SDK public API docstrings (#8057)
* add module-level docstrings

* update compiler docstrings

* update registry module docstrings

* add BaseComponent and children to public api, but discourage use

* update artifact docstrings and type annotations

* update dsl docstrings

* update client docstrings

* clean up kfp.__init__

* add dsl placeholder docstrings

* many more docstring updates

* document type aliases in dsl module
2022-07-21 19:51:18 +00:00
Connor McCarthy d8b9439ef9
add kfp v2 api reference docs (#8042)
* use __all__ in top-level modules to record public api

* add index and source files

* add kubeflow assets to _static/

* add and pin requirements

* use block quote instead of header for readme notice

* update conf.py

* delete old files
2022-07-19 15:53:54 -06:00
Connor McCarthy 13736b85b6
chore(sdk): clean up kfp code style (#8027)
* remove unused imports

* use google as isort profile

* sort imports

* format with yapf

* clean end of file new line, trailing whitespace, double quoted strings
2022-07-15 00:06:34 +00:00
Yaqi Ji 8bb0d0e025
feat(sdk)!: move v2 to main namespace (#7376)
* feat(sdk)!: move v2 to main namespace

* edit samples
2022-03-04 21:56:51 +00:00
Yaqi Ji b693909a08
feat(sdk)!: move v1 to deprecated folder (#7291)
* chore(sdk): move v1 to deprecated

* fix testsg

* fix testsg

* fix setup.py

* fix test

* s'

* fix tests

* fix test

* retore v2 test changes

* fix py3.6 test

* fix py3.6 test

* fix py3.6 import fallback error

* remove deprecated

* fix samples test

* sample test

* fix samples

* add readme

* restroe test

* python require

* remove path

* fix tests

* inteegration tests

* remove tfx tests for dependency with kfp v1

* fix e2e

* fix e2e

* fix integration tests

* fix sampe

* move client down

* change to kfp

* add import alias

* fix

* runid

* fix dsl

* only use kfp for function

* revert train_until_good

* tfx test

* kfp

* try import

* onprem
2022-03-04 00:52:59 +00:00
Chen Sun 3bbe81a463
chore(sdk): Remove redundant check in set_gpu_limit (#6866)
* remove redundant check in set_gpu_limit

* release note
2021-11-08 10:35:02 -08:00
Chen Sun dea0823fe8
fix(sdk): Make `Artifact` type be compatible with any sub-artifact types bidirectionally (#6859) 2021-11-03 16:34:21 -07:00
Toshiya Kawasaki 2f2ebf8962
fix(sdk): Fix invalid doc example of python_component (#6841) 2021-11-02 11:41:06 -07:00
Ajay Gopinathan 0be57c3880
feat(sdk): Use google.protobuf.Value in v2 for passing parameters. (#6804)
* Use google.protobuf.Value in v2 for passing parameters.

* retest samples.

* Fix tests.

* Update release, more cleanup.

* Use github.com/kubeflow/pipelines/api from same repo.

* Run go mod tidy

* chore: go mod tidy

* fix v2 compile error and clean up unused code

* pr comments.

* update goldens

* Fix metadata recording.

* Update kfp mlmd client.

* fix test again

* another try.

* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings

* fix v2 launcher + clean up

* fix a compile error

* fix a few more tests

* fix number parsing

* clean up

* disable cache_v2 test.

Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
2021-10-28 14:25:51 -07:00
dinimicky 41275b4228
fix(sdk): fixes the specified 'mlpipeline-ui-metadata','mlpipeline-metrics' path is overrided by default value (#6796)
* fix the issue:

the 'mlpipeline-ui-metadata','mlpipeline-metrics' path is replaced by the default path as '_components._generate_output_file_name(output.name)'

* reformat code

* Update _container_op.py

* Update RELEASE.md
2021-10-28 01:33:00 -07:00
Joshua Carp ec4ab2dc4c
fix(sdk): Add missing retry policy. (#6808) 2021-10-27 23:41:30 -07:00
Chen Sun 7f1e34f17d
chore(sdk.v2): Implement LoopArgument and LoopArgumentVariable v2 (#6755)
* for_loop v2

* release note

* address review comments
2021-10-19 03:09:23 -07:00
chongyouquan 14b3d6751a
fix(sdk): Fix type_utils (#6719)
* chore(kfp): Fix type_utils

* Update RELEASE.md
2021-10-12 11:17:44 -07:00
Chen Sun b466f59e2f
fix(sdk.v2): Fix a couple of ParallelFor related bugs. Fixes #6383, fixes #6628 (#6643)
* fix a couple of loop related bugs

* add release note
2021-09-29 11:27:09 -07:00
Lin Yiming 4e89973504
fix(sdk): sanitize op name. Fix #6433 (#6600)
* fix(sdk): sanitize op name. Fix #6433

* Add release note: Fix the error that kfp v1 compiler failed to provide unique name for ops of the same component.
2021-09-28 00:10:10 -07:00
Midhun R Nair 24331a9aee
feat(sdk): Enable re-use of PVC with VolumeOp (#6582) 2021-09-22 22:43:35 -07:00
Chen Sun 0fba85cfab
fix(sdk.v2): Support dict, list, bool typed input parameters from constant values and pipeline inputs. (#6523)
* fix dict list typed inputs support

* Support passing dict, list, bool typed parameter via client

* update release note

* fix test
2021-09-07 19:53:07 -07:00
Chen Sun 29226e9178
feat(sdk.v2): Support container environment variable in v2. (#6515)
* Support `set_env_variable` in v2.

* update release note
2021-09-07 13:30:07 -07:00
Chen Sun 2f19a26ffd
chore(sdk): Format all Python files under SDK folder. (#6501)
* Reformat sdk only using the new yapf config.

* Reformat docstrings using docformatter.

* update golden files to resolve diff caused by whitespaces

* fix some tests

* format .py files under sdk/python/tests using yapf

* additional docformatter

* fix some tests
2021-09-03 11:25:11 -07:00
Chen Sun cfefc6d397
feat(sdk.v2): Support `set_display_name` in v2. (#6471)
* Support `set_display_name` in v2.

* update comment in proto

* fix test
2021-08-30 12:07:25 -07:00
Yaqi Ji 65174cdd93
feat(sdk): Add BaseModel to component_spec data classes (#6372)
* pydantic setup

* switch to union types

* Fix base_component tests

* switch to pydantic for python3.6 compat and fix tests

* Fix tests

* use str for type

* fix tests

* Consistent quote sign

* Address comments

* Update type
2021-08-25 14:00:02 -07:00
Yaqi Ji 937cacd4ce
feat(sdk): add default schema_version to pipeline (#6366)
* feat(sdk): add default schema_version to pipeline

* sync api for go

* Fix tests and address comments

* Bump pipeline_spec version

* Fix v1 tests

* rebase to master

* sync api for go

* Fix tests and address comments

* Bump pipeline_spec version

* Fix v1 tests
2021-08-24 01:04:39 -07:00
Ajay Gopinathan f3f383c2ff
chore(sdk): Refactor and move all v2 related code to under the v2 namespace. (#6358)
* Refactor and move all v2 related code to under the v2 namespace.

Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
  were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.

Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.

* Update setup.py.

* update tests.

* revert accidental change of gcpc

* Fix component entrypoint.

* Update goldens.

* fix tests.

* fix merge conflict.

* revert gcpc change.

* fix tests.

* fix tests.

* Add type aliases for moved files.

* merge and update goldens.
2021-08-17 19:25:37 -07:00
Chen Sun 7559e27cfb
feat(backend/sdk): Rename `pipeline-output-directory` to `pipeline-root`. Fixes #6307 (#6329) 2021-08-13 00:41:54 -07:00
Ajay Gopinathan f743dde32f
fix(sdk): Fix compiler bug for legacy outputs mlpipeline-ui-metadata and mlpipeline-metrics. Fixes #6311 (#6325)
* Fix compiler bug for legacy outputs mlpipeline-ui-metadata and
mlpipeline-metrics.

* fix - only delete legacy outputs in non-v2 pipelines.
2021-08-12 20:05:54 -07:00
Ajay Gopinathan 911562f8d3
fix(sdk): Relax the requirement that component inputs/outputs must appear on the command line. (#6268)
* Relax the requirement that component inputs/outputs must appear
on the command line.

This change enables component builders to define their implementation
(i.e. the container arg and command) independent of the actual
component inputs/outputs defined in component.yaml. The latter will be
used when determining inputs/outputs during compilation.

Only works for tasks that turn into ContainerOps during compilation.
Naturally, does not work for ContainerOps (which have no inputs/outputs
anyway).

* fix bugs, update goldens.

* Fix issue with inputs.

* merge and update.

* Restore compiler yamls.

* Disambiguate parameters and artifacts in component bridge.

* Restore goldens from HEAD.

* restore compiler_tests.py to HEAD.

* str-ify pipeline params.

* cr comments.
2021-08-10 14:40:25 -07:00
Chen Sun 903b98654e
feat(sdk.v2): Define BaseComponent, ComponentSpec, and PipelineTask. (#6210)
* Define BaseComponent, ComponentSpec, and PipelineTask.

* fix dep

* disable positional arguments

* address feedback

* address feedback
2021-08-06 08:54:43 -07:00
Chen Sun f4657ba42a
feat(sdk.v2): Add HTML and Markdown artifact types (#6246) 2021-08-06 02:24:42 -07:00
Yuan (Bob) Gong 2181b2fb8b
feat(sdk): add_default_resource_spec container op transformer (#6203)
* feat(sdk): add_default_resource_spec container op transformer

* set memory request/limit for sampOOMed samples

* refactor

* adjust tensorboard minio memory usage

* adjust resource usage

* clean up

* address feedbacks
2021-08-05 15:28:17 +08:00
Niklas Hansson e6becd71ff
feat(sdk): add GPU runtime resource request and fix spelling in runtime_resouce_request. Fixes #4877. Fixes #1252 (#5972)
* Add runtime resource request for GPUs

* clean up

* Updated docks and add check

* updated with test

* remove from branch

* run tests

* fix gpu vendor format

* Update after feedback

* add unit tet

* remove integration test

* clean up

* Clean up

* Updated to resource_constraints instead of resource
2021-08-01 22:52:38 -07:00
Chen Sun b200e1bc7d
fix(sdk): Fix URI placeholder in v2 compatible mode. (#6040)
* fix uri placeholder in v2 compatible mode

* fix tests

* fix path generation

* fix tests

* fix test

* cleanup

* clean up

* fix test

* fix test

* fix test
2021-07-21 08:32:50 -07:00
sina chavoshi fdab2e679c
feat(sdk): Add support for executor input in dsl_util.py (#6057)
* Add support for executor input

* chagne method name to _executor_input_placeholder
2021-07-16 18:43:13 -07:00
Yaqi Ji c6cb8acf7a
feat(sdk): Add interface for enable_caching at task level. (#6007)
* feat(sdk): Add pipeline level caching options

* Update golden

* fixing caching options

* fix tests

* add a caching disabled case

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-07-13 17:33:58 -07:00
Joe Liedtke ade34542e0
chore: Updates argoproj/argo URLs to argoproj/argo-workflows (#5969)
* Updt argoproj/argo URLs to argoproj/argo-workflows

* Update link to workflows.ts

* Update license.txt to reduce # of changed lines

* Revert changes to backend Dockerfile & license.txt

* Update license.txt, keep line endings
2021-07-06 21:52:20 -07:00
Yuan (Bob) Gong a95bfd66c2
feat(v2/dsl): support minio & s3 artifacts in v2 python component. Fixes #5838 (#5909)
* feat(v2/dsl): support minio & s3 artifacts in v2 python component

* fix minio output artifact key

* fix

* fix
2021-06-24 00:47:19 -07:00
Chen Sun 3f45f78a3d
fix(sdk): Fix default value of component input not picked up. Fixes #5880. (#5903)
* Fix default value of component input not picked up

* fix v2 compat
2021-06-23 13:25:18 -07:00
Yuan (Bob) Gong 2bee21aae9
fix(sdk/compiler): v2 compat - fix mlpipeline-ui-metadata artifact. Part of #5666 (#5832)
* fix(sdk/compiler): v2 compat - fix mlpipeline-ui-metadata artifact

* fix

* add test case

* address feedback, fix the bug in less hacky way
2021-06-10 17:14:00 -07:00
Niklas Hansson 5db843102a
feat(sdk): add runtime resource requests. Fixes #1956 (#5447)
* added resource request at runtime

* fixed things

* Update to use read only parameter insteadt

* added test case and better example

* Updated again

* add the validation

* add to the test suit

* work in progress

* update after feedback

* fix the test

* clean up

* clean up

* fix the path

* add the test again

* clean up

* fix tests

* feedback fix

* comment out and clean up
2021-06-10 16:27:59 -07:00
Chen Sun 2991602cfc
feat(sdk): Better error message when compiling for v2. Fixes #5688, Fixes #5727 (#5759)
* Better error message when compiling for v2

* update samples
2021-05-31 13:39:03 -07:00
Chen Sun f9f455ed96
fix(sdk.v2): Fix the bug where string concat inside a loop may break compilation. (#5752) 2021-05-28 13:54:01 -07:00
Chen Sun 58f74d3347
feat(sdk): special handling for type annotation using `typing.Optional` (#5716)
* handle type annotation with Optional

* remove typing._GenericAlias, which is available only after Py 3.7

* fix

* fix py3.6 test

* address review comments
2021-05-25 06:57:09 -07:00
Chen Sun b7084f2906
fix(sdk.v2): fix a small bug in `io_types.is_artifact_annotation()` (#5699)
* fix is_artifact_annotation

* move tests
2021-05-19 15:58:46 -07:00
Chen Sun 2abc352ad8
feat(sdk.v2): Support bool, dict, list, JsonObject, and JsonArray type as parameter types (#5665)
* Treat bool, dict, and list type as parameter types

* fix tests

* address review comments

* update test golden files

* address review comments
2021-05-18 12:41:41 -07:00