Commit Graph

981 Commits

Author SHA1 Message Date
Chen Sun a33c8e05cb
chore(sdk): Release 1.8.8 off `sdk/release-1.8` branch (#6853) 2021-11-02 17:02:06 -07:00
Yaqi Ji 334307d6c3
chore(sdk): add deprecation warnings for v2 (#6851) 2021-11-02 16:26:06 -07:00
Trs b2414f3eb8
fix(sdk): fix cloud scheduler's job name (#6844) 2021-11-02 13:46:06 -07:00
Toshiya Kawasaki 2f2ebf8962
fix(sdk): Fix invalid doc example of python_component (#6841) 2021-11-02 11:41:06 -07:00
Chen Sun 0f2cab9b39
fix(sdk.v2): Fix display name support for groups (#6832)
* fix display name for experimental tasks_group

* release note
2021-10-29 00:30:51 -07:00
Yaqi Ji f1bb852852
feat(sdk): add load_component_from_* (#6822)
* feat(sdk): add load_component_from_*

* address comments'
:

* update release notes
2021-10-28 22:30:34 -07:00
Chen Sun 8c6843fdb8
fix(sdk.v2): Fix importer ignoring `reimport` setting, and switch to Protobuf.Value for import uri. (#6827)
* fix importer

* release note

* remove error usage in test sample

* disable importer test
2021-10-28 20:40:51 -07:00
Chen Sun ea2e5be81a
feat(sdk.v2): Implement experimental `@component` component (#6825)
* Implement experimental `@component` component

* release note
2021-10-28 18:29:51 -07:00
Chen Sun 95c145be61
chore(sdk): Fix RELEASE.md (#6824) 2021-10-28 15:29:51 -07:00
Ajay Gopinathan 0be57c3880
feat(sdk): Use google.protobuf.Value in v2 for passing parameters. (#6804)
* Use google.protobuf.Value in v2 for passing parameters.

* retest samples.

* Fix tests.

* Update release, more cleanup.

* Use github.com/kubeflow/pipelines/api from same repo.

* Run go mod tidy

* chore: go mod tidy

* fix v2 compile error and clean up unused code

* pr comments.

* update goldens

* Fix metadata recording.

* Update kfp mlmd client.

* fix test again

* another try.

* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings

* fix v2 launcher + clean up

* fix a compile error

* fix a few more tests

* fix number parsing

* clean up

* disable cache_v2 test.

Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
2021-10-28 14:25:51 -07:00
Yaqi Ji 0c40154f6d
chore: Release KFP SDK and v2 launcher 1.8.7 (#6823)
* Update RELEASE.md

* Update __init__.py

* Update v2_compat.py
2021-10-28 10:53:26 -07:00
Omar Marzouk 3e6c776360
feat(sdk): Add load_component_from_spec. Fixes #5708 #3748 (#6690)
* add load_component_from_spec

* update release.md

* update RELEASE.md

* update RELEASE.md
2021-10-28 10:27:57 -07:00
dinimicky 41275b4228
fix(sdk): fixes the specified 'mlpipeline-ui-metadata','mlpipeline-metrics' path is overrided by default value (#6796)
* fix the issue:

the 'mlpipeline-ui-metadata','mlpipeline-metrics' path is replaced by the default path as '_components._generate_output_file_name(output.name)'

* reformat code

* Update _container_op.py

* Update RELEASE.md
2021-10-28 01:33:00 -07:00
Joshua Carp 37d3234ce5
feat(sdk): Add filters to python client. (#6748)
* Add filters to python client.

* Add filter docstrings.

h/t @chensun

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-10-28 01:15:50 -07:00
Joshua Carp ec4ab2dc4c
fix(sdk): Add missing retry policy. (#6808) 2021-10-27 23:41:30 -07:00
Chen Sun 7768f28de3
fix(sdk.v2): Fix missing experimental modules in setup.py (#6817)
* fix missing experimental modules

* fix typo

* fix test
2021-10-27 10:58:39 -07:00
Ajay Gopinathan fca4c1c2fd
Fix missing experimental type module in setup.py. (#6816) 2021-10-27 09:17:03 -07:00
Chen Sun 02b96f0a63
feat(sdk.v2): Implement v2 experimental compiler. (#6803)
* Implement v2 experimental compiler.

* add release note

* Support protobuf.Value in Condition operand
2021-10-26 14:12:37 -07:00
Chen Sun ec9a8ed6cd
Update RELEASE.md (#6797) 2021-10-24 21:42:27 -07:00
Gerard Casas Saez 5dfae1d8cc
feat(sdk): Add version description optionally to Py SDK. Part of #6256 (#6472)
* add description optionally

* add release notes

* add new line

* capture if description is not accepted by server

* add todo comment

* add better description api missing exception
2021-10-24 21:24:07 -07:00
Chen Sun 61fea6ba1f
fix(sdk.v2): Fix InputPathPlaceholder changed to InputValuePlaceholder (#6794)
* fix InputPathPlaceholder changed to InputValuePlaceholder

* release note
2021-10-24 17:10:07 -07:00
Chen Sun 63d000b0f7
chore(sdk): remove dupe content in RELEASE.md (#6774) 2021-10-19 15:21:14 -07:00
Chen Sun 0fd884783f
chore: Update SDK OWNERS (#6773) 2021-10-19 15:04:14 -07:00
Chen Sun 0197e0114e
Release KFP SDK and v2 launcher 1.8.6 (#6771) 2021-10-19 14:20:20 -07:00
johnmacnamararseg ba0bb60de9
feat(sdk): Extend SDK client with disable/delete run methods (#6754)
* add disbale/delete run (job) functions to sdk client

* change run to job to accuratly reflect api

* add release note for change

* align docstrings with style guide requirements

* formatting

Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-10-19 11:40:39 -07:00
Yaqi Ji 78dfac78ac
feat(sdk): Pipeline task settings (#6746)
* task

* caching options + cpu limit

* add container ops

* remove pipelineparam

* update release notes

* address comments
2021-10-19 11:39:05 -07:00
Chen Sun 7f1e34f17d
chore(sdk.v2): Implement LoopArgument and LoopArgumentVariable v2 (#6755)
* for_loop v2

* release note

* address review comments
2021-10-19 03:09:23 -07:00
Ajay Gopinathan 77de39de75
fix(sdk): Try alternatives for obtaining pip when `ensurepip` does not exist in container. (#6737)
* Try alternatives for obtaining pip when `ensurepip` does not
exist in container.

This PR also fixes the tests that have golden files. Developers can
update golden files by setting UPDATE_GOLDENS=True before running
tests. E.g., run the following from sdk/python directory:

UPDATE_GOLDENS=True python3 -m unittest discover --verbose --pattern '*test*.py' --top-level-directory .

* Update RELEASE.
2021-10-15 14:33:52 -07:00
Ajay Gopinathan b29c906f19
fix(sdk): Require base and target images in built containers to be unique. (#6731)
* Require base and target images in built containers to be unique.

Also add a test to verify, and fix formatting in cli/components.py.

* Update release notes.

* fix formatting.
2021-10-13 11:37:07 -07:00
Yaqi Ji 6153a470fc
chore: Release KFP SDK and v2 launcher 1.8.5 (#6722)
* Update RELEASE.md

* Update RELEASE.md

* Update __init__.py

* Update v2_compat.py
2021-10-12 15:01:44 -07:00
chongyouquan 14b3d6751a
fix(sdk): Fix type_utils (#6719)
* chore(kfp): Fix type_utils

* Update RELEASE.md
2021-10-12 11:17:44 -07:00
Ajay Gopinathan fe663772c8
feat(sdk): Enable containerizing v2 Python components (#6417)
* Refactor and move all v2 related code to under the v2 namespace.

Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
  were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.

Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.

* Update setup.py.

* update tests.

* revert accidental change of gcpc

* Fix component entrypoint.

* Update goldens.

* fix tests.

* fix merge conflict.

* revert gcpc change.

* fix tests.

* fix tests.

* Add type aliases for moved files.

* merge and update goldens.

* Add a CLI command to help containerize and build v2 components.

* Flesh out the CLI for Docker.

* update comments.

* Support multiple component files in the same directory.

* address pr comments.

* Let user specify just the directory containing components.

The CLI will search for components in all Python files by default. It
can also search for a specific filepattern (as supported by
pathlib.Path objects).

Also add a bunch of tests.

* Clean up.

* fix merge error.

* Fix docker import error for tests.

* Update release notes.

* address PR comments.
2021-10-12 09:33:44 -07:00
Chen Sun d3114638f1
chore(sdk.v2): Implement PipelineTask creation (#6713)
* Implement PipelineTask __init__()

* add release note
2021-10-11 13:17:43 -07:00
Sandeep Gupta bcb0d96861
feat(sdk): Update cloudpickle to 2.0.0 (fixes #6699) (#6703)
Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-10-11 11:43:43 -07:00
Chen Sun 073c819ebc
fix(sdk.v2): Fix the issue that using `dsl.` prefix in component I/O type annotation breaks component at runtime. (#6714)
* Import kfp and kfp.v2.dsl for component runtime

* update golden

* add release note

* update golden
2021-10-11 09:59:45 -07:00
juliusvonkohout 30826061fa
chore(sdk): relax click version to click>=7.1.2,<9. Fixes #6651 (#6691)
* Update setup.py

* Update requirements.txt

* Update setup.py

* Update requirements.in

* Update RELEASE.md

* Update RELEASE.md

* Update RELEASE.md
2021-10-08 14:37:43 -07:00
Yaqi Ji 5892bf97d0
fix(sdk): update v2 yaml format (#6661)
* fix(sdk): change names to input_name and output_name

* ordered dict and schema version numbering

* Remove schema_version, annotatations and labels

* change container/dagspec format

* change to bracket

* fix tests

* fix py3.6 forwardref issue

* fix ordered dict

* address comments

* release notes + test

* address comments
2021-10-07 16:48:11 -07:00
Chen Sun 2b2e5fdd69
fix(sdk.v2): fix function-based components not preserving the namespace of GCPC artifact types. (#6702)
* Support GCPC types with function-based component

* add release note
2021-10-07 15:35:22 -07:00
Yaqi Ji a2cdf740c5
feat(sdk): add v2 placeholder variables (#6693)
* feat(sdk): add placeholder variables

* add release notes

* address comments
2021-10-06 14:17:22 -07:00
Chen Sun 7b4f28379d
chore(sdk): relax typing-extensions version to `>=3.7.4,<4`. Fixes #6651 (#6683)
* relax typing-extension version

* add release note
2021-10-05 22:50:08 -07:00
Chen Sun 82ed66ece1
fix(sdk.v2): Fix executor getting None as value when float 0 is passed in. Fixes #6636 (#6682)
* fix getting None as value when float 0 is passed in

* add release note
2021-10-05 19:03:08 -07:00
Chen Sun 0aa513eb93
Release KFP SDK and v2 launcher 1.8.4 (#6666) 2021-10-04 14:06:15 -07:00
Chen Sun 5227b76901
chore(sdk): Support artifact types under google namespace (#6648)
* support artifact types under google namespace

* add release note
2021-09-30 15:30:31 -07:00
Yaqi Ji 96a2609ac7
feat(sdk): Add IfPresentPlaceholder and ConcatPlaceholder for dsl v2 (#6639)
* feat(sdk): Add IfPresentPlaceholder and ConcatPlaceholder for v2 ComponentSpec.

* update release notes

* Fix tests

* fix tests
2021-09-30 11:20:40 -07:00
Chen Sun b466f59e2f
fix(sdk.v2): Fix a couple of ParallelFor related bugs. Fixes #6383, fixes #6628 (#6643)
* fix a couple of loop related bugs

* add release note
2021-09-29 11:27:09 -07:00
Yaqi Ji e602793d5c
chore: Release KFP SDK and v2 launcher 1.8.3 (#6631)
* Update RELEASE.md

* Update RELEASE.md

* Update __init__.py

* Update v2_compat.py
2021-09-28 23:38:47 -07:00
Lin Yiming 4e89973504
fix(sdk): sanitize op name. Fix #6433 (#6600)
* fix(sdk): sanitize op name. Fix #6433

* Add release note: Fix the error that kfp v1 compiler failed to provide unique name for ops of the same component.
2021-09-28 00:10:10 -07:00
Jack Whelpton daccc5a4b6
feat(sdk): ComponentStore - URI template support. Fixes #4708 (#5434)
* Adds URI template support for ComponentStore

* Reverts some autoformatting

* Reverts some more autoformatting

* Addresses code review comments:

* order of constructor args preserved for compatibility
* added docstrings
* added explicit uritemplate dependency

* Adds description of change

* Adds missing #
2021-09-27 18:14:10 -07:00
Nabil Abdellaoui 95162e2372
chore(sdk): Remove stack trace so it doesn't look like an error (#6590)
* Stack trace makes it look like an error

* Update RELEASE.md
2021-09-24 10:20:37 -07:00
Lin Yiming 36e627d28f
feat(sdk): Local runner supports additional docker options. Fixes #6598 (#6599)
* bump kfp-pipeline-spec to 0.1.11, tests for local runner will fail with kfp-pipeline-spec 0.1.9

* Local client supports additional docker options

* Format _local_client.py using yapf

* Add release note that local runner supports additional docker options
2021-09-24 07:58:37 -07:00