Commit Graph

895 Commits

Author SHA1 Message Date
Chen Sun d1f90b34d6
chore: Release KFP SDK and v2 launcher 1.8.0 (#6505) 2021-09-03 16:29:10 -07:00
Chen Sun 2f19a26ffd
chore(sdk): Format all Python files under SDK folder. (#6501)
* Reformat sdk only using the new yapf config.

* Reformat docstrings using docformatter.

* update golden files to resolve diff caused by whitespaces

* fix some tests

* format .py files under sdk/python/tests using yapf

* additional docformatter

* fix some tests
2021-09-03 11:25:11 -07:00
Yaqi Ji 8100c5e02c
feat(sdk): load v1 and v2 component spec (#6497)
* feat(sdk): load v1 and v2 component spec

* Update RELEASE.md
2021-09-02 12:40:18 -07:00
Yaqi Ji 5a8298ec2e
chore(sdk): fix typing issues. (#6480) 2021-09-01 16:17:37 -07:00
Joshua Carp f43ecc223d
chore(sdk): Import mock from stdlib and drop dependency. (#6456)
* Import mock from stdlib and drop dependency.

* Drop mock from requirements.

h/t @chensun
2021-08-31 16:38:44 -07:00
Chen Sun cfefc6d397
feat(sdk.v2): Support `set_display_name` in v2. (#6471)
* Support `set_display_name` in v2.

* update comment in proto

* fix test
2021-08-30 12:07:25 -07:00
Chen Sun 63913bec25
chore(sdk): Add Alpha feature notice for local client (#6462)
* alpha notice for local client

* expose `pipeline_root` parameter from the wrapper function.
2021-08-30 10:55:25 -07:00
Jaeyeon Kim 728915f4e4
feat(sdk): detail option to kfp run get. Fixes #6315 (#6404)
* [CLI]: add detail option to kfp run get command

- to get more detailed information

Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>

* merge print_run_in_detail with _display_run

Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
2021-08-30 04:21:24 -07:00
Yuan (Bob) Gong 9afe125a75
revert: "feat(sdk): Add description to upload_pipeline_version in kfp. Part of #6256" (#6468)
This reverts commit 8b51cc4f22.
2021-08-30 00:42:24 -07:00
Yaqi Ji 8f10bcdf16
chore: Release KFP SDK and v2 launcher 1.7.2 (#6442)
* Update RELEASE.md

* Update __init__.py

* Update v2_compat.py

* Update RELEASE.md
2021-08-25 19:58:04 -07:00
Chen Sun 96bb17531f
test(sdk): ignore kfp version in v2_compatible tests (#6444) 2021-08-25 17:06:05 -07:00
Chen Sun 06044dd89e
fix(sdk.v2): fix importer not taking output from upstream (#6439) 2021-08-25 15:23:50 -07:00
Yaqi Ji 65174cdd93
feat(sdk): Add BaseModel to component_spec data classes (#6372)
* pydantic setup

* switch to union types

* Fix base_component tests

* switch to pydantic for python3.6 compat and fix tests

* Fix tests

* use str for type

* fix tests

* Consistent quote sign

* Address comments

* Update type
2021-08-25 14:00:02 -07:00
Yaqi Ji 937cacd4ce
feat(sdk): add default schema_version to pipeline (#6366)
* feat(sdk): add default schema_version to pipeline

* sync api for go

* Fix tests and address comments

* Bump pipeline_spec version

* Fix v1 tests

* rebase to master

* sync api for go

* Fix tests and address comments

* Bump pipeline_spec version

* Fix v1 tests
2021-08-24 01:04:39 -07:00
Gerard Casas Saez 8b51cc4f22
feat(sdk): Add description to upload_pipeline_version in kfp. Part of #6256 (#6395) 2021-08-21 23:45:56 -07:00
Chen Sun f7c9602ca7
chore(sdk): clean up the unused arg in AIPlatformClient docstring. (#6406) 2021-08-20 18:54:55 -07:00
Toshiya Kawasaki d40985c4f2
feat(google client): Enable to specify the service account to proxy function of scheduler in google client sdk (#6013)
* google sdk: Enable to specify the service account of proxy function and cloud scheduler job

* rename service_account_for_schedule to cloud_scheduler_service_account
2021-08-17 23:26:37 -07:00
Ajay Gopinathan f3f383c2ff
chore(sdk): Refactor and move all v2 related code to under the v2 namespace. (#6358)
* Refactor and move all v2 related code to under the v2 namespace.

Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
  were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.

Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.

* Update setup.py.

* update tests.

* revert accidental change of gcpc

* Fix component entrypoint.

* Update goldens.

* fix tests.

* fix merge conflict.

* revert gcpc change.

* fix tests.

* fix tests.

* Add type aliases for moved files.

* merge and update goldens.
2021-08-17 19:25:37 -07:00
Chen Sun 7f6e11dedc
chore: Release KFP SDK and v2 launcher 1.7.1 (#6369) 2021-08-17 16:38:37 -07:00
Chen Sun 580410b04e
test(sdk): ignore kfp version in unit tests (#6371) 2021-08-17 15:51:39 -07:00
Gerard Casas Saez eb41fdd33f
feat(sdk): Surface kubernetes configuration in container builder (#6095)
* surface kubernetes configuration in container builder

* rename to k8s_client_configuration
2021-08-17 00:21:41 -07:00
Ajay Gopinathan 609e5b784f
fix(sdk): Ensure pip exists for lightweight Python-based components. (#6359)
* Ensure pip exists for lightweight Python-based components.

* Update goldens.
2021-08-16 23:40:42 -07:00
Chen Sun 434e5c3489
fix(sdk): block dsl.importer usage in KFP OSS. Fixes: #6323 (#6330)
* block dsl.importer in KFP OSS

* address cr comments
2021-08-16 01:02:06 -07:00
Chen Sun 50f7dcd5d0
chore(sdk): Warn on container component without command. Fixes #4800 (#6335)
* Warn on container component without command

* address cr comments
2021-08-14 01:14:04 -07:00
Chen Sun b0e8c7eb37
chore(sdk): Export all artifact types from io_types in `dsl` package. (#6343) 2021-08-13 15:48:04 -07:00
Yuan (Bob) Gong befd89c6ec
fix(sdk): pipeline name is required in v2 compatible mode. Fixes #6197 (#6332)
* fix(sdk): pipeline name is required in v2 compatible mode

* Update xgboost_sample.py
2021-08-13 14:00:04 -07:00
Chen Sun 7559e27cfb
feat(backend/sdk): Rename `pipeline-output-directory` to `pipeline-root`. Fixes #6307 (#6329) 2021-08-13 00:41:54 -07:00
Ajay Gopinathan f743dde32f
fix(sdk): Fix compiler bug for legacy outputs mlpipeline-ui-metadata and mlpipeline-metrics. Fixes #6311 (#6325)
* Fix compiler bug for legacy outputs mlpipeline-ui-metadata and
mlpipeline-metrics.

* fix - only delete legacy outputs in non-v2 pipelines.
2021-08-12 20:05:54 -07:00
Jiyong Jung a1f7160cf5
chore: upgrade tfx to 1.0.0 (#6316)
* test: fix sample test image build

* update to TFX 1.0.0

* update argo CLI

* Uses TFX 1.0 APIs in parameterized_tfx_oss example.

Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
2021-08-12 04:02:03 -07:00
Ajay Gopinathan 911562f8d3
fix(sdk): Relax the requirement that component inputs/outputs must appear on the command line. (#6268)
* Relax the requirement that component inputs/outputs must appear
on the command line.

This change enables component builders to define their implementation
(i.e. the container arg and command) independent of the actual
component inputs/outputs defined in component.yaml. The latter will be
used when determining inputs/outputs during compilation.

Only works for tasks that turn into ContainerOps during compilation.
Naturally, does not work for ContainerOps (which have no inputs/outputs
anyway).

* fix bugs, update goldens.

* Fix issue with inputs.

* merge and update.

* Restore compiler yamls.

* Disambiguate parameters and artifacts in component bridge.

* Restore goldens from HEAD.

* restore compiler_tests.py to HEAD.

* str-ify pipeline params.

* cr comments.
2021-08-10 14:40:25 -07:00
Yaqi Ji 0ac67d0c05
test(sdk): Ignore kfp sdk version when comparing golden (#6281) 2021-08-09 19:13:25 -07:00
Chen Sun d48792c373
test(sdk): restore a v2-compatible unit test (#6263) 2021-08-09 11:18:23 -07:00
Yaqi Ji 3044989dbf
chore: release KFP SDK and v2 launcher 1.7.0 (#6257)
* Update __init__.py

* Update v2_compat.py

* Update goldens
2021-08-06 17:18:25 -07:00
Chen Sun 903b98654e
feat(sdk.v2): Define BaseComponent, ComponentSpec, and PipelineTask. (#6210)
* Define BaseComponent, ComponentSpec, and PipelineTask.

* fix dep

* disable positional arguments

* address feedback

* address feedback
2021-08-06 08:54:43 -07:00
Chen Sun f4657ba42a
feat(sdk.v2): Add HTML and Markdown artifact types (#6246) 2021-08-06 02:24:42 -07:00
Ajay Gopinathan c5591f2652
feat(sdk): Enable the use of Executor from KFP package for v2 lightweight components (#6202)
* Enable the use of v2 Executor from KFP package.

This avoids the need to paste Executor code for v2 components into
the entrypoint of lightweight components.

* Update tests.

* quote package path.

* quote package path.

* escape fragment

* Update golden files.

* update goldens.

* Cleanup.

* More cleanup.

* update goldens.

* style updates.

* clean up sample tests.

* fix trailing comma

* Try using signed urls.

* fix

* fix presubmit

* try /bin/bash for Makefile shell

* undo signed url change.

Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
2021-08-05 14:53:03 -07:00
Yuan (Bob) Gong b1c76e0cd3
test(v2): build go images using ko. Fixes #6238 (#6240)
* test(v2): build go images using ko

* clean up

* integrate launcher license checking

* clean up

* fix go v2 test
2021-08-05 06:33:18 -07:00
Yuan (Bob) Gong 2181b2fb8b
feat(sdk): add_default_resource_spec container op transformer (#6203)
* feat(sdk): add_default_resource_spec container op transformer

* set memory request/limit for sampOOMed samples

* refactor

* adjust tensorboard minio memory usage

* adjust resource usage

* clean up

* address feedbacks
2021-08-05 15:28:17 +08:00
Toshiya Kawasaki f116e2cc85
fix(sdk): google client, Enable to provide app engine region that scheduled job is created in (#6145) 2021-08-03 11:03:12 -07:00
Gerard Casas Saez 3ec8b82fa4
feat(sdk): Allow parametrization of ui_host in kfp.Client (#6124)
* Update _client.py

* add docs
2021-08-02 14:02:10 -07:00
Niklas Hansson e6becd71ff
feat(sdk): add GPU runtime resource request and fix spelling in runtime_resouce_request. Fixes #4877. Fixes #1252 (#5972)
* Add runtime resource request for GPUs

* clean up

* Updated docks and add check

* updated with test

* remove from branch

* run tests

* fix gpu vendor format

* Update after feedback

* add unit tet

* remove integration test

* clean up

* Clean up

* Updated to resource_constraints instead of resource
2021-08-01 22:52:38 -07:00
Niklas Hansson ed2ba1a6b3
chore(sdk): Argo lint requires '--kinds=workflows' argument for offline linting. (#6135) 2021-07-26 00:40:41 -07:00
Chen Sun f4c6631e51
chore: release KFP SDK and v2 launcher 1.6.6 (#6125)
* release 1.6.6

* skip failing UT
2021-07-23 14:49:39 -07:00
Chen Sun 27051ab9d1
feat(sdk.v2): add `enable_caching` option in `create_schedule_from_job_spec` (#6119) 2021-07-22 23:00:07 -07:00
Yaqi Ji 1ad2015d90
feat(sdk): add deprecation warning for AiPlatformClient (#6106)
* feat(sdk): add deprecation warning for AiPlatformClient

* Update warning to provide more context:
2021-07-22 13:10:07 -07:00
Shuhei Fujiwara 9ebbe7b35f
fix(sdk): Fix type hints for create_run_from_job_spec. Fixes #6092 (#6098) 2021-07-22 00:21:25 -07:00
Toshiya Kawasaki 78067d1dd7
fix(sdk): google client: set ai platform endpoint by region (#6011) 2021-07-21 16:29:24 -07:00
Chen Sun b200e1bc7d
fix(sdk): Fix URI placeholder in v2 compatible mode. (#6040)
* fix uri placeholder in v2 compatible mode

* fix tests

* fix path generation

* fix tests

* fix test

* cleanup

* clean up

* fix test

* fix test

* fix test
2021-07-21 08:32:50 -07:00
Niklas Hansson 354e4251e8
chore: add offline linting (#5981)
* Added the offline linting

* update after feedback
2021-07-20 08:39:15 -07:00
Yuan (Bob) Gong ee663d9593
chore(v2): standardize MLMD data model. Fixes #5669 (#6054)
* chore(v2): standardize MLMD data model

* change context type to system namespace

* update sdk snapshots

* fix go v2 tests

* update

* update v2 compat snapshots

* fix all samples

* fix must specify pipeline root

* add artifact display name

* add UI rendering of new fields

* fix sample tests

* let ui read artifact and execution names consistently

* fix samples

* fix frontend tests

* fix sample test

* fix last sample

* address feedback
2021-07-19 22:26:15 -07:00