5227b76901
chore(sdk): Support artifact types under google namespace ( #6648 )
...
* support artifact types under google namespace
* add release note
2021-09-30 15:30:31 -07:00
96a2609ac7
feat(sdk): Add IfPresentPlaceholder and ConcatPlaceholder for dsl v2 ( #6639 )
...
* feat(sdk): Add IfPresentPlaceholder and ConcatPlaceholder for v2 ComponentSpec.
* update release notes
* Fix tests
* fix tests
2021-09-30 11:20:40 -07:00
b466f59e2f
fix(sdk.v2): Fix a couple of ParallelFor related bugs. Fixes #6383 , fixes #6628 ( #6643 )
...
* fix a couple of loop related bugs
* add release note
2021-09-29 11:27:09 -07:00
ed3453f794
chore: update comments in a pipeline sample ( #6601 )
...
* Update lightweight_python_functions_v2_pipeline.py
* Update lightweight_python_functions_v2_pipeline.py
* Update lightweight_python_functions_v2_pipeline.py
* Update mock_lightweight_python_functions_v2_pipeline.json
* Update lightweight_python_functions_v2_pipeline.json
* Update mock_lightweight_python_functions_v2_pipeline.json
2021-09-23 11:29:36 -07:00
c605c51b85
chore(sdk.v2): use `Annotated` rather than `Union` hack ( #6573 )
...
* Use `Annotated` rather than `Union` hack
* Add dependency to `setup.py`
* Add release note
Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-09-16 10:54:55 -07:00
866dfc7672
fix(sdk.v2): fixes broken output parameter type checking in `_handle_single_return_value` ( #6566 )
...
* Add tests for output parameter `list` and `dict`
* Parse parameter annotation type correctly
* Add release note
2021-09-16 10:32:55 -07:00
a52ac6d2c5
fix(sdk.v2): Component is invalid if `packages_to_install` empty and `install_kfp_package=False` ( #6527 )
...
If an empty list is provided no command should be returned to avoid a
broken command eg. `pip install `
2021-09-15 17:03:54 -07:00
5d82fa7dc8
chore(sdk): v2 compiler to throw no task defined error ( #6545 )
...
* chore(sdk): v2 compiler to throw no task defined error
* update and clean up release notes
* Update RELEASE.md
* Update RELEASE.md
2021-09-14 01:13:32 -07:00
7626abf60d
fix(sdk.v2): fix `Optional` type hint causing executor to ignore user inputs for parameters. ( #6541 )
...
* fix `Optional` type hint causing issue in executor.
* update release note
* add a pipeline test case
2021-09-10 12:16:58 -07:00
2176feeb69
feat(sdk.v2): Define PipelineParameterChannel and PipelineArtifactChannel. ( #6470 )
...
* pipeline param v2
* address cr comments
* rename to channel
* address cr comments
* update release note
2021-09-10 09:44:52 -07:00
a0b18eb9e8
fix(sdk): Get short name of complex input/output types to ensure we can map to appropriate de|serializer ( #6504 )
...
Also:
- Simplify _data_passing methods, add in type hints and docstrings.
- Remove get_deserializer_code_for_type.
2021-09-09 17:31:51 -07:00
38e826bf80
fix(sdk.v2): Fix passing in "" to a `str` parameter causes the parameter to receive it as None instead ( #6533 )
...
* fix empty string not passed through
* update release note
2021-09-08 18:13:00 -07:00
0fba85cfab
fix(sdk.v2): Support dict, list, bool typed input parameters from constant values and pipeline inputs. ( #6523 )
...
* fix dict list typed inputs support
* Support passing dict, list, bool typed parameter via client
* update release note
* fix test
2021-09-07 19:53:07 -07:00
29226e9178
feat(sdk.v2): Support container environment variable in v2. ( #6515 )
...
* Support `set_env_variable` in v2.
* update release note
2021-09-07 13:30:07 -07:00
2f19a26ffd
chore(sdk): Format all Python files under SDK folder. ( #6501 )
...
* Reformat sdk only using the new yapf config.
* Reformat docstrings using docformatter.
* update golden files to resolve diff caused by whitespaces
* fix some tests
* format .py files under sdk/python/tests using yapf
* additional docformatter
* fix some tests
2021-09-03 11:25:11 -07:00
8100c5e02c
feat(sdk): load v1 and v2 component spec ( #6497 )
...
* feat(sdk): load v1 and v2 component spec
* Update RELEASE.md
2021-09-02 12:40:18 -07:00
5a8298ec2e
chore(sdk): fix typing issues. ( #6480 )
2021-09-01 16:17:37 -07:00
f43ecc223d
chore(sdk): Import mock from stdlib and drop dependency. ( #6456 )
...
* Import mock from stdlib and drop dependency.
* Drop mock from requirements.
h/t @chensun
2021-08-31 16:38:44 -07:00
cfefc6d397
feat(sdk.v2): Support `set_display_name` in v2. ( #6471 )
...
* Support `set_display_name` in v2.
* update comment in proto
* fix test
2021-08-30 12:07:25 -07:00
06044dd89e
fix(sdk.v2): fix importer not taking output from upstream ( #6439 )
2021-08-25 15:23:50 -07:00
65174cdd93
feat(sdk): Add BaseModel to component_spec data classes ( #6372 )
...
* pydantic setup
* switch to union types
* Fix base_component tests
* switch to pydantic for python3.6 compat and fix tests
* Fix tests
* use str for type
* fix tests
* Consistent quote sign
* Address comments
* Update type
2021-08-25 14:00:02 -07:00
937cacd4ce
feat(sdk): add default schema_version to pipeline ( #6366 )
...
* feat(sdk): add default schema_version to pipeline
* sync api for go
* Fix tests and address comments
* Bump pipeline_spec version
* Fix v1 tests
* rebase to master
* sync api for go
* Fix tests and address comments
* Bump pipeline_spec version
* Fix v1 tests
2021-08-24 01:04:39 -07:00
f7c9602ca7
chore(sdk): clean up the unused arg in AIPlatformClient docstring. ( #6406 )
2021-08-20 18:54:55 -07:00
d40985c4f2
feat(google client): Enable to specify the service account to proxy function of scheduler in google client sdk ( #6013 )
...
* google sdk: Enable to specify the service account of proxy function and cloud scheduler job
* rename service_account_for_schedule to cloud_scheduler_service_account
2021-08-17 23:26:37 -07:00
f3f383c2ff
chore(sdk): Refactor and move all v2 related code to under the v2 namespace. ( #6358 )
...
* Refactor and move all v2 related code to under the v2 namespace.
Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.
Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.
* Update setup.py.
* update tests.
* revert accidental change of gcpc
* Fix component entrypoint.
* Update goldens.
* fix tests.
* fix merge conflict.
* revert gcpc change.
* fix tests.
* fix tests.
* Add type aliases for moved files.
* merge and update goldens.
2021-08-17 19:25:37 -07:00
580410b04e
test(sdk): ignore kfp version in unit tests ( #6371 )
2021-08-17 15:51:39 -07:00
609e5b784f
fix(sdk): Ensure pip exists for lightweight Python-based components. ( #6359 )
...
* Ensure pip exists for lightweight Python-based components.
* Update goldens.
2021-08-16 23:40:42 -07:00
b0e8c7eb37
chore(sdk): Export all artifact types from io_types in `dsl` package. ( #6343 )
2021-08-13 15:48:04 -07:00
911562f8d3
fix(sdk): Relax the requirement that component inputs/outputs must appear on the command line. ( #6268 )
...
* Relax the requirement that component inputs/outputs must appear
on the command line.
This change enables component builders to define their implementation
(i.e. the container arg and command) independent of the actual
component inputs/outputs defined in component.yaml. The latter will be
used when determining inputs/outputs during compilation.
Only works for tasks that turn into ContainerOps during compilation.
Naturally, does not work for ContainerOps (which have no inputs/outputs
anyway).
* fix bugs, update goldens.
* Fix issue with inputs.
* merge and update.
* Restore compiler yamls.
* Disambiguate parameters and artifacts in component bridge.
* Restore goldens from HEAD.
* restore compiler_tests.py to HEAD.
* str-ify pipeline params.
* cr comments.
2021-08-10 14:40:25 -07:00
0ac67d0c05
test(sdk): Ignore kfp sdk version when comparing golden ( #6281 )
2021-08-09 19:13:25 -07:00
3044989dbf
chore: release KFP SDK and v2 launcher 1.7.0 ( #6257 )
...
* Update __init__.py
* Update v2_compat.py
* Update goldens
2021-08-06 17:18:25 -07:00
903b98654e
feat(sdk.v2): Define BaseComponent, ComponentSpec, and PipelineTask. ( #6210 )
...
* Define BaseComponent, ComponentSpec, and PipelineTask.
* fix dep
* disable positional arguments
* address feedback
* address feedback
2021-08-06 08:54:43 -07:00
f4657ba42a
feat(sdk.v2): Add HTML and Markdown artifact types ( #6246 )
2021-08-06 02:24:42 -07:00
c5591f2652
feat(sdk): Enable the use of Executor from KFP package for v2 lightweight components ( #6202 )
...
* Enable the use of v2 Executor from KFP package.
This avoids the need to paste Executor code for v2 components into
the entrypoint of lightweight components.
* Update tests.
* quote package path.
* quote package path.
* escape fragment
* Update golden files.
* update goldens.
* Cleanup.
* More cleanup.
* update goldens.
* style updates.
* clean up sample tests.
* fix trailing comma
* Try using signed urls.
* fix
* fix presubmit
* try /bin/bash for Makefile shell
* undo signed url change.
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
2021-08-05 14:53:03 -07:00
f116e2cc85
fix(sdk): google client, Enable to provide app engine region that scheduled job is created in ( #6145 )
2021-08-03 11:03:12 -07:00
27051ab9d1
feat(sdk.v2): add `enable_caching` option in `create_schedule_from_job_spec` ( #6119 )
2021-07-22 23:00:07 -07:00
1ad2015d90
feat(sdk): add deprecation warning for AiPlatformClient ( #6106 )
...
* feat(sdk): add deprecation warning for AiPlatformClient
* Update warning to provide more context:
2021-07-22 13:10:07 -07:00
9ebbe7b35f
fix(sdk): Fix type hints for create_run_from_job_spec. Fixes #6092 ( #6098 )
2021-07-22 00:21:25 -07:00
78067d1dd7
fix(sdk): google client: set ai platform endpoint by region ( #6011 )
2021-07-21 16:29:24 -07:00
157dccbf4e
chore: Fixed typos and removed not referred private functions in google client SDK ( #6012 )
2021-07-19 18:34:15 -07:00
24c551d3fc
feat(sdk.v2): enable_caching in v2 client defaults to compile time settings. ( #6052 )
2021-07-15 14:58:37 -07:00
c6cb8acf7a
feat(sdk): Add interface for enable_caching at task level. ( #6007 )
...
* feat(sdk): Add pipeline level caching options
* Update golden
* fixing caching options
* fix tests
* add a caching disabled case
Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
2021-07-13 17:33:58 -07:00
9a6b0e8852
feat(v2,sdk): Add service_account as args to create_schedule_from_job_spec ( #5994 )
2021-07-08 14:43:22 -07:00
a95bfd66c2
feat(v2/dsl): support minio & s3 artifacts in v2 python component. Fixes #5838 ( #5909 )
...
* feat(v2/dsl): support minio & s3 artifacts in v2 python component
* fix minio output artifact key
* fix
* fix
2021-06-24 00:47:19 -07:00
3f45f78a3d
fix(sdk): Fix default value of component input not picked up. Fixes #5880 . ( #5903 )
...
* Fix default value of component input not picked up
* fix v2 compat
2021-06-23 13:25:18 -07:00
a1740b4dfa
fix(sdk compiler):Fix v2 compile for exit handler Fixes #5854 ( #5899 )
...
* punch the hole to exit handler
* add pipeline parameter to test data
2021-06-22 02:21:54 -07:00
0fc88c3748
chore: update deprecated download_as_string to download_as_bytes ( #5822 )
...
* Update deprecated download_as_string to download_as_bytes
* Add more usage
2021-06-09 15:15:51 -07:00
6b87155a33
feat(sdk.v2): Support Exit handler in v2 compiler. ( #5784 )
2021-06-03 12:58:36 -07:00
ee77fd86ec
fix(sdk.v2): fix metrics outputs populating to unblock pipeline run ( #5770 )
2021-06-02 13:18:35 -07:00
5f34603e60
fix(sdk, v2): Fix error when replica_count isn't set ( #5761 )
2021-06-01 01:48:04 -07:00
2991602cfc
feat(sdk): Better error message when compiling for v2. Fixes #5688 , Fixes #5727 ( #5759 )
...
* Better error message when compiling for v2
* update samples
2021-05-31 13:39:03 -07:00
f9f455ed96
fix(sdk.v2): Fix the bug where string concat inside a loop may break compilation. ( #5752 )
2021-05-28 13:54:01 -07:00
b7084f2906
fix(sdk.v2): fix a small bug in `io_types.is_artifact_annotation()` ( #5699 )
...
* fix is_artifact_annotation
* move tests
2021-05-19 15:58:46 -07:00
bbce172335
release(sdk): release SDK 1.6.1 with minor fix in AIPlatformClient ( #5679 )
2021-05-18 13:55:14 -07:00
bf8681b7e3
feat(sdk.v2): Add Google AIPlatformClient ( #5677 )
...
* ai platform client
* fix test
* update URL
2021-05-18 13:33:34 -07:00
2abc352ad8
feat(sdk.v2): Support bool, dict, list, JsonObject, and JsonArray type as parameter types ( #5665 )
...
* Treat bool, dict, and list type as parameter types
* fix tests
* address review comments
* update test golden files
* address review comments
2021-05-18 12:41:41 -07:00
559fe40bc9
feat(sdk): Type checking to treat "explicit" generic artifact as any artifact type. ( #5661 )
...
* allow using explicit generic artifact as any artifact type.
* address review comment
2021-05-17 20:56:40 -07:00
d892dc01a6
remove task prefix ( #5654 )
2021-05-17 13:00:04 -07:00
f20d20c41d
feat(sdk.v2): Experimental custom job support. ( #5604 )
...
* custom job
* simple vs. advanced setting (user specified raw workerpoolspecs)
* fix typo
2021-05-11 18:33:25 -07:00
00f3b3d6bf
update xgboost sample pipeline ( #5594 )
2021-05-07 22:45:20 -07:00
3324459f67
feat(sdk.v2): support using placeholder class in custom job spec. ( #5495 )
2021-05-05 18:12:05 -07:00
cc83e1089b
Assigned copyright to the project authors ( #5587 )
2021-05-05 13:53:22 +08:00
e4c6e12564
fix(sdk): Fix a bug with creating dirname for parameter files in v2 lightweight components ( #5574 )
...
* Fix a bug with creating dirname for parameter files.
* Add test as well.
2021-04-30 10:15:31 -07:00
b46f1b86f1
fix(sdk): Update syntax for lightweight components v2. ( #5568 )
...
* Update syntax for lightweight components v2.
Use Input[T] and Output[T] for representing input/output artifacts
of type T. This enables users to see these annotations as merely the
classes T, enabling autocompletion, reducing verbosity and simplifying
I/O types code. Users also no longer
Other changes include:
- no longer need to use .get() to obtain the artifact class
- no auto-creation of the `data` file for output artifacts. This enables
artifacts to be either directories or files.
* fix for Py 3.6
* Remove stray file.
* Add error message if Input/Output are used with non Artifact types.
* Fix e2e test.
2021-04-29 19:27:30 -07:00
d5a1853857
feat(sdk.v2): Raise error on artifacts vs. parameters mismatch in v2. ( #5557 )
...
* error out on passing artifact to parameter or vice versa.
* error out on missing type annotation for pipeline arguments.
* address test failures
2021-04-29 03:17:29 -07:00
5c08cf91e2
chore(sdk.v2): block recursive subgroup in v2. ( #5551 )
2021-04-28 18:51:29 -07:00
2909f259c4
fix(sdk): Fix bug where empty string was not preserved as pipeline param default value. ( #5549 )
2021-04-27 09:20:13 -07:00
3988706b7d
chore(sdk.v2): Sort compilation output json to make it deterministic. ( #5538 )
2021-04-24 19:50:44 -07:00
a2d3f234ea
chore(sdk.v2): Remove warning on missing pipeline root at compile time ( #5527 )
2021-04-22 16:48:43 -07:00
74950d620f
fix(sdk.v2): Fix loop subvar support and remove redundant loop iterator component. ( #5516 )
...
* fix loop subvar
* remove loop intermediate component
* fix subvar
* improve condition operand type handling
2021-04-22 15:13:42 -07:00
5dafda1270
feat(sdk.v2): Support explicit importer ( #5502 )
2021-04-19 13:14:09 -07:00
5f97a4502d
feat(sdk.v2): remove pipeline_root from compiler interface ( #5492 )
2021-04-16 08:49:13 -07:00
55c14ccb2d
fix(sdk): Minor API fixes to Metrics classes. ( #5494 )
...
* Minor updates to Metrics classes.
* update goldens.
* fix tests.
2021-04-15 22:31:13 -07:00
1a357a0ebe
chore(sdk.v2): Deprecate dsl.ontology_artifacts, and migrate to using dsl.io_types ( #5472 )
...
* merge ontology_artifacts to io_types
* remove tests importing ontology_artifacts
* fix tests
2021-04-15 21:38:13 -07:00
2c68229920
Fixes the Confusion Matrix storage format ( #5463 )
...
This change fixes the Confusion Matrix storage format to strictly adhere
with the schema type checking if supported.
2021-04-13 11:23:11 -07:00
85cb99173d
feat(sdk.v2): fix loop implementation and add subvar support ( #5461 )
...
* fix loop dag and add subvar support
* Bump kfp-pipeline-spec min version to 0.1.7
2021-04-12 16:59:03 -07:00
f32915e88f
feat(sdk.v2): Support setting custom job spec in v2. ( #5454 )
...
* Support setting custom job spec.
* auto fill custom job jobSpec using container spec.
* fix test
* address comments
2021-04-10 09:20:04 -07:00
a80421191d
feat(sdk.v2): surface metrics output artifacts to pipeline outputs ( #5445 )
...
* populates metrics outputs in root
* avoid kfp import in io_types.py
* populates outputs following the right path
2021-04-09 16:37:04 -07:00
0e921c3083
feat(sdk.v2): use consistent argument name in v2 compiler ( #5430 )
2021-04-06 09:05:01 -07:00
1e32f01dc4
fix(sdk): Always create output paths for parameters in addition to artifacts in v2 Python components. ( #5421 )
...
* Always create output paths for parameters in addition to artifacts in v2
lightweight function components.
Also mirror useful dsl constructs under kfp.v2.dsl to streamline imports
for end users.
* Clean up
2021-04-02 21:56:20 -07:00
eb558423ec
fix(sdk.v2): fix bug for component input parameter/artifact referencing ( #5419 )
2021-04-01 22:10:20 -07:00
a6a725560c
fix(sdk): Support returning plain tuples in lightweight function components v2. ( #5412 )
2021-04-01 13:49:20 -07:00
eb3a092b64
feat(sdk.v2): Support optional component inputs in v2. ( #5383 )
...
* support optional inputs
* fix typo
* update chicago taxi dataset component
* fix executor
2021-03-31 01:16:18 -07:00
0e7e543694
Adds classification metrics type to io_types ( #5387 )
...
Change to add support for Classification Metrics to io_types.
Also udpates IO types type_name from kfp. to system.
2021-03-30 20:46:18 -07:00
eabc10f364
chore(sdk.v2): fix runtime config wrong parameter type. ( #5389 )
2021-03-30 15:10:17 -07:00
539cbebbe3
feat(sdk): Add a component decorator for v2 components. ( #5388 )
...
* Add a component decorator for v2 components.
* Reimport missing pieces.
* reimport dsl_utils
* fix imports.
* Fix filename.
2021-03-30 14:26:17 -07:00
2fc1016cd2
feat(sdk): Support return values in lightweight Python function components v2. ( #5384 )
...
* Support return values in lightweight Python function components v2.
* Update for py-39
* fix goldens.
* address PR comments.
2021-03-30 00:21:17 -07:00
4c51b576f5
chore(sdk.v2): fix a couple of issues related to cross-DAG i/o referencing and more. ( #5382 )
...
* fix a couple of issues related to cross-DAG i/o
* address review comments
2021-03-29 19:12:17 -07:00
bf8392d914
feat(sdk): Enable InputArtifact/OutputArtifact in lightweight Python-based components ( #5378 )
...
* Enable InputArtifact/OutputArtifact in lightweight Python components.
* Add copyright notice.
* Revert bad changes.
* Add executor_test.py
* Move test alongside code.
* Undo experimental change.
* Fix presubmit test script.
* presubmit fix.
* presubmit fix.
* presubmit fix.
* Add sample for v2 compiler.
* Address PR comments.
2021-03-25 14:31:46 -07:00
e11241f135
fix(sdk): v2 - Compiler - Fixed the v2 component detection ( #5340 )
...
* SDK - v2 - Compiler - Fixed the v2 component detection
* Added line breaks
* Formatted again
2021-03-22 17:52:43 -07:00
06f0ee51d0
chore(sdk.v2): Do not inject component input for loop arguments from component output ( #5331 )
2021-03-18 14:36:16 -07:00
0d2a341bad
chore(sdk): fix metrics schema title, and missing mapping ( #5317 )
...
* fix metrics schema title, and missing mapping
* fix tests
* fix tests
2021-03-18 12:19:16 -07:00
21a012da4b
feat(sdk.v2): Parse pipeline param in argument values ( #5183 )
...
* parse pipeline param in argument values
* avoid duplicating input/output when type missing
* use double dash for compiler injected input name, and check for collision.
2021-03-18 09:48:16 -07:00
d7eb168347
chore(sdk): merge v2 dsl code back to v1. ( #5227 )
...
* Merge v2 dsl back to v1 dsl
* fix tests
* fix tests and always attach container_spec to container_op
* address review comments
* fix a bad indentation
2021-03-04 13:48:48 -08:00
3e55bd3637
Artifact class update to support protobuf Struct ( #5197 )
...
Change updates dsl Artifact class to support protobuff based metadata
removing existing implementation that uses properties/custom_properties
to handle metadata.
This change modifies the ontology artifacts with enhanced schemas for
model and datasets.
2021-03-03 16:14:48 -08:00
14af539cfb
fix(tests): Alter hardcoded sdk version in aiplatform_e2e_test.py test data before comparison. ( #5220 )
2021-03-01 11:04:16 -08:00
280cae3718
feat(sdk.v2): Support dsl.ParallelFor in v2. ( #5162 )
2021-02-25 05:12:15 -08:00
ec7201db5a
feat(sdk.v2): Implement AI Platform (Unified) Custom job component in KFP v2 SDK ( #5146 )
...
* code skeleton
* wip
* add param scaffold
* add param scaffold
* wip: add new DSL class for AiPlatformCustomJob
* wip: play with container op i/o
* wip
* wip
* enhance get_parameter_type
* add tests
* add get_ir_type and tests
* add get_ir_type and tests
* wip: implementing get_custom_job, without IO
* add op IO
* clean up
* fix issues in aiplatform module
* cmd/arg resolution
* fix
* fix nonetyped cmds
* revert _structures
* revert component_tests
* revert component_bridge
* add basic tests
* add an e2e test
* fix tests
* resolve comments.
* fix test
* fix test
2021-02-24 17:18:15 -08:00
665f3ce8ba
chore(sdk.v2): Error out on input value being a combination of constant value and PipelineParam reference. ( #5185 )
...
* parse pipeline param in argument values
* error out on input value using `.format()`
2021-02-24 14:40:14 -08:00
a2e2e6ca6f
feat(sdk.v2): support dsl.Condition in v2. ( #5127 )
...
* support condition in sdk.v2
* address review comments
2021-02-12 15:05:57 -08:00
Chen Sun
Yaqi Ji
Judah Rand
Alex Latchford
Joshua Carp
Toshiya Kawasaki
Ajay Gopinathan
Shuhei Fujiwara
Wakana Nogami
Yuan (Bob) Gong
Trs
Wakana Nogami
Alexey Volkov
dushyanthsc
Jiaxiao Zheng