* Reformat sdk only using the new yapf config.
* Reformat docstrings using docformatter.
* update golden files to resolve diff caused by whitespaces
* fix some tests
* format .py files under sdk/python/tests using yapf
* additional docformatter
* fix some tests
* feat(sdk): add default schema_version to pipeline
* sync api for go
* Fix tests and address comments
* Bump pipeline_spec version
* Fix v1 tests
* rebase to master
* sync api for go
* Fix tests and address comments
* Bump pipeline_spec version
* Fix v1 tests
* Refactor and move all v2 related code to under the v2 namespace.
Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.
Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.
* Update setup.py.
* update tests.
* revert accidental change of gcpc
* Fix component entrypoint.
* Update goldens.
* fix tests.
* fix merge conflict.
* revert gcpc change.
* fix tests.
* fix tests.
* Add type aliases for moved files.
* merge and update goldens.
* Enable the use of v2 Executor from KFP package.
This avoids the need to paste Executor code for v2 components into
the entrypoint of lightweight components.
* Update tests.
* quote package path.
* quote package path.
* escape fragment
* Update golden files.
* update goldens.
* Cleanup.
* More cleanup.
* update goldens.
* style updates.
* clean up sample tests.
* fix trailing comma
* Try using signed urls.
* fix
* fix presubmit
* try /bin/bash for Makefile shell
* undo signed url change.
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
* Add runtime resource request for GPUs
* clean up
* Updated docks and add check
* updated with test
* remove from branch
* run tests
* fix gpu vendor format
* Update after feedback
* add unit tet
* remove integration test
* clean up
* Clean up
* Updated to resource_constraints instead of resource
* fix uri placeholder in v2 compatible mode
* fix tests
* fix path generation
* fix tests
* fix test
* cleanup
* clean up
* fix test
* fix test
* fix test
* fix(launcher): handle parameter values with special characters stably
* include new test case
* add tensorboard minio test case
* fix go unit tests
* update test golden
* address feedback
* fix tests
* feat(sdk/dsl/compiler): support --mode flag which can turn on v2 compatible mode
* override compiler default mode using KF_PIPELINES_COMPILER_MODE env var
* update V1_LEGACY to V1
* add unit tests
* address feedback
* clean up
* cleanup again
* use absl.testing.parameterized for table driven tests
* update
* Update deprecated download_as_string to download_as_bytes
* Add more usage
* chore: Change python/kfp/cli from relative to absolute path
* chore: Change python/kfp/compiler from relative to absolute path
* added resource request at runtime
* fixed things
* Update to use read only parameter insteadt
* added test case and better example
* Updated again
* add the validation
* add to the test suit
* work in progress
* update after feedback
* fix the test
* clean up
* clean up
* fix the path
* add the test again
* clean up
* fix tests
* feedback fix
* comment out and clean up
* Escape strings in RuntimeInfo.
This allows strings to be serialized dictionaries etc.
Also, re-enable a couple of v2 tests.
* update goldens.
* Update exit_handler.py
* add comment.
* keep last error
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
* Support v2 components in KFP v2-compatible mode.
Adds support in v2-compatible mode for the following
* v2 lightweight function components
* support for passing ExecutorInput to component
* support for parsing ExecutorOutput from component
Also renamed several variables in RuntimeInfo as follow ups to previous
PR.
* Remove newly irrelevant variables in runtimeInfo
* update goldens.
* Renaming in runtimeInfo.
* Add v2 component pipeline tests and update goldens.
* create output directory for parameter files.
* Fix output schemas.
* Support concat placeholder.
* Use Path and MetadataPath
* update launcher.go too with new names.
* move functions around.
* add some TODOs, replace cmd placeholder.
* fix tests
* SDK - Refactored command-line resolving
Moved the execution engine specific code to the component bridge.
* Added placeholder_resolver
This simplifies adding custom placeholder resolving logic.
* updated the validation default test
* updated the version
* Update compiler.py
* Update compiler.py
* Update compiler.py
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
Chen Sun