* fix v2 sample tests for cuj3
* separate sh -c
* fix two step pipeline
* fix piping error
* fix nit
* change path to uri to correct cat command
* correct bash script
* remove display name
* change directory reference in commands
* remove old files
* change output type of importer to artifact_class.TYPE_NAME
* use the same logic as component_factory._annotation_to_type_struct for determining type_name
* format using yapf
* fix type_name
* format using yapf
* add yaml file
* changes with regards to PR comments
* format pipeline_with_importer_and_gcpc_types.py
* remove unused imports
* change name of test component
* add pipeline_with_importer_and_gcpc_types to config
* update yaml file
* feat(sdk): add support for ParallelFor parallelism setting
* add parallelism value check
* Add unit tests
* Adding a compiler test
* used None as default parallelism input signature, other minor fixes on format and tests
* fix import statements
* add release.md message
* update ParallelFor docstring
* fixed docstring comments
* removed 'optional' in docstrings
* test if and concat placeholder and validate artifact channel usage
* fix nit in read write test config
* resolve comments
* resolve comments and move test cases
* resolve comments and add placeholder validation
* remove unused imports
* update release file
* add compiler test pipeline with multiple exit handlers
* remove blocker of multiple exit handlers
* move exit handler builder logic to pipeline_spec_builder
* build all exit handlers per pipeline
* add compiler test with IR inspection
* prevent usage of cross-pipeline after
* test cross-pipeline after is prevented
* update existing task dependency logic and tests
* add v2 sample test
* remove cross-pipeline .after
* prevent cross-dag data dependency for dsl features
* add compiler test pipeline with nested exit handlers
* add support for nested exit handlers
* clean up pipeline with nested exit handlers
* remove sample with multiple exit handlers
* remove compiler test with nested exit handlers
* add compilation guard against nested exit handlers in subdag
* update release notes
* support container_component decorator for function with no inputs
* resolve review comments
* add sample tests for milestone 1
* modify compiler test data
* resolve reviews
* resolve reviews
* WIP
* implementation of function of no inputs
* fixed sample test
* re-fix sample test
* fix rebase merge conflict
* resolve formatting
* resolve isort error for test data
* resolve comments
* fix nit
* resolve nit
* add implementation for placeholders i/o, sample and compiler tests
* resolve comments and merge logic for constructing container component
* resolve comments
* resolve comments
* fix assertion messages
* add error handling for accessing artifact by itself
* add test for raising error for accessing artifact by itself
* add module-level docstrings
* update compiler docstrings
* update registry module docstrings
* add BaseComponent and children to public api, but discourage use
* update artifact docstrings and type annotations
* update dsl docstrings
* update client docstrings
* clean up kfp.__init__
* add dsl placeholder docstrings
* many more docstring updates
* document type aliases in dsl module
* use __all__ in top-level modules to record public api
* add index and source files
* add kubeflow assets to _static/
* add and pin requirements
* use block quote instead of header for readme notice
* update conf.py
* delete old files
* remove unused imports
* use google as isort profile
* sort imports
* format with yapf
* clean end of file new line, trailing whitespace, double quoted strings
* move placeholders to placeholders module; implement if-present and concat placeholders
* update module name and method names throughout codebase
* rename placeholder abc
* add back compat support for CEL-style concat string
* add copyright
* add docstrings; clean up
* remove duplicate code
* clean up relevant files
* use context manager for temporary directory cleanup
* add component compilation methods
* add component compilation tests
* add more complete pipeline_func argument handling
* use context manager for handling type check mode
* improve checks on inputs for component compilation
* correct pipeline name for primitive component IR
* simplify component compilation logic
* extract compiler methods
* move compiler helpers to helpers file to avoid circular imports
* remove underscore prefix from functions in hidden module
* move helpers tests to helper_test module
* put to_pipeline_spec on ComponentSpec
* fix inputs reference
* use same naming regex for pipelines and components
* permit compilation of all BaseComponent subclasses
* clean up docstrings
* merge compiler.helpers into pipeline_spec_builder
* modify docstrings; refactor
* modify docstring
* change serialization format from json to yaml
* refactor: use guard clause for clarity
* convert test data from json to yaml
* add and update tests
* permit deprecated compile to json
* re-add json test data
* update compiler tests
* refactor Compiler._write_pipeline_spec_file
* update golden snapshots
* update sample output extensions
* test file extension exceptions and warnings
* update release notes
* remove json golden snapshots
* Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest
* Update test for TTL
* Declare fix in release note of SDK
* update syntaxe
* Update RELEASE.md
Add Braking change due to incompatibility with KFP pre 1.7 due to Argo 2.X
* Use google.protobuf.Value in v2 for passing parameters.
* retest samples.
* Fix tests.
* Update release, more cleanup.
* Use github.com/kubeflow/pipelines/api from same repo.
* Run go mod tidy
* chore: go mod tidy
* fix v2 compile error and clean up unused code
* pr comments.
* update goldens
* Fix metadata recording.
* Update kfp mlmd client.
* fix test again
* another try.
* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings
* fix v2 launcher + clean up
* fix a compile error
* fix a few more tests
* fix number parsing
* clean up
* disable cache_v2 test.
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
* Try alternatives for obtaining pip when `ensurepip` does not
exist in container.
This PR also fixes the tests that have golden files. Developers can
update golden files by setting UPDATE_GOLDENS=True before running
tests. E.g., run the following from sdk/python directory:
UPDATE_GOLDENS=True python3 -m unittest discover --verbose --pattern '*test*.py' --top-level-directory .
* Update RELEASE.
* Fix podSpecPatch bug
* wrong structure for nodeselector and bring back integreation test
* updated python compiler test
* updated to use the correct GPU type
* missed to commit the updated test config
* Updated release notes
* remove test to see if it solved the issue
* Reformat sdk only using the new yapf config.
* Reformat docstrings using docformatter.
* update golden files to resolve diff caused by whitespaces
* fix some tests
* format .py files under sdk/python/tests using yapf
* additional docformatter
* fix some tests
* feat(sdk): add default schema_version to pipeline
* sync api for go
* Fix tests and address comments
* Bump pipeline_spec version
* Fix v1 tests
* rebase to master
* sync api for go
* Fix tests and address comments
* Bump pipeline_spec version
* Fix v1 tests
* Refactor and move all v2 related code to under the v2 namespace.
Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.
Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.
* Update setup.py.
* update tests.
* revert accidental change of gcpc
* Fix component entrypoint.
* Update goldens.
* fix tests.
* fix merge conflict.
* revert gcpc change.
* fix tests.
* fix tests.
* Add type aliases for moved files.
* merge and update goldens.
* Enable the use of v2 Executor from KFP package.
This avoids the need to paste Executor code for v2 components into
the entrypoint of lightweight components.
* Update tests.
* quote package path.
* quote package path.
* escape fragment
* Update golden files.
* update goldens.
* Cleanup.
* More cleanup.
* update goldens.
* style updates.
* clean up sample tests.
* fix trailing comma
* Try using signed urls.
* fix
* fix presubmit
* try /bin/bash for Makefile shell
* undo signed url change.
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
* Add runtime resource request for GPUs
* clean up
* Updated docks and add check
* updated with test
* remove from branch
* run tests
* fix gpu vendor format
* Update after feedback
* add unit tet
* remove integration test
* clean up
* Clean up
* Updated to resource_constraints instead of resource
* fix uri placeholder in v2 compatible mode
* fix tests
* fix path generation
* fix tests
* fix test
* cleanup
* clean up
* fix test
* fix test
* fix test
* fix(launcher): handle parameter values with special characters stably
* include new test case
* add tensorboard minio test case
* fix go unit tests
* update test golden
* address feedback
* fix tests
* feat(sdk/dsl/compiler): support --mode flag which can turn on v2 compatible mode
* override compiler default mode using KF_PIPELINES_COMPILER_MODE env var
* update V1_LEGACY to V1
* add unit tests
* address feedback
* clean up
* cleanup again
* use absl.testing.parameterized for table driven tests
* update
* Update deprecated download_as_string to download_as_bytes
* Add more usage
* chore: Change python/kfp/cli from relative to absolute path
* chore: Change python/kfp/compiler from relative to absolute path
* added resource request at runtime
* fixed things
* Update to use read only parameter insteadt
* added test case and better example
* Updated again
* add the validation
* add to the test suit
* work in progress
* update after feedback
* fix the test
* clean up
* clean up
* fix the path
* add the test again
* clean up
* fix tests
* feedback fix
* comment out and clean up
* Escape strings in RuntimeInfo.
This allows strings to be serialized dictionaries etc.
Also, re-enable a couple of v2 tests.
* update goldens.
* Update exit_handler.py
* add comment.
* keep last error
Co-authored-by: Yuan (Bob) Gong <4957653+Bobgy@users.noreply.github.com>
Connor McCarthy
Chen Sun
Scott_Xu
chongyouquan
Bo
Deepyaman Datta
Yaqi Ji
Seb
Yuan (Bob) Gong
juliusvonkohout
Ajay Gopinathan
Niklas Hansson
capri-xiyue
Alexey Volkov