* support placeholder in metadata dict
* add unit tests
* add compilation test
* fix tests
* update compiler test
* use 'metadata' as input key
* update compiler test and golden snapshot
* add int input to compiler test
* change constant name
* use util function
* support f-strings in dynamic import metadata; add and update tests
* update method name after rebase
* make compile consistent across python versions
* add runtime artifact instance creation logic
* refactor executor
* add executor tests
* add custom artifact type import handling and tests
* fix artifact class construction
* fix custom artifact type in tests
* add typing extensions dependency for all python versions
* use mock google namespace artifact for tests
* remove print statement
* update google artifact golden snapshot
* resolve some review feedback
* remove handling for OutputPath and InputPath custom artifact types; update function names and tests
* clarify named tuple tests
* update executor tests
* add artifact return and named tuple support; refactor; clean tests
* implement review feedback; clean up artifact names
* move test method
* remove dependency of deprecated on v2
* use new artifact identification logic
* add custom artifact type compiler test
* compile schema_version from non-kfp artifacts
* update tests
* add importer to compiler test
* update golden snapshots
* fix type checking code
* undo executor_test diff
* remove gcpc types compiler test
* add compilation test comment
* update google artifact handling
* update compiler test
* only permit system. and google. artifact types; add tests
* update compiler test
* simplify type checking logic
* use bundled artifact types in PipelineArtifactChannel
* raise error on unknown v1 input or output
* add type utils and move functions
* add type utils tests
* remove schema_version from inputspec and outputspec
* update artifact proto creation and tests
* propogate bundled artifact type string changes throughout sdk
* update golden snapshot
* remove unused v1 compat and tests
* update type checking and tests to validate schema_version using semantic versioning
* update importer sample
* update test data config
* update test_data config
* clean up sample
* clean up sample
* fix v2 sample tests for cuj3
* separate sh -c
* fix two step pipeline
* fix piping error
* fix nit
* change path to uri to correct cat command
* correct bash script
* remove display name
* change directory reference in commands
* remove old files
* change output type of importer to artifact_class.TYPE_NAME
* use the same logic as component_factory._annotation_to_type_struct for determining type_name
* format using yapf
* fix type_name
* format using yapf
* add yaml file
* changes with regards to PR comments
* format pipeline_with_importer_and_gcpc_types.py
* remove unused imports
* change name of test component
* add pipeline_with_importer_and_gcpc_types to config
* update yaml file
* feat(sdk): add support for ParallelFor parallelism setting
* add parallelism value check
* Add unit tests
* Adding a compiler test
* used None as default parallelism input signature, other minor fixes on format and tests
* fix import statements
* add release.md message
* update ParallelFor docstring
* fixed docstring comments
* removed 'optional' in docstrings
* test if and concat placeholder and validate artifact channel usage
* fix nit in read write test config
* resolve comments
* resolve comments and move test cases
* resolve comments and add placeholder validation
* remove unused imports
* update release file
* add compiler test pipeline with multiple exit handlers
* remove blocker of multiple exit handlers
* move exit handler builder logic to pipeline_spec_builder
* build all exit handlers per pipeline
* add compiler test with IR inspection
* prevent usage of cross-pipeline after
* test cross-pipeline after is prevented
* update existing task dependency logic and tests
* add v2 sample test
* remove cross-pipeline .after
* prevent cross-dag data dependency for dsl features
* add compiler test pipeline with nested exit handlers
* add support for nested exit handlers
* clean up pipeline with nested exit handlers
* remove sample with multiple exit handlers
* remove compiler test with nested exit handlers
* add compilation guard against nested exit handlers in subdag
* update release notes
* support container_component decorator for function with no inputs
* resolve review comments
* add sample tests for milestone 1
* modify compiler test data
* resolve reviews
* resolve reviews
* WIP
* implementation of function of no inputs
* fixed sample test
* re-fix sample test
* fix rebase merge conflict
* resolve formatting
* resolve isort error for test data
* resolve comments
* fix nit
* resolve nit
* add implementation for placeholders i/o, sample and compiler tests
* resolve comments and merge logic for constructing container component
* resolve comments
* resolve comments
* fix assertion messages
* add error handling for accessing artifact by itself
* add test for raising error for accessing artifact by itself
* add module-level docstrings
* update compiler docstrings
* update registry module docstrings
* add BaseComponent and children to public api, but discourage use
* update artifact docstrings and type annotations
* update dsl docstrings
* update client docstrings
* clean up kfp.__init__
* add dsl placeholder docstrings
* many more docstring updates
* document type aliases in dsl module
* use __all__ in top-level modules to record public api
* add index and source files
* add kubeflow assets to _static/
* add and pin requirements
* use block quote instead of header for readme notice
* update conf.py
* delete old files
* remove unused imports
* use google as isort profile
* sort imports
* format with yapf
* clean end of file new line, trailing whitespace, double quoted strings
* implement override caching options for client
* use absolute path in test for overriding caching
* format code and add unit tests from pipeline
* delete typo
* add two-step pipeline test and review changes
* fix formatting
* fix typo on deprecated code
* update release file
* resolve comments on docstring and testing
* update release file
* Add default registry context
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
* Address review comments
* Reinstate `_is_ar_host()` check
* Remove `view=FULL` flag for default list and get
version URLs
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
* Change outdated test name
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
* move placeholders to placeholders module; implement if-present and concat placeholders
* update module name and method names throughout codebase
* rename placeholder abc
* add back compat support for CEL-style concat string
* add copyright
* add docstrings; clean up
* remove duplicate code
* Handle corner case when list is empty
* add validation for tags and versions
* add tests for empty lists
* add tests for version and tag validation
* fix formatting and optional arguments
* fix type annotation
* clean up relevant files
* use context manager for temporary directory cleanup
* add component compilation methods
* add component compilation tests
* add more complete pipeline_func argument handling
* use context manager for handling type check mode
* improve checks on inputs for component compilation
* correct pipeline name for primitive component IR
* simplify component compilation logic
* extract compiler methods
* move compiler helpers to helpers file to avoid circular imports
* remove underscore prefix from functions in hidden module
* move helpers tests to helper_test module
* put to_pipeline_spec on ComponentSpec
* fix inputs reference
* use same naming regex for pipelines and components
* permit compilation of all BaseComponent subclasses
* clean up docstrings
* merge compiler.helpers into pipeline_spec_builder
* modify docstrings; refactor
* modify docstring
* clean up client.py
* update client method return values
* update other types
* use preferred path to kfp_server_api classes
* fix incorrect module name
* other type annotation cleanup
* make docstrings consistent
* use correct type when option is path
* fix options on create recurring-run
* make output logging more consistent
* show deprecation warning for dsl-compile on invocation
* fix unarchive experiment
* fix broken f-string
* convert camelcase methods to snake case
* add deprecated alias group
* add deprecated alias group tests
* implement deprecated alias group
* clean up other alias tests
* clean up docstring
* fix type annotations / mypy
* format with yapf
* add docstring parsing utils
* implement docstring parsing for help text
* implement smoke test of all cli commands
* clean up use of optional in docstring
* make command descriptions more consistent
* fix disable_type_check
* update release notes
* allow both singular and plural versions of nouns
* use command table for client creation
* cleanup
* simplify docker mock
* add docker to test requirements; clean up existing requirements
* change serialization format from json to yaml
* refactor: use guard clause for clarity
* convert test data from json to yaml
* add and update tests
* permit deprecated compile to json
* re-add json test data
* update compiler tests
* refactor Compiler._write_pipeline_spec_file
* update golden snapshots
* update sample output extensions
* test file extension exceptions and warnings
* update release notes
* remove json golden snapshots
* AIP-5661: Add in filter to list_pipeline_versions SDK method
* AIP-5661: Update type hints + docstrings to conform to standard. Include message in release notes
* Support per workflow TTL (ttl_seconds_after_finished) with new format of Argo workflow manifest
* Update test for TTL
* Declare fix in release note of SDK
* update syntaxe
* Update RELEASE.md
Add Braking change due to incompatibility with KFP pre 1.7 due to Argo 2.X
* Use google.protobuf.Value in v2 for passing parameters.
* retest samples.
* Fix tests.
* Update release, more cleanup.
* Use github.com/kubeflow/pipelines/api from same repo.
* Run go mod tidy
* chore: go mod tidy
* fix v2 compile error and clean up unused code
* pr comments.
* update goldens
* Fix metadata recording.
* Update kfp mlmd client.
* fix test again
* another try.
* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings
* fix v2 launcher + clean up
* fix a compile error
* fix a few more tests
* fix number parsing
* clean up
* disable cache_v2 test.
Co-authored-by: Yuan Gong <gongyuan94@gmail.com>
* fix the issue:
the 'mlpipeline-ui-metadata','mlpipeline-metrics' path is replaced by the default path as '_components._generate_output_file_name(output.name)'
* reformat code
* Update _container_op.py
* Update RELEASE.md
* add description optionally
* add release notes
* add new line
* capture if description is not accepted by server
* add todo comment
* add better description api missing exception
* add disbale/delete run (job) functions to sdk client
* change run to job to accuratly reflect api
* add release note for change
* align docstrings with style guide requirements
* formatting
Co-authored-by: Chen Sun <chensun@users.noreply.github.com>
* Try alternatives for obtaining pip when `ensurepip` does not
exist in container.
This PR also fixes the tests that have golden files. Developers can
update golden files by setting UPDATE_GOLDENS=True before running
tests. E.g., run the following from sdk/python directory:
UPDATE_GOLDENS=True python3 -m unittest discover --verbose --pattern '*test*.py' --top-level-directory .
* Update RELEASE.
* Require base and target images in built containers to be unique.
Also add a test to verify, and fix formatting in cli/components.py.
* Update release notes.
* fix formatting.
* Refactor and move all v2 related code to under the v2 namespace.
Most of the changes are around imports and restructuring of the
codebase. While it looks like a lot of code was added, most of the code
already existed and was simply moved or copied over to v2. The only
exceptions are:
- under kfp/v2/components/component_factory.py: some helper functions
were copied with simplification from _python_op.py
- we no longer strip the `_path` suffix in v2 components.
Note: there is still some duplication of code (particularly between
component_factory.py and _python_op.py), but it's ok for now since we
intend to replace some of this with v2 ComponentSpec + BaseComponent.
* Update setup.py.
* update tests.
* revert accidental change of gcpc
* Fix component entrypoint.
* Update goldens.
* fix tests.
* fix merge conflict.
* revert gcpc change.
* fix tests.
* fix tests.
* Add type aliases for moved files.
* merge and update goldens.
* Add a CLI command to help containerize and build v2 components.
* Flesh out the CLI for Docker.
* update comments.
* Support multiple component files in the same directory.
* address pr comments.
* Let user specify just the directory containing components.
The CLI will search for components in all Python files by default. It
can also search for a specific filepattern (as supported by
pathlib.Path objects).
Also add a bunch of tests.
* Clean up.
* fix merge error.
* Fix docker import error for tests.
* Update release notes.
* address PR comments.
Chen Sun