* Fix bug where delete resource op should not have success_condition, failure_condition, and output parameters
* remove unnecessary whitespace
* compiler test for delete resource ops should retrieve templates from spec instead of root
* Collecting coiverage when running python tests
* Added coveralls to python unit tests
* Try removing the PATH modification
* Specifying coverage run --source
* Using the installed package
* Try getting the correct coverage paths
* Lint Python code for undefined names
* Lint Python code for undefined names
* Exclude tfdv.py to workaround an overzealous pytest
* Fixup for tfdv.py
* Fixup for tfdv.py
* Fixup for tfdv.py
* SDK - Refactoring - Serialized PipelineParam does not need type
Only the types in non-serialized PipelineParams are ever used.
* SDK - Refactoring - Serialized PipelineParam does not need value
Default values are only relevant when PipelineParam is used in the pipeline function signature and even in this case compiler captures them explicitly from the pipelineParam objects in the signature.
There is no other uses for them.
* avoid istio injector in the container builder
* find the correct namespace
* configure default ns to kubeflow if out of cluster; fix unit tests
* container build default gcs bucket
* resolve comments
* code refactor; add create_bucket_if_not_exist in containerbuilder
* support load kube config and output error, good for ai platform notebooks/local notebooks
* remove create_bucket_if_not_exist param
* SDK - Containers - Returning image name with digest
Image building functions now return image name with digest: image_repo@sha256:digest
Fixes https://github.com/kubeflow/pipelines/issues/1715
* Added comments
* Remove redundant import.
* Simplify sample_test.yaml by using withItem syntax.
* Simplify sample_test.yaml by using withItem syntax.
* Change dict to str in withItems.
* Add image pull secret sample.
* Move imagepullsecret sample from test dir to sample dir. Waiting on corresponding unit test infra refactoring.
* Update the location of imagepullsecrets so that it can serve as an example.
* Add minimal comments documenting usage.
* Remove redundant import.
* Simplify sample_test.yaml by using withItem syntax.
* Simplify sample_test.yaml by using withItem syntax.
* Change dict to str in withItems.
* Add preemptible gpu tpu sample and unittest
* Update a test utility function.
* Seperate the location of sample and gold .yaml for testing purpose.
* Added support for mulitple outputs
* Added test for multiple output
* Adding sample for multiple outputs
* func_signature now shorter form
* Added parameters tag
* Fixed func_signature mistake
* refactor component build code
* remove unnecessary import
* minor changes
* fix unit tests
* separate the container build from the component build; add support for directories in the containerbuilder
* minor fixes
* fix unit test
* fix tarball error
* revert changes
* unit test fix
* minor fix
* addressing comments
* removing the check_gcs_path function
* move namespace to the contructor of containerbuilder
* fix bugs
* refactor component build code
* remove unnecessary import
* minor changes
* fix unit tests
* fix sample test bug
* revert the change of the dependency orders
* add -u to disable python stdout buffering
* address the comments
* separate lines to look clean
* fix unit tests
* fix
* SDK - Lightweight - Added support for "None" default values
Previously it was impossible to pass None to components since it was being converted to the string "None".
* is_required = not input.optional for now
As asked by @gaoning777
* Add PipelineConf method to set ttlSecondsAfterFinished in argo workflow spec
* remove unnecessary compile test for ttl. add unit test for ttl instead.
* Configure gcp connectors in dsl
* Make configure_gcp_connector more extensible
* Add add_pod_env op handler.
* Only apply add_pod_env on ContainerOp
* Update license header
I've introduced code pickling to capture dependencies in https://github.com/kubeflow/pipelines/pull/1372
Later I've discovered that there is a serious opcode incompatibility between python versions 3.5 and 3.6+. See my analysis of the issue: https://github.com/cloudpipe/cloudpickle/issues/293
Dues to this issue I decided to switch back to using source code copying by default and to continue improving it.
Until we stop supporting python 3.5 (https://github.com/kubeflow/pipelines/pull/668) it's too dangerous to use code pickling by default.
Code pickling can be enabled by specifying `pickle_code=True` when calling `func_to_container_op`
* SDK/DSL: Make 'name' argument of a PipelineVolume omittable
Also remove unused imports from _pipeline_volume module
Signed-off-by: Ilias Katsakioris <elikatsis@arrikto.com>
* Use hashlib.sha256() instead of id()
* Fix not maintaining provided name
* SDK - Controlling which modules are captured with Lightweight components
All func_to_* functions now accept the modules_to_capture parameter: List of module names that will be captured (instead of just referencing) during the dependency scan. By default the func.__module__ is captured.
* Described the behavior more in depth.
* Added a test to check that only dependencies are captured
* Frontend - Show customized task display names
* Added customized name test
* Added ContainerOp.set_display_name(name) method
* Stopped writing human_name to display_name annotation for now
Reason: It's a change to existing pipelines.
* Added test for op.set_display_name
* Fix for tests that have workflows with status nodes, but without any spec or templates
* Fixed the test workflow
* Fix linter error
Error: "The key 'metadata' is not sorted alphabetically"
* add default value type checking
* add jsonschema dependency
* fix unit test error
* workaround for travis python package installation
* add back jsonschema version
* fix sample test error in type checking sample
* add jsonschema in requirements such that sphinx works fine
* Transitively capturing code dependencies
Using cloudpickle.
* Got rid of func_type_declarations_code variable
* Extracted the function code extraction functions
* Improved support for capturing module-level dependencies
* Added test for capturing module-level dependencies
* Removed the _capture_function_code_using_source_copy function
As requested by Ning
* SDK/Compiler - Added op and template transformers
They can be used to apply some functions (e.g. to add secrets) to all pipeline ops.
* Removed the template_transformers for now
* Moved the op_transformers to PipelineConf
* Added op_transformers test
Jiaxiao Zheng
Eterna2
Alexey Volkov
Christian Clauss
Ning
Zane Durante
IronPan
hongye-sun
Derek Hao Hu
Ilias Katsakioris
Krassimir Valev
Alexey Volkov
Hamed