* Reformat sdk only using the new yapf config.
* Reformat docstrings using docformatter.
* update golden files to resolve diff caused by whitespaces
* fix some tests
* format .py files under sdk/python/tests using yapf
* additional docformatter
* fix some tests
* Prepare SDK docs environment so its easier to understand how to build the docs locally so theyre consistent with ReadTheDocs.
* Clean up docstrings for kfp.Client
* Add in updates to the docs for compiler and components
* Update components area to add in code references and make formatting a little more consistent.
* Clean up containers, add in custom CSS to ensure we do not overflow on inline code blocks
* Clean up containers, add in custom CSS to ensure we do not overflow on inline code blocks
* Remove unused kfp.notebook package links
* Clean up a few more errant references
* Clean up the DSL docs some more
* Update SDK docs for KFP extensions to follow Sphinx guidelines
* Clean up formatting of docstrings after Ark-Kuns comments
* SDK - Components refactoring
This change is a pure refactoring of the implementation of component task creation.
For pipelines compiled using the DSL compiler (the compile() function or the command-line program) nothing should change.
The main goal of the refactoring is to change the way the component instantiation can be customized.
Previously, the flow was like this:
`ComponentSpec` + arguments --> `TaskSpec` --resolving+transform--> `ContainerOp`
This PR changes it to more direct path:
`ComponentSpec` + arguments --constructor--> `ContainerOp`
or
`ComponentSpec` + arguments --constructor--> `TaskSpec`
or
`ComponentSpec` + arguments --constructor--> `SomeCustomTask`
The original approach where the flow always passes through `TaskSpec` had some issues since TaskSpec only accepts string arguments (and two
other reference classes). This made it harder to handle custom types of arguments like PipelineParam or Channel.
Low-level refactoring changes:
Resolving of command-line argument placeholders has been extracted into a function usable by different task constructors.
Changed `_components._created_task_transformation_handler` to `_components._container_task_constructor`. Previously, the handler was receiving a `TaskSpec` instance. Now it receives `ComponentSpec` + arguments [+ `ComponentReference`].
Moved the `ContainerOp` construction handler setup to the `kfp.dsl.Pipeline` context class as planned.
Extracted `TaskSpec` creation to `_components._create_task_spec_from_component_and_arguments`.
Refactored `_dsl_bridge.create_container_op_from_task` to `_components._resolve_command_line_and_paths` which returns `_ResolvedCommandLineAndPaths`.
Renamed `_dsl_bridge._create_container_op_from_resolved_task` to `_dsl_bridge._create_container_op_from_component_and_arguments`.
The signature of `_components._resolve_graph_task` was changed and it now returns `_ResolvedGraphTask` instead of modified `TaskSpec`.
Some of the component tests still expect ContainerOp and its attributes.
These tests will be changed later.
* Adapted the _python_op tests
* Fixed linter failure
I do not want to add any top-level kfp imports in this file to prevent circular references.
* Added docstrings
* FIxed the return type forward reference
This makes the graph input references consistent with task output references.
This is a breaking change, but the graph components are not exposed in the documentation or samples yet.
* SDK/Components - Creating graph components from python pipeline function
`create_graph_component_from_pipeline_func` converts python pipeline function to a graph component object that can be saved, shared, composed or submitted for execution.
Example:
producer_op = load_component(component_with_0_inputs_and_2_outputs)
processor_op = load_component(component_with_2_inputs_and_2_outputs)
def pipeline1(pipeline_param_1: int):
producer_task = producer_op()
processor_task = processor_op(pipeline_param_1, producer_task.outputs['Output 2'])
return OrderedDict([
('Pipeline output 1', producer_task.outputs['Output 1']),
('Pipeline output 2', processor_task.outputs['Output 2']),
])
graph_component = create_graph_component_from_pipeline_func(pipeline1)
* Changed the signatures of exported functions
Non-public create_graph_component_spec_from_pipeline_func creates ComponentSpec
Public create_graph_component_from_pipeline_func creates component and writes it to file.
* Switched to using _extract_component_interface to analyze function signature
Stopped humanizing the input names for now. I think it's benefitial to extract interface from function signature the same way for both container and graph python components.
* Support outputs declared using pipeline function's return annotation
* Cleaned up the test
* Stop including the whole parent tasks in task output references
* By default, do not include task component specs in the graph component
Remove the component spec from component reference unless it will make the reference empty or unless explicitly asked by the user
* Exported the create_graph_component_from_pipeline_func function
* Fixed imports
* Updated the copyright year.
Chen Sun
Alexey Volkov
Alex Latchford
Alexey Volkov