Commit Graph

4 Commits

Author SHA1 Message Date
Alexey Volkov be4f5851ed SDK - Components - Creating graph components from python pipeline function (#2273)
* SDK/Components - Creating graph components from python pipeline function

`create_graph_component_from_pipeline_func` converts python pipeline function to a graph component object that can be saved, shared, composed or submitted for execution.

Example:

    producer_op = load_component(component_with_0_inputs_and_2_outputs)
    processor_op = load_component(component_with_2_inputs_and_2_outputs)

    def pipeline1(pipeline_param_1: int):
        producer_task = producer_op()
        processor_task = processor_op(pipeline_param_1, producer_task.outputs['Output 2'])

        return OrderedDict([
            ('Pipeline output 1', producer_task.outputs['Output 1']),
            ('Pipeline output 2', processor_task.outputs['Output 2']),
        ])

    graph_component = create_graph_component_from_pipeline_func(pipeline1)

* Changed the signatures of exported functions

Non-public create_graph_component_spec_from_pipeline_func creates ComponentSpec
Public create_graph_component_from_pipeline_func creates component and writes it to file.

* Switched to using _extract_component_interface to analyze function signature

Stopped humanizing the input names for now. I think it's benefitial to extract interface from function signature the same way for both container and graph python components.

* Support outputs declared using pipeline function's return annotation

* Cleaned up the test

* Stop including the whole parent tasks in task output references

* By default, do not include task component specs in the graph component

Remove the component spec from component reference unless it will make the reference empty or unless explicitly asked by the user

* Exported the create_graph_component_from_pipeline_func function

* Fixed imports

* Updated the copyright year.
2019-10-02 16:20:07 -07:00
Alexey Volkov a7635f1cd4 SDK - Using Airflow ops in Pipelines (#1483)
* SDK - Using Airflow ops in Pipelines

* Documented the create_component_from_airflow_op function

* Need to set use_code_pickling=True now

* Using the original operator name as the component name

* Filtering out `*args` and `**kwargs` parameters that some operators have

* Fixed the function call

* Changed the default airflow base image
Airflow has removed most of the old images and tags.
See https://issues.apache.org/jira/browse/AIRFLOW-5093 and  https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+CI+image#AIP-10Multi-layeredandmulti-stageofficialAirflowCIimage-ProposedsetupoftheDockerHubandTravisCI .
2019-08-02 19:53:52 -07:00
Alexey Volkov 17582a8fab SDK/Components - Made component search locations configurable (#519)
* SDK/Components - Made component search locations configurable

* Raise proper error on existing, but malformed components instead of saying the component was not found.
Component name must be non-empty.
Addressed PR feedback.
2018-12-12 17:08:51 -08:00
Pascal Vicaire 633e2ddcc8 Initial commit of the kubeflow/pipeline project. 2018-11-02 14:02:31 -07:00