pipelines

Commit Graph

Author	SHA1	Message	Date
Alexey Volkov	9adf16301d	SDK - Airflow - Fixed bug in airflow op creation (#1911 ) This PR fixes a bug in AirFlow op creation. The `_run_airflow_op` helper function was not captured along with the `_run_airflow_op_closure` function, because they belong to different modules (`_run_airflow_op_closure` was module-less). This was not discovered during the notebook testing of the code since in that environment the `_run_airflow_op` was also module-less as it was defined in a notebook (not in .py file).	2019-08-21 16:29:54 -07:00
Alexey Volkov	a7635f1cd4	SDK - Using Airflow ops in Pipelines (#1483 ) * SDK - Using Airflow ops in Pipelines * Documented the create_component_from_airflow_op function * Need to set use_code_pickling=True now * Using the original operator name as the component name * Filtering out `args` and `kwargs` parameters that some operators have Fixed the function call * Changed the default airflow base image Airflow has removed most of the old images and tags. See https://issues.apache.org/jira/browse/AIRFLOW-5093 and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+CI+image#AIP-10Multi-layeredandmulti-stageofficialAirflowCIimage-ProposedsetupoftheDockerHubandTravisCI .	2019-08-02 19:53:52 -07:00

Author

SHA1

Message

Date

Alexey Volkov

9adf16301d

SDK - Airflow - Fixed bug in airflow op creation (#1911 )

This PR fixes a bug in AirFlow op creation.
The `_run_airflow_op` helper function was not captured along with the `_run_airflow_op_closure` function, because they belong to different modules (`_run_airflow_op_closure` was module-less).
This was not discovered during the notebook testing of the code since in that environment the `_run_airflow_op` was also module-less as it was defined in a notebook (not in .py file).

2019-08-21 16:29:54 -07:00

Alexey Volkov

a7635f1cd4

SDK - Using Airflow ops in Pipelines (#1483 )

* SDK - Using Airflow ops in Pipelines

* Documented the create_component_from_airflow_op function

* Need to set use_code_pickling=True now

* Using the original operator name as the component name

* Filtering out `*args` and `**kwargs` parameters that some operators have

* Fixed the function call

* Changed the default airflow base image
Airflow has removed most of the old images and tags.
See https://issues.apache.org/jira/browse/AIRFLOW-5093 and  https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+CI+image#AIP-10Multi-layeredandmulti-stageofficialAirflowCIimage-ProposedsetupoftheDockerHubandTravisCI .

2019-08-02 19:53:52 -07:00

2 Commits