pipelines/components/nuclio

Nuclio (Serverless) Components

Nuclio is a native and high performance serverless platform over Kubernetes which automate the process of build, deployment, monitoring, and auto-scaling of micro-services. Nuclio support variety of data and data-science related features (e.g. stream processing, GPUs, volume/DB mounts, high concurrency, etc.)

To install Nuclio over Kubernetes follow the instruction in Github, or this interactive tutorial.

Nuclio functions can be used in the following ML pipline tasks:

• Data collectors, ETL, stream processing
• Data preparation and analysis
• Hyper parameter model training
• Real-time model serving
• Feature vector assembly (real-time data preparation)

Read more on the use of Nuclio in data-science here. Nuclio functions can be generated automatically from 8 code languages, from Jupyter Notebooks, Zip, Git, Docker, etc. The nuclio-jupyter repo provide guidance and many examples.

Components

There are currently 3 components in this package:

• deploy - Automatically build and deploy/re-deploy functions from code/zip/notebooks/git/.. and/or override various deployment configurations such as setting cpu/mem/gpu resources, scaling, environment variables, triggers, etc.
• delete - Delete a function
• invoker - invoke a function and return the results/logs

Additional components and examples will be added soon for parallel batch/stream processing

Examples

Deploy a function (from Github)

nuclio_dep = kfp.components.load_component_from_file('deploy/component.yaml')

def my_pipeline():
    new_func = nuclio_dep(url='git://github.com/nuclio/nuclio#master:/hack/examples/python/helloworld', name='myfunc', project='myproj', tag='0.11') 
    
    ...