{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Train and deploy with FfDL and Seldon demo\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### A simple IBM OSS pipeline demonstrates how to train a model using Fabric for Deep Learning and then deploy it with Seldon.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Model for this pipeline\n", "\n", "We will be training a PyTorch model that can classify the gender of a human face image. This PyTorch model is a simple convolutional neural network (CNN) with 3 convolutional layers and 2 fully connected layers using the [UTKFace](https://susanqq.github.io/UTKFace/) dataset. We will be training for 5 epochs for the purpose of this demo.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define the necessary environment variables and install the KubeFlow Pipeline SDK\n", "We assume this notebook kernel has access to Python's site-packages and is in Python3.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "config_file_url = ''\n", "github_token = ''" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Install the necessary python packages\n", "\n", "Note: Please change pip to the package manager that's used for this Notebook Kernel." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install kfp --upgrade\n", "!pip install ai_pipeline_params --upgrade" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import the KubeFlow Pipeline library and define the client and experiment " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import kfp\n", "from kfp import compiler\n", "import kfp\n", "from kfp import components\n", "from kfp import dsl\n", "from kfp import notebook\n", "\n", "# Run client with KUBEFLOW_PIPELINE_LINK if this notebook server\n", "# is running on localhost without enterprise gateway.\n", "\n", "# KUBEFLOW_PIPELINE_LINK = ''\n", "# client = kfp.Client(KUBEFLOW_PIPELINE_LINK)\n", "\n", "client = kfp.Client()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Define pipeline tasks using the kfp library. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# define secret name that contains the credentials for this pipeline, and load components\n", "secret_name = 'kfp-creds'\n", "configuration_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/eb830cd73ca148e5a1a6485a9374c2dc068314bc/components/ibm-components/commons/config/component.yaml')\n", "train_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/eb830cd73ca148e5a1a6485a9374c2dc068314bc/components/ibm-components/ffdl/train/component.yaml')\n", "serve_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/eb830cd73ca148e5a1a6485a9374c2dc068314bc/components/ibm-components/ffdl/serve/component.yaml')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import kfp.dsl as dsl\n", "import ai_pipeline_params as params\n", "\n", "\n", "# create pipeline\n", "@dsl.pipeline(\n", " name='FfDL pipeline',\n", " description='A pipeline for machine learning workflow using Fabric for Deep Learning and Seldon.'\n", ")\n", "def ffdlPipeline(\n", " GITHUB_TOKEN=github_token,\n", " CONFIG_FILE_URL=config_file_url,\n", " model_def_file_path='gender-classification.zip',\n", " manifest_file_path='manifest.yml',\n", " model_deployment_name='gender-classifier',\n", " model_class_name='ThreeLayerCNN',\n", " model_class_file='gender_classification.py'\n", "):\n", " \"\"\"A pipeline for end to end machine learning workflow.\"\"\"\n", "\n", " get_configuration = configuration_op(\n", " token = GITHUB_TOKEN,\n", " url = CONFIG_FILE_URL,\n", " name = secret_name\n", " )\n", "\n", " train = train_op(\n", " model_def_file_path,\n", " manifest_file_path\n", " ).apply(params.use_ai_pipeline_params(secret_name))\n", "\n", " serve = serve_op(\n", " train.output, \n", " model_deployment_name, \n", " model_class_name, \n", " model_class_file\n", " ).apply(params.use_ai_pipeline_params(secret_name))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Below are the default parameters for the above pipeline, \n", "# you can customize these parameters for each pipeline run.\n", "\n", "parameters={'config-file-url': config_file_url,\n", " 'github-token': github_token,\n", " 'model-def-file-path': 'gender-classification.zip',\n", " 'manifest-file-path': 'manifest.yml',\n", " 'model-deployment-name': 'gender-classifier',\n", " 'model-class-name': 'ThreeLayerCNN',\n", " 'model-class-file': 'gender_classification.py'}\n", "\n", "\n", "run = client.create_run_from_pipeline_func(ffdlPipeline, arguments=parameters).run_info\n", "\n", "import IPython\n", "html = ('

'\n", " % (client._get_url_prefix(), run.id))\n", "IPython.display.HTML(html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }