{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and deploy with FfDL and Seldon demo\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### A simple IBM OSS pipeline demonstrates how to train a model using Fabric for Deep Learning and then deploy it with Seldon.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model for this pipeline\n",
    "\n",
    "We will be training a PyTorch model that can classify the gender of a human face image. This PyTorch model is a simple convolutional neural network (CNN) with 3 convolutional layers and 2 fully connected layers using the [UTKFace](https://susanqq.github.io/UTKFace/) dataset. We will be training for 5 epochs for the purpose of this demo.\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the necessary environment variables and install the KubeFlow Pipeline SDK\n",
    "We assume this notebook kernel has access to Python's site-packages and is in Python3.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_file_url = ''\n",
    "github_token = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install the necessary python packages\n",
    "\n",
    "Note: Please change pip to the package manager that's used for this Notebook Kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install kfp --upgrade\n",
    "!pip install ai_pipeline_params --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the KubeFlow Pipeline library and define the client and experiment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kfp\n",
    "from kfp import compiler\n",
    "import kfp\n",
    "from kfp import components\n",
    "from kfp import dsl\n",
    "from kfp import notebook\n",
    "\n",
    "# Run client with KUBEFLOW_PIPELINE_LINK if this notebook server\n",
    "# is running on localhost without enterprise gateway.\n",
    "\n",
    "# KUBEFLOW_PIPELINE_LINK = ''\n",
    "# client = kfp.Client(KUBEFLOW_PIPELINE_LINK)\n",
    "\n",
    "client = kfp.Client()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Define pipeline tasks using the kfp library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define secret name that contains the credentials for this pipeline, and load components\n",
    "secret_name = 'kfp-creds'\n",
    "configuration_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/eb830cd73ca148e5a1a6485a9374c2dc068314bc/components/ibm-components/commons/config/component.yaml')\n",
    "train_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/eb830cd73ca148e5a1a6485a9374c2dc068314bc/components/ibm-components/ffdl/train/component.yaml')\n",
    "serve_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/eb830cd73ca148e5a1a6485a9374c2dc068314bc/components/ibm-components/ffdl/serve/component.yaml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kfp.dsl as dsl\n",
    "import ai_pipeline_params as params\n",
    "\n",
    "\n",
    "# create pipeline\n",
    "@dsl.pipeline(\n",
    "  name='FfDL pipeline',\n",
    "  description='A pipeline for machine learning workflow using Fabric for Deep Learning and Seldon.'\n",
    ")\n",
    "def ffdlPipeline(\n",
    "    GITHUB_TOKEN=github_token,\n",
    "    CONFIG_FILE_URL=config_file_url,\n",
    "    model_def_file_path='gender-classification.zip',\n",
    "    manifest_file_path='manifest.yml',\n",
    "    model_deployment_name='gender-classifier',\n",
    "    model_class_name='ThreeLayerCNN',\n",
    "    model_class_file='gender_classification.py'\n",
    "):\n",
    "    \"\"\"A pipeline for end to end machine learning workflow.\"\"\"\n",
    "\n",
    "    get_configuration = configuration_op(\n",
    "                   token = GITHUB_TOKEN,\n",
    "                   url = CONFIG_FILE_URL,\n",
    "                   name = secret_name\n",
    "    )\n",
    "\n",
    "    train = train_op(\n",
    "                   model_def_file_path,\n",
    "                   manifest_file_path\n",
    "    ).apply(params.use_ai_pipeline_params(secret_name))\n",
    "\n",
    "    serve = serve_op(\n",
    "                   train.output, \n",
    "                   model_deployment_name, \n",
    "                   model_class_name, \n",
    "                   model_class_file\n",
    "    ).apply(params.use_ai_pipeline_params(secret_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Below are the default parameters for the above pipeline, \n",
    "# you can customize these parameters for each pipeline run.\n",
    "\n",
    "parameters={'config-file-url': config_file_url,\n",
    "            'github-token': github_token,\n",
    "            'model-def-file-path': 'gender-classification.zip',\n",
    "            'manifest-file-path': 'manifest.yml',\n",
    "            'model-deployment-name': 'gender-classifier',\n",
    "            'model-class-name': 'ThreeLayerCNN',\n",
    "            'model-class-file': 'gender_classification.py'}\n",
    "\n",
    "\n",
    "run = client.create_run_from_pipeline_func(ffdlPipeline, arguments=parameters).run_info\n",
    "\n",
    "import IPython\n",
    "html = ('<p id=\"link\"> </p> <script> document.getElementById(\"link\").innerHTML = \"Actual Run link <a href=//\" + location.hostname + \"%s/#/runs/details/%s target=_blank >here</a>\"; </script>'\n",
    "          % (client._get_url_prefix(), run.id))\n",
    "IPython.display.HTML(html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
