{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Seldon Deployment Pipeline For CIFAR10 Image Recognition\n",
    "\n",
    "In this example (and Kale Kubeflow pipeline) we create a deployment pipeline for a pretrained CIFAR10 image model. \n",
    "The following steps will be run:\n",
    "\n",
    " * Setup Minio client\n",
    " * Download and test CIFAR10 model and train an Anchors Images explainer on it. Save both model and explainer to Minio.\n",
    " * Deploy model and explainer using Seldon and test\n",
    " * Train outlier detector\n",
    " * Train drift detector\n",
    " * Deploy knative eventing display to show asynchronous results from outlier and drift detectors. \n",
    " * Deploy outlier detector and test\n",
    " * deploy drift detector and test\n",
    " \n",
    " ### Setup\n",
    " \n",
    " You will need a kubeflow cluster >= 1.0 with \n",
    "   \n",
    "  * Knative eventing \n",
    "  * Seldon >= 1.2.1\n",
    " \n",
    " ### Kubeflow Jupyter Notebook Server\n",
    " \n",
    " To run this notebook inside kubeflow. Create a Jupyter notebook server using the image `seldonio/jupyter-lab-alibi-kale:0.11`\n",
    " \n",
    " ### GCP Setup\n",
    " \n",
    " On GCP If you use Kale to save this notebook as a pipeline you will need to add the storage_class of the `VolumeOp` to `nfs-client` if you have followed the steps to create a NFS RWX PV on GCP. e.g.:\n",
    " \n",
    " ```\n",
    "    marshal_vop = dsl.VolumeOp(\n",
    "        name=\"kale-marshal-volume\",\n",
    "        resource_name=\"kale-marshal-pvc\",\n",
    "        storage_class=\"nfs-client\",\n",
    "        modes=dsl.VOLUME_MODE_RWM,\n",
    "        size=\"1Gi\"\n",
    "    )\n",
    " ```\n",
    " \n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "imports"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from alibi.explainers import AnchorImage\n",
    "from alibi.datasets import fetch_adult\n",
    "from minio import Minio\n",
    "from minio.error import ResponseError\n",
    "from joblib import dump, load\n",
    "import dill\n",
    "from subprocess import run, Popen, PIPE\n",
    "from alibi_detect.utils.data import create_outlier_batch\n",
    "from alibi_detect.utils.fetching import fetch_tf_model\n",
    "import json\n",
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "tf.keras.backend.clear_session()\n",
    "from tensorflow.keras.layers import Conv2D, Conv2DTranspose, Dense, Layer, Reshape, InputLayer\n",
    "from tqdm import tqdm\n",
    "\n",
    "from alibi_detect.models.losses import elbo\n",
    "from alibi_detect.od import OutlierVAE\n",
    "from alibi_detect.utils.fetching import fetch_detector\n",
    "from alibi_detect.utils.perturbation import apply_mask\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.utils.visualize import plot_instance_score, plot_feature_outlier_image\n",
    "import time\n",
    "\n",
    "logger = tf.get_logger()\n",
    "logger.setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup Pipeline Paramers\n",
    "\n",
    "The following global variables can be set. These will be used as Pipeline parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pipeline-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "MINIO_HOST=\"minio-service.kubeflow:9000\"\n",
    "MINIO_ACCESS_KEY=\"minio\"\n",
    "MINIO_SECRET_KEY=\"minio123\"\n",
    "MINIO_MODEL_BUCKET=\"seldon\"\n",
    "CIFAR10_MODEL_PATH=\"tfserving/cifar10/model\"\n",
    "EXPLAINER_MODEL_PATH=\"tfserving/cifar10/explainer\"\n",
    "OUTLIER_MODEL_PATH=\"tfserving/cifar10/outlier\"\n",
    "DRIFT_MODEL_PATH=\"tfserving/cifar10/drift\"\n",
    "DEPLOY_NAMESPACE=\"admin\"\n",
    "TRAIN_OUTLIER_DETECTOR=False\n",
    "TRAIN_DRIFT_DETECTOR=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "functions"
    ]
   },
   "outputs": [],
   "source": [
    "def get_minio():\n",
    "    return Minio(MINIO_HOST,\n",
    "                    access_key=MINIO_ACCESS_KEY,\n",
    "                    secret_key=MINIO_SECRET_KEY,\n",
    "                    secure=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:setup"
    ]
   },
   "outputs": [],
   "source": [
    "minioClient = get_minio()\n",
    "buckets = minioClient.list_buckets()\n",
    "for bucket in buckets:\n",
    "    print(bucket.name, bucket.creation_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not minioClient.bucket_exists(MINIO_MODEL_BUCKET):\n",
    "    minioClient.make_bucket(MINIO_MODEL_BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test and save Model\n",
    "\n",
    "For simplicity we will use a pretrained Resnet32 CIFAR10 tensorflow model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_model_and_explainer",
     "prev:setup"
    ]
   },
   "outputs": [],
   "source": [
    "model = fetch_tf_model('cifar10', 'resnet32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train, test = tf.keras.datasets.cifar10.load_data()\n",
    "X_train, y_train = train\n",
    "X_test, y_test = test\n",
    "\n",
    "X_train = X_train.astype('float32') / 255\n",
    "X_test = X_test.astype('float32') / 255\n",
    "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n",
    "               'dog', 'frog', 'horse', 'ship', 'truck']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Test model locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx = 1\n",
    "X = X_test[idx].reshape(1, 32, 32, 3)\n",
    "plt.imshow(X.reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "print(\"class:\",class_names[y_test[idx][0]])\n",
    "print(\"prediction:\",class_names[model.predict(X_test[idx:idx+1])[0].argmax()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "modelfilepath=\"resnet\"\n",
    "tf.saved_model.save(model, modelfilepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "model_filepath=\"resnet\"\n",
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{CIFAR10_MODEL_PATH}/1/saved_model.pb\", modelfilepath+\"/saved_model.pb\"))\n",
    "variable_filepath = modelfilepath+\"/variables\"\n",
    "onlyfiles = [f for f in listdir(variable_filepath) if isfile(join(variable_filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{CIFAR10_MODEL_PATH}/1/variables/{filename}\", join(variable_filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "def predict_fn(x):\n",
    "    return model.predict(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "\n",
    "image_shape = (32, 32, 3)\n",
    "segmentation_fn = 'slic'\n",
    "kwargs = {'n_segments': 5, 'compactness': 20, 'sigma': .5}\n",
    "explainer = AnchorImage(predict_fn, image_shape, segmentation_fn=segmentation_fn, \n",
    "                        segmentation_kwargs=kwargs, images_background=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx=0\n",
    "image = X_test[0]\n",
    "np.random.seed(0)\n",
    "explanation = explainer.explain(image, threshold=.95, p_sample=.5, tau=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = X_test[idx].reshape(1, 32, 32, 3)\n",
    "plt.imshow(X.reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "print(\"class:\",class_names[y_test[idx][0]])\n",
    "print(\"prediction:\",class_names[model.predict(X_test[idx:idx+1])[0].argmax()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt.imshow(explanation[\"anchor\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"explainer.dill\", \"wb\") as dill_file:\n",
    "    dill.dump(explainer, dill_file)    \n",
    "    dill_file.close()\n",
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{EXPLAINER_MODEL_PATH}/explainer.dill\", 'explainer.dill'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Outlier Detector\n",
    "\n",
    "For further details and extended notebook see [Alibi-Detect Documentation](https://docs.seldon.io/projects/alibi-detect/en/stable/). These steps were derived from [Alibi-Detect CIFAR10 Example](https://docs.seldon.io/projects/alibi-detect/en/stable/examples/od_vae_cifar10.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_outlier_detector",
     "prev:train_model_and_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "tf.keras.backend.clear_session()\n",
    "from tensorflow.keras.layers import Conv2D, Conv2DTranspose, Dense, Layer, Reshape, InputLayer\n",
    "from tqdm import tqdm\n",
    "\n",
    "from alibi_detect.models.losses import elbo\n",
    "from alibi_detect.od import OutlierVAE\n",
    "from alibi_detect.utils.fetching import fetch_detector\n",
    "from alibi_detect.utils.perturbation import apply_mask\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.utils.visualize import plot_instance_score, plot_feature_outlier_image\n",
    "\n",
    "logger = tf.get_logger()\n",
    "logger.setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if TRAIN_OUTLIER_DETECTOR:\n",
    "    latent_dim = 1024\n",
    "    \n",
    "    encoder_net = tf.keras.Sequential(\n",
    "        [\n",
    "            InputLayer(input_shape=(32, 32, 3)),\n",
    "            Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu)\n",
    "        ])\n",
    "\n",
    "    decoder_net = tf.keras.Sequential(\n",
    "        [\n",
    "            InputLayer(input_shape=(latent_dim,)),\n",
    "            Dense(4*4*128),\n",
    "            Reshape(target_shape=(4, 4, 128)),\n",
    "            Conv2DTranspose(256, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2DTranspose(64, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "            Conv2DTranspose(3, 4, strides=2, padding='same', activation='sigmoid')\n",
    "        ])\n",
    "    \n",
    "     # initialize outlier detector\n",
    "    od = OutlierVAE(threshold=.015,  # threshold for outlier score\n",
    "                score_type='mse',  # use MSE of reconstruction error for outlier detection\n",
    "                encoder_net=encoder_net,  # can also pass VAE model instead\n",
    "                decoder_net=decoder_net,  # of separate encoder and decoder\n",
    "                latent_dim=latent_dim,\n",
    "                samples=2)\n",
    "    # train\n",
    "    od.fit(X_train, \n",
    "            loss_fn=elbo,\n",
    "            cov_elbo=dict(sim=.05),\n",
    "            epochs=50,\n",
    "            verbose=True)\n",
    "else:\n",
    "    od = load_detector(\"/home/models/samples/od/cifar10\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx = 8\n",
    "X = X_train[idx].reshape(1, 32, 32, 3)\n",
    "X_recon = od.vae(X)\n",
    "plt.imshow(X.reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()\n",
    "plt.imshow(X_recon.numpy().reshape(32, 32, 3))\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = X_train[:500]\n",
    "print(X.shape)\n",
    "od_preds = od.predict(X,\n",
    "                      outlier_type='instance',    # use 'feature' or 'instance' level\n",
    "                      return_feature_score=True,  # scores used to determine outliers\n",
    "                      return_instance_score=True)\n",
    "print(list(od_preds['data'].keys()))\n",
    "target = np.zeros(X.shape[0],).astype(int)  # all normal CIFAR10 training instances\n",
    "labels = ['normal', 'outlier']\n",
    "plot_instance_score(od_preds, target, labels, od.threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "filepath=\"cifar10outlier\"\n",
    "save_detector(od, filepath) \n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{OUTLIER_MODEL_PATH}/{filename}\", join(filepath, filename)))\n",
    "filepath=\"cifar10outlier/model\"\n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{OUTLIER_MODEL_PATH}/model/{filename}\", join(filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train a Drift Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_drift_detector",
     "prev:train_model_and_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Conv2D, Dense, Flatten, InputLayer, Reshape\n",
    "\n",
    "from alibi_detect.cd import KSDrift\n",
    "from alibi_detect.cd.preprocess import uae, hidden_output\n",
    "from alibi_detect.models.resnet import scale_by_instance\n",
    "from alibi_detect.utils.fetching import fetch_tf_model, fetch_detector\n",
    "from alibi_detect.utils.prediction import predict_batch\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from alibi_detect.datasets import fetch_cifar10c, corruption_types_cifar10c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(0)\n",
    "\n",
    "if True:\n",
    "    np.random.seed(0)\n",
    "    n_test = X_test.shape[0]\n",
    "    idx = np.random.choice(n_test, size=n_test // 2, replace=False)\n",
    "    idx_h0 = np.delete(np.arange(n_test), idx, axis=0)\n",
    "    X_ref,y_ref = X_test[idx], y_test[idx]\n",
    "    X_h0, y_h0 = X_test[idx_h0], y_test[idx_h0]\n",
    "    print(X_ref.shape, X_h0.shape)\n",
    "    # define encoder\n",
    "    encoding_dim = 32\n",
    "    encoder_net = tf.keras.Sequential(\n",
    "      [\n",
    "          InputLayer(input_shape=(32, 32, 3)),\n",
    "          Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "          Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "          Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "          Flatten(),\n",
    "          Dense(encoding_dim,)\n",
    "      ]\n",
    "    )\n",
    "\n",
    "    # initialise drift detector\n",
    "    p_val = .05\n",
    "    cd = KSDrift(\n",
    "        p_val=p_val,        # p-value for K-S test\n",
    "        X_ref=X_ref,       # test against original test set\n",
    "        preprocess_fn=uae,  # UAE for dimensionality reduction\n",
    "        preprocess_kwargs={'encoder_net': encoder_net, 'batch_size': 128},\n",
    "        alternative='two-sided'  # other options: 'less', 'greater'\n",
    "    )\n",
    "else:\n",
    "    cd = load_detector(\"/home/models/samples/cd/cifar10\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "filepath=\"cifar10Drift\"\n",
    "save_detector(cd, filepath) \n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{DRIFT_MODEL_PATH}/{filename}\", join(filepath, filename)))\n",
    "filepath=\"cifar10Drift/model\"\n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{DRIFT_MODEL_PATH}/model/{filename}\", join(filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy Seldon Core Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_model",
     "prev:train_model_and_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "secret = f\"\"\"apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: seldon-init-container-secret\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "type: Opaque\n",
    "stringData:\n",
    "  AWS_ACCESS_KEY_ID: {MINIO_ACCESS_KEY}\n",
    "  AWS_SECRET_ACCESS_KEY: {MINIO_SECRET_KEY}\n",
    "  AWS_ENDPOINT_URL: http://{MINIO_HOST}\n",
    "  USE_SSL: \"false\"\n",
    "\"\"\"\n",
    "with open(\"secret.yaml\",\"w\") as f:\n",
    "    f.write(secret)\n",
    "run(\"cat secret.yaml | kubectl apply -f -\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sa = f\"\"\"apiVersion: v1\n",
    "kind: ServiceAccount\n",
    "metadata:\n",
    "  name: minio-sa\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "secrets:\n",
    "  - name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"sa.yaml\",\"w\") as f:\n",
    "    f.write(sa)\n",
    "run(\"kubectl apply -f sa.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_yaml=f\"\"\"apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: cifar10-classifier\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  protocol: tensorflow\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    graph:\n",
    "      implementation: TENSORFLOW_SERVER\n",
    "      modelUri: s3://{MINIO_MODEL_BUCKET}/{CIFAR10_MODEL_PATH}\n",
    "      envSecretRefName: seldon-init-container-secret\n",
    "      name: classifier\n",
    "      logger:\n",
    "         mode: all\n",
    "    explainer:\n",
    "      type: AnchorImages\n",
    "    name: default\n",
    "    replicas: 1\n",
    "\"\"\"\n",
    "with open(\"model.yaml\",\"w\") as f:\n",
    "    f.write(model_yaml)\n",
    "run(\"kubectl apply -f model.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l seldon-deployment-id=cifar10-classifier -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l seldon-deployment-id=cifar10-classifier -o jsonpath='{{.items[1].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Make a prediction request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def test_model():\n",
    " idx=10\n",
    " test_example=X_test[idx:idx+1].tolist()\n",
    " payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    " cmd=f\"\"\"curl -d '{payload}' \\\n",
    "   http://cifar10-classifier-default.{DEPLOY_NAMESPACE}:8000/v1/models/classifier/:predict \\\n",
    "   -H \"Content-Type: application/json\"\n",
    " \"\"\"\n",
    " ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    " raw = ret.stdout.read().decode(\"utf-8\")\n",
    " print(raw)\n",
    " res=json.loads(raw)\n",
    " arr=np.array(res[\"predictions\"])\n",
    " X = X_test[idx].reshape(1, 32, 32, 3)\n",
    " plt.imshow(X.reshape(32, 32, 3))\n",
    " plt.axis('off')\n",
    " plt.show()\n",
    " print(\"class:\",class_names[y_test[idx][0]])\n",
    " print(\"prediction:\",class_names[arr[0].argmax()])\n",
    "\n",
    "ok = False\n",
    "while not ok:\n",
    "    try:\n",
    "        test_model()\n",
    "        ok = True\n",
    "    except:\n",
    "        print(\"Failed calling model, sleeping\")\n",
    "        time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Make an explanation request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx=10\n",
    "test_example=X_test[idx:idx+1].tolist()\n",
    "payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    "cmd=f\"\"\"curl -d '{payload}' \\\n",
    "   http://cifar10-classifier-default-explainer.{DEPLOY_NAMESPACE}:9000/v1/models/cifar10-classifier/:explain \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "\"\"\"\n",
    "ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "raw = ret.stdout.read().decode(\"utf-8\")\n",
    "print(raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KNative Eventing Event Display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_event_display",
     "prev:train_drift_detector",
     "prev:train_outlier_detector",
     "prev:deploy_model"
    ]
   },
   "outputs": [],
   "source": [
    "event_display=f\"\"\"apiVersion: apps/v1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}          \n",
    "spec:\n",
    "  replicas: 1\n",
    "  selector:\n",
    "    matchLabels: &labels\n",
    "      app: event-display\n",
    "  template:\n",
    "    metadata:\n",
    "      labels: *labels\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: helloworld-go\n",
    "          # Source code: https://github.com/knative/eventing-contrib/tree/master/cmd/event_display\n",
    "          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display@sha256:f4628e97a836c77ed38bd3b6fd3d0b06de4d5e7db6704772fe674d48b20bd477\n",
    "---\n",
    "kind: Service\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  selector:\n",
    "    app: event-display\n",
    "  ports:\n",
    "    - protocol: TCP\n",
    "      port: 80\n",
    "      targetPort: 8080\n",
    "---\n",
    "apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-outlier-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: io.seldon.serving.inference.outlier\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: v1\n",
    "      kind: Service\n",
    "      name: event-display\n",
    "---\n",
    "apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-drift-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: io.seldon.serving.inference.drift\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: v1\n",
    "      kind: Service\n",
    "      name: event-display\n",
    "\"\"\"\n",
    "with open(\"event_display.yaml\",\"w\") as f:\n",
    "    f.write(event_display)\n",
    "run(\"kubectl apply -f event_display.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy Seldon Outlier Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_outlier_detector",
     "prev:deploy_event_display"
    ]
   },
   "outputs": [],
   "source": [
    "outlier_yaml=f\"\"\"apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: cifar10-outlier\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:1.2.1\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - cifar10od\n",
    "        - --protocol\n",
    "        - tensorflow.http\n",
    "        - --storage_uri\n",
    "        - s3://{MINIO_MODEL_BUCKET}/{OUTLIER_MODEL_PATH}\n",
    "        - --reply_url\n",
    "        - http://default-broker       \n",
    "        - --event_type\n",
    "        - io.seldon.serving.inference.outlier\n",
    "        - --event_source\n",
    "        - io.seldon.serving.cifar10od\n",
    "        - OutlierDetector\n",
    "        envFrom:\n",
    "        - secretRef:\n",
    "            name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"outlier.yaml\",\"w\") as f:\n",
    "    f.write(outlier_yaml)\n",
    "run(\"kubectl apply -f outlier.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trigger_outlier_yaml=f\"\"\"apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-outlier-trigger\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  filter:\n",
    "    sourceAndType:\n",
    "      type: io.seldon.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1\n",
    "      kind: Service\n",
    "      name: cifar10-outlier\n",
    "\"\"\"\n",
    "with open(\"outlier_trigger.yaml\",\"w\") as f:\n",
    "    f.write(trigger_outlier_yaml)\n",
    "run(\"kubectl apply -f outlier_trigger.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l serving.knative.dev/service=cifar10-outlier -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test Seldon Outlier Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_oulier_detection",
     "prev:deploy_outlier_detector"
    ]
   },
   "outputs": [],
   "source": [
    "idx = 1\n",
    "X = X_train[idx:idx+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0) \n",
    "X_mask, mask = apply_mask(X.reshape(1, 32, 32, 3),\n",
    "                                  mask_size=(10,10),\n",
    "                                  n_masks=1,\n",
    "                                  channels=[0,1,2],\n",
    "                                  mask_type='normal',\n",
    "                                  noise_distr=(0,1),\n",
    "                                  clip_rng=(0,1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def predict():\n",
    "    test_example=X_mask.tolist()\n",
    "    payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    "    cmd=f\"\"\"curl -d '{payload}' \\\n",
    "       http://cifar10-classifier-default.{DEPLOY_NAMESPACE}:8000/v1/models/classifier/:predict \\\n",
    "       -H \"Content-Type: application/json\"\n",
    "    \"\"\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    raw = ret.stdout.read().decode(\"utf-8\")\n",
    "    print(raw)\n",
    "    res=json.loads(raw)\n",
    "    arr=np.array(res[\"predictions\"])\n",
    "    plt.imshow(X_mask.reshape(32, 32, 3))\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "    print(\"class:\",class_names[y_train[idx][0]])\n",
    "    print(\"prediction:\",class_names[arr[0].argmax()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_outlier_event_display_logs():\n",
    "    cmd=f\"kubectl logs $(kubectl get pod -l app=event-display -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE}) -n {DEPLOY_NAMESPACE}\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    res = ret.stdout.read().decode(\"utf-8\").split(\"\\n\")\n",
    "    data= []\n",
    "    for i in range(0,len(res)):\n",
    "        if res[i] == 'Data,':\n",
    "            j = json.loads(json.loads(res[i+1]))\n",
    "            if \"is_outlier\"in j[\"data\"].keys():\n",
    "                data.append(j)\n",
    "    if len(data) > 0:\n",
    "        return data[-1]\n",
    "    else:\n",
    "        return None\n",
    "j = None\n",
    "while j is None:\n",
    "    predict()\n",
    "    print(\"Waiting for outlier logs, sleeping\")\n",
    "    time.sleep(2)\n",
    "    j = get_outlier_event_display_logs()\n",
    "    \n",
    "print(j)\n",
    "print(\"Outlier\",j[\"data\"][\"is_outlier\"]==[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy Seldon Drift Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_drift_detector",
     "prev:test_oulier_detection"
    ]
   },
   "outputs": [],
   "source": [
    "drift_yaml=f\"\"\"apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: cifar10-drift\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:1.2.2-dev\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - cifar10cd\n",
    "        - --protocol\n",
    "        - tensorflow.http\n",
    "        - --storage_uri\n",
    "        - s3://{MINIO_MODEL_BUCKET}/{DRIFT_MODEL_PATH}\n",
    "        - --reply_url\n",
    "        - http://default-broker\n",
    "        - --event_type\n",
    "        - io.seldon.serving.inference.drift\n",
    "        - --event_source\n",
    "        - io.seldon.serving.cifar10cd\n",
    "        - DriftDetector\n",
    "        - --drift_batch_size\n",
    "        - '500'\n",
    "        envFrom:\n",
    "        - secretRef:\n",
    "            name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"drift.yaml\",\"w\") as f:\n",
    "    f.write(drift_yaml)\n",
    "run(\"kubectl apply -f drift.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trigger_outlier_yaml=f\"\"\"apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: cifar10-drift-trigger\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  filter:\n",
    "    sourceAndType:\n",
    "      type: io.seldon.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1\n",
    "      kind: Service\n",
    "      name: cifar10-drift\n",
    "\"\"\"\n",
    "with open(\"outlier_trigger.yaml\",\"w\") as f:\n",
    "    f.write(trigger_outlier_yaml)\n",
    "run(\"kubectl apply -f outlier_trigger.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l serving.knative.dev/service=cifar10-drift -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test Seldon Drift Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_drift_detector",
     "prev:deploy_drift_detector"
    ]
   },
   "outputs": [],
   "source": [
    "def show(X):\n",
    "    plt.imshow(X.reshape(32, 32, 3))\n",
    "    plt.axis('off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.datasets import fetch_cifar10c, corruption_types_cifar10c\n",
    "corruption = ['motion_blur']\n",
    "X_corr, y_corr = fetch_cifar10c(corruption=corruption, severity=5, return_X_y=True)\n",
    "X_corr = X_corr.astype('float32') / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "show(X_corr[0])\n",
    "show(X_corr[1])\n",
    "show(X_corr[2])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def predict(X):\n",
    "    test_example=X.tolist()\n",
    "    payload='{\"instances\":'+f\"{test_example}\"+' }'\n",
    "    with open(\"payload.json\",\"w\") as f:\n",
    "        f.write(payload)\n",
    "    cmd=f\"\"\"curl -d @./payload.json \\\n",
    "   http://cifar10-classifier-default.{DEPLOY_NAMESPACE}:8000/v1/models/classifier/:predict \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "    \"\"\"\n",
    "    run(cmd, shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_drift_event_display_logs():\n",
    "    cmd=f\"kubectl logs $(kubectl get pod -l app=event-display -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE}) -n {DEPLOY_NAMESPACE}\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    res = ret.stdout.read().decode(\"utf-8\").split(\"\\n\")\n",
    "    data= []\n",
    "    for i in range(0,len(res)):\n",
    "        if res[i] == 'Data,':\n",
    "            j = json.loads(json.loads(res[i+1]))\n",
    "            if \"is_drift\"in j[\"data\"].keys():\n",
    "                data.append(j)\n",
    "    if len(data) > 0:\n",
    "        return data[-1]\n",
    "    else:\n",
    "        return None\n",
    "j = None\n",
    "for i in range(0,1000,50):\n",
    "    X = X_corr[i:i+50]\n",
    "    predict(X)\n",
    "    print(\"Waiting for drift logs, sleeping\")\n",
    "    time.sleep(2)\n",
    "    j = get_drift_event_display_logs()\n",
    "    if j is not None:\n",
    "        break\n",
    "    \n",
    "print(j)\n",
    "print(\"Drift\",j[\"data\"][\"is_drift\"]==1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl delete sdep cifar10-classifier -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete ksvc cifar10-outlier -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete ksvc cifar10-drift -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete trigger --all -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete sa  minio-sa -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete secret seldon-init-container-secret -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete deployment event-display -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete svc event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kubeflow_notebook": {
   "docker_image": "seldonio/jupyter-lab-alibi-kale:0.11",
   "experiment": {
    "id": "new",
    "name": "seldon-e2e-cifar10"
   },
   "experiment_name": "seldon-e2e-cifar10",
   "katib_metadata": {
    "algorithm": {
     "algorithmName": "grid"
    },
    "maxFailedTrialCount": 3,
    "maxTrialCount": 12,
    "objective": {
     "objectiveMetricName": "",
     "type": "minimize"
    },
    "parallelTrialCount": 3,
    "parameters": []
   },
   "katib_run": false,
   "pipeline_description": "Seldon CIFAR10 Example",
   "pipeline_name": "seldon-e2e-cifar10",
   "volumes": []
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
