{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# End to End Machine Learning Pipeline for Income Prediction\n",
    "\n",
    "We use [demographic features from the 1996 US census](https://archive.ics.uci.edu/ml/datasets/census+income) to build an end to end machine learning pipeline. The pipeline is also annotated so it can be run as a [Kubeflow Pipeline](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/) using the [Kale](https://github.com/kubeflow-kale/kale) pipeline generator.\n",
    "\n",
    "The notebook/pipeline stages are:\n",
    "\n",
    " 1. Setup \n",
    "   * Imports\n",
    "   * pipeline-parameters\n",
    "   * minio client test\n",
    " 1. Train a simple sklearn model and push to minio\n",
    " 1. Prepare an Anchors explainer for model and push to minio\n",
    " 1. Test Explainer\n",
    " 1. Train an isolation forest outlier detector for model and push to minio\n",
    " 1. Deploy a Seldon model and test\n",
    " 1. Deploy a KfServing model and test\n",
    " 1. Deploy an outlier detector \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "imports"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from alibi.explainers import AnchorTabular\n",
    "from alibi.datasets import fetch_adult\n",
    "from minio import Minio\n",
    "from minio.error import ResponseError\n",
    "from joblib import dump, load\n",
    "import dill\n",
    "import time\n",
    "import json\n",
    "from subprocess import run, Popen, PIPE\n",
    "from alibi_detect.utils.data import create_outlier_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pipeline-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "MINIO_HOST=\"minio-service.kubeflow:9000\"\n",
    "MINIO_ACCESS_KEY=\"minio\"\n",
    "MINIO_SECRET_KEY=\"minio123\"\n",
    "MINIO_MODEL_BUCKET=\"seldon\"\n",
    "INCOME_MODEL_PATH=\"sklearn/income/model\"\n",
    "EXPLAINER_MODEL_PATH=\"sklearn/income/explainer\"\n",
    "OUTLIER_MODEL_PATH=\"sklearn/income/outlier\"\n",
    "DEPLOY_NAMESPACE=\"admin\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "functions"
    ]
   },
   "outputs": [],
   "source": [
    "def get_minio():\n",
    "    return Minio(MINIO_HOST,\n",
    "                    access_key=MINIO_ACCESS_KEY,\n",
    "                    secret_key=MINIO_SECRET_KEY,\n",
    "                    secure=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:setup"
    ]
   },
   "outputs": [],
   "source": [
    "minioClient = get_minio()\n",
    "buckets = minioClient.list_buckets()\n",
    "for bucket in buckets:\n",
    "    print(bucket.name, bucket.creation_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not minioClient.bucket_exists(MINIO_MODEL_BUCKET):\n",
    "    minioClient.make_bucket(MINIO_MODEL_BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:build_model",
     "prev:setup"
    ]
   },
   "outputs": [],
   "source": [
    "adult = fetch_adult()\n",
    "adult.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "data = adult.data\n",
    "target = adult.target\n",
    "feature_names = adult.feature_names\n",
    "category_map = adult.category_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Note that for your own datasets you can use our utility function [gen_category_map](../api/alibi.utils.data.rst) to create the category map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi.utils.data import gen_category_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Define shuffled training and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "data_perm = np.random.permutation(np.c_[data, target])\n",
    "data = data_perm[:,:-1]\n",
    "target = data_perm[:,-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx = 30000\n",
    "X_train,Y_train = data[:idx,:], target[:idx]\n",
    "X_test, Y_test = data[idx+1:,:], target[idx+1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Create feature transformation pipeline\n",
    "Create feature pre-processor. Needs to have 'fit' and 'transform' methods. Different types of pre-processing can be applied to all or part of the features. In the example below we will standardize ordinal features and apply one-hot-encoding to categorical features.\n",
    "\n",
    "Ordinal features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ordinal_features = [x for x in range(len(feature_names)) if x not in list(category_map.keys())]\n",
    "ordinal_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),\n",
    "                                      ('scaler', StandardScaler())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Categorical features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "categorical_features = list(category_map.keys())\n",
    "categorical_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),\n",
    "                                          ('onehot', OneHotEncoder(handle_unknown='ignore'))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Combine and fit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer(transformers=[('num', ordinal_transformer, ordinal_features),\n",
    "                                               ('cat', categorical_transformer, categorical_features)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Train Random Forest model\n",
    "\n",
    "Fit on pre-processed (imputing, OHE, standardizing) data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "clf = RandomForestClassifier(n_estimators=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model=Pipeline(steps=[(\"preprocess\",preprocessor),(\"model\",clf)])\n",
    "model.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Define predict function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "def predict_fn(x):\n",
    "    return model.predict(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:",
     "prev:build_model"
    ]
   },
   "outputs": [],
   "source": [
    "#predict_fn = lambda x: clf.predict(preprocessor.transform(x))\n",
    "print('Train accuracy: ', accuracy_score(Y_train, predict_fn(X_train)))\n",
    "print('Test accuracy: ', accuracy_score(Y_test, predict_fn(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dump(model, 'model.joblib') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{INCOME_MODEL_PATH}/model.joblib\", 'model.joblib'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_explainer",
     "prev:build_model"
    ]
   },
   "outputs": [],
   "source": [
    "model.predict(X_train)\n",
    "explainer = AnchorTabular(predict_fn, feature_names, categorical_names=category_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Discretize the ordinal features into quartiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "explainer.fit(X_train, disc_perc=[25, 50, 75])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"explainer.dill\", \"wb\") as dill_file:\n",
    "    dill.dump(explainer, dill_file)    \n",
    "    dill_file.close()\n",
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{EXPLAINER_MODEL_PATH}/explainer.dill\", 'explainer.dill'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Get Explanation\n",
    "\n",
    "Below, we get an anchor for the prediction of the first observation in the test set. An anchor is a sufficient condition - that is, when the anchor holds, the prediction should be the same as the prediction for this instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:explain",
     "prev:train_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "model.predict(X_train)\n",
    "idx = 0\n",
    "class_names = adult.target_names\n",
    "print('Prediction: ', class_names[explainer.predict_fn(X_test[idx].reshape(1, -1))[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "We set the precision threshold to 0.95. This means that predictions on observations where the anchor holds will be the same as the prediction on the explained instance at least 95% of the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "explanation = explainer.explain(X_test[idx], threshold=0.95)\n",
    "print('Anchor: %s' % (' AND '.join(explanation['names'])))\n",
    "print('Precision: %.2f' % explanation['precision'])\n",
    "print('Coverage: %.2f' % explanation['coverage'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Outlier Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:build_outlier",
     "prev:build_model"
    ]
   },
   "outputs": [],
   "source": [
    "from alibi_detect.od import IForest\n",
    "\n",
    "od = IForest(\n",
    "    threshold=0.,\n",
    "    n_estimators=200,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "od.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "perc_outlier = 5\n",
    "threshold_batch = create_outlier_batch(X_train, Y_train, n_samples=1000, perc_outlier=perc_outlier)\n",
    "X_threshold, y_threshold = threshold_batch.data.astype('float'), threshold_batch.target\n",
    "#X_threshold = (X_threshold - mean) / stdev\n",
    "print('{}% outliers'.format(100 * y_threshold.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "od.infer_threshold(X_threshold, threshold_perc=100-perc_outlier)\n",
    "print('New threshold: {}'.format(od.threshold))\n",
    "threshold = od.threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_outlier = [[300,  4,  4,  2,  1,  4,  4,  0,  0,  0, 600,  9]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "od.predict(\n",
    "    X_outlier\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "filepath=\"ifoutlier\"\n",
    "save_detector(od, filepath) \n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{OUTLIER_MODEL_PATH}/{filename}\", join(filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy Seldon Core Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_seldon",
     "prev:train_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "secret = f\"\"\"apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: seldon-init-container-secret\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "type: Opaque\n",
    "stringData:\n",
    "  AWS_ACCESS_KEY_ID: {MINIO_ACCESS_KEY}\n",
    "  AWS_SECRET_ACCESS_KEY: {MINIO_SECRET_KEY}\n",
    "  AWS_ENDPOINT_URL: http://{MINIO_HOST}\n",
    "  USE_SSL: \"false\"\n",
    "\"\"\"\n",
    "with open(\"secret.yaml\",\"w\") as f:\n",
    "    f.write(secret)\n",
    "run(\"cat secret.yaml | kubectl apply -f -\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sa = f\"\"\"apiVersion: v1\n",
    "kind: ServiceAccount\n",
    "metadata:\n",
    "  name: minio-sa\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "secrets:\n",
    "  - name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"sa.yaml\",\"w\") as f:\n",
    "    f.write(sa)\n",
    "run(\"kubectl apply -f sa.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_yaml=f\"\"\"apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: income-classifier\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    graph:\n",
    "      implementation: SKLEARN_SERVER\n",
    "      modelUri: s3://{MINIO_MODEL_BUCKET}/{INCOME_MODEL_PATH}\n",
    "      envSecretRefName: seldon-init-container-secret\n",
    "      name: classifier\n",
    "      logger:\n",
    "         mode: all\n",
    "    explainer:\n",
    "      type: AnchorTabular\n",
    "      modelUri: s3://{MINIO_MODEL_BUCKET}/{EXPLAINER_MODEL_PATH}\n",
    "      envSecretRefName: seldon-init-container-secret\n",
    "    name: default\n",
    "    replicas: 1\n",
    "\"\"\"\n",
    "with open(\"model.yaml\",\"w\") as f:\n",
    "    f.write(model_yaml)\n",
    "run(\"kubectl apply -f model.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l seldon-deployment-id=income-classifier -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l seldon-deployment-id=income-classifier -o jsonpath='{{.items[1].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Make a prediction request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_model",
     "prev:deploy_seldon"
    ]
   },
   "outputs": [],
   "source": [
    "payload='{\"data\": {\"ndarray\": [[53,4,0,2,8,4,4,0,0,0,60,9]]}}'\n",
    "cmd=f\"\"\"curl -d '{payload}' \\\n",
    "   http://income-classifier-default.{DEPLOY_NAMESPACE}:8000/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "\"\"\"\n",
    "ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "raw = ret.stdout.read().decode(\"utf-8\")\n",
    "print(raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Make an explanation request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "payload='{\"data\": {\"ndarray\": [[53,4,0,2,8,4,4,0,0,0,60,9]]}}'\n",
    "cmd=f\"\"\"curl -d '{payload}' \\\n",
    "   http://income-classifier-default-explainer.{DEPLOY_NAMESPACE}:9000/api/v1.0/explain \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "\"\"\"\n",
    "ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "raw = ret.stdout.read().decode(\"utf-8\")\n",
    "print(raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy Outier Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_outlier",
     "prev:build_outlier",
     "prev:test_model"
    ]
   },
   "outputs": [],
   "source": [
    "outlier_yaml=f\"\"\"apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: income-outlier\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:1.2.2-dev_alibidetect\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - adultod\n",
    "        - --http_port\n",
    "        - '8080'\n",
    "        - --protocol\n",
    "        - seldon.http\n",
    "        - --storage_uri\n",
    "        - s3://{MINIO_MODEL_BUCKET}/{OUTLIER_MODEL_PATH}\n",
    "        - --reply_url\n",
    "        - http://default-broker       \n",
    "        - --event_type\n",
    "        - io.seldon.serving.inference.outlier\n",
    "        - --event_source\n",
    "        - io.seldon.serving.incomeod\n",
    "        - OutlierDetector\n",
    "        envFrom:\n",
    "        - secretRef:\n",
    "            name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"outlier.yaml\",\"w\") as f:\n",
    "    f.write(outlier_yaml)\n",
    "run(\"kubectl apply -f outlier.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trigger_outlier_yaml=f\"\"\"apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: income-outlier-trigger\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  filter:\n",
    "    sourceAndType:\n",
    "      type: io.seldon.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1alpha1\n",
    "      kind: Service\n",
    "      name: income-outlier\n",
    "\"\"\"\n",
    "with open(\"outlier_trigger.yaml\",\"w\") as f:\n",
    "    f.write(trigger_outlier_yaml)\n",
    "run(\"kubectl apply -f outlier_trigger.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l serving.knative.dev/service=income-outlier -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KNative Eventing Event Display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_event_display",
     "prev:deploy_outlier"
    ]
   },
   "outputs": [],
   "source": [
    "event_display=f\"\"\"apiVersion: apps/v1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}          \n",
    "spec:\n",
    "  replicas: 1\n",
    "  selector:\n",
    "    matchLabels: &labels\n",
    "      app: event-display\n",
    "  template:\n",
    "    metadata:\n",
    "      labels: *labels\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: helloworld-go\n",
    "          # Source code: https://github.com/knative/eventing-contrib/tree/master/cmd/event_display\n",
    "          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display@sha256:f4628e97a836c77ed38bd3b6fd3d0b06de4d5e7db6704772fe674d48b20bd477\n",
    "---\n",
    "kind: Service\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  selector:\n",
    "    app: event-display\n",
    "  ports:\n",
    "    - protocol: TCP\n",
    "      port: 80\n",
    "      targetPort: 8080\n",
    "---\n",
    "apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: income-outlier-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: io.seldon.serving.inference.outlier\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: v1\n",
    "      kind: Service\n",
    "      name: event-display\n",
    "\"\"\"\n",
    "with open(\"event_display.yaml\",\"w\") as f:\n",
    "    f.write(event_display)\n",
    "run(\"kubectl apply -f event_display.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test Outlier Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_outliers",
     "prev:deploy_event_display"
    ]
   },
   "outputs": [],
   "source": [
    "def predict():\n",
    "    payload='{\"data\": {\"ndarray\": [[300,  4,  4,  2,  1,  4,  4,  0,  0,  0, 600,  9]]}}'\n",
    "    cmd=f\"\"\"curl -d '{payload}' \\\n",
    "       http://income-classifier-default.{DEPLOY_NAMESPACE}:8000/api/v1.0/predictions \\\n",
    "       -H \"Content-Type: application/json\"\n",
    "    \"\"\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    raw = ret.stdout.read().decode(\"utf-8\")\n",
    "    print(raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_outlier_event_display_logs():\n",
    "    cmd=f\"kubectl logs $(kubectl get pod -l app=event-display -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE}) -n {DEPLOY_NAMESPACE}\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    res = ret.stdout.read().decode(\"utf-8\").split(\"\\n\")\n",
    "    data= []\n",
    "    for i in range(0,len(res)):\n",
    "        if res[i] == 'Data,':\n",
    "            j = json.loads(json.loads(res[i+1]))\n",
    "            if \"is_outlier\"in j[\"data\"].keys():\n",
    "                data.append(j)\n",
    "    if len(data) > 0:\n",
    "        return data[-1]\n",
    "    else:\n",
    "        return None\n",
    "j = None\n",
    "while j is None:\n",
    "    predict()\n",
    "    print(\"Waiting for outlier logs, sleeping\")\n",
    "    time.sleep(2)\n",
    "    j = get_outlier_event_display_logs()\n",
    "    \n",
    "print(j)\n",
    "print(\"Outlier\",j[\"data\"][\"is_outlier\"]==[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Clean Up Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl delete sdep income-classifier -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete ksvc income-outlier -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete sa  minio-sa -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete secret seldon-init-container-secret -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete deployment event-display -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete svc event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kubeflow_notebook": {
   "docker_image": "seldonio/jupyter-lab-alibi-kale:0.11",
   "experiment": {
    "id": "new",
    "name": "seldon-e2e-adult"
   },
   "experiment_name": "seldon-e2e-adult",
   "katib_metadata": {
    "algorithm": {
     "algorithmName": "grid"
    },
    "maxFailedTrialCount": 3,
    "maxTrialCount": 12,
    "objective": {
     "objectiveMetricName": "",
     "type": "minimize"
    },
    "parallelTrialCount": 3,
    "parameters": []
   },
   "katib_run": false,
   "pipeline_description": "Seldon e2e adult",
   "pipeline_name": "seldon-e2e-adult",
   "volumes": []
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
