{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# End to End Machine Learning Pipeline for Income Prediction\n",
    "\n",
    "We use [demographic features from the 1996 US census](https://archive.ics.uci.edu/ml/datasets/census+income) to build an end to end machine learning pipeline. The pipeline is also annotated so it can be run as a [Kubeflow Pipeline](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/) using the [Kale](https://github.com/kubeflow-kale/kale) pipeline generator.\n",
    "\n",
    "The notebook/pipeline stages are:\n",
    "\n",
    " 1. Setup \n",
    "   * Imports\n",
    "   * pipeline-parameters\n",
    "   * minio client test\n",
    " 1. Train a simple sklearn model and push to minio\n",
    " 1. Prepare an Anchors explainer for model and push to minio\n",
    " 1. Test Explainer\n",
    " 1. Train an isolation forest outlier detector for model and push to minio\n",
    " 1. Deploy a KfSering model and test\n",
    " 1. Deploy an outlier detector and test\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "imports"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from alibi.explainers import AnchorTabular\n",
    "from alibi.datasets import fetch_adult\n",
    "from minio import Minio\n",
    "from minio.error import ResponseError\n",
    "from joblib import dump, load\n",
    "import dill\n",
    "import time\n",
    "import json\n",
    "from subprocess import run, Popen, PIPE\n",
    "from alibi_detect.utils.data import create_outlier_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pipeline-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "MINIO_HOST=\"minio-service.kubeflow:9000\"\n",
    "MINIO_ACCESS_KEY=\"minio\"\n",
    "MINIO_SECRET_KEY=\"minio123\"\n",
    "MINIO_MODEL_BUCKET=\"seldon\"\n",
    "INCOME_MODEL_PATH=\"sklearn/income/model\"\n",
    "EXPLAINER_MODEL_PATH=\"sklearn/income/explainer\"\n",
    "OUTLIER_MODEL_PATH=\"sklearn/income/outlier\"\n",
    "DEPLOY_NAMESPACE=\"admin\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "functions"
    ]
   },
   "outputs": [],
   "source": [
    "def get_minio():\n",
    "    return Minio(MINIO_HOST,\n",
    "                    access_key=MINIO_ACCESS_KEY,\n",
    "                    secret_key=MINIO_SECRET_KEY,\n",
    "                    secure=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:setup"
    ]
   },
   "outputs": [],
   "source": [
    "minioClient = get_minio()\n",
    "buckets = minioClient.list_buckets()\n",
    "for bucket in buckets:\n",
    "    print(bucket.name, bucket.creation_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not minioClient.bucket_exists(MINIO_MODEL_BUCKET):\n",
    "    minioClient.make_bucket(MINIO_MODEL_BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:build_model",
     "prev:setup"
    ]
   },
   "outputs": [],
   "source": [
    "adult = fetch_adult()\n",
    "adult.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "data = adult.data\n",
    "target = adult.target\n",
    "feature_names = adult.feature_names\n",
    "category_map = adult.category_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Note that for your own datasets you can use our utility function [gen_category_map](../api/alibi.utils.data.rst) to create the category map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi.utils.data import gen_category_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Define shuffled training and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "data_perm = np.random.permutation(np.c_[data, target])\n",
    "data = data_perm[:,:-1]\n",
    "target = data_perm[:,-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "idx = 30000\n",
    "X_train,Y_train = data[:idx,:], target[:idx]\n",
    "X_test, Y_test = data[idx+1:,:], target[idx+1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Create feature transformation pipeline\n",
    "Create feature pre-processor. Needs to have 'fit' and 'transform' methods. Different types of pre-processing can be applied to all or part of the features. In the example below we will standardize ordinal features and apply one-hot-encoding to categorical features.\n",
    "\n",
    "Ordinal features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ordinal_features = [x for x in range(len(feature_names)) if x not in list(category_map.keys())]\n",
    "ordinal_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),\n",
    "                                      ('scaler', StandardScaler())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Categorical features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "categorical_features = list(category_map.keys())\n",
    "categorical_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),\n",
    "                                          ('onehot', OneHotEncoder(handle_unknown='ignore'))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Combine and fit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer(transformers=[('num', ordinal_transformer, ordinal_features),\n",
    "                                               ('cat', categorical_transformer, categorical_features)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Train Random Forest model\n",
    "\n",
    "Fit on pre-processed (imputing, OHE, standardizing) data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "clf = RandomForestClassifier(n_estimators=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model=Pipeline(steps=[(\"preprocess\",preprocessor),(\"model\",clf)])\n",
    "model.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Define predict function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [],
   "source": [
    "def predict_fn(x):\n",
    "    return model.predict(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:",
     "prev:build_model"
    ]
   },
   "outputs": [],
   "source": [
    "#predict_fn = lambda x: clf.predict(preprocessor.transform(x))\n",
    "print('Train accuracy: ', accuracy_score(Y_train, predict_fn(X_train)))\n",
    "print('Test accuracy: ', accuracy_score(Y_test, predict_fn(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dump(model, 'model.joblib') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{INCOME_MODEL_PATH}/model.joblib\", 'model.joblib'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:train_explainer",
     "prev:build_model"
    ]
   },
   "outputs": [],
   "source": [
    "model.predict(X_train)\n",
    "explainer = AnchorTabular(predict_fn, feature_names, categorical_names=category_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Discretize the ordinal features into quartiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "explainer.fit(X_train, disc_perc=[25, 50, 75])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"explainer.dill\", \"wb\") as dill_file:\n",
    "    dill.dump(explainer, dill_file)    \n",
    "    dill_file.close()\n",
    "print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{EXPLAINER_MODEL_PATH}/explainer.dill\", 'explainer.dill'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Get Explanation\n",
    "\n",
    "Below, we get an anchor for the prediction of the first observation in the test set. An anchor is a sufficient condition - that is, when the anchor holds, the prediction should be the same as the prediction for this instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:explain",
     "prev:train_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "model.predict(X_train)\n",
    "idx = 0\n",
    "class_names = adult.target_names\n",
    "print('Prediction: ', class_names[explainer.predict_fn(X_test[idx].reshape(1, -1))[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "We set the precision threshold to 0.95. This means that predictions on observations where the anchor holds will be the same as the prediction on the explained instance at least 95% of the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "explanation = explainer.explain(X_test[idx], threshold=0.95)\n",
    "print('Anchor: %s' % (' AND '.join(explanation['names'])))\n",
    "print('Precision: %.2f' % explanation['precision'])\n",
    "print('Coverage: %.2f' % explanation['coverage'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train Outlier Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:build_outlier",
     "prev:build_model"
    ]
   },
   "outputs": [],
   "source": [
    "from alibi_detect.od import IForest\n",
    "\n",
    "od = IForest(\n",
    "    threshold=0.,\n",
    "    n_estimators=200,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "od.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "perc_outlier = 5\n",
    "threshold_batch = create_outlier_batch(X_train, Y_train, n_samples=1000, perc_outlier=perc_outlier)\n",
    "X_threshold, y_threshold = threshold_batch.data.astype('float'), threshold_batch.target\n",
    "#X_threshold = (X_threshold - mean) / stdev\n",
    "print('{}% outliers'.format(100 * y_threshold.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "od.infer_threshold(X_threshold, threshold_perc=100-perc_outlier)\n",
    "print('New threshold: {}'.format(od.threshold))\n",
    "threshold = od.threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_outlier = [[300,  4,  4,  2,  1,  4,  4,  0,  0,  0, 600,  9]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "od.predict(\n",
    "    X_outlier\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "\n",
    "filepath=\"ifoutlier\"\n",
    "save_detector(od, filepath) \n",
    "onlyfiles = [f for f in listdir(filepath) if isfile(join(filepath, f))]\n",
    "for filename in onlyfiles:\n",
    "    print(filename)\n",
    "    print(get_minio().fput_object(MINIO_MODEL_BUCKET, f\"{OUTLIER_MODEL_PATH}/{filename}\", join(filepath, filename)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KFServing Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_kfserving",
     "prev:train_explainer"
    ]
   },
   "outputs": [],
   "source": [
    "secret=f\"\"\"apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: income-kf-secret\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "  annotations:\n",
    "     serving.kubeflow.org/s3-endpoint: {MINIO_HOST} # replace with your s3 endpoint\n",
    "     serving.kubeflow.org/s3-usehttps: \"0\" # by default 1, for testing with minio you need to set to 0\n",
    "type: Opaque\n",
    "stringData:\n",
    "  awsAccessKeyID: {MINIO_ACCESS_KEY}\n",
    "  awsSecretAccessKey: {MINIO_SECRET_KEY}\n",
    "\"\"\"\n",
    "with open(\"secret.yaml\",\"w\") as f:\n",
    "    f.write(secret)\n",
    "run(\"kubectl apply -f secret.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "secret = f\"\"\"apiVersion: v1\n",
    "kind: Secret\n",
    "metadata:\n",
    "  name: seldon-init-container-secret\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "type: Opaque\n",
    "stringData:\n",
    "  AWS_ACCESS_KEY_ID: {MINIO_ACCESS_KEY}\n",
    "  AWS_SECRET_ACCESS_KEY: {MINIO_SECRET_KEY}\n",
    "  AWS_ENDPOINT_URL: http://{MINIO_HOST}\n",
    "  USE_SSL: \"false\"\n",
    "\"\"\"\n",
    "with open(\"secret.yaml\",\"w\") as f:\n",
    "    f.write(secret)\n",
    "run(\"cat secret.yaml | kubectl apply -f -\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sa = f\"\"\"apiVersion: v1\n",
    "kind: ServiceAccount\n",
    "metadata:\n",
    "  name: minio-kf-sa\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "secrets:\n",
    "  - name: income-kf-secret\n",
    "\"\"\"\n",
    "with open(\"sa.yaml\",\"w\") as f:\n",
    "    f.write(sa)\n",
    "run(\"kubectl apply -f sa.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from kubernetes import client\n",
    "from kfserving import KFServingClient\n",
    "from kfserving import constants\n",
    "from kfserving import utils\n",
    "from kfserving import V1alpha2EndpointSpec\n",
    "from kfserving import V1alpha2PredictorSpec\n",
    "from kfserving import V1alpha2ExplainerSpec\n",
    "from kfserving import V1alpha2AlibiExplainerSpec\n",
    "from kfserving import V1alpha2SKLearnSpec\n",
    "from kfserving import V1alpha2InferenceServiceSpec\n",
    "from kfserving import V1alpha2InferenceService\n",
    "from kfserving import V1alpha2Logger\n",
    "from kubernetes.client import V1ResourceRequirements\n",
    "\n",
    "api_version = constants.KFSERVING_GROUP + '/' + constants.KFSERVING_VERSION\n",
    "default_endpoint_spec = V1alpha2EndpointSpec(\n",
    "                          predictor=V1alpha2PredictorSpec(\n",
    "                            service_account_name='minio-kf-sa',\n",
    "                            sklearn=V1alpha2SKLearnSpec(\n",
    "                              storage_uri='s3://'+MINIO_MODEL_BUCKET+'/'+ INCOME_MODEL_PATH,\n",
    "                              resources=V1ResourceRequirements(\n",
    "                                  requests={'cpu':'100m','memory':'1Gi'},\n",
    "                                  limits={'cpu':'100m', 'memory':'1Gi'})),\n",
    "                            logger=V1alpha2Logger(\n",
    "                                mode='all'\n",
    "                            )),\n",
    "                            explainer=V1alpha2ExplainerSpec(\n",
    "                              service_account_name='minio-kf-sa',\n",
    "                            alibi=V1alpha2AlibiExplainerSpec(\n",
    "                              type='AnchorTabular',\n",
    "                              storage_uri='s3://'+MINIO_MODEL_BUCKET+'/'+ EXPLAINER_MODEL_PATH,\n",
    "                              resources=V1ResourceRequirements(\n",
    "                                  requests={'cpu':'100m','memory':'1Gi'},\n",
    "                                  limits={'cpu':'100m', 'memory':'1Gi'}))))\n",
    "    \n",
    "isvc = V1alpha2InferenceService(api_version=api_version,\n",
    "                          kind=constants.KFSERVING_KIND,\n",
    "                          metadata=client.V1ObjectMeta(\n",
    "                              name='kf-income', namespace=DEPLOY_NAMESPACE),\n",
    "                          spec=V1alpha2InferenceServiceSpec(default=default_endpoint_spec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "KFServing = KFServingClient()\n",
    "KFServing.create(isvc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "KFServing.get('kf-cifar10', namespace=DEPLOY_NAMESPACE, watch=True, timeout_seconds=120)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test Model and explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_model",
     "prev:deploy_kfserving"
    ]
   },
   "outputs": [],
   "source": [
    "payload='{\"instances\": [[53,4,0,2,8,4,4,0,0,0,60,9]]}'\n",
    "cmd=f\"\"\"curl -v -d '{payload}' \\\n",
    "   -H \"Host: kf-income.admin.example.com\" \\\n",
    "   -H \"Content-Type: application/json\" \\\n",
    "   http://kfserving-ingressgateway.istio-system/v1/models/kf-income:predict\n",
    "\"\"\"\n",
    "ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "raw = ret.stdout.read().decode(\"utf-8\")\n",
    "print(raw)\n",
    "res=json.loads(raw)\n",
    "arr=np.array(res[\"predictions\"])\n",
    "if arr[0] > 0:\n",
    "    print(\"Prediction: High Income\")\n",
    "else:\n",
    "    print(\"Prediction: Low Income\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Make an explanation request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "payload='{\"instances\": [[53,4,0,2,8,4,4,0,0,0,60,9]]}'\n",
    "cmd=f\"\"\"curl -v -d '{payload}' \\\n",
    "   -H \"Host: kf-income.admin.example.com\" \\\n",
    "   -H \"Content-Type: application/json\" \\\n",
    "   http://kfserving-ingressgateway.istio-system/v1/models/kf-income:explain\n",
    "\"\"\"\n",
    "ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "raw = ret.stdout.read().decode(\"utf-8\")\n",
    "res=json.loads(raw)\n",
    "print(res[\"names\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy Outier Detector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_outlier",
     "prev:build_outlier",
     "prev:test_model"
    ]
   },
   "outputs": [],
   "source": [
    "outlier_yaml=f\"\"\"apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: income-outlier\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:1.2.2-dev_alibidetect\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - adultod\n",
    "        - --http_port\n",
    "        - '8080'\n",
    "        - --protocol\n",
    "        - tensorflow.http\n",
    "        - --storage_uri\n",
    "        - s3://{MINIO_MODEL_BUCKET}/{OUTLIER_MODEL_PATH}\n",
    "        - --reply_url\n",
    "        - http://default-broker       \n",
    "        - --event_type\n",
    "        - org.kubeflow.serving.inference.outlier\n",
    "        - --event_source\n",
    "        - org.kubeflow.serving.incomeod\n",
    "        - OutlierDetector\n",
    "        envFrom:\n",
    "        - secretRef:\n",
    "            name: seldon-init-container-secret\n",
    "\"\"\"\n",
    "with open(\"outlier.yaml\",\"w\") as f:\n",
    "    f.write(outlier_yaml)\n",
    "run(\"kubectl apply -f outlier.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trigger_outlier_yaml=f\"\"\"apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: income-outlier-trigger\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  filter:\n",
    "    sourceAndType:\n",
    "      type: org.kubeflow.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1alpha1\n",
    "      kind: Service\n",
    "      name: income-outlier\n",
    "\"\"\"\n",
    "with open(\"outlier_trigger.yaml\",\"w\") as f:\n",
    "    f.write(trigger_outlier_yaml)\n",
    "run(\"kubectl apply -f outlier_trigger.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/$(kubectl get deploy -l serving.knative.dev/service=income-outlier -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE})\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy KNative Eventing Event Display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:deploy_event_display",
     "prev:deploy_outlier"
    ]
   },
   "outputs": [],
   "source": [
    "event_display=f\"\"\"apiVersion: apps/v1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}          \n",
    "spec:\n",
    "  replicas: 1\n",
    "  selector:\n",
    "    matchLabels: &labels\n",
    "      app: event-display\n",
    "  template:\n",
    "    metadata:\n",
    "      labels: *labels\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: helloworld-go\n",
    "          # Source code: https://github.com/knative/eventing-contrib/tree/master/cmd/event_display\n",
    "          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display@sha256:f4628e97a836c77ed38bd3b6fd3d0b06de4d5e7db6704772fe674d48b20bd477\n",
    "---\n",
    "kind: Service\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: event-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  selector:\n",
    "    app: event-display\n",
    "  ports:\n",
    "    - protocol: TCP\n",
    "      port: 80\n",
    "      targetPort: 8080\n",
    "---\n",
    "apiVersion: eventing.knative.dev/v1alpha1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: income-outlier-display\n",
    "  namespace: {DEPLOY_NAMESPACE}\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: org.kubeflow.serving.inference.outlier\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: v1\n",
    "      kind: Service\n",
    "      name: event-display\n",
    "\"\"\"\n",
    "with open(\"event_display.yaml\",\"w\") as f:\n",
    "    f.write(event_display)\n",
    "run(\"kubectl apply -f event_display.yaml\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl rollout status -n {DEPLOY_NAMESPACE} deploy/event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Test Outlier Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "block:test_outliers",
     "prev:deploy_event_display"
    ]
   },
   "outputs": [],
   "source": [
    "def predict():\n",
    "    payload='{\"instances\": [[300,  4,  4,  2,  1,  4,  4,  0,  0,  0, 600,  9]]}'\n",
    "    cmd=f\"\"\"curl -v -d '{payload}' \\\n",
    "       -H \"Host: kf-income.admin.example.com\" \\\n",
    "       -H \"Content-Type: application/json\" \\\n",
    "       http://kfserving-ingressgateway.istio-system/v1/models/kf-income:predict\n",
    "    \"\"\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    raw = ret.stdout.read().decode(\"utf-8\")\n",
    "    print(raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_outlier_event_display_logs():\n",
    "    cmd=f\"kubectl logs $(kubectl get pod -l app=event-display -o jsonpath='{{.items[0].metadata.name}}' -n {DEPLOY_NAMESPACE}) -n {DEPLOY_NAMESPACE}\"\n",
    "    ret = Popen(cmd, shell=True,stdout=PIPE)\n",
    "    res = ret.stdout.read().decode(\"utf-8\").split(\"\\n\")\n",
    "    data= []\n",
    "    for i in range(0,len(res)):\n",
    "        if res[i] == 'Data,':\n",
    "            j = json.loads(json.loads(res[i+1]))\n",
    "            if \"is_outlier\"in j[\"data\"].keys():\n",
    "                data.append(j)\n",
    "    if len(data) > 0:\n",
    "        return data[-1]\n",
    "    else:\n",
    "        return None\n",
    "j = None\n",
    "while j is None:\n",
    "    predict()\n",
    "    print(\"Waiting for outlier logs, sleeping\")\n",
    "    time.sleep(2)\n",
    "    j = get_outlier_event_display_logs()\n",
    "    \n",
    "print(j)\n",
    "print(\"Outlier\",j[\"data\"][\"is_outlier\"]==[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Clean Up Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "run(f\"kubectl delete inferenceservice kf-income -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete ksvc income-outlier -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete sa  minio-kf-sa -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete secret seldon-init-container-secret -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete secret income-kf-secret -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete deployment event-display -n {DEPLOY_NAMESPACE}\", shell=True)\n",
    "run(f\"kubectl delete svc event-display -n {DEPLOY_NAMESPACE}\", shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kubeflow_notebook": {
   "docker_image": "seldonio/jupyter-lab-alibi-kale:0.11",
   "experiment": {
    "id": "new",
    "name": "kfserving-e2e-adult"
   },
   "experiment_name": "kfserving-e2e-adult",
   "katib_metadata": {
    "algorithm": {
     "algorithmName": "grid"
    },
    "maxFailedTrialCount": 3,
    "maxTrialCount": 12,
    "objective": {
     "objectiveMetricName": "",
     "type": "minimize"
    },
    "parallelTrialCount": 3,
    "parameters": []
   },
   "katib_run": false,
   "pipeline_description": "KFServing e2e adult",
   "pipeline_name": "kfserving-e2e-adult",
   "volumes": []
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
