{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tfp_anomaly_detection.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "iXDk02sFYgJy"
      },
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QZo6Stv1Z2Nf"
      },
      "source": [
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/experimental/tensorflow_probability/anomaly_detection/tfp_anomaly_detection.ipynb\"\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/experimental/tensorflow_probability/anomaly_detection/tfp_anomaly_detection.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YRvy0UhwcjOm"
      },
      "source": [
        "# Anomaly Detection with TensorFlow Probability STS on Kubeflow Pipelines"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0oCL7T67aa02"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates how to use [TensorFlow Probability](https://www.tensorflow.org/probability) and [Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/) for anomaly detection in time series data. It uses structural time series (STS), a class of Bayesian statistical models, to decompose a time series into interpretable seasonal and trend components. This algorithm fits an STS model to the time series, generates a forecast of acceptable values for each timestep, and flags any points outside of the forecast as an anomaly. To learn more about STS models, check out this demo on [Structural Time Series Modeling Case Studies](https://www.tensorflow.org/probability/examples/Structural_Time_Series_Modeling_Case_Studies_Atmospheric_CO2_and_Electricity_Demand).\n",
        "\n",
        "This demo is most relevant for those who would like to automatically flag anomalies in time series data and can be used for applications like network monitoring, infrastructure maintenance, and sales tracking.\n",
        "\n",
        "### Dataset\n",
        "\n",
        "This demo uses the [Numenta Anomaly Benchmark](https://github.com/numenta/NAB), a popular benchmark of time series data with labeled anomalies. More specifically, our demo uses [nyc_taxi.csv](https://github.com/numenta/NAB/blob/d2854d17a3feb9e143b1e9a715c5af67da2c1888/data/realKnownCause/nyc_taxi.csv) which reports the total number of passengers in NYC taxis from July 2014 to January 2015 in 30-minute increments.\n",
        "\n",
        "### Objective\n",
        "\n",
        "You will go through the following steps: \n",
        "* Define and launch an anomaly detection algorithm on Kubeflow Pipelines.\n",
        "* Retrieve and visualize results.\n",
        "* Benchmark predictions using the Numenta Anomaly Benchmark scoring method.\n",
        "\n",
        "### Costs\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UsKeVhtHbYeU"
      },
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
        "all the requirements to run this notebook. You can skip this step.\n",
        "\n",
        "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
        "You need the following:\n",
        "\n",
        "* The Google Cloud SDK\n",
        "* Git\n",
        "* Python 3\n",
        "* virtualenv\n",
        "* Jupyter notebook running in a virtual environment with Python 3\n",
        "\n",
        "The Google Cloud guide to [Setting up a Python development\n",
        "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
        "installation guide](https://jupyter.org/install) provide detailed instructions\n",
        "for meeting these requirements. The following steps provide a condensed set of\n",
        "instructions:\n",
        "\n",
        "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
        "\n",
        "1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
        "\n",
        "1. [Install\n",
        "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
        "   and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
        "\n",
        "1. To install Jupyter, run `pip3 install jupyter` on the\n",
        "command-line in a terminal shell.\n",
        "\n",
        "1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
        "\n",
        "1. Open this notebook in the Jupyter Notebook Dashboard."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install additional packages\n",
        "\n",
        "Install additional package dependencies not installed in your notebook environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "source": [
        "import os\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
        "USER_FLAG = \"\"\n",
        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    USER_FLAG = \"--user\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8EO5Ar64fidZ"
      },
      "source": [
        "! pip3 install {USER_FLAG} --upgrade kfp\n",
        "! pip3 install {USER_FLAG} --upgrade google-cloud-pipeline-components\n",
        "! pip3 install {USER_FLAG} --upgrade tensorflow\n",
        "! pip3 install {USER_FLAG} --upgrade matplotlib\n",
        "! pip3 install {USER_FLAG} --upgrade numpy\n",
        "! pip3 install {USER_FLAG} --upgrade pandas"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wIibh1qxkc_G"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zGbkBMWPPvPY"
      },
      "source": [
        "# Automatically restart kernel after installs\n",
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No37uLsjk1UN"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "1. [Enable the Vertex AI API, Cloud Build API, Cloud Storage API, and Container Registry API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,cloudbuild.googleapis.com,storage.googleapis.com,containerregistry.googleapis.com).\n",
        "\n",
        "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
        "\n",
        "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ff394ff4"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "1988b81c",
        "jupyter": {
          "outputs_hidden": true
        },
        "outputId": "6a6488f6-6883-4246-a197-b3c5816dab0d"
      },
      "source": [
        "import os\n",
        "\n",
        "# Get your Google Cloud project ID from gcloud\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "  shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "  PROJECT_ID = shell_output[0]\n",
        "  print(\"Project ID: \", PROJECT_ID)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Project ID:  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5adca8fc"
      },
      "source": [
        "Otherwise, set your project ID here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "939b4566",
        "jupyter": {
          "outputs_hidden": true
        }
      },
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
        "  PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ypoM_48NJCuU",
        "outputId": "82b0e9ed-0728-4ce8-dcb8-8acfac3ca6e5"
      },
      "source": [
        "!gcloud config set project {PROJECT_ID}"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Updated property [core/project].\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2df8c231"
      },
      "source": [
        "#### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "84ba4ab1",
        "jupyter": {
          "outputs_hidden": true
        }
      },
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dr--iN2kAylZ"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "**If you are using Google Cloud Notebooks**, your environment is already\n",
        "authenticated. Skip this step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "**If you are using Colab**, run the cell below and follow the instructions\n",
        "when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "1. In the Cloud Console, go to the [**Create service account key**\n",
        "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
        "\n",
        "2. Click **Create service account**.\n",
        "\n",
        "3. In the **Service account name** field, enter a name, and\n",
        "   click **Create**.\n",
        "\n",
        "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
        "into the filter box, and select\n",
        "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
        "\n",
        "5. Click *Create*. A JSON file that contains your key downloads to your\n",
        "local environment.\n",
        "\n",
        "6. Enter the path to your service account key as the\n",
        "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "# The Google Cloud Notebook product has specific requirements\n",
        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
        "\n",
        "# If on Google Cloud Notebooks, then don't execute this code\n",
        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
        "    if \"google.colab\" in sys.modules:\n",
        "        from google.colab import auth as google_auth\n",
        "\n",
        "        google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    elif not os.getenv(\"IS_TESTING\"):\n",
        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "When you submit a training job, Vertex AI saves all resources to the given GCS bucket. We will also use the same bucket to download and host the input data. \n",
        "\n",
        "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
        "Cloud Storage buckets.\n",
        "\n",
        "You may also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
        "available](https://cloud.google.com/vertex-ai/docs/general/locations#available_regions). You may not use a Multi-Regional Storage bucket for training with Vertex AI."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}\n",
        "REGION = \"[your-region]\"  # @param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cf221059d072"
      },
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ucvCsknMCims"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vhOb7YnwClBb"
      },
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h01b41bGpu6l"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4NUcY9w0pidh"
      },
      "source": [
        "PIPELINE_NAME = '{0}-{1}'.format('tfp-anomaly-detection', TIMESTAMP)\n",
        "PIPELINE_ROOT = '{0}/{1}'.format(BUCKET_NAME, PIPELINE_NAME)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6MsrEnjUJMDB"
      },
      "source": [
        "from typing import Callable, Optional, Mapping, Any\n",
        "\n",
        "import kfp\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2 import dsl\n",
        "from kfp.v2.google.client import AIPlatformClient\n",
        "from kfp.v2.dsl import Input, Output, Dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8aSrF6n1noLh"
      },
      "source": [
        "### Define the anomaly detection components\n",
        "\n",
        "Here you will load components from the [anomaly_detection](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud/google_cloud_pipeline_components/experimental/tensorflow_probability/anomaly_detection) folder in the [Google Cloud Pipeline Components SDK](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud).\n",
        "\n",
        "You can also save and modify the original Python component file. For example, for [tfp_anomaly_detection.py](https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/experimental/tensorflow_probability/anomaly_detection/tfp_anomaly_detection.py):\n",
        "\n",
        "* Call `generate_component_file()` which creates a yaml file.\n",
        "* Replace the next cell with `anomaly_detection_op = kfp.components.load_component_from_file('component.yaml')`\n",
        "\n",
        "The components do the following:\n",
        "* `preprocess`: Regularizes and resamples a time series.\n",
        "* `tfp_anomaly_detection`: Infers the structure of the time series, fits the model, and identifies anomalies based on the predictive distribution of acceptable values at each timestep.\n",
        "* `postprocess`: Fills missing values from regularizing and resampling."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZeMp71LgJYMF"
      },
      "source": [
        "preprocess_op = kfp.components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/master/components/google-cloud/google_cloud_pipeline_components/experimental/tensorflow_probability/anomaly_detection/preprocess.yaml')\n",
        "anomaly_detection_op = kfp.components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/master/components/google-cloud/google_cloud_pipeline_components/experimental/tensorflow_probability/anomaly_detection/component.yaml')\n",
        "postprocess_op = kfp.components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/master/components/google-cloud/google_cloud_pipeline_components/experimental/tensorflow_probability/anomaly_detection/postprocess.yaml')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFFIz_X0n7p2"
      },
      "source": [
        "### Define the pipeline\n",
        "\n",
        "Here you will define the relationship between the components and how data is passed. In this pipeline a Google Cloud Storage csv is imported, the data is preprocessed, anomalies are flagged, and the results are postprocessed so that the output csv is scoreable by the Numenta Anomaly Benchmark."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s1vqx-VGJrzT"
      },
      "source": [
        "@dsl.pipeline(\n",
        "    pipeline_root=PIPELINE_ROOT, name=PIPELINE_NAME)\n",
        "def pipeline(input_url: str, memory_limit: str, seed: int) -> None:\n",
        "  \"\"\"\n",
        "      Train model and return detected anomalies.\n",
        "  \"\"\"\n",
        "  input_task = kfp.dsl.importer(\n",
        "        artifact_uri=input_url,\n",
        "        artifact_class=Dataset)\n",
        "  preprocess_task = preprocess_op(input_dataset=input_task.output)\n",
        "  anomaly_detection_task = anomaly_detection_op(input_dataset=preprocess_task.output, seed=seed).set_memory_limit(memory_limit)\n",
        "  postprocess_op(input_dataset=input_task.output, predictions_dataset=anomaly_detection_task.output)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dpZB1t1CJJCT"
      },
      "source": [
        "def run_pipeline(pipeline: Callable,\n",
        "                 parameter_values: Optional[Mapping[str, Any]] = {},\n",
        "                 enable_caching: bool = False) -> None:\n",
        "  \"\"\"Runs a given pipeline function using Kubeflow Pipelines.\n",
        "\n",
        "  Args:\n",
        "   pipeline: The function to run.\n",
        "   parameter_values: Parameters passed to the pipeline function when run.\n",
        "   enable_caching: Whether to used cached results from previous runs.\n",
        "  \"\"\"\n",
        "  compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline,\n",
        "    package_path='{}_pipeline.json'.format(PIPELINE_NAME))\n",
        "\n",
        "  api_client = AIPlatformClient(\n",
        "    project_id=PROJECT_ID,\n",
        "    region=REGION,\n",
        "  )\n",
        "\n",
        "  _ = api_client.create_run_from_job_spec(\n",
        "    job_spec_path='{}_pipeline.json'.format(PIPELINE_NAME),\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values=parameter_values,\n",
        "    enable_caching=enable_caching)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BO65SDC8qMt"
      },
      "source": [
        "### Download the data\n",
        "\n",
        "Here you will download the Numenta Anomaly Benchmark and upload the dataset to your GCS bucket. We will then find the exact GCS file url associated with the chosen task to pass as the input url into the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4VWXRyy-8pE8"
      },
      "source": [
        "import os\n",
        "\n",
        "NAB_DATA_BLOB = '{0}/NAB'.format(BUCKET_NAME)\n",
        "if not os.path.exists('content/NAB'):\n",
        "  !git clone https://github.com/numenta/NAB\n",
        "!gsutil cp -r NAB/data $NAB_DATA_BLOB"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I0qjQy908uQb"
      },
      "source": [
        "# Find the full file path in gcs for the chosen task\n",
        "import tensorflow as tf\n",
        "\n",
        "chosen_task_folder = 'realKnownCause'\n",
        "chosen_task = 'nyc_taxi'\n",
        "nab_files = tf.io.gfile.glob('{0}/*/*.csv'.format(NAB_DATA_BLOB))\n",
        "chosen_task_file = [file for file in nab_files if chosen_task in file][0]\n",
        "print('The pipeline will be run on the task: {0}'.format(chosen_task))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0zQBTufbGuPl"
      },
      "source": [
        "### Run the pipeline\n",
        "\n",
        "Finally, we run the pipeline. Please wait until the run has completed before proceeding to the next steps."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BLwijes3q1Qf"
      },
      "source": [
        "parameter_values = {\n",
        "  'input_url': chosen_task_file,\n",
        "  'memory_limit': '50G',\n",
        "  'seed': 0,\n",
        "}\n",
        "run_pipeline(pipeline, parameter_values=parameter_values)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "skkJf62AGIe8"
      },
      "source": [
        "### Download the results locally\n",
        "\n",
        "Copy the GCS file path from the final `postprocess` step of the pipeline below. Here we will save this output locally for visualization and scoring."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-GQhGfxh6bvv"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import json"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pzvNg8qeDU0p"
      },
      "source": [
        "gcs_file = '[your-pipeline-output]' # @param {type:'string'}\n",
        "output_file = '/content/{0}-{1}.csv'.format(chosen_task, TIMESTAMP)\n",
        "!gsutil cp $gcs_file $output_file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6uBGVh4rDgc3"
      },
      "source": [
        "# Collect targets specifically for the chosen task\n",
        "targets = json.load(open('/content/NAB/labels/combined_labels.json'))\n",
        "chosen_task_targets = [targets[key] for key in targets if chosen_task in key][0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JittvMUhDYgO"
      },
      "source": [
        "### Visualize the results\n",
        "\n",
        "Here we will plot the forecast distribution outputted by the pipeline, the points flagged as anomalies (red), and the ground truth targets (green). The graph is plotted with daily granularity due to the resampling done during preprocessing.\n",
        "\n",
        "Note how the algorithm correctly identifies December 25th as an anomaly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "MsiYGrpFDeMS"
      },
      "source": [
        "#@title Plotting setup\n",
        "from matplotlib import pylab as plt\n",
        "from matplotlib.lines import Line2D\n",
        "\n",
        "def plot_predictions(predictions: pd.DataFrame, annotation_fn: Callable = lambda timestamp: timestamp) -> None:\n",
        "  \"\"\"\n",
        "    Plots the time series, forecast, detected anomalies, and residuals.\n",
        "\n",
        "    Args:\n",
        "      predictions: The output of the anomaly detection algorithm.\n",
        "  \"\"\"\n",
        "  # Drop NaN values during plotting\n",
        "  predictions = predictions.dropna(how='any')\n",
        "  predictions = predictions.reset_index()\n",
        "\n",
        "  timestamp = pd.to_datetime(predictions['timestamp'], format='%Y-%m-%d')\n",
        "  # Plot the value from predictions which may be\n",
        "  # an aggregation of the original value\n",
        "  value = np.array(predictions['value_predictions'])\n",
        "  lower_limit = np.array(predictions['lower_limit'])\n",
        "  upper_limit = np.array(predictions['upper_limit'])\n",
        "  mean = np.array(predictions['mean'])\n",
        "  anomalies = np.array(predictions['label']).nonzero()[0]\n",
        "  targets = []\n",
        "  if 'target' in predictions:\n",
        "    targets = np.array(predictions['target']).nonzero()[0]\n",
        "\n",
        "  fig = plt.figure(figsize=(10, 5), constrained_layout=True)\n",
        "  spec = fig.add_gridspec(ncols=1, nrows=2, height_ratios=[2., 1.])\n",
        "  series_ax = fig.add_subplot(spec[0, 0])\n",
        "  residuals_ax = fig.add_subplot(spec[1, 0], sharex=series_ax)\n",
        "\n",
        "  # Plot anomalies on series_ax\n",
        "  series_ax.plot(\n",
        "      timestamp,\n",
        "      value,\n",
        "      color='black',\n",
        "      alpha=0.6)\n",
        "  series_ax.fill_between(\n",
        "      timestamp,\n",
        "      lower_limit,\n",
        "      upper_limit,\n",
        "      color='tab:blue',\n",
        "      alpha=0.3)\n",
        "\n",
        "  for anomaly_idx in anomalies:\n",
        "    x = timestamp[anomaly_idx]\n",
        "    y = value[anomaly_idx]\n",
        "    series_ax.scatter(x, y, s=100, alpha=0.4, c='red')\n",
        "  \n",
        "  for target_idx in targets:\n",
        "    x = timestamp[target_idx]\n",
        "    y = value[target_idx]\n",
        "    series_ax.scatter(x, y, s=100, alpha=0.4, c='green')\n",
        "    series_ax.annotate(annotation_fn(x), (x, y))\n",
        "\n",
        "  # Plot residuals on residuals_ax\n",
        "  time_delta = timestamp[1] - timestamp[0]\n",
        "  residuals_ax.bar(\n",
        "      timestamp,\n",
        "      height=upper_limit - lower_limit,\n",
        "      bottom=lower_limit - mean,\n",
        "      width=time_delta,\n",
        "      align='center',\n",
        "      color='tab:blue',\n",
        "      alpha=0.3)\n",
        "  residuals_ax.bar(\n",
        "      timestamp,\n",
        "      width=time_delta,\n",
        "      height=value - mean,\n",
        "      align='center',\n",
        "      color='black',\n",
        "      alpha=0.6)\n",
        "\n",
        "  # Set up grid styling\n",
        "  series_ax.set_ylabel('Original series')\n",
        "  residuals_ax.set_ylabel('Residuals')\n",
        "  series_ax.grid(True, color='whitesmoke')\n",
        "  residuals_ax.grid(True, color='whitesmoke')\n",
        "  series_ax.set_axisbelow(True)\n",
        "  residuals_ax.set_axisbelow(True)\n",
        "\n",
        "  # Add title and legend\n",
        "  series_ax.set_title('TFP STS model forecast, anomalies, and residuals for {0}'.format(chosen_task))\n",
        "  create_legend_label = lambda label, color: Line2D([0], [0], marker='o', color='w', label=label, markerfacecolor=color, markersize=10)\n",
        "  legend_elements = [create_legend_label(label, color) for label, color in [('predicted anomaly', 'red'), ('target', 'green')]]\n",
        "  series_ax.legend(handles=legend_elements, loc='lower right')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QETIRiPb6q1o"
      },
      "source": [
        "# Round target timestamps to day for plotting\n",
        "round_to_day = lambda timestamp: timestamp.split()[0]\n",
        "rounded_targets = [round_to_day(timestamp) for timestamp in chosen_task_targets]\n",
        "rounded_targets = set(rounded_targets)\n",
        "predictions = pd.read_csv(output_file)\n",
        "predictions['target'] = predictions.apply(lambda df: round_to_day(df['timestamp']) in rounded_targets, axis=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "im04k2PeDku0"
      },
      "source": [
        "# Change the start and end to view different slices of the prediction\n",
        "start, end = 8000, 9000\n",
        "round_annotation = lambda timestamp: timestamp.date()\n",
        "plot_predictions(predictions.iloc[start:end], round_annotation)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QPpqq3-EFg3r"
      },
      "source": [
        "### Run scoring\n",
        "\n",
        "Here we quantitatively score the algorithm's performance on the Numenta Anomaly Benchmark. The benchmark uses a custom scoring mechanism described in [their paper](https://arxiv.org/abs/1510.03336). Unlike precision and recall which do not reward for early detection, this scoring mechanism rewards based on windows around anomalous points rather than the exact points themselves.\n",
        "\n",
        "We will run the scoring script with the `--optimize` flag, which uses the `anomaly_scores` column to score and optimizes the decision threshold. If this flag is omitted, then the script will only use the `label` column originally outputted by the component."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B-SImAPlFgaA"
      },
      "source": [
        "# Set up NAB folder for running scoring\n",
        "%cd /content/NAB\n",
        "!pip install . --user\n",
        "!python scripts/create_new_detector.py --detector $PIPELINE_NAME"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_BzHLJ1oFjai"
      },
      "source": [
        "# Move gcs output into the NAB results folder structure\n",
        "results_file = 'results/{0}/{1}/{0}_{2}.csv'.format(PIPELINE_NAME, chosen_task_folder, chosen_task)\n",
        "!cp $output_file $results_file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pq5_w9NlFo4W"
      },
      "source": [
        "# Run the scoring script\n",
        "!python run.py -d $PIPELINE_NAME --optimize --score --normalize"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QYx3JvlhFqgc"
      },
      "source": [
        "#@title Score collection and normalization setup\n",
        "import glob\n",
        "\n",
        "def collect_scores(profile_name: str, chosen_task: str) -> pd.DataFrame:\n",
        "  \"\"\"Crawls through results files for all detectors in NAB to get results for the chosen task.\n",
        "  \n",
        "  Args:\n",
        "    profile_name: One of 'standard', 'low_FP_rate', 'low_FN_rate'.\n",
        "    chosen_task: The chosen benchmark task.\n",
        "  \n",
        "  Returns:\n",
        "    all_scores_df: A pandas DataFrame of results for the task sorted by highest to lowest score.\n",
        "  \"\"\"\n",
        "  all_scores = []\n",
        "  for scores_file in glob.glob('/content/NAB/results/**/*_{0}_scores.csv'.format(profile_name)):\n",
        "    scores_df = pd.read_csv(scores_file)\n",
        "    chosen_task_row = scores_df[scores_df['File'].str.contains(chosen_task).fillna(False)]\n",
        "    all_scores.append(chosen_task_row)\n",
        "  all_scores_df = pd.concat(all_scores)\n",
        "  all_scores_df = all_scores_df.sort_values(by=['Score'], ascending=False)\n",
        "  all_scores_df = all_scores_df.reset_index().drop('index', axis=1)\n",
        "  return all_scores_df\n",
        "\n",
        "def normalize_scores(results: pd.DataFrame, profile_name: str, profiles: dict,\n",
        "                     tpCount: int) -> pd.DataFrame:\n",
        "  \"\"\"Normalizes scores with the max from a perfect detector and the min from a null detector.\n",
        "\n",
        "  Args:\n",
        "    results: Pandas DataFrame with score results.\n",
        "    profile_name: One of 'standard', 'low_FP_rate', 'low_FN_rate'.\n",
        "    profiles: Dictionary containing cost matrix for each profile.\n",
        "    tpCount: The number of true positives in the ground truth targets.\n",
        "\n",
        "  Returns:\n",
        "    The results DataFrame with an added column of normalized scores.\n",
        "  \"\"\"\n",
        "  perfect = tpCount * profiles[profile_name][\"CostMatrix\"][\"tpWeight\"]\n",
        "  # Note that the null detector's name is NaN in the `Detector` column\n",
        "  base = results[pd.isna(results['Detector'])]['Score'].iloc[0]\n",
        "  scores = results['Score']\n",
        "  results['Normalized_Score'] = 100 * (scores - base) / (perfect - base)\n",
        "  \n",
        "  # Reindex column order for more organized table\n",
        "  columns = results.columns.to_list()\n",
        "  columns.remove('Score')\n",
        "  columns.remove('Normalized_Score')\n",
        "  columns += ['Score', 'Normalized_Score']\n",
        "  results = results.reindex(columns=columns)\n",
        "  print('Normalization used min raw score: {0} and max raw score: {1}'.format(base, perfect))\n",
        "  return results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2trwsXI24zCp"
      },
      "source": [
        "NAB also provides scores for three profile settings: `standard`, `reward_low_FN_rate`, and `reward_low_FP_rate`. If you run the cell below you can see the cost matrix for each profile, where `reward_low_FN_rate` penalizes false negatives more and `reward_low_FP_rate` penalizes false positives more. For example, if for the NYC Taxi & Limousine Commission it is worse to not have enough taxis during a big event than it is to have too many, then they may want to score based on a `reward_low_FN_rate` profile.\n",
        "\n",
        "For the purposes of this demo we will only display results for the `standard` profile.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4eNbITfx41VU"
      },
      "source": [
        "tpCount = len(chosen_task_targets)\n",
        "profile_name = 'standard'\n",
        "profiles = json.load(open('/content/NAB/config/profiles.json'))\n",
        "profiles"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vz_wAGOZFuxZ"
      },
      "source": [
        "results = collect_scores(profile_name, chosen_task)\n",
        "results = normalize_scores(results, profile_name, profiles, tpCount)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "74drnCcN-y1t"
      },
      "source": [
        "results"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}