ipython-notebook/API Sample/Batch anomaly detection with the Anomaly Detector API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch anomaly detection with the Anomaly Detector API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use this Jupyter notebook to start visualizing anomalies as a batch with the Anomaly Detector API in Python.\n",
    "\n",
    "This notebook shows you how to send a batch anomaly detection request, and vizualize the anomalies found throughout the example data set. The graph created at the end of this notebook will display the following:\n",
    "* Anomalies found throughout the data set, highlighted.\n",
    "* The expected values versus the values contained in the data set.\n",
    "* Anomaly detection boundaries \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start sending requests to the Anomaly Detector API, paste your Anomaly Detector resource access key below,\n",
    "and replace the endpoint variable with the endpoint for your region or your on-premise container endpoint. \n",
    "\n",
    "Endpoint examples:\n",
    "\n",
    "`https://westus2.api.cognitive.microsoft.com/anomalydetector/v1.0/timeseries/entire/detect`\n",
    "\n",
    "`http://127.0.0.1:5000/anomalydetector/v1.0/timeseries/entire/detect`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "apikey = '[Placeholder: Your Anomaly Detector resource access key]' \n",
    "endpoint = '[Placeholder: Your Anomaly Detector resource endpoint]/anomalydetector/v1.0/timeseries/entire/detect'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Import library to display results\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.plotting import figure,output_notebook, show\n",
    "from bokeh.palettes import Blues4\n",
    "from bokeh.models import ColumnDataSource,Slider\n",
    "import datetime\n",
    "from bokeh.io import push_notebook\n",
    "from dateutil import parser\n",
    "from ipywidgets import interact, widgets, fixed\n",
    "output_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect(endpoint, apikey, request_data):\n",
    "    headers = {'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': apikey}\n",
    "    response = requests.post(endpoint, data=json.dumps(request_data), headers=headers)\n",
    "    if response.status_code == 200:\n",
    "        return json.loads(response.content.decode(\"utf-8\"))\n",
    "    else:\n",
    "        print(response.status_code)\n",
    "        raise Exception(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_figure(sample_data, sensitivity):\n",
    "    sample_data['sensitivity'] = sensitivity\n",
    "    result = detect(endpoint, apikey, sample_data)\n",
    "    columns = {'expectedValues': result['expectedValues'], 'isAnomaly': result['isAnomaly'], 'isNegativeAnomaly': result['isNegativeAnomaly'],\n",
    "          'isPositiveAnomaly': result['isPositiveAnomaly'], 'upperMargins': result['upperMargins'], 'lowerMargins': result['lowerMargins'],\n",
    "          'timestamp': [parser.parse(x['timestamp']) for x in sample_data['series']], \n",
    "          'value': [x['value'] for x in sample_data['series']]}\n",
    "    response = pd.DataFrame(data=columns)\n",
    "    values = response['value']\n",
    "    label = response['timestamp']\n",
    "    anomalies = []\n",
    "    anomaly_labels = []\n",
    "    index = 0\n",
    "    anomaly_indexes = []\n",
    "    p = figure(x_axis_type='datetime', title=\"Batch Anomaly Detection ({0} Sensitvity)\".format(sensitivity), width=800, height=600)\n",
    "    for anom in response['isAnomaly']:\n",
    "        if anom == True and (values[index] > response.iloc[index]['expectedValues'] + response.iloc[index]['upperMargins'] or \n",
    "                         values[index] < response.iloc[index]['expectedValues'] - response.iloc[index]['lowerMargins']):\n",
    "            anomalies.append(values[index])\n",
    "            anomaly_labels.append(label[index])\n",
    "            anomaly_indexes.append(index)\n",
    "        index = index+1\n",
    "    upperband = response['expectedValues'] + response['upperMargins']\n",
    "    lowerband = response['expectedValues'] -response['lowerMargins']\n",
    "    band_x = np.append(label, label[::-1])\n",
    "    band_y = np.append(lowerband, upperband[::-1])\n",
    "    boundary = p.patch(band_x, band_y, color=Blues4[2], fill_alpha=0.5, line_width=1, legend_label='Boundary')\n",
    "    p.line(label, values, legend_label='Value', color=\"#2222aa\", line_width=1)\n",
    "    p.line(label, response['expectedValues'], legend_label='ExpectedValue',  line_width=1, line_dash=\"dotdash\", line_color='olivedrab')\n",
    "    anom_source = ColumnDataSource(dict(x=anomaly_labels, y=anomalies))\n",
    "    anoms = p.circle('x', 'y', size=5, color='tomato', source=anom_source)\n",
    "    p.legend.border_line_width = 1\n",
    "    p.legend.background_fill_alpha  = 0.1\n",
    "    show(p, notebook_handle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vizualizing anomalies throughout your data\n",
    "\n",
    "The following cells call the Anomaly Detector API with two different example time series data sets, and different sensitivities for anomaly detection. Varying the sensitivity of the Anomaly Detector API can improve how well the response fits your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: time series with an hourly sampling frequency\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hourly Sample\n",
    "sample_data = json.load(open('../../sampledata/univariate/univariate_sample_hourly.json'))\n",
    "sample_data['granularity'] = 'hourly'\n",
    "sample_data['period'] = 24\n",
    "# 95 sensitivity\n",
    "build_figure(sample_data,95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 90 sensitivity\n",
    "build_figure(sample_data,90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#85 sensitivity\n",
    "build_figure(sample_data,85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: time series with an daily sampling frequency\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#daily sample\n",
    "sample_data = json.load(open('../../sampledata/univariate/univariate_sample_daily.json'))\n",
    "sample_data['granularity'] = 'daily'\n",
    "# 95 sensitivity\n",
    "build_figure(sample_data,95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 90 sensitivity\n",
    "build_figure(sample_data,90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 85 sensitivity\n",
    "build_figure(sample_data,80)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "530dd268f570c5cdc0ceeb4c9b04e15d56fa20bfd35f8d335952cdbe40d1c280"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}