notebooks/data-sources/TestGrid/metrics/metric_template.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# {Metric name}\n",
    "- Write the name of the metric or the name of a group of metrics that will be implemented in this notebook.\n",
    "\n",
    "## Description:\n",
    "The desciption of the metrics should explain their definition and purpose. It can include the following points:\n",
    "\n",
    "    - A list of all the metrics implemented in this notebook.\n",
    "    - What are these metrics calculating? and how?\n",
    "    - How to interpret the metrics? \n",
    "    - Link to the ocp-ci-analysis repository issue.\n",
    "    - Additional notes and information.\n",
    "    \n",
    "\n",
    "\n",
    "# Example: _Flake metrics_\n",
    "\n",
    "_One of the key perfomance indicators that we would like to create greater visbility into and track over time is overall number and percent of flakes that occur. Individual test runs are flagged a \"flake\" if they are run mulitple times with a mix of passes and failes without any changes to the code being tested. Although they occur for individual test runs, there are a number of aggregate views that developers may want to look at to assess the overall health of their project or testing platform. Through this notebook, we will be able to compute:_\n",
    "* Percentage of flakes on platform each day\n",
    "* Percentage of flakes by tab each week\n",
    "* Percentage of flakes by grid each month\n",
    "* Percentage of flakes by test overall (this can also be seen as a severity level = overall flake rate of test)\n",
    "* Severity of flakes\n",
    "\n",
    "_In order to provide maximum flexibility for the end-user of this work, instead of creating a number of dataframes to answer each of these specifc questions, we will define a long and narrow data structure (a list of tuples saved as a csv for now) that contains only 5 columns (\"timestamp\", \"tab\",\"grid\",\"test\",\"flake\"). This allows superset (or pandas) to perform the last filter and/or aggreagtion of interest to an end user. Which is to say, there may appear to be a lot of repetion within the final dataset, but each row should be unique, and it should provide the simpelest useability for an end-user._  \n",
    "\n",
    "_Linked issues: [Issue 1](https://github.com/aicoe-aiops/ocp-ci-analysis/issues/104), [Issue 2](https://github.com/aicoe-aiops/ocp-ci-analysis/issues/138)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.147422Z",
     "start_time": "2021-03-24T18:48:18.242365Z"
    }
   },
   "outputs": [],
   "source": [
    "## Import libraries\n",
    "from enum import Enum\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datetime\n",
    "import os\n",
    "from io import BytesIO\n",
    "from pathlib import Path\n",
    "import gzip\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import warnings\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "import boto3\n",
    "\n",
    "load_dotenv(find_dotenv())\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.154367Z",
     "start_time": "2021-03-24T18:48:19.149668Z"
    }
   },
   "outputs": [],
   "source": [
    "## Specify variables\n",
    "\n",
    "METRIC_NAME = \"example\"\n",
    "\n",
    "# Specify the path for input grid data\n",
    "INPUT_DATA_PATH = \"../../../../data/raw/testgrid_183.json.gz\"\n",
    "\n",
    "# Specify the path for output metric data\n",
    "OUTPUT_DATA_PATH = f\"../../../../data/processed/metrics/{METRIC_NAME}\"\n",
    "\n",
    "## CEPH Bucket variables\n",
    "## Create a .env file on your local with the correct configs\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "s3_path = os.getenv(\"S3_PROJECT_KEY\", \"ai4ci/testgrid/metrics\")\n",
    "s3_input_data_path = \"raw_data\"\n",
    "\n",
    "# Specify whether or not we are running this as a notebook or part of an automation pipeline.\n",
    "AUTOMATION = os.getenv(\"IN_AUTOMATION\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CephCommunication:\n",
    "    \"\"\"\n",
    "    Class to establish communication with a ceph s3 bucket.\n",
    "    It connects with the bucket and provides methods to read and write data in the parquet format.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self, s3_endpoint_url, aws_access_key_id, aws_secret_access_key, s3_bucket\n",
    "    ):\n",
    "        self.s3_endpoint_url = s3_endpoint_url\n",
    "        self.aws_access_key_id = aws_access_key_id\n",
    "        self.aws_secret_access_key = aws_secret_access_key\n",
    "        self.s3_resource = boto3.resource(\n",
    "            \"s3\",\n",
    "            endpoint_url=self.s3_endpoint_url,\n",
    "            aws_access_key_id=self.aws_access_key_id,\n",
    "            aws_secret_access_key=self.aws_secret_access_key,\n",
    "        )\n",
    "        self.bucket = s3_bucket\n",
    "        ## Todo: Add try catch\n",
    "\n",
    "    def upload_to_ceph(self, dataframe, s3_path, filename):\n",
    "        \"\"\"\n",
    "        This helper function takes as input the data frame to be uploaded, and the output filename.\n",
    "        It then saves the data frame in the defined ceph bucket.\n",
    "        \"\"\"\n",
    "        parquet_buffer = BytesIO()\n",
    "        dataframe.to_parquet(parquet_buffer)\n",
    "        s3_obj = self.s3_resource.Object(self.bucket, f\"{s3_path}/{filename}\")\n",
    "        status = s3_obj.put(Body=parquet_buffer.getvalue())\n",
    "        return status\n",
    "\n",
    "    def read_from_ceph(self, s3_path, filename):\n",
    "        \"\"\"\n",
    "        Helper function to read from ceph and see if the saved data is correct.\n",
    "        \"\"\"\n",
    "        buffer = BytesIO()\n",
    "        s3_object = self.s3_resource.Object(self.bucket, f\"{s3_path}/{filename}\")\n",
    "        s3_object.download_fileobj(buffer)\n",
    "        df_temp = pd.read_parquet(buffer)\n",
    "        return df_temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_to_disk(dataframe, path, filename):\n",
    "    \"\"\"\n",
    "    Helper function to save the dataframe\n",
    "    as a parquet file to disk.\n",
    "    \"\"\"\n",
    "    dataset_base_path = Path(path)\n",
    "    dataset_base_path.mkdir(parents=True, exist_ok=True)\n",
    "    dataframe.to_parquet(f\"{path}/{filename}\")\n",
    "    return True\n",
    "\n",
    "\n",
    "def read_from_disk(path, filename):\n",
    "    \"\"\"\n",
    "    Helper function to read from disk and see if the saved data is the same.\n",
    "    \"\"\"\n",
    "    return pd.read_parquet(f\"{path}/{filename}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.201948Z",
     "start_time": "2021-03-24T18:48:19.156754Z"
    }
   },
   "outputs": [],
   "source": [
    "## Import data\n",
    "timestamp = datetime.datetime.today()\n",
    "\n",
    "if AUTOMATION:\n",
    "    filename = f\"testgrid_{timestamp.day}{timestamp.month}.json\"\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    s3_object = cc.s3_resource.Object(s3_bucket, f\"{s3_input_data_path}/{filename}\")\n",
    "    file_content = s3_object.get()[\"Body\"].read().decode(\"utf-8\")\n",
    "    testgrid_data = json.loads(file_content)\n",
    "\n",
    "else:\n",
    "    with gzip.open(INPUT_DATA_PATH, \"rb\") as read_file:\n",
    "        testgrid_data = json.load(read_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.229548Z",
     "start_time": "2021-03-24T18:48:19.204428Z"
    }
   },
   "outputs": [],
   "source": [
    "## Helper classes and functions\n",
    "## In your metric notebook, you could just\n",
    "## load the template notebook functions\n",
    "## If your metric requires specific helper\n",
    "## functions, write them here, otherwise if these\n",
    "## functions are general, update the template.\n",
    "## Author: AIOps\n",
    "\n",
    "\n",
    "class TestStatus(Enum):\n",
    "    \"\"\"\n",
    "    Enum to encode what test status each value in testgrid corresponds to\n",
    "\n",
    "    Basically python equivalent of the enum here:\n",
    "    https://github.com/GoogleCloudPlatform/testgrid/blob/a18fe953cf98174c215c43e0258b0515e37c283b/pb/test_status/test_status.proto#L3\n",
    "    \"\"\"\n",
    "\n",
    "    NO_RESULT = 0\n",
    "    PASS = 1\n",
    "    PASS_WITH_ERRORS = 2\n",
    "    PASS_WITH_SKIPS = 3\n",
    "    RUNNING = 4\n",
    "    CATEGORIZED_ABORT = 5\n",
    "    UNKNOWN = 6\n",
    "    CANCEL = 7\n",
    "    BLOCKED = 8\n",
    "    TIMED_OUT = 9\n",
    "    CATEGORIZED_FAIL = 10\n",
    "    BUILD_FAIL = 11\n",
    "    FAIL = 12\n",
    "    FLAKY = 13\n",
    "    TOOL_FAIL = 14\n",
    "    BUILD_PASSED = 15\n",
    "\n",
    "\n",
    "def decode_run_length(x):\n",
    "    \"\"\"\n",
    "    Decodes the run length encoded data into an unrolled form.\n",
    "    Returns a list of values.\n",
    "\n",
    "    E.g. takes in [{\"value\":12, \"count\":3}, {\"value\":1, \"count\":2}]\n",
    "    and gives [12, 12, 12, 1, 1]\n",
    "    \"\"\"\n",
    "    lst = []\n",
    "    for run_length in x:\n",
    "        extension = [run_length[\"value\"]] * run_length[\"count\"]\n",
    "        lst.extend(extension)\n",
    "    return lst\n",
    "\n",
    "\n",
    "def testgrid_labelwise_encoding(data, label, overall_only=True):\n",
    "\n",
    "    \"\"\"\n",
    "    Run length encode the dataset and unroll the dataset into a list.\n",
    "    Return flattened list after encoding specified value as\n",
    "    True and rest as False\n",
    "    \"\"\"\n",
    "\n",
    "    percent_label_by_grid_csv = []\n",
    "\n",
    "    for tab in data.keys():\n",
    "\n",
    "        for grid in data[tab].keys():\n",
    "            current_grid = data[tab][grid]\n",
    "\n",
    "            if len(current_grid[\"grid\"]) == 0:\n",
    "                pass\n",
    "            else:\n",
    "                # get all timestamps for this grid (x-axis of grid)\n",
    "                timestamps = [\n",
    "                    datetime.datetime.fromtimestamp(x // 1000)\n",
    "                    for x in current_grid[\"timestamps\"]\n",
    "                ]\n",
    "                # get all test names for this grid (y-axis of grid)\n",
    "                tests = [\n",
    "                    current_grid[\"grid\"][i][\"name\"]\n",
    "                    for i in range(len(current_grid[\"grid\"]))\n",
    "                ]\n",
    "\n",
    "                graphs = [\n",
    "                    current_grid[\"grid\"][i][\"graphs\"]\n",
    "                    for i in range(len(current_grid[\"grid\"]))\n",
    "                ]\n",
    "\n",
    "                # unroll the run-length encoding and set bool for flake or not (x==13)\n",
    "                decoded = [\n",
    "                    (\n",
    "                        np.array(decode_run_length(current_grid[\"grid\"][i][\"statuses\"]))\n",
    "                        == label\n",
    "                    ).tolist()\n",
    "                    for i in range(len(current_grid[\"grid\"]))\n",
    "                ]\n",
    "\n",
    "                # add the timestamp to bool value\n",
    "                decoded = [list(zip(timestamps, g)) for g in decoded]\n",
    "                # add the test, tab and grid name to each entry\n",
    "                # TODO: any ideas for avoiding this quad-loop\n",
    "                # if the label is passed as an arg, add the timestamp, tab,\n",
    "                # grid, tests, graphs metric and the bool values\n",
    "                if label:\n",
    "                    for i, d in enumerate(decoded):\n",
    "                        for j, k in enumerate(d):\n",
    "                            # here we are fetching the test duration values for the tests\n",
    "                            # however,since not all tests contain time duration values,\n",
    "                            # we are only considering the 'Overall' test and fetching the\n",
    "                            # time duration values for this test and setting it to 'None'\n",
    "                            # for all the other tests in each grid\n",
    "                            if overall_only:\n",
    "                                if \"Overall\" in tests[i]:\n",
    "                                    try:\n",
    "                                        test_duration = graphs[i][0][\"values\"][0][j]\n",
    "                                    except IndexError:\n",
    "                                        test_duration = None\n",
    "                                else:\n",
    "                                    test_duration = None\n",
    "                            else:\n",
    "                                try:\n",
    "                                    graphs[i][0].keys()\n",
    "                                    try:\n",
    "                                        graphs[i][0][\"values\"][0][j]\n",
    "                                        test_duration = graphs[i][0][\"values\"][0][j]\n",
    "                                    except IndexError:\n",
    "                                        test_duration = None\n",
    "                                except TypeError:\n",
    "                                    test_duration = None\n",
    "\n",
    "                            decoded[i][j] = (\n",
    "                                k[0],\n",
    "                                tab,\n",
    "                                grid,\n",
    "                                tests[i],\n",
    "                                test_duration,\n",
    "                                k[1],\n",
    "                            )\n",
    "                    # accumulate the results\n",
    "                    percent_label_by_grid_csv.append(decoded)\n",
    "\n",
    "                # if label is 'None', add only the timestamp, tab, grid, tests and test\n",
    "                # duration values\n",
    "                else:\n",
    "                    for i, d in enumerate(decoded):\n",
    "                        for j, k in enumerate(d):\n",
    "                            # here we are fetching the time duration values for the tests\n",
    "                            # however,since not all tests contain time duration values,\n",
    "                            # we are only considering the 'Overall' test and fetching the time duration\n",
    "                            # values for this test in each grid\n",
    "                            if overall_only:\n",
    "                                if \"Overall\" in tests[i]:\n",
    "                                    try:\n",
    "                                        test_duration = graphs[i][0][\"values\"][0][j]\n",
    "                                    except IndexError:\n",
    "                                        print(\n",
    "                                            \"Test duration value does not exist for all \\\n",
    "                                            timestamps for test Overall in grid \",\n",
    "                                            grid,\n",
    "                                            \"in tab \",\n",
    "                                            tab,\n",
    "                                        )\n",
    "                                        test_duration = None\n",
    "                                else:\n",
    "                                    test_duration = None\n",
    "                            else:\n",
    "                                try:\n",
    "                                    graphs[i][0].keys()\n",
    "                                    try:\n",
    "                                        graphs[i][0][\"values\"][0][j]\n",
    "                                        test_duration = graphs[i][0][\"values\"][0][j]\n",
    "                                    except IndexError:\n",
    "                                        test_duration = None\n",
    "                                except TypeError:\n",
    "                                    test_duration = None\n",
    "\n",
    "                            decoded[i][j] = (k[0], tab, grid, tests[i], test_duration)\n",
    "                    percent_label_by_grid_csv.append(decoded)\n",
    "\n",
    "    # output above leaves us with a doubly nested list. Flatten\n",
    "    flat_list = [item for sublist in percent_label_by_grid_csv for item in sublist]\n",
    "    flatter_list = [item for sublist in flat_list for item in sublist]\n",
    "\n",
    "    return flatter_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculation\n",
    "- In this section, calculate the metric values from the data.\n",
    "- For example, we shall use flake metrics calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.331427Z",
     "start_time": "2021-03-24T18:48:19.231697Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50999978\n"
     ]
    }
   ],
   "source": [
    "unrolled_list = testgrid_labelwise_encoding(testgrid_data, 13)\n",
    "print(len(unrolled_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.379108Z",
     "start_time": "2021-03-24T18:48:19.333623Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>test_duration</th>\n",
       "      <th>flake</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-10-08 00:18:00</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>118.866667</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-10-07 20:37:44</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>113.750000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-10-05 16:04:16</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>91.416667</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-10-04 20:36:36</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>117.950000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-10-04 00:00:45</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-10-08 00:18:00  \"redhat-assisted-installer\"   \n",
       "1 2021-10-07 20:37:44  \"redhat-assisted-installer\"   \n",
       "2 2021-10-05 16:04:16  \"redhat-assisted-installer\"   \n",
       "3 2021-10-04 20:36:36  \"redhat-assisted-installer\"   \n",
       "4 2021-10-04 00:00:45  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid  \\\n",
       "0  periodic-ci-openshift-release-master-nightly-4...   \n",
       "1  periodic-ci-openshift-release-master-nightly-4...   \n",
       "2  periodic-ci-openshift-release-master-nightly-4...   \n",
       "3  periodic-ci-openshift-release-master-nightly-4...   \n",
       "4  periodic-ci-openshift-release-master-nightly-4...   \n",
       "\n",
       "                                                test  test_duration  flake  \n",
       "0  periodic-ci-openshift-release-master-nightly-4...     118.866667  False  \n",
       "1  periodic-ci-openshift-release-master-nightly-4...     113.750000  False  \n",
       "2  periodic-ci-openshift-release-master-nightly-4...      91.416667  False  \n",
       "3  periodic-ci-openshift-release-master-nightly-4...     117.950000  False  \n",
       "4  periodic-ci-openshift-release-master-nightly-4...            NaN  False  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Next, print the data frame head with the metric values.\n",
    "df = pd.DataFrame(\n",
    "    unrolled_list[0:200],\n",
    "    columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\", \"flake\"],\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.409135Z",
     "start_time": "2021-03-24T18:48:19.381155Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test</th>\n",
       "      <th>flake_severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Overall</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TestInstall_test_install.prepare_for_installat...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TestInstall_test_install.start_install_and_wai...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>download_logs_download_logs_suite.download_logs</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>download_logs_download_logs_suite.download_mus...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>openshift-tests.[k8s.io] Probing container wit...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] CustomReso...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>openshift-tests.[sig-api-machinery] Watchers s...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>operator.Import the release payload \"latest\" f...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>operator.Run multi-stage test e2e-metal-assist...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>operator.Run multi-stage test e2e-metal-assist...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>operator.Run multi-stage test e2e-metal-assist...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>operator.Run multi-stage test e2e-metal-assist...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 test  flake_severity\n",
       "0                                             Overall             0.0\n",
       "1   TestInstall_test_install.prepare_for_installat...             0.0\n",
       "2   TestInstall_test_install.start_install_and_wai...             0.0\n",
       "3     download_logs_download_logs_suite.download_logs             0.0\n",
       "4   download_logs_download_logs_suite.download_mus...             0.0\n",
       "5   openshift-tests.[k8s.io] Probing container wit...             0.0\n",
       "6   openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "7   openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "8   openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "9   openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "10  openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "11  openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "12  openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "13  openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "14  openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "15  openshift-tests.[sig-api-machinery] CustomReso...             0.0\n",
       "16  openshift-tests.[sig-api-machinery] Watchers s...             0.0\n",
       "17  operator.Import the release payload \"latest\" f...             0.0\n",
       "18  operator.Run multi-stage test e2e-metal-assist...             0.0\n",
       "19  operator.Run multi-stage test e2e-metal-assist...             0.0\n",
       "20  operator.Run multi-stage test e2e-metal-assist...             0.0\n",
       "21  operator.Run multi-stage test e2e-metal-assist...             0.0\n",
       "22  periodic-ci-openshift-release-master-nightly-4...             0.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Calculate flake severity or flake rate by tests\n",
    "## Moving forward, this will be aggregated in Superset\n",
    "## For the sake of completeness, it is implmented here\n",
    "flake_severity = df.groupby(\"test\").flake.mean().reset_index()\n",
    "flake_severity.rename(columns={\"flake\": \"flake_severity\"}, inplace=True)\n",
    "flake_severity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization \n",
    "- In this optional section, you can provide a quick visualization of the metric that you computed.\n",
    "- Although, the main visualization will be done in Superset, the graphs helps to understand the metric better.\n",
    "- The following plot shows the flake severity metric for a snapshot of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-24T18:48:19.784725Z",
     "start_time": "2021-03-24T18:48:19.412228Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3sAAAJWCAYAAADhiJ6/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAk20lEQVR4nO3da5SV5X3/4e/M4KANUBwEMqjR6gqUVvFEio3V5QIVDxATrWIxjXF5jC3GE4k1FOIJO4bqiopFG2O0ZXmOmkEUNbaR2MTWqssYEk8BjUBAAeUURGbm/8KV+UtBHJHZm7m9rlczez/P3j/gXsN85rn3npq2tra2AAAAUJTaag8AAADAlif2AAAACiT2AAAACiT2AAAACiT2AAAACiT2AAAACtStEk+ybNmyfOMb38hrr72W+vr67LLLLrnkkkvS0NCQQYMGZeDAgamtfa87r7zyygwaNKgSYwEAABSrphK/Z++tt97KCy+8kGHDhiVJmpqa8vbbb2fy5MkZNGhQnn766XzqU5/q7DEAAAA+MSqyjbN3797toZcke++9dxYsWFCJpwYAAPhEqsg2zvdrbW3NbbfdluHDh7ff9rd/+7dpaWnJQQcdlHHjxqW+vr7SYwEAABSlIts43+/iiy/OokWLct1116W2tjYLFy5MY2NjVq5cmfHjx2fgwIE599xzKzkSAABAcSp6Za+pqSmvvvpqpk2b1v6GLI2NjUmSHj165LjjjsvNN9/8kR932bJVaW2taLOyhfXp0yNLlqys9hgUyvqiM1lfdCbri85mjXV9tbU12X77jb//ScVi76qrrsrzzz+fG2+8sX2b5ttvv53u3btn2223zbp16zJr1qwMHjz4Iz92a2ub2CuAf0M6k/VFZ7K+6EzWF53NGitXRWLvpZdeyg033JBdd901J5xwQpJkp512yqmnnpqJEyempqYm69atyz777JOvf/3rlRgJAACgaBWJvc9+9rN54YUXNnpfc3NzJUYAAAD4RKnIr14AAACgssQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgcQeAABAgbpVewAA6MrWvtuSvn17bta5a95ZlxXLf7+FJwKA94g9APgY6repy+jz79+sc5v/+eis2MLzAMAf2MYJAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQoG6VeJJly5blG9/4Rl577bXU19dnl112ySWXXJKGhoY8++yzmThxYt55553suOOO+c53vpM+ffpUYiwAAIBiVeTKXk1NTU499dTMmjUrzc3N2XnnnTNlypS0trZm/PjxmThxYmbNmpWhQ4dmypQplRgJAACgaBWJvd69e2fYsGHtn++9995ZsGBBnn/++XTv3j1Dhw5Nkpxwwgl56KGHKjESAABA0SqyjfP9Wltbc9ttt2X48OFZuHBhBgwY0H5fQ0NDWltb89Zbb6V3794dfsw+fXp0wqRUWt++Pas9AgWzvthaWZt8GGuEzmaNlavisXfppZfmj/7oj/LlL385jzzyyBZ5zCVLVqa1tW2LPBbV0bdvz7zxxopqj0GhrC8608f9JsnaZFN8/aKzWWNdX21tzQde/Kpo7DU1NeXVV1/NtGnTUltbm8bGxixYsKD9/qVLl6a2tvYjXdUDAABgQxX71QtXXXVVnn/++UydOjX19fVJkj322CNr1qzJU089lSS5/fbbc/jhh1dqJAAAgGJV5MreSy+9lBtuuCG77rprTjjhhCTJTjvtlKlTp+bKK6/MpEmT1vvVCwAAAHw8FYm9z372s3nhhRc2et++++6b5ubmSowBAADwiVGxbZwAAABUjtgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAokNgDAAAoULdKPVFTU1NmzZqV+fPnp7m5OQMHDkySDB8+PPX19enevXuS5IILLsiBBx5YqbEAAACKVLHYGzFiRL7yla/kxBNP3OC+a665pj3+AAAA+PgqFntDhw6t1FMBAAB84lUs9jblggsuSFtbW/bbb7+cd9556dWr10c6v0+fHp00GZXUt2/Pao9AwawvtlbWJh/GGqGzWWPlqnrsTZ8+PY2NjVm7dm0uv/zyXHLJJZkyZcpHeowlS1amtbWtkyakEvr27Zk33lhR7TEolPVFZ/q43yRZm2yKr190Nmus66utrfnAi19VfzfOxsbGJEl9fX3Gjh2bp59+usoTAQAAdH1Vjb3Vq1dnxYr3fpLQ1taWmTNnZvDgwdUcCQAAoAgV28Z52WWX5eGHH86bb76Zk08+Ob179860adMybty4tLS0pLW1NbvvvnsmTZpUqZEAAACKVbHYmzBhQiZMmLDB7ffdd1+lRgAAAPjEqPpr9gAAANjyxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBunXkoJdffjm9e/fODjvskFWrVuWmm25KbW1tTjnllGy33XadPSMAAAAfUYeu7J133nlZvnx5kqSpqSn/8z//k2effTYTJ07s1OEAAADYPB26sjd//vzstttuaWtryyOPPJIHHngg2267bUaMGNHZ8wEAALAZOhR73bt3z8qVK/PKK6+ksbExDQ0NWbduXd55553Ong8AAIDN0KHYGzVqVE466aSsWrUqX/7yl5Mkc+bMyU477dSpwwEAALB5OhR7F110UX7605+mW7du2X///ZMkNTU1+Yd/+IdOHQ4AAIDN06E3aLnsssvyV3/1V+2hlyR77rlnHnvssU4bDAAAgM3Xodj74Q9/uNHbf/SjH23RYQAAANgyNrmN8+67706StLS0tH/8B7/97W/Tu3fvThsMAACAzbfJ2Lv//vuTJO+++277x8l7r9fbYYcd0tTU1LnTAQAAsFk2GXv/9m//liS5+uqrc+6551ZkIAAAAD6+Dr1m7w+/diF5b0vnPffck/vuuy+tra2dOhwAAACbp0Oxd8YZZ+TVV19Nklx11VX5/ve/n5tvvjn/9E//1KnDAQAAsHk6FHvz5s3L4MGDkyTNzc3513/919xyyy2ZOXNmpw4HAADA5unQL1Wvra3Nu+++m7lz56Znz54ZMGBAWltb27d2AgAAsHXpUOwddNBB+frXv5633norRx55ZJLk5ZdfTv/+/Tt1OAAAADZPh2Lv8ssvz7333ptu3brl6KOPTpIsW7Ys48aN69ThAAAA2Dwdir36+vqMGTMmra2tefPNN9OvX78MGzass2cDAABgM3XoDVqWL1+e888/P0OGDMlhhx2WJPnxj3+cq6++ulOHAwAAYPN0KPYmTZqUHj165LHHHss222yTJNlnn33y4IMPdupwAAAAbJ4ObeP82c9+ltmzZ2ebbbZJTU1NkqShoSFLlizp1OEAAADYPB26stezZ88sW7ZsvdsWLFiQvn37dspQAAAAfDybjL0ZM2YkSY477ricffbZ+fnPf57W1tY888wz+eY3v5kTTjihIkMCAADw0Wwy9iZOnJgkOe2003LEEUfkkksuybp163LRRRdlxIgROemkkyoyJAAAAB/NJl+z19bWliSpqanJSSedJO4AAAC6iE3GXmtra37+85+3R9/G/OVf/uUWHwoAAICPZ5Oxt3bt2nzrW9/6wNirqanJj3/8404ZDAAAgM23ydjbbrvtxBwAAEAX1KFfvQAAAEDXssnY29Rr9QAAANh6bTL2nnnmmUrNAQAAwBZkGycAAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBxB4AAECBKhJ7TU1NGT58eAYNGpQXX3yx/fa5c+dmzJgxGTlyZMaMGZN58+ZVYhwAAIDiVST2RowYkenTp2fHHXdc7/ZJkyZl7NixmTVrVsaOHZuJEydWYhwAAIDiVST2hg4dmsbGxvVuW7JkSebMmZNRo0YlSUaNGpU5c+Zk6dKllRgJAACgaFV7zd7ChQvTv3//1NXVJUnq6urSr1+/LFy4sFojAQAAFKNbtQfYEvr06VHtEdgC+vbtWe0RKJj1xdbK2uTDWCN0NmusXFWLvcbGxixatCgtLS2pq6tLS0tLFi9evMF2z45YsmRlWlvbOmFKKqVv3555440V1R6DQllfdKaP+02Stcmm+PpFZ7PGur7a2poPvPhVtW2cffr0yeDBgzNjxowkyYwZMzJ48OA0NDRUayQAAIBiVOTK3mWXXZaHH344b775Zk4++eT07t07DzzwQL797W/nwgsvzPXXX59evXqlqampEuMAAAAUryKxN2HChEyYMGGD23fffffcddddlRgBAADgE6Vq2zgBAADoPGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQN2qPUCSDB8+PPX19enevXuS5IILLsiBBx5Y5akAAAC6rq0i9pLkmmuuycCBA6s9BgAAQBFs4wQAACjQVnNl74ILLkhbW1v222+/nHfeeenVq1e1RwIAAOiyatra2tqqPcTChQvT2NiYtWvX5vLLL8+qVasyZcqUao8FAB0y+vz7N+u85n8+egtPAgD/31ZxZa+xsTFJUl9fn7Fjx+ZrX/vaRzp/yZKVaW2terPyMfTt2zNvvLGi2mNQKOuLztS3b8+Pdb61yab4+kVns8a6vtramvTp02Pj91V4lg2sXr06K1a8t8Da2toyc+bMDB48uMpTAQAAdG1Vv7K3ZMmSjBs3Li0tLWltbc3uu++eSZMmVXssAACALq3qsbfzzjvnvvvuq/YYAAAARan6Nk4AAAC2PLEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQILEHAABQoK0i9ubOnZsxY8Zk5MiRGTNmTObNm1ftkQAAALq0rSL2Jk2alLFjx2bWrFkZO3ZsJk6cWO2RAAAAurRu1R5gyZIlmTNnTm6++eYkyahRo3LppZdm6dKlaWho6NBj1NbWdOaIVIh/RzqT9UVn6rf9dpt9rrXJh7FG6GzWWNe2qX+/qsfewoUL079//9TV1SVJ6urq0q9fvyxcuLDDsbf99p/qzBGpkD59elR7BApmfdGZbppw2Gafa23yYawROps1Vq6tYhsnAAAAW1bVY6+xsTGLFi1KS0tLkqSlpSWLFy9OY2NjlScDAADouqoee3369MngwYMzY8aMJMmMGTMyePDgDm/hBAAAYEM1bW1tbdUe4pVXXsmFF16Y5cuXp1evXmlqaspuu+1W7bEAAAC6rK0i9gAAANiyqr6NEwAAgC1P7AEAABRI7AEAABRI7AEAABRI7FEVv//973POOefk0EMPzeGHH57/+I//2OTx77zzTo466qgcc8wxFZqQrqyj6+vRRx/NMccck1GjRuWoo47K97///QpPSlcxd+7cjBkzJiNHjsyYMWMyb968DY5paWnJxRdfnEMOOSSHHnpo7rrrrsoPSpfVkTU2derUHHXUURk9enSOOeaYzJ49u/KD0iV1ZH39wW9+85vstddeaWpqqtyAdBqxR1XcdNNN6dGjRx555JFMmzYtEyZMyKpVqz7w+Kuvvjp77bVXBSekK+vo+urbt2/+5V/+JTNmzMjtt9+e2267LU899VQVJmZrN2nSpIwdOzazZs3K2LFjM3HixA2OaW5uzmuvvZaHH344d9xxR6699tq8/vrrVZiWrqgja2zIkCG5++6709zcnMmTJ+fcc8/NmjVrqjAtXU1H1lfy3g+tJk2alEMOOaTCE9JZxB5V8eCDD2bMmDFJkl133TV77LFHHn/88Y0e+9RTT2XevHk5+uijKzkiXVhH19dee+2V/v37J0l69uyZ3XffPfPnz6/orGz9lixZkjlz5mTUqFFJklGjRmXOnDlZunTpesfNnDkzxx13XGpra9PQ0JBDDjkkDz30UDVGpovp6Bo78MADs9122yVJBg0alLa2trz11luVHpcupqPrK0luvPHGHHzwwdl1110rPCWdRexRFQsWLMiOO+7Y/nljY2N+97vfbXDc6tWrM3ny5Fx88cWVHI8urqPr6/1eeeWVPPvss9l///07ezy6mIULF6Z///6pq6tLktTV1aVfv35ZuHDhBscNGDCg/fOOrDtIOr7G3u++++7LZz7zmXz605+u1Jh0UR1dX7/+9a/z05/+NF/96lerMCWdpVu1B6BMX/rSl7JgwYKN3vdf//VfHX6cK6+8MmPHjk3//v03ub+cT5Yttb7+YPHixTnrrLMyadKk9it9AFur//7v/853v/tdrzNmi3n33Xfzj//4j7niiivao5AyiD06xb333rvJ+wcMGJD58+enoaEhyXs/dRo2bNgGx/3v//5vHn/88Vx//fV555138vbbb2f06NFpbm7ulLnpGrbU+kre295y8skn59RTT80RRxyxxWel62tsbMyiRYvS0tKSurq6tLS0ZPHixWlsbNzguAULFmTIkCFJNrzSBx+ko2ssSZ555pmMHz8+119/fXbbbbcqTEtX05H19cYbb+S1117L6aefniRZvnx52trasnLlylx66aXVGp0twDZOquLwww/PHXfckSSZN29efvGLX+TAAw/c4Ljm5uY89thjeeyxx3LVVVdl4MCBQo8P1dH1tWzZspx88sk58cQTc9xxx1V6TLqIPn36ZPDgwZkxY0aSZMaMGRk8eHD7DxP+4PDDD89dd92V1tbWLF26NI8++mhGjhxZjZHpYjq6xp577rmce+65ueaaa/Lnf/7n1RiVLqgj62vAgAF58skn27/nOumkk3L88ccLvQLUtLW1tVV7CD55Vq9enQsvvDC/+tWvUltbm/Hjx7e/89N3v/vd9OvXL3/zN3+z3jlPPvlkmpqa8sMf/rAaI9OFdHR9NTU1Zfr06fmTP/mT9nO/8pWv5Nhjj63W6GylXnnllVx44YVZvnx5evXqlaampuy222457bTTcvbZZ2fPPfdMS0tLLrnkkjzxxBNJktNOO639jYLgw3RkjR177LGZP3/+etvNr7zyygwaNKiKk9MVdGR9vd+1116b1atX55vf/GaVJmZLEXsAAAAFso0TAACgQGIPAACgQGIPAACgQGIPAACgQGIPAACgQGIPgC7n9ddfz6BBg7Ju3boPPfbJJ5/MQQcdVIGpKmefffbJb3/722qPAcBWrlu1BwCADzJ8+PC8+eabqaura7/toYceqsosa9euzVVXXZWZM2dmxYoV2X777TNixIh861vfqvgszzzzTPvHF154Yfr3759zzz234nMAsHUTewBs1aZNm5bPf/7z6932+uuvV3yOG2+8Mc8//3zuuuuu9OvXL/Pnz89TTz1V0RnWrVuXbt381w1Ax9jGCUCXd8899+SII47IPvvskxEjRuT222//wGNvvfXWHHnkkfnd736XtWvXpqmpKQcffHA+//nPZ+LEiVmzZs1Gz/vFL36RQw45JP37909NTU122mmnfPGLX2y/f9GiRRk3blz233//DB8+PLfeemv77UOGDMlbb73VfuycOXMybNiwvPvuu0mSu+++O0cccUQ+97nP5ZRTTsn8+fPbjx00aFCmT5+eww47LIcddlj7ba+++mruuOOONDc356abbso+++yTM888M9/73vcybty49Wa/7LLLctlll32kv1MAuj6xB0CX16dPn9xwww15+umnc8UVV+SKK67IL3/5yw2Ou+6663Lvvffm3//93/PpT386U6ZMydy5c3Pffffl4YcfzuLFizN16tSNPsdee+2VH/zgB5k+fXpeeOGFtLW1td/X2tqar33taxk0aFAef/zx3HLLLbnlllsye/bs9O/fP3vvvXcefvjh9uObm5szcuTIbLPNNnn00Udzww035LrrrsvPfvaz7Lfffjn//PPXe+5HH300d955Z2bOnLne7WPGjMno0aNzyimn5Jlnnsm0adPyhS98IbNnz87y5cuTvHc18IEHHlgvTAH4ZBB7AGzV/u7v/i5Dhw7N0KFDc9ZZZ230mIMPPjif+cxnUlNTk7/4i7/IAQccsN4Wy7a2tlxxxRV54okncuutt6ahoSFtbW258847c9FFF6V3797p0aNHzjjjjDzwwAMbfY4zzjgjp512Wpqbm3PsscfmwAMPzL333pvkvat+S5cuzd///d+nvr4+O++8c44//vj2OBs9enRmzJjRPsvMmTMzevToJMntt9+e008/Pbvvvnu6deuWM888M7/61a/Wu7p3+umnp3fv3tl2220/9O+rX79+GTp0aPtrG2fPnp3tt98+e+yxx4eeC0BZbPwHYKs2derUDV6z93/95Cc/ydSpUzNv3ry0trZmzZo1GThwYPv9K1asyJ133pmrr746PXv2TJIsXbo0v//973PMMce0H9fW1pbW1taNPkddXV1OPPHEnHjiiVmzZk3uueeeXHTRRRkyZEjmz5+fxYsXZ+jQoe3Ht7S0tH9+2GGH5dJLL83ixYszb9681NbWtt+3YMGCTJ48OU1NTevNsWjRouy4445JksbGxo/yV5YvfelLue2223L88cfnRz/6UY4++uiPdD4AZRB7AHRpa9euzdlnn52mpqaMGDEi22yzTc4666z1tln26tUr3/nOd3LOOefkuuuuy3777Zftt98+2267bR544IH079//Iz3ntttumxNPPDHXXHNNXn755TQ2NmannXZab6vm+/3xH/9xDjjggMycOTO/+c1vcuSRR6ampibJeyF35pln5gtf+MIHPt8fju3ofYcccki+/e1v58UXX8x//ud/Zvz48R/pzwdAGWzjBKBLW7t2bdauXZuGhoZ069YtP/nJT/LEE09scNywYcMyZcqUjBs3Ls8991xqa2tz3HHHZfLkyVmyZEmS995MZfbs2Rt9nh/84Ad58skns2bNmqxbty733ntvVq1alT/7sz/LkCFD8qlPfSo33nhj1qxZk5aWlrz44ot57rnn2s8fPXp07r///syaNat9C2eSnHDCCbnxxhvz0ksvJXnvKuSDDz7Y4T9/nz59Nnh30u7du2fkyJE5//zzs+eee2bAgAEdfjwAyiH2AOjSevTokQkTJuScc87J5z73ucyYMSPDhw/f6LEHHHBAJk+enDPPPDO//OUvM378+Oyyyy45/vjjs+++++arX/1q5s6du9Fzt9tuuzQ1NeWAAw7I/vvvn+nTp+faa6/NzjvvnLq6ukybNi2//vWvM2LEiOy///6ZMGFCVq5c2X7+8OHDM2/evOywww750z/90/bbDz300Jx66qk577zzsu+++2bUqFF5/PHHO/zn/+u//uu8/PLLG7ym8Ytf/GJefPFFWzgBPsFq2t6/zwUAKMKCBQtyxBFH5IknnkiPHj2qPQ4AVeDKHgAUprW1NTfffHOOPPJIoQfwCeYNWgCgIKtXr84BBxyQAQMG5Hvf+161xwGgimzjBAAAKJBtnAAAAAUSewAAAAUSewAAAAUSewAAAAUSewAAAAUSewAAAAX6f568NvcEGl/mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set(rc={\"figure.figsize\": (15, 10)})\n",
    "flake_severity.flake_severity.hist(bins=50)\n",
    "plt.ylabel(\"Tests\")\n",
    "plt.xlabel(\"Flake Severity\")\n",
    "plt.ylim((0, 25))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save results to Ceph or locally\n",
    "* Use the following helper function to save the data frame in a parquet format on the Ceph bucket if we are running in automation, and locally if not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp = datetime.datetime.now()\n",
    "\n",
    "if AUTOMATION:\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    cc.upload_to_ceph(\n",
    "        df,\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    save_to_disk(\n",
    "        df,\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>test_duration</th>\n",
       "      <th>flake</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-10-08 00:18:00</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>118.866667</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-10-07 20:37:44</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>113.750000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-10-05 16:04:16</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>91.416667</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-10-04 20:36:36</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>117.950000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-10-04 00:00:45</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>2021-09-30 20:16:50</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[k8s.io] Probing container wit...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>2021-09-28 11:26:39</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[k8s.io] Probing container wit...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>2021-09-27 00:00:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[k8s.io] Probing container wit...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>2021-10-08 00:18:00</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[sig-api-machinery] Watchers s...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>2021-10-07 20:37:44</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>openshift-tests.[sig-api-machinery] Watchers s...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              timestamp                          tab  \\\n",
       "0   2021-10-08 00:18:00  \"redhat-assisted-installer\"   \n",
       "1   2021-10-07 20:37:44  \"redhat-assisted-installer\"   \n",
       "2   2021-10-05 16:04:16  \"redhat-assisted-installer\"   \n",
       "3   2021-10-04 20:36:36  \"redhat-assisted-installer\"   \n",
       "4   2021-10-04 00:00:45  \"redhat-assisted-installer\"   \n",
       "..                  ...                          ...   \n",
       "195 2021-09-30 20:16:50  \"redhat-assisted-installer\"   \n",
       "196 2021-09-28 11:26:39  \"redhat-assisted-installer\"   \n",
       "197 2021-09-27 00:00:32  \"redhat-assisted-installer\"   \n",
       "198 2021-10-08 00:18:00  \"redhat-assisted-installer\"   \n",
       "199 2021-10-07 20:37:44  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                  grid  \\\n",
       "0    periodic-ci-openshift-release-master-nightly-4...   \n",
       "1    periodic-ci-openshift-release-master-nightly-4...   \n",
       "2    periodic-ci-openshift-release-master-nightly-4...   \n",
       "3    periodic-ci-openshift-release-master-nightly-4...   \n",
       "4    periodic-ci-openshift-release-master-nightly-4...   \n",
       "..                                                 ...   \n",
       "195  periodic-ci-openshift-release-master-nightly-4...   \n",
       "196  periodic-ci-openshift-release-master-nightly-4...   \n",
       "197  periodic-ci-openshift-release-master-nightly-4...   \n",
       "198  periodic-ci-openshift-release-master-nightly-4...   \n",
       "199  periodic-ci-openshift-release-master-nightly-4...   \n",
       "\n",
       "                                                  test  test_duration  flake  \n",
       "0    periodic-ci-openshift-release-master-nightly-4...     118.866667  False  \n",
       "1    periodic-ci-openshift-release-master-nightly-4...     113.750000  False  \n",
       "2    periodic-ci-openshift-release-master-nightly-4...      91.416667  False  \n",
       "3    periodic-ci-openshift-release-master-nightly-4...     117.950000  False  \n",
       "4    periodic-ci-openshift-release-master-nightly-4...            NaN  False  \n",
       "..                                                 ...            ...    ...  \n",
       "195  openshift-tests.[k8s.io] Probing container wit...            NaN  False  \n",
       "196  openshift-tests.[k8s.io] Probing container wit...            NaN  False  \n",
       "197  openshift-tests.[k8s.io] Probing container wit...            NaN  False  \n",
       "198  openshift-tests.[sig-api-machinery] Watchers s...            NaN  False  \n",
       "199  openshift-tests.[sig-api-machinery] Watchers s...            NaN  False  \n",
       "\n",
       "[200 rows x 6 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Sanity check to see if the dataset is the same\n",
    "if AUTOMATION:\n",
    "    sanity_check = cc.read_from_ceph(\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    sanity_check = read_from_disk(\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "\n",
    "sanity_check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "- Report the status of the calculation that you aimed to complete in this notebook.\n",
    "- Highlight any difficulties in the calculation of the metric.\n",
    "- Optional: Add any side notes, future efforts\n",
    "\n",
    "### _Example Conclusion_ \n",
    "_This notebook computed number of flakes and the flake severity metric. The dataframe saved on ceph can be used to generate aggregated views and visualizations._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}