README.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:8ffe3f1937cce3967967f5c0e40f02ab9431fdd76865cd7af51da63bc653bd41"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "KDD Cup 99 - PySpark"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This is my try with the *KDD Cup of 1999* using Python, Scikit-learn, and Spark.  \n",
      "The dataset for this data mining competition can be found [here](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Task description summary"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can find the complete description of the task [here](http://kdd.ics.uci.edu/databases/kddcup99/task.html).  \n",
      "\n",
      "Software to detect network intrusions protects a computer network from unauthorized users, including perhaps insiders.  The intrusion detector learning task is to build a predictive model (i.e. a classifier) capable of distinguishing between *bad connections*, called intrusions or attacks, and *good normal connections*.  \n",
      "\n",
      "A connection is a sequence of TCP packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol.  Each connection is labeled as either normal, or as an attack, with exactly one specific attack type.  Each connection record consists of about 100 bytes.  \n",
      "\n",
      "Attacks fall into four main categories:  \n",
      "\n",
      "- DOS: denial-of-service, e.g. syn flood;  \n",
      "- R2L: unauthorized access from a remote machine, e.g. guessing password;  \n",
      "- U2R:  unauthorized access to local superuser (root) privileges, e.g., various ``buffer overflow'' attacks;  \n",
      "- probing: surveillance and other probing, e.g., port scanning.  \n",
      "\n",
      "It is important to note that the test data is not from the same probability distribution as the training data, and it includes specific attack types not in the training data. This makes the task more realistic. The datasets contain a total of 24 training attack types, with an additional 14 types in the test data only.    \n",
      "\n",
      "Some intrusion experts believe that most novel attacks are variants of known attacks and the \"signature\" of known attacks can be sufficient to catch novel variants. Based on this idea, we will experiment with different machine learning approaches.  "
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Approach"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We will start by working on a reduced dataset (the 10 percent dataset provided).  \n",
      "\n",
      "There we will do some exploratory data analysis using `Pandas`. Then we will build a classifier using `Scikit-learn`. Our classifier will just classify entries into `normal` or `attack`. By doing so, we can generalise the model to new attack types.    \n",
      "\n",
      "However, in our final approach we want to use clustering and anomality detection. We want our model to be able to work well with unknown attack types and also to give an approchimation of the closest attack type. Initially we will do clustering using `Scikit-learn` again and see if we can beat our previous classifier.  \n",
      "\n",
      "Finally, we will use `Spark` to implement the clustering approach on the complete dataset containing around 5 million interactions.  "
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Loading the data"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas\n",
      "from time import time\n",
      "col_names = [\"duration\",\"protocol_type\",\"service\",\"flag\",\"src_bytes\",\n",
      "    \"dst_bytes\",\"land\",\"wrong_fragment\",\"urgent\",\"hot\",\"num_failed_logins\",\n",
      "    \"logged_in\",\"num_compromised\",\"root_shell\",\"su_attempted\",\"num_root\",\n",
      "    \"num_file_creations\",\"num_shells\",\"num_access_files\",\"num_outbound_cmds\",\n",
      "    \"is_host_login\",\"is_guest_login\",\"count\",\"srv_count\",\"serror_rate\",\n",
      "    \"srv_serror_rate\",\"rerror_rate\",\"srv_rerror_rate\",\"same_srv_rate\",\n",
      "    \"diff_srv_rate\",\"srv_diff_host_rate\",\"dst_host_count\",\"dst_host_srv_count\",\n",
      "    \"dst_host_same_srv_rate\",\"dst_host_diff_srv_rate\",\"dst_host_same_src_port_rate\",\n",
      "    \"dst_host_srv_diff_host_rate\",\"dst_host_serror_rate\",\"dst_host_srv_serror_rate\",\n",
      "    \"dst_host_rerror_rate\",\"dst_host_srv_rerror_rate\",\"label\"]\n",
      "kdd_data_10percent = pandas.read_csv(\"/nfs/data/KDD99/kddcup.data_10_percent\", header=None, names = col_names)\n",
      "kdd_data_10percent.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>duration</th>\n",
        "      <th>src_bytes</th>\n",
        "      <th>dst_bytes</th>\n",
        "      <th>land</th>\n",
        "      <th>wrong_fragment</th>\n",
        "      <th>urgent</th>\n",
        "      <th>hot</th>\n",
        "      <th>num_failed_logins</th>\n",
        "      <th>logged_in</th>\n",
        "      <th>num_compromised</th>\n",
        "      <th>...</th>\n",
        "      <th>dst_host_count</th>\n",
        "      <th>dst_host_srv_count</th>\n",
        "      <th>dst_host_same_srv_rate</th>\n",
        "      <th>dst_host_diff_srv_rate</th>\n",
        "      <th>dst_host_same_src_port_rate</th>\n",
        "      <th>dst_host_srv_diff_host_rate</th>\n",
        "      <th>dst_host_serror_rate</th>\n",
        "      <th>dst_host_srv_serror_rate</th>\n",
        "      <th>dst_host_rerror_rate</th>\n",
        "      <th>dst_host_srv_rerror_rate</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 4.940210e+05</td>\n",
        "      <td>  494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td>...</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td>     47.979302</td>\n",
        "      <td> 3.025610e+03</td>\n",
        "      <td>     868.532425</td>\n",
        "      <td>      0.000045</td>\n",
        "      <td>      0.006433</td>\n",
        "      <td>      0.000014</td>\n",
        "      <td>      0.034519</td>\n",
        "      <td>      0.000152</td>\n",
        "      <td>      0.148247</td>\n",
        "      <td>      0.010212</td>\n",
        "      <td>...</td>\n",
        "      <td>    232.470778</td>\n",
        "      <td>    188.665670</td>\n",
        "      <td>      0.753780</td>\n",
        "      <td>      0.030906</td>\n",
        "      <td>      0.601935</td>\n",
        "      <td>      0.006684</td>\n",
        "      <td>      0.176754</td>\n",
        "      <td>      0.176443</td>\n",
        "      <td>      0.058118</td>\n",
        "      <td>      0.057412</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>    707.746472</td>\n",
        "      <td> 9.882181e+05</td>\n",
        "      <td>   33040.001252</td>\n",
        "      <td>      0.006673</td>\n",
        "      <td>      0.134805</td>\n",
        "      <td>      0.005510</td>\n",
        "      <td>      0.782103</td>\n",
        "      <td>      0.015520</td>\n",
        "      <td>      0.355345</td>\n",
        "      <td>      1.798326</td>\n",
        "      <td>...</td>\n",
        "      <td>     64.745380</td>\n",
        "      <td>    106.040437</td>\n",
        "      <td>      0.410781</td>\n",
        "      <td>      0.109259</td>\n",
        "      <td>      0.481309</td>\n",
        "      <td>      0.042133</td>\n",
        "      <td>      0.380593</td>\n",
        "      <td>      0.380919</td>\n",
        "      <td>      0.230590</td>\n",
        "      <td>      0.230140</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 0.000000e+00</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 4.500000e+01</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>     46.000000</td>\n",
        "      <td>      0.410000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 5.200000e+02</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 1.032000e+03</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.040000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td>  58329.000000</td>\n",
        "      <td> 6.933756e+08</td>\n",
        "      <td> 5155468.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      3.000000</td>\n",
        "      <td>      3.000000</td>\n",
        "      <td>     30.000000</td>\n",
        "      <td>      5.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>    884.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>8 rows \u00d7 38 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 2,
       "text": [
        "            duration     src_bytes       dst_bytes           land  \\\n",
        "count  494021.000000  4.940210e+05   494021.000000  494021.000000   \n",
        "mean       47.979302  3.025610e+03      868.532425       0.000045   \n",
        "std       707.746472  9.882181e+05    33040.001252       0.006673   \n",
        "min         0.000000  0.000000e+00        0.000000       0.000000   \n",
        "25%         0.000000  4.500000e+01        0.000000       0.000000   \n",
        "50%         0.000000  5.200000e+02        0.000000       0.000000   \n",
        "75%         0.000000  1.032000e+03        0.000000       0.000000   \n",
        "max     58329.000000  6.933756e+08  5155468.000000       1.000000   \n",
        "\n",
        "       wrong_fragment         urgent            hot  num_failed_logins  \\\n",
        "count   494021.000000  494021.000000  494021.000000      494021.000000   \n",
        "mean         0.006433       0.000014       0.034519           0.000152   \n",
        "std          0.134805       0.005510       0.782103           0.015520   \n",
        "min          0.000000       0.000000       0.000000           0.000000   \n",
        "25%          0.000000       0.000000       0.000000           0.000000   \n",
        "50%          0.000000       0.000000       0.000000           0.000000   \n",
        "75%          0.000000       0.000000       0.000000           0.000000   \n",
        "max          3.000000       3.000000      30.000000           5.000000   \n",
        "\n",
        "           logged_in  num_compromised       ...         dst_host_count  \\\n",
        "count  494021.000000    494021.000000       ...          494021.000000   \n",
        "mean        0.148247         0.010212       ...             232.470778   \n",
        "std         0.355345         1.798326       ...              64.745380   \n",
        "min         0.000000         0.000000       ...               0.000000   \n",
        "25%         0.000000         0.000000       ...             255.000000   \n",
        "50%         0.000000         0.000000       ...             255.000000   \n",
        "75%         0.000000         0.000000       ...             255.000000   \n",
        "max         1.000000       884.000000       ...             255.000000   \n",
        "\n",
        "       dst_host_srv_count  dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
        "count       494021.000000           494021.000000           494021.000000   \n",
        "mean           188.665670                0.753780                0.030906   \n",
        "std            106.040437                0.410781                0.109259   \n",
        "min              0.000000                0.000000                0.000000   \n",
        "25%             46.000000                0.410000                0.000000   \n",
        "50%            255.000000                1.000000                0.000000   \n",
        "75%            255.000000                1.000000                0.040000   \n",
        "max            255.000000                1.000000                1.000000   \n",
        "\n",
        "       dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
        "count                494021.000000                494021.000000   \n",
        "mean                      0.601935                     0.006684   \n",
        "std                       0.481309                     0.042133   \n",
        "min                       0.000000                     0.000000   \n",
        "25%                       0.000000                     0.000000   \n",
        "50%                       1.000000                     0.000000   \n",
        "75%                       1.000000                     0.000000   \n",
        "max                       1.000000                     1.000000   \n",
        "\n",
        "       dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
        "count         494021.000000             494021.000000         494021.000000   \n",
        "mean               0.176754                  0.176443              0.058118   \n",
        "std                0.380593                  0.380919              0.230590   \n",
        "min                0.000000                  0.000000              0.000000   \n",
        "25%                0.000000                  0.000000              0.000000   \n",
        "50%                0.000000                  0.000000              0.000000   \n",
        "75%                0.000000                  0.000000              0.000000   \n",
        "max                1.000000                  1.000000              1.000000   \n",
        "\n",
        "       dst_host_srv_rerror_rate  \n",
        "count             494021.000000  \n",
        "mean                   0.057412  \n",
        "std                    0.230140  \n",
        "min                    0.000000  \n",
        "25%                    0.000000  \n",
        "50%                    0.000000  \n",
        "75%                    0.000000  \n",
        "max                    1.000000  \n",
        "\n",
        "[8 rows x 38 columns]"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we have our data loaded into a `Pandas` data frame. In order to get familiar with our data, let's have a look at how the labels are distributed.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "kdd_data_10percent['label'].value_counts()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "smurf.              280790\n",
        "neptune.            107201\n",
        "normal.              97278\n",
        "back.                 2203\n",
        "satan.                1589\n",
        "ipsweep.              1247\n",
        "portsweep.            1040\n",
        "warezclient.          1020\n",
        "teardrop.              979\n",
        "pod.                   264\n",
        "nmap.                  231\n",
        "guess_passwd.           53\n",
        "buffer_overflow.        30\n",
        "land.                   21\n",
        "warezmaster.            20\n",
        "imap.                   12\n",
        "rootkit.                10\n",
        "loadmodule.              9\n",
        "ftp_write.               8\n",
        "multihop.                7\n",
        "phf.                     4\n",
        "perl.                    3\n",
        "spy.                     2\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Feature selection"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Initially, we will use all features. We need to do something with our categorical variables. For now, we will not include them in the training features.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "num_features = [\n",
      "    \"duration\",\"src_bytes\",\n",
      "    \"dst_bytes\",\"land\",\"wrong_fragment\",\"urgent\",\"hot\",\"num_failed_logins\",\n",
      "    \"logged_in\",\"num_compromised\",\"root_shell\",\"su_attempted\",\"num_root\",\n",
      "    \"num_file_creations\",\"num_shells\",\"num_access_files\",\"num_outbound_cmds\",\n",
      "    \"is_host_login\",\"is_guest_login\",\"count\",\"srv_count\",\"serror_rate\",\n",
      "    \"srv_serror_rate\",\"rerror_rate\",\"srv_rerror_rate\",\"same_srv_rate\",\n",
      "    \"diff_srv_rate\",\"srv_diff_host_rate\",\"dst_host_count\",\"dst_host_srv_count\",\n",
      "    \"dst_host_same_srv_rate\",\"dst_host_diff_srv_rate\",\"dst_host_same_src_port_rate\",\n",
      "    \"dst_host_srv_diff_host_rate\",\"dst_host_serror_rate\",\"dst_host_srv_serror_rate\",\n",
      "    \"dst_host_rerror_rate\",\"dst_host_srv_rerror_rate\"\n",
      "]\n",
      "features = kdd_data_10percent[num_features].astype(float)\n",
      "features.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>duration</th>\n",
        "      <th>src_bytes</th>\n",
        "      <th>dst_bytes</th>\n",
        "      <th>land</th>\n",
        "      <th>wrong_fragment</th>\n",
        "      <th>urgent</th>\n",
        "      <th>hot</th>\n",
        "      <th>num_failed_logins</th>\n",
        "      <th>logged_in</th>\n",
        "      <th>num_compromised</th>\n",
        "      <th>...</th>\n",
        "      <th>dst_host_count</th>\n",
        "      <th>dst_host_srv_count</th>\n",
        "      <th>dst_host_same_srv_rate</th>\n",
        "      <th>dst_host_diff_srv_rate</th>\n",
        "      <th>dst_host_same_src_port_rate</th>\n",
        "      <th>dst_host_srv_diff_host_rate</th>\n",
        "      <th>dst_host_serror_rate</th>\n",
        "      <th>dst_host_srv_serror_rate</th>\n",
        "      <th>dst_host_rerror_rate</th>\n",
        "      <th>dst_host_srv_rerror_rate</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 4.940210e+05</td>\n",
        "      <td>  494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td>...</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td>     47.979302</td>\n",
        "      <td> 3.025610e+03</td>\n",
        "      <td>     868.532425</td>\n",
        "      <td>      0.000045</td>\n",
        "      <td>      0.006433</td>\n",
        "      <td>      0.000014</td>\n",
        "      <td>      0.034519</td>\n",
        "      <td>      0.000152</td>\n",
        "      <td>      0.148247</td>\n",
        "      <td>      0.010212</td>\n",
        "      <td>...</td>\n",
        "      <td>    232.470778</td>\n",
        "      <td>    188.665670</td>\n",
        "      <td>      0.753780</td>\n",
        "      <td>      0.030906</td>\n",
        "      <td>      0.601935</td>\n",
        "      <td>      0.006684</td>\n",
        "      <td>      0.176754</td>\n",
        "      <td>      0.176443</td>\n",
        "      <td>      0.058118</td>\n",
        "      <td>      0.057412</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>    707.746472</td>\n",
        "      <td> 9.882181e+05</td>\n",
        "      <td>   33040.001252</td>\n",
        "      <td>      0.006673</td>\n",
        "      <td>      0.134805</td>\n",
        "      <td>      0.005510</td>\n",
        "      <td>      0.782103</td>\n",
        "      <td>      0.015520</td>\n",
        "      <td>      0.355345</td>\n",
        "      <td>      1.798326</td>\n",
        "      <td>...</td>\n",
        "      <td>     64.745380</td>\n",
        "      <td>    106.040437</td>\n",
        "      <td>      0.410781</td>\n",
        "      <td>      0.109259</td>\n",
        "      <td>      0.481309</td>\n",
        "      <td>      0.042133</td>\n",
        "      <td>      0.380593</td>\n",
        "      <td>      0.380919</td>\n",
        "      <td>      0.230590</td>\n",
        "      <td>      0.230140</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 0.000000e+00</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 4.500000e+01</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>     46.000000</td>\n",
        "      <td>      0.410000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 5.200000e+02</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 1.032000e+03</td>\n",
        "      <td>       0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.040000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td>  58329.000000</td>\n",
        "      <td> 6.933756e+08</td>\n",
        "      <td> 5155468.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      3.000000</td>\n",
        "      <td>      3.000000</td>\n",
        "      <td>     30.000000</td>\n",
        "      <td>      5.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>    884.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>    255.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>8 rows \u00d7 38 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "            duration     src_bytes       dst_bytes           land  \\\n",
        "count  494021.000000  4.940210e+05   494021.000000  494021.000000   \n",
        "mean       47.979302  3.025610e+03      868.532425       0.000045   \n",
        "std       707.746472  9.882181e+05    33040.001252       0.006673   \n",
        "min         0.000000  0.000000e+00        0.000000       0.000000   \n",
        "25%         0.000000  4.500000e+01        0.000000       0.000000   \n",
        "50%         0.000000  5.200000e+02        0.000000       0.000000   \n",
        "75%         0.000000  1.032000e+03        0.000000       0.000000   \n",
        "max     58329.000000  6.933756e+08  5155468.000000       1.000000   \n",
        "\n",
        "       wrong_fragment         urgent            hot  num_failed_logins  \\\n",
        "count   494021.000000  494021.000000  494021.000000      494021.000000   \n",
        "mean         0.006433       0.000014       0.034519           0.000152   \n",
        "std          0.134805       0.005510       0.782103           0.015520   \n",
        "min          0.000000       0.000000       0.000000           0.000000   \n",
        "25%          0.000000       0.000000       0.000000           0.000000   \n",
        "50%          0.000000       0.000000       0.000000           0.000000   \n",
        "75%          0.000000       0.000000       0.000000           0.000000   \n",
        "max          3.000000       3.000000      30.000000           5.000000   \n",
        "\n",
        "           logged_in  num_compromised       ...         dst_host_count  \\\n",
        "count  494021.000000    494021.000000       ...          494021.000000   \n",
        "mean        0.148247         0.010212       ...             232.470778   \n",
        "std         0.355345         1.798326       ...              64.745380   \n",
        "min         0.000000         0.000000       ...               0.000000   \n",
        "25%         0.000000         0.000000       ...             255.000000   \n",
        "50%         0.000000         0.000000       ...             255.000000   \n",
        "75%         0.000000         0.000000       ...             255.000000   \n",
        "max         1.000000       884.000000       ...             255.000000   \n",
        "\n",
        "       dst_host_srv_count  dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
        "count       494021.000000           494021.000000           494021.000000   \n",
        "mean           188.665670                0.753780                0.030906   \n",
        "std            106.040437                0.410781                0.109259   \n",
        "min              0.000000                0.000000                0.000000   \n",
        "25%             46.000000                0.410000                0.000000   \n",
        "50%            255.000000                1.000000                0.000000   \n",
        "75%            255.000000                1.000000                0.040000   \n",
        "max            255.000000                1.000000                1.000000   \n",
        "\n",
        "       dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
        "count                494021.000000                494021.000000   \n",
        "mean                      0.601935                     0.006684   \n",
        "std                       0.481309                     0.042133   \n",
        "min                       0.000000                     0.000000   \n",
        "25%                       0.000000                     0.000000   \n",
        "50%                       1.000000                     0.000000   \n",
        "75%                       1.000000                     0.000000   \n",
        "max                       1.000000                     1.000000   \n",
        "\n",
        "       dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
        "count         494021.000000             494021.000000         494021.000000   \n",
        "mean               0.176754                  0.176443              0.058118   \n",
        "std                0.380593                  0.380919              0.230590   \n",
        "min                0.000000                  0.000000              0.000000   \n",
        "25%                0.000000                  0.000000              0.000000   \n",
        "50%                0.000000                  0.000000              0.000000   \n",
        "75%                0.000000                  0.000000              0.000000   \n",
        "max                1.000000                  1.000000              1.000000   \n",
        "\n",
        "       dst_host_srv_rerror_rate  \n",
        "count             494021.000000  \n",
        "mean                   0.057412  \n",
        "std                    0.230140  \n",
        "min                    0.000000  \n",
        "25%                    0.000000  \n",
        "50%                    0.000000  \n",
        "75%                    0.000000  \n",
        "max                    1.000000  \n",
        "\n",
        "[8 rows x 38 columns]"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As we mentioned, we are going to reduce the outputs to `normal` and `attack`. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.neighbors import KNeighborsClassifier\n",
      "labels = kdd_data_10percent['label'].copy()\n",
      "labels[labels!='normal.'] = 'attack.'\n",
      "labels.value_counts()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "attack.    396743\n",
        "normal.     97278\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Feature scaling"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We are going to use a lot of distance-based methods here. In order to avoid some features distances dominate others, we need to scale all of them."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.preprocessing import MinMaxScaler\n",
      "features.apply(lambda x: MinMaxScaler().fit_transform(x))\n",
      "features.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>duration</th>\n",
        "      <th>src_bytes</th>\n",
        "      <th>dst_bytes</th>\n",
        "      <th>land</th>\n",
        "      <th>wrong_fragment</th>\n",
        "      <th>urgent</th>\n",
        "      <th>hot</th>\n",
        "      <th>num_failed_logins</th>\n",
        "      <th>logged_in</th>\n",
        "      <th>num_compromised</th>\n",
        "      <th>...</th>\n",
        "      <th>dst_host_count</th>\n",
        "      <th>dst_host_srv_count</th>\n",
        "      <th>dst_host_same_srv_rate</th>\n",
        "      <th>dst_host_diff_srv_rate</th>\n",
        "      <th>dst_host_same_src_port_rate</th>\n",
        "      <th>dst_host_srv_diff_host_rate</th>\n",
        "      <th>dst_host_serror_rate</th>\n",
        "      <th>dst_host_srv_serror_rate</th>\n",
        "      <th>dst_host_rerror_rate</th>\n",
        "      <th>dst_host_srv_rerror_rate</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 4.940210e+05</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td>...</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "      <td> 494021.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td>      0.000823</td>\n",
        "      <td> 4.363595e-06</td>\n",
        "      <td>      0.000168</td>\n",
        "      <td>      0.000045</td>\n",
        "      <td>      0.002144</td>\n",
        "      <td>      0.000005</td>\n",
        "      <td>      0.001151</td>\n",
        "      <td>      0.000030</td>\n",
        "      <td>      0.148247</td>\n",
        "      <td>      0.000012</td>\n",
        "      <td>...</td>\n",
        "      <td>      0.911650</td>\n",
        "      <td>      0.739865</td>\n",
        "      <td>      0.753780</td>\n",
        "      <td>      0.030906</td>\n",
        "      <td>      0.601935</td>\n",
        "      <td>      0.006684</td>\n",
        "      <td>      0.176754</td>\n",
        "      <td>      0.176443</td>\n",
        "      <td>      0.058118</td>\n",
        "      <td>      0.057412</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>      0.012134</td>\n",
        "      <td> 1.425228e-03</td>\n",
        "      <td>      0.006409</td>\n",
        "      <td>      0.006673</td>\n",
        "      <td>      0.044935</td>\n",
        "      <td>      0.001837</td>\n",
        "      <td>      0.026070</td>\n",
        "      <td>      0.003104</td>\n",
        "      <td>      0.355345</td>\n",
        "      <td>      0.002034</td>\n",
        "      <td>...</td>\n",
        "      <td>      0.253903</td>\n",
        "      <td>      0.415845</td>\n",
        "      <td>      0.410781</td>\n",
        "      <td>      0.109259</td>\n",
        "      <td>      0.481309</td>\n",
        "      <td>      0.042133</td>\n",
        "      <td>      0.380593</td>\n",
        "      <td>      0.380919</td>\n",
        "      <td>      0.230590</td>\n",
        "      <td>      0.230140</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 0.000000e+00</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 6.489989e-08</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.180392</td>\n",
        "      <td>      0.410000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 7.499542e-07</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td>      0.000000</td>\n",
        "      <td> 1.488371e-06</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.040000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "      <td>      0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td>      1.000000</td>\n",
        "      <td> 1.000000e+00</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>...</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "      <td>      1.000000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>8 rows \u00d7 38 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "            duration     src_bytes      dst_bytes           land  \\\n",
        "count  494021.000000  4.940210e+05  494021.000000  494021.000000   \n",
        "mean        0.000823  4.363595e-06       0.000168       0.000045   \n",
        "std         0.012134  1.425228e-03       0.006409       0.006673   \n",
        "min         0.000000  0.000000e+00       0.000000       0.000000   \n",
        "25%         0.000000  6.489989e-08       0.000000       0.000000   \n",
        "50%         0.000000  7.499542e-07       0.000000       0.000000   \n",
        "75%         0.000000  1.488371e-06       0.000000       0.000000   \n",
        "max         1.000000  1.000000e+00       1.000000       1.000000   \n",
        "\n",
        "       wrong_fragment         urgent            hot  num_failed_logins  \\\n",
        "count   494021.000000  494021.000000  494021.000000      494021.000000   \n",
        "mean         0.002144       0.000005       0.001151           0.000030   \n",
        "std          0.044935       0.001837       0.026070           0.003104   \n",
        "min          0.000000       0.000000       0.000000           0.000000   \n",
        "25%          0.000000       0.000000       0.000000           0.000000   \n",
        "50%          0.000000       0.000000       0.000000           0.000000   \n",
        "75%          0.000000       0.000000       0.000000           0.000000   \n",
        "max          1.000000       1.000000       1.000000           1.000000   \n",
        "\n",
        "           logged_in  num_compromised       ...         dst_host_count  \\\n",
        "count  494021.000000    494021.000000       ...          494021.000000   \n",
        "mean        0.148247         0.000012       ...               0.911650   \n",
        "std         0.355345         0.002034       ...               0.253903   \n",
        "min         0.000000         0.000000       ...               0.000000   \n",
        "25%         0.000000         0.000000       ...               1.000000   \n",
        "50%         0.000000         0.000000       ...               1.000000   \n",
        "75%         0.000000         0.000000       ...               1.000000   \n",
        "max         1.000000         1.000000       ...               1.000000   \n",
        "\n",
        "       dst_host_srv_count  dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
        "count       494021.000000           494021.000000           494021.000000   \n",
        "mean             0.739865                0.753780                0.030906   \n",
        "std              0.415845                0.410781                0.109259   \n",
        "min              0.000000                0.000000                0.000000   \n",
        "25%              0.180392                0.410000                0.000000   \n",
        "50%              1.000000                1.000000                0.000000   \n",
        "75%              1.000000                1.000000                0.040000   \n",
        "max              1.000000                1.000000                1.000000   \n",
        "\n",
        "       dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
        "count                494021.000000                494021.000000   \n",
        "mean                      0.601935                     0.006684   \n",
        "std                       0.481309                     0.042133   \n",
        "min                       0.000000                     0.000000   \n",
        "25%                       0.000000                     0.000000   \n",
        "50%                       1.000000                     0.000000   \n",
        "75%                       1.000000                     0.000000   \n",
        "max                       1.000000                     1.000000   \n",
        "\n",
        "       dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
        "count         494021.000000             494021.000000         494021.000000   \n",
        "mean               0.176754                  0.176443              0.058118   \n",
        "std                0.380593                  0.380919              0.230590   \n",
        "min                0.000000                  0.000000              0.000000   \n",
        "25%                0.000000                  0.000000              0.000000   \n",
        "50%                0.000000                  0.000000              0.000000   \n",
        "75%                0.000000                  0.000000              0.000000   \n",
        "max                1.000000                  1.000000              1.000000   \n",
        "\n",
        "       dst_host_srv_rerror_rate  \n",
        "count             494021.000000  \n",
        "mean                   0.057412  \n",
        "std                    0.230140  \n",
        "min                    0.000000  \n",
        "25%                    0.000000  \n",
        "50%                    0.000000  \n",
        "75%                    0.000000  \n",
        "max                    1.000000  \n",
        "\n",
        "[8 rows x 38 columns]"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Visualising data using Principal Components"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "By using Principal Component Analysis, we can reduce the dimensionality of our data and plot it into a two-dimensional space. The PCA will capture those dimensions with the maximum variance, reducing the information loss.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Training a classifier"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Following the idea that new attack types will be similar to known types, let's start by trying a k-nearest neighbours classifier. We must to avoid brute force comparisons in the Nxd space at all costs. Being N the number of samples in our data more than 400K, and d the number of features 38, we will end up with an unfeasible modeling process. For this reason we pass `algorithm = 'ball_tree'`. For more on kNN performance, check [here](http://scikit-learn.org/stable/modules/neighbors.html#choice-of-nearest-neighbors-algorithm).      "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf = KNeighborsClassifier(n_neighbors = 5, algorithm = 'ball_tree', leaf_size=500)\n",
      "t0 = time()\n",
      "clf.fit(features,labels)\n",
      "tt = time()-t0\n",
      "print \"Classifier trained in {} seconds\".format(round(tt,3))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Classifier trained in 2405.17 seconds\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now let's try the classifier with the testing data. First we need to load the labeled test data. We wil also sample 10 percent of the entries. For that. we will take advantage of the `train_test_split` function in `sklearn`.       "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "kdd_data_corrected = pandas.read_csv(\"/nfs/data/KDD99/corrected\", header=None, names = col_names)\n",
      "kdd_data_corrected['label'].value_counts()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "smurf.              164091\n",
        "normal.              60593\n",
        "neptune.             58001\n",
        "snmpgetattack.        7741\n",
        "mailbomb.             5000\n",
        "guess_passwd.         4367\n",
        "snmpguess.            2406\n",
        "satan.                1633\n",
        "warezmaster.          1602\n",
        "back.                 1098\n",
        "mscan.                1053\n",
        "apache2.               794\n",
        "processtable.          759\n",
        "saint.                 736\n",
        "portsweep.             354\n",
        "ipsweep.               306\n",
        "httptunnel.            158\n",
        "pod.                    87\n",
        "nmap.                   84\n",
        "buffer_overflow.        22\n",
        "multihop.               18\n",
        "named.                  17\n",
        "sendmail.               17\n",
        "ps.                     16\n",
        "xterm.                  13\n",
        "rootkit.                13\n",
        "teardrop.               12\n",
        "xlock.                   9\n",
        "land.                    9\n",
        "xsnoop.                  4\n",
        "ftp_write.               3\n",
        "sqlattack.               2\n",
        "loadmodule.              2\n",
        "worm.                    2\n",
        "perl.                    2\n",
        "phf.                     2\n",
        "udpstorm.                2\n",
        "imap.                    1\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can see that we have new attack labels. In any case, we will convert all of the to the `attack.` label.    "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "kdd_data_corrected['label'][kdd_data_corrected['label']!='normal.'] = 'attack.'\n",
      "kdd_data_corrected['label'].value_counts()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "attack.    250436\n",
        "normal.     60593\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Again we select features and scale.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.cross_validation import train_test_split\n",
      "kdd_data_corrected[num_features] = kdd_data_corrected[num_features].astype(float)\n",
      "kdd_data_corrected[num_features].apply(lambda x: MinMaxScaler().fit_transform(x))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>duration</th>\n",
        "      <th>src_bytes</th>\n",
        "      <th>dst_bytes</th>\n",
        "      <th>land</th>\n",
        "      <th>wrong_fragment</th>\n",
        "      <th>urgent</th>\n",
        "      <th>hot</th>\n",
        "      <th>num_failed_logins</th>\n",
        "      <th>logged_in</th>\n",
        "      <th>num_compromised</th>\n",
        "      <th>...</th>\n",
        "      <th>dst_host_count</th>\n",
        "      <th>dst_host_srv_count</th>\n",
        "      <th>dst_host_same_srv_rate</th>\n",
        "      <th>dst_host_diff_srv_rate</th>\n",
        "      <th>dst_host_same_src_port_rate</th>\n",
        "      <th>dst_host_srv_diff_host_rate</th>\n",
        "      <th>dst_host_serror_rate</th>\n",
        "      <th>dst_host_srv_serror_rate</th>\n",
        "      <th>dst_host_rerror_rate</th>\n",
        "      <th>dst_host_srv_rerror_rate</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.039216</td>\n",
        "      <td> 0.011765</td>\n",
        "      <td> 0.30</td>\n",
        "      <td> 0.30</td>\n",
        "      <td> 0.30</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.992157</td>\n",
        "      <td> 0.99</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9     </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000004</td>\n",
        "      <td> 0.000036</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.278431</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000004</td>\n",
        "      <td> 0.000050</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.011765</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.33</td>\n",
        "      <td> 0.07</td>\n",
        "      <td> 0.33</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>12    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.988235</td>\n",
        "      <td> 0.99</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>14    </th>\n",
        "      <td> 0.000017</td>\n",
        "      <td> 0.000050</td>\n",
        "      <td> 0.000063</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.211765</td>\n",
        "      <td> 0.152941</td>\n",
        "      <td> 0.72</td>\n",
        "      <td> 0.11</td>\n",
        "      <td> 0.02</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.02</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.09</td>\n",
        "      <td> 0.13</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>15    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000005</td>\n",
        "      <td> 0.002650</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.694118</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>16    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000005</td>\n",
        "      <td> 0.000681</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.733333</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>17    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000005</td>\n",
        "      <td> 0.000145</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.768627</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>18    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>19    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>20    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000004</td>\n",
        "      <td> 0.001775</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.227451</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.02</td>\n",
        "      <td> 0.05</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>21    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.992157</td>\n",
        "      <td> 0.99</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000004</td>\n",
        "      <td> 0.000036</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>23    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000004</td>\n",
        "      <td> 0.001699</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>24    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000004</td>\n",
        "      <td> 0.003760</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000012</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.301961</td>\n",
        "      <td> 0.129412</td>\n",
        "      <td> 0.34</td>\n",
        "      <td> 0.08</td>\n",
        "      <td> 0.34</td>\n",
        "      <td> 0.06</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>26    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>27    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000560</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.360784</td>\n",
        "      <td> 0.172549</td>\n",
        "      <td> 0.43</td>\n",
        "      <td> 0.07</td>\n",
        "      <td> 0.43</td>\n",
        "      <td> 0.05</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>28    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000133</td>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 0.403922</td>\n",
        "      <td> 0.211765</td>\n",
        "      <td> 0.49</td>\n",
        "      <td> 0.06</td>\n",
        "      <td> 0.49</td>\n",
        "      <td> 0.04</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>29    </th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>...</th>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>310999</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311000</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 0.996078</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311001</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311002</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311003</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311004</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311005</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311006</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311007</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311008</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311009</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311010</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311011</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311012</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311013</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311014</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311015</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311016</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311017</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311018</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311019</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311020</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311021</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311022</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311023</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000020</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311024</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311025</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311026</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311027</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>311028</th>\n",
        "      <td> 0.000000</td>\n",
        "      <td> 0.000002</td>\n",
        "      <td> 0.000028</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0</td>\n",
        "      <td>...</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.000000</td>\n",
        "      <td> 1.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.01</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0</td>\n",
        "      <td> 0.00</td>\n",
        "      <td> 0.00</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>311029 rows \u00d7 38 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "        duration  src_bytes  dst_bytes  land  wrong_fragment  urgent  hot  \\\n",
        "0       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "1       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "2       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "3       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "4       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "5       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "6       0.000000   0.000000   0.000000     0               0       0    0   \n",
        "7       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "8       0.000000   0.000002   0.000028     0               0       0    0   \n",
        "9       0.000000   0.000004   0.000036     0               0       0    0   \n",
        "10      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "11      0.000000   0.000004   0.000050     0               0       0    0   \n",
        "12      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "13      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "14      0.000017   0.000050   0.000063     0               0       0    0   \n",
        "15      0.000000   0.000005   0.002650     0               0       0    0   \n",
        "16      0.000000   0.000005   0.000681     0               0       0    0   \n",
        "17      0.000000   0.000005   0.000145     0               0       0    0   \n",
        "18      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "19      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "20      0.000000   0.000004   0.001775     0               0       0    0   \n",
        "21      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "22      0.000000   0.000004   0.000036     0               0       0    0   \n",
        "23      0.000000   0.000004   0.001699     0               0       0    0   \n",
        "24      0.000000   0.000004   0.003760     0               0       0    0   \n",
        "25      0.000000   0.000012   0.000000     0               0       0    0   \n",
        "26      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "27      0.000000   0.000560   0.000000     0               0       0    0   \n",
        "28      0.000000   0.000133   0.000000     0               0       0    0   \n",
        "29      0.000000   0.000002   0.000028     0               0       0    0   \n",
        "...          ...        ...        ...   ...             ...     ...  ...   \n",
        "310999  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311000  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311001  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311002  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311003  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311004  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311005  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311006  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311007  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311008  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311009  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311010  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311011  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311012  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311013  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311014  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311015  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311016  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311017  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311018  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311019  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311020  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311021  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311022  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311023  0.000000   0.000002   0.000020     0               0       0    0   \n",
        "311024  0.000000   0.000002   0.000028     0               0       0    0   \n",
        "311025  0.000000   0.000002   0.000028     0               0       0    0   \n",
        "311026  0.000000   0.000002   0.000028     0               0       0    0   \n",
        "311027  0.000000   0.000002   0.000028     0               0       0    0   \n",
        "311028  0.000000   0.000002   0.000028     0               0       0    0   \n",
        "\n",
        "        num_failed_logins  logged_in  num_compromised       ...         \\\n",
        "0                       0          0                0       ...          \n",
        "1                       0          0                0       ...          \n",
        "2                       0          0                0       ...          \n",
        "3                       0          0                0       ...          \n",
        "4                       0          0                0       ...          \n",
        "5                       0          0                0       ...          \n",
        "6                       0          0                0       ...          \n",
        "7                       0          0                0       ...          \n",
        "8                       0          0                0       ...          \n",
        "9                       0          1                0       ...          \n",
        "10                      0          0                0       ...          \n",
        "11                      0          1                0       ...          \n",
        "12                      0          0                0       ...          \n",
        "13                      0          0                0       ...          \n",
        "14                      0          1                0       ...          \n",
        "15                      0          1                0       ...          \n",
        "16                      0          1                0       ...          \n",
        "17                      0          1                0       ...          \n",
        "18                      0          0                0       ...          \n",
        "19                      0          0                0       ...          \n",
        "20                      0          1                0       ...          \n",
        "21                      0          0                0       ...          \n",
        "22                      0          1                0       ...          \n",
        "23                      0          1                0       ...          \n",
        "24                      0          1                0       ...          \n",
        "25                      0          0                0       ...          \n",
        "26                      0          0                0       ...          \n",
        "27                      0          0                0       ...          \n",
        "28                      0          0                0       ...          \n",
        "29                      0          0                0       ...          \n",
        "...                   ...        ...              ...       ...          \n",
        "310999                  0          0                0       ...          \n",
        "311000                  0          0                0       ...          \n",
        "311001                  0          0                0       ...          \n",
        "311002                  0          0                0       ...          \n",
        "311003                  0          0                0       ...          \n",
        "311004                  0          0                0       ...          \n",
        "311005                  0          0                0       ...          \n",
        "311006                  0          0                0       ...          \n",
        "311007                  0          0                0       ...          \n",
        "311008                  0          0                0       ...          \n",
        "311009                  0          0                0       ...          \n",
        "311010                  0          0                0       ...          \n",
        "311011                  0          0                0       ...          \n",
        "311012                  0          0                0       ...          \n",
        "311013                  0          0                0       ...          \n",
        "311014                  0          0                0       ...          \n",
        "311015                  0          0                0       ...          \n",
        "311016                  0          0                0       ...          \n",
        "311017                  0          0                0       ...          \n",
        "311018                  0          0                0       ...          \n",
        "311019                  0          0                0       ...          \n",
        "311020                  0          0                0       ...          \n",
        "311021                  0          0                0       ...          \n",
        "311022                  0          0                0       ...          \n",
        "311023                  0          0                0       ...          \n",
        "311024                  0          0                0       ...          \n",
        "311025                  0          0                0       ...          \n",
        "311026                  0          0                0       ...          \n",
        "311027                  0          0                0       ...          \n",
        "311028                  0          0                0       ...          \n",
        "\n",
        "        dst_host_count  dst_host_srv_count  dst_host_same_srv_rate  \\\n",
        "0             1.000000            0.996078                    1.00   \n",
        "1             1.000000            0.996078                    1.00   \n",
        "2             1.000000            0.996078                    1.00   \n",
        "3             1.000000            0.996078                    1.00   \n",
        "4             1.000000            0.996078                    1.00   \n",
        "5             1.000000            1.000000                    1.00   \n",
        "6             0.039216            0.011765                    0.30   \n",
        "7             1.000000            0.992157                    0.99   \n",
        "8             1.000000            0.996078                    1.00   \n",
        "9             0.278431            1.000000                    1.00   \n",
        "10            1.000000            0.996078                    1.00   \n",
        "11            0.011765            1.000000                    1.00   \n",
        "12            1.000000            0.996078                    1.00   \n",
        "13            1.000000            0.988235                    0.99   \n",
        "14            0.211765            0.152941                    0.72   \n",
        "15            0.694118            1.000000                    1.00   \n",
        "16            0.733333            1.000000                    1.00   \n",
        "17            0.768627            1.000000                    1.00   \n",
        "18            1.000000            0.996078                    1.00   \n",
        "19            1.000000            0.996078                    1.00   \n",
        "20            0.227451            1.000000                    1.00   \n",
        "21            1.000000            0.992157                    0.99   \n",
        "22            1.000000            1.000000                    1.00   \n",
        "23            1.000000            1.000000                    1.00   \n",
        "24            1.000000            1.000000                    1.00   \n",
        "25            0.301961            0.129412                    0.34   \n",
        "26            1.000000            0.996078                    1.00   \n",
        "27            0.360784            0.172549                    0.43   \n",
        "28            0.403922            0.211765                    0.49   \n",
        "29            1.000000            0.996078                    1.00   \n",
        "...                ...                 ...                     ...   \n",
        "310999        1.000000            0.996078                    1.00   \n",
        "311000        1.000000            0.996078                    1.00   \n",
        "311001        1.000000            1.000000                    1.00   \n",
        "311002        1.000000            1.000000                    1.00   \n",
        "311003        1.000000            1.000000                    1.00   \n",
        "311004        1.000000            1.000000                    1.00   \n",
        "311005        1.000000            1.000000                    1.00   \n",
        "311006        1.000000            1.000000                    1.00   \n",
        "311007        1.000000            1.000000                    1.00   \n",
        "311008        1.000000            1.000000                    1.00   \n",
        "311009        1.000000            1.000000                    1.00   \n",
        "311010        1.000000            1.000000                    1.00   \n",
        "311011        1.000000            1.000000                    1.00   \n",
        "311012        1.000000            1.000000                    1.00   \n",
        "311013        1.000000            1.000000                    1.00   \n",
        "311014        1.000000            1.000000                    1.00   \n",
        "311015        1.000000            1.000000                    1.00   \n",
        "311016        1.000000            1.000000                    1.00   \n",
        "311017        1.000000            1.000000                    1.00   \n",
        "311018        1.000000            1.000000                    1.00   \n",
        "311019        1.000000            1.000000                    1.00   \n",
        "311020        1.000000            1.000000                    1.00   \n",
        "311021        1.000000            1.000000                    1.00   \n",
        "311022        1.000000            1.000000                    1.00   \n",
        "311023        1.000000            1.000000                    1.00   \n",
        "311024        1.000000            1.000000                    1.00   \n",
        "311025        1.000000            1.000000                    1.00   \n",
        "311026        1.000000            1.000000                    1.00   \n",
        "311027        1.000000            1.000000                    1.00   \n",
        "311028        1.000000            1.000000                    1.00   \n",
        "\n",
        "        dst_host_diff_srv_rate  dst_host_same_src_port_rate  \\\n",
        "0                         0.01                         0.00   \n",
        "1                         0.01                         0.00   \n",
        "2                         0.01                         0.00   \n",
        "3                         0.01                         0.00   \n",
        "4                         0.01                         0.01   \n",
        "5                         0.00                         0.01   \n",
        "6                         0.30                         0.30   \n",
        "7                         0.01                         0.00   \n",
        "8                         0.01                         0.00   \n",
        "9                         0.00                         0.01   \n",
        "10                        0.01                         0.00   \n",
        "11                        0.00                         0.33   \n",
        "12                        0.01                         0.01   \n",
        "13                        0.01                         0.00   \n",
        "14                        0.11                         0.02   \n",
        "15                        0.00                         0.01   \n",
        "16                        0.00                         0.01   \n",
        "17                        0.00                         0.01   \n",
        "18                        0.01                         0.01   \n",
        "19                        0.01                         0.00   \n",
        "20                        0.00                         0.02   \n",
        "21                        0.01                         0.00   \n",
        "22                        0.00                         0.00   \n",
        "23                        0.00                         0.00   \n",
        "24                        0.00                         0.00   \n",
        "25                        0.08                         0.34   \n",
        "26                        0.01                         0.00   \n",
        "27                        0.07                         0.43   \n",
        "28                        0.06                         0.49   \n",
        "29                        0.01                         0.00   \n",
        "...                        ...                          ...   \n",
        "310999                    0.01                         0.00   \n",
        "311000                    0.01                         0.00   \n",
        "311001                    0.00                         0.00   \n",
        "311002                    0.00                         0.00   \n",
        "311003                    0.00                         0.00   \n",
        "311004                    0.00                         0.00   \n",
        "311005                    0.00                         0.00   \n",
        "311006                    0.00                         0.00   \n",
        "311007                    0.00                         0.00   \n",
        "311008                    0.00                         0.00   \n",
        "311009                    0.00                         0.00   \n",
        "311010                    0.00                         0.00   \n",
        "311011                    0.00                         0.00   \n",
        "311012                    0.00                         0.00   \n",
        "311013                    0.00                         0.00   \n",
        "311014                    0.00                         0.00   \n",
        "311015                    0.00                         0.00   \n",
        "311016                    0.00                         0.00   \n",
        "311017                    0.00                         0.00   \n",
        "311018                    0.00                         0.00   \n",
        "311019                    0.00                         0.00   \n",
        "311020                    0.00                         0.00   \n",
        "311021                    0.00                         0.00   \n",
        "311022                    0.00                         0.00   \n",
        "311023                    0.00                         0.00   \n",
        "311024                    0.00                         0.01   \n",
        "311025                    0.00                         0.01   \n",
        "311026                    0.00                         0.01   \n",
        "311027                    0.00                         0.01   \n",
        "311028                    0.00                         0.01   \n",
        "\n",
        "        dst_host_srv_diff_host_rate  dst_host_serror_rate  \\\n",
        "0                              0.00                  0.00   \n",
        "1                              0.00                  0.00   \n",
        "2                              0.00                  0.00   \n",
        "3                              0.00                  0.00   \n",
        "4                              0.00                  0.00   \n",
        "5                              0.00                  0.00   \n",
        "6                              0.00                  0.00   \n",
        "7                              0.00                  0.00   \n",
        "8                              0.00                  0.00   \n",
        "9                              0.01                  0.00   \n",
        "10                             0.00                  0.00   \n",
        "11                             0.07                  0.33   \n",
        "12                             0.00                  0.00   \n",
        "13                             0.00                  0.00   \n",
        "14                             0.00                  0.02   \n",
        "15                             0.01                  0.00   \n",
        "16                             0.01                  0.00   \n",
        "17                             0.01                  0.00   \n",
        "18                             0.00                  0.00   \n",
        "19                             0.00                  0.00   \n",
        "20                             0.05                  0.00   \n",
        "21                             0.00                  0.00   \n",
        "22                             0.00                  0.00   \n",
        "23                             0.00                  0.00   \n",
        "24                             0.00                  0.00   \n",
        "25                             0.06                  0.00   \n",
        "26                             0.00                  0.00   \n",
        "27                             0.05                  0.00   \n",
        "28                             0.04                  0.00   \n",
        "29                             0.00                  0.00   \n",
        "...                             ...                   ...   \n",
        "310999                         0.00                  0.00   \n",
        "311000                         0.00                  0.00   \n",
        "311001                         0.00                  0.00   \n",
        "311002                         0.00                  0.00   \n",
        "311003                         0.00                  0.00   \n",
        "311004                         0.00                  0.00   \n",
        "311005                         0.00                  0.00   \n",
        "311006                         0.00                  0.00   \n",
        "311007                         0.00                  0.00   \n",
        "311008                         0.00                  0.00   \n",
        "311009                         0.00                  0.00   \n",
        "311010                         0.00                  0.00   \n",
        "311011                         0.00                  0.00   \n",
        "311012                         0.00                  0.00   \n",
        "311013                         0.00                  0.00   \n",
        "311014                         0.00                  0.00   \n",
        "311015                         0.00                  0.00   \n",
        "311016                         0.00                  0.00   \n",
        "311017                         0.00                  0.00   \n",
        "311018                         0.00                  0.00   \n",
        "311019                         0.00                  0.00   \n",
        "311020                         0.00                  0.00   \n",
        "311021                         0.00                  0.00   \n",
        "311022                         0.00                  0.00   \n",
        "311023                         0.00                  0.00   \n",
        "311024                         0.00                  0.00   \n",
        "311025                         0.00                  0.00   \n",
        "311026                         0.00                  0.00   \n",
        "311027                         0.00                  0.00   \n",
        "311028                         0.00                  0.00   \n",
        "\n",
        "        dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
        "0                              0                  0.00   \n",
        "1                              0                  0.00   \n",
        "2                              0                  0.00   \n",
        "3                              0                  0.00   \n",
        "4                              0                  0.00   \n",
        "5                              0                  0.00   \n",
        "6                              0                  0.00   \n",
        "7                              0                  0.00   \n",
        "8                              0                  0.00   \n",
        "9                              0                  0.00   \n",
        "10                             0                  0.00   \n",
        "11                             0                  0.00   \n",
        "12                             0                  0.00   \n",
        "13                             0                  0.00   \n",
        "14                             0                  0.09   \n",
        "15                             0                  0.00   \n",
        "16                             0                  0.00   \n",
        "17                             0                  0.00   \n",
        "18                             0                  0.00   \n",
        "19                             0                  0.00   \n",
        "20                             0                  0.00   \n",
        "21                             0                  0.00   \n",
        "22                             0                  0.00   \n",
        "23                             0                  0.00   \n",
        "24                             0                  0.00   \n",
        "25                             0                  0.00   \n",
        "26                             0                  0.00   \n",
        "27                             0                  0.00   \n",
        "28                             0                  0.00   \n",
        "29                             0                  0.00   \n",
        "...                          ...                   ...   \n",
        "310999                         0                  0.00   \n",
        "311000                         0                  0.00   \n",
        "311001                         0                  0.00   \n",
        "311002                         0                  0.00   \n",
        "311003                         0                  0.00   \n",
        "311004                         0                  0.00   \n",
        "311005                         0                  0.00   \n",
        "311006                         0                  0.00   \n",
        "311007                         0                  0.00   \n",
        "311008                         0                  0.00   \n",
        "311009                         0                  0.00   \n",
        "311010                         0                  0.00   \n",
        "311011                         0                  0.00   \n",
        "311012                         0                  0.00   \n",
        "311013                         0                  0.00   \n",
        "311014                         0                  0.00   \n",
        "311015                         0                  0.00   \n",
        "311016                         0                  0.00   \n",
        "311017                         0                  0.00   \n",
        "311018                         0                  0.00   \n",
        "311019                         0                  0.00   \n",
        "311020                         0                  0.00   \n",
        "311021                         0                  0.00   \n",
        "311022                         0                  0.00   \n",
        "311023                         0                  0.00   \n",
        "311024                         0                  0.00   \n",
        "311025                         0                  0.00   \n",
        "311026                         0                  0.00   \n",
        "311027                         0                  0.00   \n",
        "311028                         0                  0.00   \n",
        "\n",
        "        dst_host_srv_rerror_rate  \n",
        "0                           0.00  \n",
        "1                           0.00  \n",
        "2                           0.00  \n",
        "3                           0.00  \n",
        "4                           0.00  \n",
        "5                           0.00  \n",
        "6                           0.00  \n",
        "7                           0.00  \n",
        "8                           0.00  \n",
        "9                           0.00  \n",
        "10                          0.00  \n",
        "11                          0.00  \n",
        "12                          0.00  \n",
        "13                          0.00  \n",
        "14                          0.13  \n",
        "15                          0.00  \n",
        "16                          0.00  \n",
        "17                          0.00  \n",
        "18                          0.00  \n",
        "19                          0.00  \n",
        "20                          0.00  \n",
        "21                          0.00  \n",
        "22                          0.00  \n",
        "23                          0.00  \n",
        "24                          0.00  \n",
        "25                          0.00  \n",
        "26                          0.00  \n",
        "27                          0.00  \n",
        "28                          0.00  \n",
        "29                          0.00  \n",
        "...                          ...  \n",
        "310999                      0.00  \n",
        "311000                      0.00  \n",
        "311001                      0.00  \n",
        "311002                      0.00  \n",
        "311003                      0.00  \n",
        "311004                      0.00  \n",
        "311005                      0.00  \n",
        "311006                      0.00  \n",
        "311007                      0.00  \n",
        "311008                      0.00  \n",
        "311009                      0.00  \n",
        "311010                      0.00  \n",
        "311011                      0.00  \n",
        "311012                      0.00  \n",
        "311013                      0.00  \n",
        "311014                      0.00  \n",
        "311015                      0.00  \n",
        "311016                      0.00  \n",
        "311017                      0.00  \n",
        "311018                      0.00  \n",
        "311019                      0.00  \n",
        "311020                      0.00  \n",
        "311021                      0.00  \n",
        "311022                      0.00  \n",
        "311023                      0.00  \n",
        "311024                      0.00  \n",
        "311025                      0.00  \n",
        "311026                      0.00  \n",
        "311027                      0.00  \n",
        "311028                      0.00  \n",
        "\n",
        "[311029 rows x 38 columns]"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can sample the 10 percent of the test data (after we scale it). Although we also get training data, we don't need it in this case.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "features_train, features_test, labels_train, labels_test = train_test_split(\n",
      "    kdd_data_corrected[num_features], \n",
      "    kdd_data_corrected['label'], \n",
      "    test_size=0.1, \n",
      "    random_state=42)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, do predictions using our classifier and the test data. kNN classifiers are slow compared to other methods due to all the comparisons required in order to make predictions.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "t0 = time()\n",
      "pred = clf.predict(features_test)\n",
      "tt = time() - t0\n",
      "print \"Predicted in {} seconds\".format(round(tt,3))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Predicted in 902.673 seconds\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "That took a lot of time. Actually, the more training data we use with a k-means classifier, the slower it gets to predict. It needs to compare the new data with all the points. Definitively we want some centroid-based classifier if we plan to use it in real-time detection.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And finally, calculate the R squared value using the test labels.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.metrics import accuracy_score\n",
      "acc = accuracy_score(pred, labels_test)\n",
      "print \"R squared is {}.\".format(round(acc,4))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "R squared is 0.8202.\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Clustering"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "So finally, let's try our anomaly detection approach in the reduced dataset. We will start by doing **k-means clustering**. Once we have the cluster centers, we will use them to determine the labels of the test data (unlabeled).  \n",
      "\n",
      "Based on the assumption that new attack types will resemble old type, we will be able to detect those. Moreover, anything that falls too far from any cluster, will be considered anomalous and therefore a possible attack.  "
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "KMeans clustering"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.cluster import KMeans\n",
      "k = 30\n",
      "km = KMeans(n_clusters = k)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "t0 = time()\n",
      "km.fit(features)\n",
      "tt = time()-t0\n",
      "print \"Clustered in {} seconds\".format(round(tt,3))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Clustered in 331.394 seconds\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can check cluster sizes.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pandas.Series(km.labels_).value_counts()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 18,
       "text": [
        "0     262807\n",
        "1      48555\n",
        "15     38319\n",
        "5      24427\n",
        "3      20528\n",
        "12     19508\n",
        "28     17879\n",
        "8      11524\n",
        "26      9162\n",
        "7       4941\n",
        "17      4272\n",
        "10      4215\n",
        "25      3996\n",
        "13      3513\n",
        "2       2845\n",
        "29      1951\n",
        "11      1686\n",
        "4       1640\n",
        "24      1557\n",
        "9       1341\n",
        "14      1239\n",
        "16      1230\n",
        "6       1201\n",
        "21      1116\n",
        "18      1020\n",
        "19       970\n",
        "23       949\n",
        "27       775\n",
        "20       680\n",
        "22       175\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Get labels for each cluster. Here, we go back to use the complete set of labels.    "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "labels = kdd_data_10percent['label']\n",
      "label_names = map(\n",
      "    lambda x: pandas.Series([labels[i] for i in range(len(km.labels_)) if km.labels_[i]==x]), \n",
      "    range(k))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Print labels for each cluster.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for i in range(k):\n",
      "    print \"Cluster {} labels:\".format(i)\n",
      "    print label_names[i].value_counts()\n",
      "    print"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Cluster 0 labels:\n",
        "smurf.     262805\n",
        "normal.         2\n",
        "dtype: int64\n",
        "\n",
        "Cluster 1 labels:\n",
        "neptune.      48551\n",
        "portsweep.        4\n",
        "dtype: int64\n",
        "\n",
        "Cluster 2 labels:\n",
        "normal.    2845\n",
        "dtype: int64\n",
        "\n",
        "Cluster 3 labels:\n",
        "neptune.      20456\n",
        "portsweep.       58\n",
        "satan.           10\n",
        "normal.           4\n",
        "dtype: int64\n",
        "\n",
        "Cluster 4 labels:\n",
        "normal.      1510\n",
        "pod.           60\n",
        "smurf.         36\n",
        "satan.         22\n",
        "teardrop.       7\n",
        "rootkit.        3\n",
        "spy.            1\n",
        "nmap.           1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 5 labels:\n",
        "normal.    23165\n",
        "back.       1258\n",
        "phf.           3\n",
        "satan.         1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 6 labels:\n",
        "normal.    1201\n",
        "dtype: int64\n",
        "\n",
        "Cluster 7 labels:\n",
        "normal.         4873\n",
        "warezclient.      52\n",
        "rootkit.           4\n",
        "satan.             4\n",
        "perl.              3\n",
        "ipsweep.           2\n",
        "loadmodule.        1\n",
        "spy.               1\n",
        "imap.              1\n",
        "dtype: int64"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Cluster 8 labels:\n",
        "normal.    11424\n",
        "smurf.        98\n",
        "nmap.          2\n",
        "dtype: int64\n",
        "\n",
        "Cluster 9 labels:\n",
        "normal.       1230\n",
        "satan.         109\n",
        "teardrop.        1\n",
        "portsweep.       1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 10 labels:\n",
        "normal.          4153\n",
        "guess_passwd.      49\n",
        "back.               7\n",
        "ipsweep.            3\n",
        "portsweep.          2\n",
        "neptune.            1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 11 labels:\n",
        "normal.          1177\n",
        "ipsweep.          339\n",
        "pod.               94\n",
        "nmap.              22\n",
        "warezmaster.       18\n",
        "satan.             12\n",
        "imap.               8\n",
        "smurf.              3\n",
        "land.               3\n",
        "ftp_write.          2\n",
        "rootkit.            2\n",
        "guess_passwd.       2\n",
        "multihop.           2\n",
        "loadmodule.         1\n",
        "portsweep.          1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 12 labels:\n",
        "normal.    19508\n",
        "dtype: int64\n",
        "\n",
        "Cluster 13 labels:\n",
        "normal.         3500\n",
        "back.             12\n",
        "warezclient.       1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 14 labels:\n",
        "warezclient.        661\n",
        "normal.             537\n",
        "buffer_overflow.     22\n",
        "ftp_write.            6\n",
        "back.                 5\n",
        "loadmodule.           5\n",
        "multihop.             2\n",
        "rootkit.              1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 15 labels:\n",
        "neptune.         38189\n",
        "nmap.              103\n",
        "portsweep.          19\n",
        "normal.              6\n",
        "land.                1\n",
        "guess_passwd.        1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 16 labels:\n",
        "satan.        1222\n",
        "portsweep.       8\n",
        "dtype: int64\n",
        "\n",
        "Cluster 17 labels:\n",
        "normal.       4237\n",
        "satan.          26\n",
        "portsweep.       9\n",
        "dtype: int64\n",
        "\n",
        "Cluster 18 labels:\n",
        "portsweep.    934\n",
        "ipsweep.       83\n",
        "normal.         3\n",
        "dtype: int64\n",
        "\n",
        "Cluster 19 labels:\n",
        "teardrop.    970\n",
        "dtype: int64\n",
        "\n",
        "Cluster 20 labels:\n",
        "normal.         370\n",
        "warezclient.    306\n",
        "multihop.         2\n",
        "warezmaster.      2\n",
        "dtype: int64\n",
        "\n",
        "Cluster 21 labels:\n",
        "ipsweep.     813\n",
        "normal.      118\n",
        "nmap.         99\n",
        "pod.          71\n",
        "land.         14\n",
        "multihop.      1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 22 labels:\n",
        "satan.        172\n",
        "portsweep.      3\n",
        "dtype: int64\n",
        "\n",
        "Cluster 23 labels:\n",
        "normal.    949\n",
        "dtype: int64\n",
        "\n",
        "Cluster 24 labels:\n",
        "normal.       1497\n",
        "pod.            29\n",
        "smurf.           7\n",
        "ipsweep.         7\n",
        "satan.           6\n",
        "nmap.            4\n",
        "imap.            3\n",
        "neptune.         2\n",
        "teardrop.        1\n",
        "portsweep.       1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 25 labels:\n",
        "normal.             3521\n",
        "back.                464\n",
        "buffer_overflow.       8\n",
        "loadmodule.            2\n",
        "guess_passwd.          1\n",
        "dtype: int64\n",
        "\n",
        "Cluster 26 labels:\n",
        "normal.    9091\n",
        "back.        71\n",
        "dtype: int64\n",
        "\n",
        "Cluster 27 labels:\n",
        "normal.     760\n",
        "pod.         10\n",
        "land.         3\n",
        "neptune.      2\n",
        "dtype: int64\n",
        "\n",
        "Cluster 28 labels:\n",
        "smurf.     17841\n",
        "normal.       33\n",
        "satan.         5\n",
        "dtype: int64\n",
        "\n",
        "Cluster 29 labels:\n",
        "normal.    1564\n",
        "back.       386\n",
        "phf.          1\n",
        "dtype: int64\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Cluster description  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can see that, in most clusters, there is a dominant label. It would be interesting to go cluster by cluster and analyise mayority labels, or how labels are split between different clusters (some with more dominance than others). All that would help us understand each type of attack! This is also a benefit of using a clustering-based approach.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### TODO:\n",
      "- Get dominant labels\n",
      "- Analyse cluster centers, specially for heterogeneous clusters containing `normal`. This will discover conflictive interactions.  "
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Predictions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can now predict using our test data.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "t0 = time()\n",
      "pred = km.predict(kdd_data_corrected[num_features])\n",
      "tt = time() - t0\n",
      "print \"Assigned clusters in {} seconds\".format(round(tt,3))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Assigned clusters in 0.693 seconds\n"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can see that the assignment process is much faster than the prediction process with our kNN. But we still need to assign labels.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO: get mayority label for each cluster assignment (we have labels from the previous step)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO: check these labels with those in the corrected test data in order to calculate accuracy"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 23
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Using the complete dataset with Spark"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The script [KDDCup99.py](KDDCup99.py) runds through a series of steps to perform k-means clustering over the complete dataset using `PySpark` with different K values in order to find the best one.  \n",
      "\n",
      "The clustering results are stored in a `CSV` file. This file is very convenient for visualisation purposes. It would be very hard to cluster and visualise results of the complete dataset using `Scikit-learn`.  \n",
      "\n",
      "The following chart depicts the **first two pincipal components** for the clustering results.  \n",
      "\n",
      "![](clusters.png)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Remember that we have up to 24 different labels in our complete dataset. However we have generated up to 80 different clusters. As a result of this, some of the clusters appear very close in the first principal component. This is due to the variability of interactions for a given type of attack (or label).   "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO: follow the same approach for label assignment in the test data as before"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Clustering using Spark"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In order to show how we use `Spark` to do k-means clustering in our dataset, let's perform here a single clustering run with the complete dataset, for a K value of 80 that showed to be particulary good."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Some imports we will use\n",
      "from collections import OrderedDict\n",
      "from time import time"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First we need to load the data, using the complete dataset file stored in NFS."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data_file = \"/nfs/data/KDD99/kddcup.data\"\n",
      "raw_data = sc.textFile(data_file)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As a warm up, let's count the number of interactions by label."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# count by all different labels and print them decreasingly\n",
      "print \"Counting all different labels\"\n",
      "labels = raw_data.map(lambda line: line.strip().split(\",\")[-1])\n",
      "\n",
      "t0 = time()\n",
      "label_counts = labels.countByValue()\n",
      "tt = time()-t0\n",
      "\n",
      "sorted_labels = OrderedDict(sorted(label_counts.items(), key=lambda t: t[1], reverse=True))\n",
      "for label, count in sorted_labels.items():\n",
      "    print label, count\n",
      "    \n",
      "print \"Counted in {} seconds\".format(round(tt,3))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Counting all different labels\n",
        "smurf."
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 2807886\n",
        "neptune. 1072017\n",
        "normal. 972781\n",
        "satan. 15892\n",
        "ipsweep. 12481\n",
        "portsweep. 10413\n",
        "nmap. 2316\n",
        "back. 2203\n",
        "warezclient. 1020\n",
        "teardrop. 979\n",
        "pod. 264\n",
        "guess_passwd. 53\n",
        "buffer_overflow. 30\n",
        "land. 21\n",
        "warezmaster. 20\n",
        "imap. 12\n",
        "rootkit. 10\n",
        "loadmodule. 9\n",
        "ftp_write. 8\n",
        "multihop. 7\n",
        "phf. 4\n",
        "perl. 3\n",
        "spy. 2\n",
        "Counted in 9.12 seconds\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we prepare the data for clustering input. The data contains non-numeric features, and we want to exclude them since k-means works just with numeric features. These are the first three and the last column in each data row that is the label.  \n",
      "In order to do that, we define a function that we apply to the *RDD* as a `Spark` **transformation** by using `map`. Remember that we can apply as many transofmrations as we want without making `Spark` start any processing. Is is when we trigger an action when all the transformations are applied.    "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def parse_interaction(line):\n",
      "    \"\"\"\n",
      "    Parses a network data interaction.\n",
      "    \"\"\"\n",
      "    line_split = line.split(\",\")\n",
      "    clean_line_split = [line_split[0]]+line_split[4:-1]\n",
      "    return (line_split[-1], array([float(x) for x in clean_line_split]))\n",
      "\n",
      "parsed_data = raw_data.map(parse_interaction)\n",
      "parsed_data_values = parsed_data.values().cache()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Additionally, we have used `cache` in order to keep the results at hand once they are calculated. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We will also standardise our data as we have done so far when performing distance-based clustering."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pyspark.mllib.feature import StandardScaler\n",
      "standardizer = StandardScaler(True, True)\n",
      "t0 = time()\n",
      "standardizer_model = standardizer.fit(parsed_data_values)\n",
      "tt = time() - t0\n",
      "standardized_data_values = standardizer_model.transform(parsed_data_values)\n",
      "print \"Data standardized in {} seconds\".format(round(tt,3))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Data standardized in 9.54 seconds\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can now perform k-means clustering.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pyspark.mllib.clustering import KMeans\n",
      "t0 = time()\n",
      "clusters = KMeans.train(standardized_data_values, 80, \n",
      "                        maxIterations=10, runs=5, \n",
      "                        initializationMode=\"random\")\n",
      "tt = time() - t0\n",
      "print \"Data clustered in {} seconds\".format(round(tt,3))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Data clustered in 137.496 seconds\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Once we have our clusters, we can use them to label test data and test accuracy."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}