AFML 4.3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Weights\n",
    "\n",
    "This notebook will cover exercise answer.\n",
    "\n",
    "* Exercise 4.6\n",
    "* Exercise 4.7\n",
    "\n",
    "As we go along, there will be some explanations.\n",
    "\n",
    "More importantly, this method can be applied not just within mean-reversion strategy but also other strategies as well.\n",
    "\n",
    "Most of the functions below can be found under research/Sampling.\n",
    "\n",
    "Contact: boyboi86@gmail.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num of CPU core:  4\n",
      "Machine info:  Windows-10-10.0.18362-SP0\n",
      "Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]\n",
      "Numpy 1.17.3\n",
      "Pandas 1.0.3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Anaconda3\\lib\\site-packages\\statsmodels\\tools\\_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
      "  import pandas.util.testing as tm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1500x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import research as rs\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "dollar = pd.read_csv('./research/Sample_data/dollar_bars.txt', \n",
    "                 sep=',', \n",
    "                 header=0, \n",
    "                 parse_dates = True, \n",
    "                 index_col=['date_time'])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bollinger Band results:\n",
      "\n",
      "Num of times upper limit touched: 548\n",
      "Num of times lower limit touched: 674\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Desktop\\Python\\research\\Labels\\triple_barrier_method.py:75: UserWarning: Data and events index shape must be same, reindex data to fit events\n",
      "  warnings.warn('Data and events index shape must be same, reindex data to fit events')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[                                             t1                      sl  pt\n",
      "2015-01-08 12:23:50.674 2015-01-12 14:56:49.380 2015-01-08 15:59:00.235 NaT\n",
      "2015-01-08 14:38:13.115 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:42:32.124 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:47:20.893 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:54:32.974 2015-01-12 14:56:49.380                     NaT NaT\n",
      "...                                         ...                     ...  ..\n",
      "2015-09-09 19:48:41.756 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 19:53:21.700 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 19:57:58.827 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 22:44:56.773 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-10 12:36:39.559 2015-09-17 18:46:53.905                     NaT NaT\n",
      "\n",
      "[393 rows x 3 columns]] this out\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-05-26 21:15:32.942036 33.33% _pt_sl_t1 done after 0.11 minutes. Remaining 0.23 minutes.\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[                                             t1                      sl  pt\n",
      "2015-01-08 12:23:50.674 2015-01-12 14:56:49.380 2015-01-08 15:59:00.235 NaT\n",
      "2015-01-08 14:38:13.115 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:42:32.124 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:47:20.893 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:54:32.974 2015-01-12 14:56:49.380                     NaT NaT\n",
      "...                                         ...                     ...  ..\n",
      "2015-09-09 19:48:41.756 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 19:53:21.700 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 19:57:58.827 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 22:44:56.773 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-10 12:36:39.559 2015-09-17 18:46:53.905                     NaT NaT\n",
      "\n",
      "[393 rows x 3 columns],                                              t1  sl  pt\n",
      "2015-09-17 18:46:53.905 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-17 18:48:33.362 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-17 18:51:12.314 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-18 10:42:15.964 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-18 11:15:21.515 2015-09-22 08:32:53.406 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-01-20 17:05:12.492 2016-01-21 17:10:35.564 NaT NaT\n",
      "2016-01-20 17:09:40.041 2016-01-21 17:10:35.564 NaT NaT\n",
      "2016-01-20 17:16:12.118 2016-01-21 17:30:08.064 NaT NaT\n",
      "2016-01-20 17:22:26.639 2016-01-21 17:30:08.064 NaT NaT\n",
      "2016-01-20 17:25:16.158 2016-01-21 17:30:08.064 NaT NaT\n",
      "\n",
      "[393 rows x 3 columns]] this out\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-05-26 21:15:33.375586 66.67% _pt_sl_t1 done after 0.12 minutes. Remaining 0.06 minutes.\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[                                             t1                      sl  pt\n",
      "2015-01-08 12:23:50.674 2015-01-12 14:56:49.380 2015-01-08 15:59:00.235 NaT\n",
      "2015-01-08 14:38:13.115 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:42:32.124 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:47:20.893 2015-01-12 14:56:49.380                     NaT NaT\n",
      "2015-01-08 14:54:32.974 2015-01-12 14:56:49.380                     NaT NaT\n",
      "...                                         ...                     ...  ..\n",
      "2015-09-09 19:48:41.756 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 19:53:21.700 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 19:57:58.827 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-09 22:44:56.773 2015-09-17 18:46:53.905                     NaT NaT\n",
      "2015-09-10 12:36:39.559 2015-09-17 18:46:53.905                     NaT NaT\n",
      "\n",
      "[393 rows x 3 columns],                                              t1  sl  pt\n",
      "2015-09-17 18:46:53.905 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-17 18:48:33.362 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-17 18:51:12.314 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-18 10:42:15.964 2015-09-22 08:32:53.406 NaT NaT\n",
      "2015-09-18 11:15:21.515 2015-09-22 08:32:53.406 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-01-20 17:05:12.492 2016-01-21 17:10:35.564 NaT NaT\n",
      "2016-01-20 17:09:40.041 2016-01-21 17:10:35.564 NaT NaT\n",
      "2016-01-20 17:16:12.118 2016-01-21 17:30:08.064 NaT NaT\n",
      "2016-01-20 17:22:26.639 2016-01-21 17:30:08.064 NaT NaT\n",
      "2016-01-20 17:25:16.158 2016-01-21 17:30:08.064 NaT NaT\n",
      "\n",
      "[393 rows x 3 columns],                                              t1  sl  pt\n",
      "2016-01-20 17:29:50.227 2016-01-21 17:30:08.064 NaT NaT\n",
      "2016-01-20 17:34:47.179 2016-01-22 07:55:50.836 NaT NaT\n",
      "2016-01-20 17:43:20.462 2016-01-22 07:55:50.836 NaT NaT\n",
      "2016-01-20 17:50:05.543 2016-01-22 07:55:50.836 NaT NaT\n",
      "2016-01-20 20:18:15.345 2016-01-22 07:55:50.836 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-11-09 16:58:13.840                     NaT NaT NaT\n",
      "2016-11-09 17:03:36.650                     NaT NaT NaT\n",
      "2016-11-09 19:41:40.425                     NaT NaT NaT\n",
      "2016-11-10 08:20:36.695                     NaT NaT NaT\n",
      "2016-11-10 08:41:06.102                     NaT NaT NaT\n",
      "\n",
      "[393 rows x 3 columns]] this out\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-05-26 21:15:33.650509 100.0% _pt_sl_t1 done after 0.12 minutes. Remaining 0.0 minutes.\n"
     ]
    }
   ],
   "source": [
    "dollar = rs.bband_as_side(data = dollar, \n",
    "                          window = 50, \n",
    "                          width = 0.009)\n",
    "\n",
    "dollar['volatility'] = rs.vol(dollar['close'], span0 = 50)\n",
    "\n",
    "events = rs.cs_filter(dollar['close'], \n",
    "                    limit = dollar['volatility'].mean())\n",
    "\n",
    "vb = rs.vert_barrier(data = dollar['close'], \n",
    "                 events = events, \n",
    "                 period = 'days', \n",
    "                 freq = 1)\n",
    "\n",
    "tb = rs.tri_barrier(data = dollar['close'], \n",
    "                    events = events, \n",
    "                    trgt = dollar['volatility'] * 5, \n",
    "                    min_req = 0.002, \n",
    "                    num_threads = 3, \n",
    "                    ptSl = [0,2],\n",
    "                    t1 = vb, \n",
    "                    side = dollar['side'])\n",
    "\n",
    "mlabel = rs.meta_label(data = dollar['close'], \n",
    "                       events = tb, \n",
    "                       drop = False) # when you have a side binary, you won't have rare labels usually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t1</th>\n",
       "      <th>trgt</th>\n",
       "      <th>side</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-01-08 12:23:50.674</th>\n",
       "      <td>2015-01-08 15:59:00.235</td>\n",
       "      <td>0.002936</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:38:13.115</th>\n",
       "      <td>2015-01-12 14:56:49.380</td>\n",
       "      <td>0.003550</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:42:32.124</th>\n",
       "      <td>2015-01-12 14:56:49.380</td>\n",
       "      <td>0.004632</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:47:20.893</th>\n",
       "      <td>2015-01-12 14:56:49.380</td>\n",
       "      <td>0.005816</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:54:32.974</th>\n",
       "      <td>2015-01-12 14:56:49.380</td>\n",
       "      <td>0.006257</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 06:25:45.452</th>\n",
       "      <td>2016-11-10 08:20:36.695</td>\n",
       "      <td>0.084342</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 06:47:55.499</th>\n",
       "      <td>2016-11-10 08:20:36.695</td>\n",
       "      <td>0.083149</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 07:05:15.916</th>\n",
       "      <td>2016-11-10 08:20:36.695</td>\n",
       "      <td>0.081748</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 08:13:18.700</th>\n",
       "      <td>2016-11-10 08:20:36.695</td>\n",
       "      <td>0.081405</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 08:21:05.644</th>\n",
       "      <td>2016-11-10 08:41:06.102</td>\n",
       "      <td>0.080937</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1138 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             t1      trgt  side\n",
       "2015-01-08 12:23:50.674 2015-01-08 15:59:00.235  0.002936  -1.0\n",
       "2015-01-08 14:38:13.115 2015-01-12 14:56:49.380  0.003550  -1.0\n",
       "2015-01-08 14:42:32.124 2015-01-12 14:56:49.380  0.004632  -1.0\n",
       "2015-01-08 14:47:20.893 2015-01-12 14:56:49.380  0.005816  -1.0\n",
       "2015-01-08 14:54:32.974 2015-01-12 14:56:49.380  0.006257  -1.0\n",
       "...                                         ...       ...   ...\n",
       "2016-11-09 06:25:45.452 2016-11-10 08:20:36.695  0.084342   1.0\n",
       "2016-11-09 06:47:55.499 2016-11-10 08:20:36.695  0.083149   1.0\n",
       "2016-11-09 07:05:15.916 2016-11-10 08:20:36.695  0.081748   1.0\n",
       "2016-11-09 08:13:18.700 2016-11-10 08:20:36.695  0.081405  -1.0\n",
       "2016-11-09 08:21:05.644 2016-11-10 08:41:06.102  0.080937  -1.0\n",
       "\n",
       "[1138 rows x 3 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1128</th>\n",
       "      <th>1129</th>\n",
       "      <th>1130</th>\n",
       "      <th>1131</th>\n",
       "      <th>1132</th>\n",
       "      <th>1133</th>\n",
       "      <th>1134</th>\n",
       "      <th>1135</th>\n",
       "      <th>1136</th>\n",
       "      <th>1137</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date_time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-01-08 12:23:50.674</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:38:13.115</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:42:32.124</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:47:20.893</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 14:54:32.974</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 16:58:13.840</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 17:03:36.650</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-09 19:41:40.425</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-10 08:20:36.695</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-10 08:41:06.102</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1215 rows × 1138 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         0     1     2     3     4     5     6     7     8     \\\n",
       "date_time                                                                       \n",
       "2015-01-08 12:23:50.674   1.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:38:13.115   1.0   1.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:42:32.124   1.0   1.0   1.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:47:20.893   1.0   1.0   1.0   1.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:54:32.974   1.0   1.0   1.0   1.0   1.0   0.0   0.0   0.0   0.0   \n",
       "...                       ...   ...   ...   ...   ...   ...   ...   ...   ...   \n",
       "2016-11-09 16:58:13.840   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2016-11-09 17:03:36.650   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2016-11-09 19:41:40.425   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2016-11-10 08:20:36.695   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2016-11-10 08:41:06.102   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "\n",
       "                         9     ...  1128  1129  1130  1131  1132  1133  1134  \\\n",
       "date_time                      ...                                             \n",
       "2015-01-08 12:23:50.674   0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:38:13.115   0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:42:32.124   0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:47:20.893   0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "2015-01-08 14:54:32.974   0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "...                       ...  ...   ...   ...   ...   ...   ...   ...   ...   \n",
       "2016-11-09 16:58:13.840   0.0  ...   1.0   1.0   1.0   1.0   1.0   1.0   1.0   \n",
       "2016-11-09 17:03:36.650   0.0  ...   1.0   1.0   1.0   1.0   1.0   1.0   1.0   \n",
       "2016-11-09 19:41:40.425   0.0  ...   1.0   1.0   1.0   1.0   1.0   1.0   1.0   \n",
       "2016-11-10 08:20:36.695   0.0  ...   1.0   1.0   1.0   1.0   1.0   1.0   1.0   \n",
       "2016-11-10 08:41:06.102   0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   \n",
       "\n",
       "                         1135  1136  1137  \n",
       "date_time                                  \n",
       "2015-01-08 12:23:50.674   0.0   0.0   0.0  \n",
       "2015-01-08 14:38:13.115   0.0   0.0   0.0  \n",
       "2015-01-08 14:42:32.124   0.0   0.0   0.0  \n",
       "2015-01-08 14:47:20.893   0.0   0.0   0.0  \n",
       "2015-01-08 14:54:32.974   0.0   0.0   0.0  \n",
       "...                       ...   ...   ...  \n",
       "2016-11-09 16:58:13.840   1.0   1.0   1.0  \n",
       "2016-11-09 17:03:36.650   1.0   1.0   1.0  \n",
       "2016-11-09 19:41:40.425   1.0   1.0   1.0  \n",
       "2016-11-10 08:20:36.695   1.0   1.0   1.0  \n",
       "2016-11-10 08:41:06.102   0.0   0.0   1.0  \n",
       "\n",
       "[1215 rows x 1138 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#this is a modified func using mp_pandas_obj, run up to 700x faster depending on data size.\n",
    "\n",
    "idxM0 = rs.mp_idx_matrix(data = dollar['close'], events = tb)\n",
    "\n",
    "idxM0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequential bootstrap vs Standard Bootstrap\n",
    "\n",
    "**A short introduction to the methods**\n",
    "\n",
    "For standard bootstrap, we will randomly choose columns given their size. As the law of large number goes, as number of sample increase beyond 30.\n",
    "\n",
    "It will start to display normal-like distribution. So more loops equals to higher reliability.\n",
    "\n",
    "For sequential bootstrap, we choose based on known uniquness, therefore we will likely experience higher overall uniqueness.\n",
    "\n",
    "However when samples get large enough, there will be a gradual drop in uniqueness. \n",
    "\n",
    "Given updated probability, we are less likely to choose the same column twice.\n",
    "\n",
    "As sequential bootstrap develop into normal-like distribution, it will start to \"converge\" to mean with samples that are more unique compared to standard bootstrap method.\n",
    "\n",
    "Hence for sequential bootstrap to be reliable, another method call monte-carlos was introduced.\n",
    "\n",
    "**Note**\n",
    "\n",
    "Due to the lack of resource, the Monte-Carlos Sequential Bootstrap (MC_seq_bts) was implemented but was not used. It will take high computational power to run this algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unique_check(idxM0, iter_num):\n",
    "    i = 0\n",
    "    total = 0\n",
    "    while i < (iter_num + 1):\n",
    "        phi_ = np.random.choice(idxM0.columns, size = idxM0.shape[1])\n",
    "        stdU = rs.av_unique(idxM0[phi_]).mean()\n",
    "        if stdU > total:\n",
    "            total = stdU\n",
    "        i += 1\n",
    "    print (\"Highest Standard Uniqueness after {0} loops: {1}\".format(iter_num, total))\n",
    "    phi = rs.mp_seq_bts(idxM = idxM0, sample_len = iter_num)\n",
    "    seqU = rs.av_unique(idxM0[phi]).mean()\n",
    "    print (\"Ave Sequential Uniqueness after {0} loops: {1}\".format(iter_num, seqU))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Highest Standard Uniqueness after 20 loops: 0.09038045162842588\n",
      "Ave Sequential Uniqueness after 20 loops: 0.9488218390804597\n"
     ]
    }
   ],
   "source": [
    "# As n < 30\n",
    "\n",
    "unique_check(idxM0, 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Highest Standard Uniqueness after 50 loops: 0.08840470735702916\n",
      "Ave Sequential Uniqueness after 50 loops: 0.8097207229881863\n"
     ]
    }
   ],
   "source": [
    "# As n > 30\n",
    "\n",
    "unique_check(idxM0, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Due to resource constraint, we used func unique_check.\n",
    "\n",
    "However, this is not a fair comparison since phi_ randomly choose all the columns with replacement.\n",
    "\n",
    "As such, there won't be much difference in their uniqueness for standard method, hence we will use the highest unique sample based on standard bootstrap to compare against sequential bootstrap sample's mean uniqueness.\n",
    "\n",
    "The idea however is to let you know the difference when we use bootstrap in sklearn vs if we were to implement a sequential bootstrap for sklearn classifier to perform sampling.\n",
    "\n",
    "For sequential bootstrap, we will randomly choose columns but we use the previous columns choosen to compare (in-the-bag). In that way, only samples that are unique will be used to train ML models.\n",
    "\n",
    "As a result, it will not report an inflated OOB score.\n",
    "\n",
    "**Interesting facts**\n",
    "\n",
    "In most cases given a large data sample, choosen sample's mean uniqueness can remain 1.0 (highest uniqueness) for 3 - 10 loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1    2\n",
       "0  1.0  0.0  0.0\n",
       "1  1.0  0.0  0.0\n",
       "2  1.0  1.0  0.0\n",
       "3  0.0  1.0  0.0\n",
       "4  0.0  0.0  1.0\n",
       "5  0.0  0.0  1.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Exercise 4.6\n",
    "\n",
    "def idx_matrix(data: pd.Series, events: pd.DataFrame):\n",
    "    '''\n",
    "    AFML pg 63 snippet 4.3\n",
    "    Calculates the number of times events sample overlaps each other\n",
    "    Some simple moification included\n",
    "    \n",
    "    logic is still based on initial func stated in AFML pg 63\n",
    "    \n",
    "    This func has been modified to fit the example given\n",
    "       \n",
    "    This version of idx_matrix will take up to 30 minutes when tested against (DataFrame.shape(4025, 1840))\n",
    "    '''\n",
    "    indM_ = pd.DataFrame(0, index = data, columns=np.arange(events.t1.shape[0]))\n",
    "    for i,(t0,t1) in enumerate(events.t1.items()):\n",
    "        indM_.loc[t0:t1,i] = 1.\n",
    "    \n",
    "    idxM = indM_[indM_.sum(axis = 1) != 0]\n",
    "    return idxM\n",
    "\n",
    "def seq_bts(idxM: pd.DataFrame, phi: list, sample_len = None):\n",
    "    '''\n",
    "    Modified to fit the example\n",
    "    '''\n",
    "    # Generate a sample via sequential bootstrap\n",
    "    if sample_len is None:\n",
    "        sample_len=idxM.shape[1]\n",
    "    \n",
    "    avgU=pd.Series(0., dtype=float)\n",
    "    for i in idxM:\n",
    "        indM_ = idxM[phi + [i]] # reduce indM\n",
    "        avgU.loc[i] = rs.av_unique(indM_).iloc[-1] #get the last value, if it is very unique it's value will be high\n",
    "    prob = avgU/ avgU.sum() # this part lets you know the uniqueness of the entire series based on it's pick, hence assign prob\n",
    "    print(prob)\n",
    "    phi.append(np.random.choice(idxM.columns, p=prob))\n",
    "    return phi\n",
    "\n",
    "\n",
    "t1 = pd.Series([2,3,5], index=[0,2,4], name = 't1')\n",
    "bar = np.arange(t1.max() + 1)\n",
    "event = pd.Series([2,3,5], index=[0,2,4], name = 't1').to_frame()\n",
    "idxM1 = idx_matrix(data = bar, events = event)\n",
    "\n",
    "idxM1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    0.454545\n",
      "1    0.272727\n",
      "2    0.272727\n",
      "dtype: float64\n",
      "\n",
      "Current samples: [1, 2, 2]\n",
      "Next pick would most likely be column 2\n"
     ]
    }
   ],
   "source": [
    "#if you run this a few more times, but if your dataset is limited, it will still not achieve the goal.\n",
    "phi = [1,2]\n",
    "out = seq_bts(idxM = idxM1, sample_len = None, phi = phi )\n",
    "\n",
    "print(\"\\nCurrent samples: {0}\\nNext pick would most likely be column {1}\".format(out, out[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    0.5\n",
      "1    0.3\n",
      "2    0.2\n",
      "dtype: float64\n",
      "\n",
      "Current samples: [1, 2, 2, 0]\n",
      "Next pick would most likely be column 0\n"
     ]
    }
   ],
   "source": [
    "# Exercise 4.7\n",
    "\n",
    "phi = [1,2,2]\n",
    "out = seq_bts(idxM = idxM1, sample_len = None, phi = phi )\n",
    "\n",
    "#As mentioned above, if the same column was picked twice even if it is unique, it is unlikely to be picked twice.\n",
    "print(\"\\nCurrent samples: {0}\\nNext pick would most likely be column {1}\".format(out, out[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "As mentioned previously, the func will keep picking what it considered as unique.\n",
    "\n",
    "If you notice, both exercise outcome favors column 0 as the next pick.\n",
    "\n",
    "Because it is considered unique to the current list, since it was the only one that was not picked.\n",
    "\n",
    "As mentioned earlier, sequential boostrap will reduce the probability of a sample being repick. \n",
    "\n",
    "But given a large dataset, its overall uniqueness will gradually drop since getting a highly unique column is not possible, if the most unique ones already exist within the sample group.\n",
    "\n",
    "This is to maintain it's uniqueness of all samples choosen.\n",
    "\n",
    "**Note**\n",
    "\n",
    "Probability of choosing maybe high, but it is still random hence it may not be reflected if the func was to be run a few more times."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}