AFML 6.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ensemble Methods\n",
    "\n",
    "This notebook will cover exercise answer.\n",
    "\n",
    "* Exercise 6.1\n",
    "* Exercise 6.2\n",
    "* Exercise 6.3\n",
    "* Exercise 6.4\n",
    "* Exercise 6.5\n",
    "\n",
    "As we go along, there will be some explanations.\n",
    "\n",
    "ML models are generally smart enough to deduce key features and perform forecast, however training such models to produce effective and reliable outcome is the key.\n",
    "\n",
    "Most of the functions below can be found under research/Ensemble\n",
    "\n",
    "Contact: boyboi86@gmail.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num of CPU core:  4\n",
      "Machine info:  Windows-10-10.0.18362-SP0\n",
      "Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]\n",
      "Numpy 1.17.3\n",
      "Pandas 1.0.3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Anaconda3\\lib\\site-packages\\statsmodels\\tools\\_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
      "  import pandas.util.testing as tm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1500x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import research as rs\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "individual learners are not considered poor classifier\n",
      "0.3333333333333333 0.4407356602143977\n",
      "individual learners are not considered poor classifier\n",
      "0.3333333333333333 0.4811966952738904\n",
      "individual learners are not considered poor classifier\n",
      "0.3333333333333333 0.5029710233411802\n"
     ]
    }
   ],
   "source": [
    "from scipy.special import comb\n",
    "\n",
    "def cls_accuracy(N = 100, p = 1./3, k = 3.):\n",
    "    #N, p, k = 100, 1./3, 3.\n",
    "    p_ = 0\n",
    "\n",
    "    for i in np.arange(0, int(N/k)+1):\n",
    "        p_ += comb(N,i)*p**i*(1-p)**(N-i)\n",
    "    if p > 1-p_ : \n",
    "        print(\"individual learners are considered poor classifier\")\n",
    "    else: \n",
    "        print(\"individual learners are not considered poor classifier\")\n",
    "    print(p, 1-p_)\n",
    "\n",
    "cls_accuracy(N = 10, p = 1./3, k = 3.) # cls = 3, N estimate = 10\n",
    "cls_accuracy(N = 100, p = 1./3, k = 3.) # cls = 3, N estimate = 100\n",
    "cls_accuracy(N = 1001, p = 1./3, k = 3.) # cls = 3, N estimate = 1001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "individual learners are considered poor classifier\n",
      "0.5 0.46020538130641064\n",
      "individual learners are not considered poor classifier\n",
      "0.5 0.5000000000001502\n"
     ]
    }
   ],
   "source": [
    "#optional comparison with 2 class only\n",
    "\n",
    "cls_accuracy(N = 100, p = 1./2, k = 2.) # cls = 2, N estimate = 100\n",
    "cls_accuracy(N = 1001, p = 1./2, k = 2.) # cls = 2, N estimate = 1001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy improvement\n",
    "\n",
    "Based on the above using only 2 classes, if a classifer were to be poor.\n",
    "\n",
    "There is always a much higher chance bagging ensemble will not help to improve bias, unless a sufficiently large estimates were used (N = 1001).\n",
    "\n",
    "On the other hand if the classifer is considered good, a small amount of estimators can already provide good outcome (class k = 3).\n",
    "\n",
    "In short, identified relevant features does improve overall accuracy of classifers. All of it would be before employing any ML ensemble method.\n",
    "\n",
    "### Variance reduction\n",
    "\n",
    "Bagging is sampling with replacement, each subset of the sample can be used mulitple times. This may introduce more randomness, will slightly higher bias (If it was already bias, it will probability be worst).\n",
    "\n",
    "Pasting is sampling without replacement, each subset of the sample can be used once at most (Requires large dataset to work and more computering power).\n",
    "\n",
    "In the case of financial application, samples drawn with replacement are more likely to be virtually correlated (almost 1.0), bagging will not reduce variance. (Bagging reduce variance is always under assumption that observations are IID, this is not true in financial applications)\n",
    "\n",
    "> \"In chapter 4 we studied why financial observations cannot be assumed to be IID..\n",
    "> \n",
    "> ..and Bagging will not reduce variance regardless of number of N.\"\n",
    ">\n",
    "> AFML chapter 6, page 97, section 6.3.3\n",
    "\n",
    "As a result, OOB score will always be inflated. As tested in previous notebook. [juypter notebook](https://github.com/boyboi86/AFML/blob/master/AFML%204.1.ipynb)\n",
    "\n",
    "**Note**\n",
    "\n",
    "If you realised when we tried to run a binomial expansion on accuracy for classes vs num of estimators. The most effective way to reduce bias is actually before running any ML ensemble methods.\n",
    "\n",
    "As for variance, pasting ensemble does provide a better solution however would be considered expensive. The other way which was introduced by Dr Marco Lopez would be sequential bootstrap ensemble method. [juypter notebook 5.4](http://localhost:8888/notebooks/AFML%205.4.ipynb#)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bagging as a method\n",
    "\n",
    "As long as samples are considered redundant (Non-IID) or virtually correlated (almost 1.0), bagging will be ineffective and still prone to overfit.\n",
    "\n",
    "In the case where samples are lowly unique, observations are most likely virtually identical to each other. As a result, bagging will still be ineffective.\n",
    "\n",
    "Since bagging based on the above lowly unique samples will lead to overfitting problem discussed, Out-of-bag (OOB) score will naturally be inflated hence unreliable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>close</th>\n",
       "      <th>ffd_series</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-01-04 23:20:12.567</th>\n",
       "      <td>2040.75</td>\n",
       "      <td>-0.003825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-07 10:55:23.194</th>\n",
       "      <td>2008.00</td>\n",
       "      <td>0.004166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-07 15:10:50.900</th>\n",
       "      <td>2014.50</td>\n",
       "      <td>-0.002360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 01:48:57.964</th>\n",
       "      <td>2037.25</td>\n",
       "      <td>0.006857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 05:47:32.006</th>\n",
       "      <td>2032.75</td>\n",
       "      <td>-0.003119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-28 01:31:48.252</th>\n",
       "      <td>2205.75</td>\n",
       "      <td>-0.099969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-04 23:32:49.403</th>\n",
       "      <td>2183.50</td>\n",
       "      <td>-0.102717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-05 02:06:52.025</th>\n",
       "      <td>2189.00</td>\n",
       "      <td>-0.096321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-08 12:46:05.346</th>\n",
       "      <td>2233.00</td>\n",
       "      <td>-0.101778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-14 20:09:50.768</th>\n",
       "      <td>2248.25</td>\n",
       "      <td>-0.103682</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>839 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                           close  ffd_series\n",
       "2015-01-04 23:20:12.567  2040.75   -0.003825\n",
       "2015-01-07 10:55:23.194  2008.00    0.004166\n",
       "2015-01-07 15:10:50.900  2014.50   -0.002360\n",
       "2015-01-08 01:48:57.964  2037.25    0.006857\n",
       "2015-01-08 05:47:32.006  2032.75   -0.003119\n",
       "...                          ...         ...\n",
       "2016-11-28 01:31:48.252  2205.75   -0.099969\n",
       "2016-12-04 23:32:49.403  2183.50   -0.102717\n",
       "2016-12-05 02:06:52.025  2189.00   -0.096321\n",
       "2016-12-08 12:46:05.346  2233.00   -0.101778\n",
       "2016-12-14 20:09:50.768  2248.25   -0.103682\n",
       "\n",
       "[839 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exercise 6.3\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import BaggingClassifier, RandomForestClassifier\n",
    "\n",
    "#from sklearn.datasets import make_classification # create dataset\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "dollar = pd.read_csv('./research/Sample_data/dollar_bars.txt', \n",
    "                 sep=',', \n",
    "                 header=0, \n",
    "                 parse_dates = True, \n",
    "                 index_col=['date_time'])\n",
    "\n",
    "# For most part of the func we only use 'close'\n",
    "\n",
    "close = dollar['close'].to_frame()\n",
    "\n",
    "ffd_series = close.apply(np.log).cumsum()\n",
    "ffd_series = rs.fracDiff_FFD(ffd_series, \n",
    "                    d = 1.99999889, \n",
    "                    thres=1e-5\n",
    "                   ).dropna()\n",
    "\n",
    "cs_event = rs.cs_filter(data = ffd_series, limit=(ffd_series.std() * 0.2))\n",
    "\n",
    "df_mtx = pd.DataFrame(index = cs_event).assign(close = close,\n",
    "                                                ffd_series = ffd_series).drop_duplicates().dropna()\n",
    "df_mtx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Desktop\\Python\\research\\Labels\\triple_barrier_method.py:75: UserWarning: Data and events index shape must be same, reindex data to fit events\n",
      "  warnings.warn('Data and events index shape must be same, reindex data to fit events')\n",
      "C:\\Users\\Wei_X\\Desktop\\Python\\research\\Labels\\triple_barrier_method.py:112: UserWarning: Not Recommended: No side prediction provided\n",
      "  warnings.warn('Not Recommended: No side prediction provided')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[                                             t1                      sl  \\\n",
      "2015-01-07 15:10:50.900 2015-01-12 16:02:08.112                     NaT   \n",
      "2015-01-08 01:48:57.964 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-08 05:47:32.006 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-09 14:48:46.704 2015-01-14 19:14:20.771 2015-01-14 04:31:40.468   \n",
      "2015-01-12 14:36:34.243 2015-01-19 09:36:49.301 2015-01-14 19:14:20.771   \n",
      "...                                         ...                     ...   \n",
      "2015-09-01 19:46:17.742 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-01 22:05:09.069 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 00:20:04.277 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 02:09:17.333 2015-09-07 04:40:25.376                     NaT   \n",
      "2015-09-02 08:08:50.931 2015-09-08 01:27:51.915                     NaT   \n",
      "\n",
      "                                             pt  \n",
      "2015-01-07 15:10:50.900 2015-01-08 01:48:57.964  \n",
      "2015-01-08 01:48:57.964                     NaT  \n",
      "2015-01-08 05:47:32.006                     NaT  \n",
      "2015-01-09 14:48:46.704                     NaT  \n",
      "2015-01-12 14:36:34.243                     NaT  \n",
      "...                                         ...  \n",
      "2015-09-01 19:46:17.742                     NaT  \n",
      "2015-09-01 22:05:09.069                     NaT  \n",
      "2015-09-02 00:20:04.277                     NaT  \n",
      "2015-09-02 02:09:17.333                     NaT  \n",
      "2015-09-02 08:08:50.931                     NaT  \n",
      "\n",
      "[279 rows x 3 columns]] this out\n",
      "[                                             t1                      sl  \\\n",
      "2015-01-07 15:10:50.900 2015-01-12 16:02:08.112                     NaT   \n",
      "2015-01-08 01:48:57.964 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-08 05:47:32.006 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-09 14:48:46.704 2015-01-14 19:14:20.771 2015-01-14 04:31:40.468   \n",
      "2015-01-12 14:36:34.243 2015-01-19 09:36:49.301 2015-01-14 19:14:20.771   \n",
      "...                                         ...                     ...   \n",
      "2015-09-01 19:46:17.742 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-01 22:05:09.069 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 00:20:04.277 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 02:09:17.333 2015-09-07 04:40:25.376                     NaT   \n",
      "2015-09-02 08:08:50.931 2015-09-08 01:27:51.915                     NaT   \n",
      "\n",
      "                                             pt  \n",
      "2015-01-07 15:10:50.900 2015-01-08 01:48:57.964  \n",
      "2015-01-08 01:48:57.964                     NaT  \n",
      "2015-01-08 05:47:32.006                     NaT  \n",
      "2015-01-09 14:48:46.704                     NaT  \n",
      "2015-01-12 14:36:34.243                     NaT  \n",
      "...                                         ...  \n",
      "2015-09-01 19:46:17.742                     NaT  \n",
      "2015-09-01 22:05:09.069                     NaT  \n",
      "2015-09-02 00:20:04.277                     NaT  \n",
      "2015-09-02 02:09:17.333                     NaT  \n",
      "2015-09-02 08:08:50.931                     NaT  \n",
      "\n",
      "[279 rows x 3 columns],                                              t1  sl  pt\n",
      "2015-09-03 00:05:02.346 2015-09-08 01:27:51.915 NaT NaT\n",
      "2015-09-03 06:00:48.819 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 07:52:14.021 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 09:57:43.822 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 12:33:59.011 2015-09-09 03:06:59.837 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-01-21 16:00:11.522 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 18:20:55.061 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:14:12.535 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:36:37.429 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 20:07:23.306 2016-01-27 00:04:03.812 NaT NaT\n",
      "\n",
      "[279 rows x 3 columns]] this out\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-05-31 12:45:26.856440 33.33% _pt_sl_t1 done after 0.11 minutes. Remaining 0.21 minutes.\r",
      "2020-05-31 12:45:26.864467 66.67% _pt_sl_t1 done after 0.11 minutes. Remaining 0.05 minutes.\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[                                             t1                      sl  \\\n",
      "2015-01-07 15:10:50.900 2015-01-12 16:02:08.112                     NaT   \n",
      "2015-01-08 01:48:57.964 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-08 05:47:32.006 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-09 14:48:46.704 2015-01-14 19:14:20.771 2015-01-14 04:31:40.468   \n",
      "2015-01-12 14:36:34.243 2015-01-19 09:36:49.301 2015-01-14 19:14:20.771   \n",
      "...                                         ...                     ...   \n",
      "2015-09-01 19:46:17.742 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-01 22:05:09.069 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 00:20:04.277 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 02:09:17.333 2015-09-07 04:40:25.376                     NaT   \n",
      "2015-09-02 08:08:50.931 2015-09-08 01:27:51.915                     NaT   \n",
      "\n",
      "                                             pt  \n",
      "2015-01-07 15:10:50.900 2015-01-08 01:48:57.964  \n",
      "2015-01-08 01:48:57.964                     NaT  \n",
      "2015-01-08 05:47:32.006                     NaT  \n",
      "2015-01-09 14:48:46.704                     NaT  \n",
      "2015-01-12 14:36:34.243                     NaT  \n",
      "...                                         ...  \n",
      "2015-09-01 19:46:17.742                     NaT  \n",
      "2015-09-01 22:05:09.069                     NaT  \n",
      "2015-09-02 00:20:04.277                     NaT  \n",
      "2015-09-02 02:09:17.333                     NaT  \n",
      "2015-09-02 08:08:50.931                     NaT  \n",
      "\n",
      "[279 rows x 3 columns],                                              t1  sl  pt\n",
      "2015-09-03 00:05:02.346 2015-09-08 01:27:51.915 NaT NaT\n",
      "2015-09-03 06:00:48.819 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 07:52:14.021 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 09:57:43.822 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 12:33:59.011 2015-09-09 03:06:59.837 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-01-21 16:00:11.522 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 18:20:55.061 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:14:12.535 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:36:37.429 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 20:07:23.306 2016-01-27 00:04:03.812 NaT NaT\n",
      "\n",
      "[279 rows x 3 columns],                                              t1  sl  pt\n",
      "2016-01-21 20:15:15.382 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-22 01:19:34.937 2016-01-27 13:41:54.306 NaT NaT\n",
      "2016-01-22 04:21:15.807 2016-01-27 13:41:54.306 NaT NaT\n",
      "2016-01-22 06:19:21.200 2016-01-27 13:41:54.306 NaT NaT\n",
      "2016-01-22 08:48:09.889 2016-01-27 13:41:54.306 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-11-28 01:31:48.252 2016-12-04 23:32:49.403 NaT NaT\n",
      "2016-12-04 23:32:49.403 2016-12-14 20:09:50.768 NaT NaT\n",
      "2016-12-05 02:06:52.025 2016-12-14 20:09:50.768 NaT NaT\n",
      "2016-12-08 12:46:05.346 2016-12-14 20:09:50.768 NaT NaT\n",
      "2016-12-14 20:09:50.768                     NaT NaT NaT\n",
      "\n",
      "[279 rows x 3 columns]] this out\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-05-31 12:45:27.126982 100.0% _pt_sl_t1 done after 0.11 minutes. Remaining 0.0 minutes.\n"
     ]
    }
   ],
   "source": [
    "df_mtx['volatility'] = rs.vol(df_mtx.close, span0 = 50) #one of our features, since we do not have a side\n",
    "\n",
    "df_mtx.dropna(inplace = True)\n",
    "\n",
    "vb = rs.vert_barrier(data = df_mtx.close, events = cs_event, period = 'days', freq = 5)\n",
    "\n",
    "# triple barrier events based on filter while data is also based on filtered index\n",
    "tb = rs.tri_barrier(data = df_mtx.close, \n",
    "                    events = cs_event, \n",
    "                    trgt = df_mtx['volatility'], \n",
    "                    min_req = 0.0002, \n",
    "                    num_threads = 3, \n",
    "                    ptSl= [2,2], #2x barriers\n",
    "                    t1 = vb, \n",
    "                    side = None)\n",
    "\n",
    "mlabel = rs.meta_label(data = df_mtx.close, \n",
    "                       events = tb, \n",
    "                       drop = 0.05) # because we do not have side, we need to drop rare labels\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 1.0    489\n",
       "-1.0    345\n",
       "Name: bin, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlabel['bin'].value_counts() #834"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ave Uniqueness of Observations 0.1201464235230114\n"
     ]
    }
   ],
   "source": [
    "X = df_mtx.reindex(mlabel.index)\n",
    "Z = tb.reindex(mlabel.index)\n",
    "y = mlabel['bin']\n",
    "\n",
    "idx_Mat0 = rs.mp_idx_matrix(data = X.close, events = Z)\n",
    "\n",
    "avgU = rs.av_unique(idx_Mat0).mean() #get ave uniqueness\n",
    "print(\"Ave Uniqueness of Observations\", avgU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default rf Out-of-bag score: 0.8027444253859348\n",
      "\n",
      "Default dt Out-of-bag score: 0.7873070325900514\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Split data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, shuffle=True, stratify=None)\n",
    "\n",
    "# Benchmarks\n",
    "rf_clf0 = RandomForestClassifier(n_estimators = 1000, \n",
    "                                bootstrap=True, \n",
    "                                n_jobs=1,\n",
    "                                random_state=42,\n",
    "                                oob_score=True)\n",
    "\n",
    "base_estimate0 = DecisionTreeClassifier()\n",
    "\n",
    "\n",
    "bag_clf0 = BaggingClassifier(base_estimator = base_estimate0,\n",
    "                                n_estimators = 1000,\n",
    "                                bootstrap=True, \n",
    "                                n_jobs=1, \n",
    "                                random_state=42,\n",
    "                                oob_score=True)\n",
    "\n",
    "\n",
    "rf_clf0.fit(X_train, y_train)\n",
    "bag_clf0.fit(X_train, y_train)\n",
    "\n",
    "print('Default rf Out-of-bag score: {}\\n'.format(rf_clf0.oob_score_))\n",
    "print('Default dt Out-of-bag score: {}\\n'.format(bag_clf0.oob_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rf_clf rf Out-of-bag score: 0.7701543739279588\n",
      "\n",
      "bag_clf Out-of-bag score: 0.758147512864494\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#based on book recommendation\n",
    "rf_clf = RandomForestClassifier(n_estimators = 1000,\n",
    "                                criterion = \"entropy\",\n",
    "                                max_samples=avgU, #note averge unique used\n",
    "                                bootstrap=True, \n",
    "                                n_jobs=1,\n",
    "                                random_state=42,\n",
    "                                class_weight=\"balanced_subsample\",\n",
    "                                oob_score=True)\n",
    "\n",
    "clf = DecisionTreeClassifier(criterion = \"entropy\", \n",
    "                             max_features=\"auto\", \n",
    "                             class_weight=\"balanced\")\n",
    "\n",
    "bag_clf = BaggingClassifier(base_estimator = clf,\n",
    "                            n_estimators = 1000, \n",
    "                            max_samples=avgU, #note averge unique used\n",
    "                            bootstrap=True, \n",
    "                            n_jobs=1,\n",
    "                            random_state=42,\n",
    "                            oob_score=True)\n",
    "\n",
    "rf_clf.fit(X_train, y_train)\n",
    "bag_clf.fit(X_train, y_train)\n",
    "\n",
    "print('rf_clf rf Out-of-bag score: {}\\n'.format(rf_clf.oob_score_))\n",
    "print('bag_clf Out-of-bag score: {}\\n'.format(bag_clf.oob_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Classifer vs Decision Tree Classifer (bagging)\n",
    "\n",
    "The OOB scores for both types of classifers regardless of bagging or not, proves to have a lower OOB score when compared.\n",
    "\n",
    "* criterion = \"entropy\" for the information gain\n",
    "* max_sample = avg uniqueness of observations\n",
    "* class_weight (depending on type of classifer)\n",
    "* max_features should be \"auto\" (Decision Tree Ensemble) and 1.0 (Random Forest Ensemble)\n",
    "* n_estimate has to be large enough (Refer to accuracy improvement at the top)\n",
    "\n",
    "The above parameters will have an impact on your overall OOB score. Less inflated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rf_clf rf Out-of-bag score: 0.79073756432247\n",
      "\n",
      "bag_clf Out-of-bag score: 0.79073756432247\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#based on book recommendation\n",
    "rf_clf1 = RandomForestClassifier(n_estimators = 1000,\n",
    "                                criterion = \"entropy\",\n",
    "                                bootstrap=True, \n",
    "                                n_jobs=1,\n",
    "                                random_state=42,\n",
    "                                class_weight=\"balanced_subsample\",\n",
    "                                oob_score=True)\n",
    "\n",
    "clf1 = DecisionTreeClassifier(criterion = \"entropy\",\n",
    "                             splitter=\"random\", #added random as splitter, which was in rf but not in dt\n",
    "                             max_features=None, \n",
    "                             class_weight=\"balanced\")\n",
    "\n",
    "bag_clf1 = BaggingClassifier(base_estimator = clf1,\n",
    "                            n_estimators = 850, \n",
    "                            max_samples=avgU, #note averge unique used\n",
    "                            bootstrap=True, \n",
    "                            n_jobs=1,\n",
    "                            random_state=42,\n",
    "                            oob_score=True)\n",
    "\n",
    "rf_clf1.fit(X_train, y_train)\n",
    "bag_clf1.fit(X_train, y_train)\n",
    "\n",
    "print('rf_clf rf Out-of-bag score: {}\\n'.format(rf_clf1.oob_score_))\n",
    "print('bag_clf Out-of-bag score: {}\\n'.format(bag_clf1.oob_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest vs Decision Tree Ensemble\n",
    "\n",
    "**Key difference**\n",
    "\n",
    "The max_sample for Decision Tree was set to average uniqueness of observations (only using a fraction of dataset) while Random Forest was using default (entire X dataset).\n",
    "\n",
    "**The changes made to Decision tree**\n",
    "\n",
    "* splitter = \"random\"\n",
    "* max_features = None (which will affect split)\n",
    "\n",
    "**The changes made to Bagging Classifer**\n",
    "\n",
    "* n_estimate = 850\n",
    "\n",
    "After the changes, OOB score for both Decision Tree Ensemble and Random Forest became identical: 0.79073756432247\n",
    "\n",
    "**Note**\n",
    "\n",
    "Initutively this is my guess.\n",
    "\n",
    "The overall uniqueness of the samples used will affect the number of estimators required. Inverse relationship.\n",
    "\n",
    "If random forest classifer was employed instead of decision tree classifer. We can have randomness incorporated into our algo and reduced number of samples required (reduce variance without overfitting). \n",
    "\n",
    "More importantly, a reduced number of estimators required (less expensive).\n",
    "\n",
    "Hence, if possible we should modify Random Forest Classifer to fit it with sequential bootstrap and use it with bagging for optimal results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "individual learners are considered poor classifier\n",
      "0.3333333333333333 0.33333333333333326\n",
      "individual learners are considered poor classifier\n",
      "0.2 0.19999999999999996\n",
      "individual learners are considered poor classifier\n",
      "0.1 0.09999999999999998\n"
     ]
    }
   ],
   "source": [
    "# if N is too small\n",
    "cls_accuracy(N = 1, p = 1./3, k = 3.)\n",
    "cls_accuracy(N = 1, p = 1./5, k = 5.)\n",
    "cls_accuracy(N = 1, p = 1./10, k = 10.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "individual learners are considered poor classifier\n",
      "0.3333333333333333 -inf\n",
      "individual learners are not considered poor classifier\n",
      "0.2 0.48273446631083894\n",
      "individual learners are not considered poor classifier\n",
      "0.1 0.475713305287882\n"
     ]
    }
   ],
   "source": [
    "# if N is too large\n",
    "cls_accuracy(N = 1200, p = 1./3, k = 3.)\n",
    "cls_accuracy(N = 1200, p = 1./5, k = 5.)\n",
    "cls_accuracy(N = 1200, p = 1./10, k = 10.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of Trees vs Number of Features\n",
    "\n",
    "With reference to the above.\n",
    "\n",
    "When we were using binomial expansion on N (number of trees) against k (number of classes).\n",
    "\n",
    "The more classes means less trees required. Likewise, if there is more relevant features less tree will be required.\n",
    "\n",
    "However, these features are under the assumption of relevant or what we would considered elements which will provide information gains.\n",
    "\n",
    "Using the same binomial formula, under the same condition where every features are equal weighted with binary labels only. To derive below results.\n",
    "\n",
    "**If Number of Trees are too small**\n",
    "\n",
    "The number of trees will never be too small for classes available. (See above for mathematical proof)\n",
    "\n",
    "But if N <= 1, the outcome will be a poor classifier even with relevant features available.\n",
    "\n",
    "**If Number of Trees are too large**\n",
    "\n",
    "However, the number of trees can be too large for features available. (Notice as N = 1200, k = 3 will yield -infinite probabilities.)\n",
    "\n",
    "#### Conclusion\n",
    "\n",
    "In order to attain high accuracy, the proportion of relevant classes must be \"inline\" with the number of trees generated to yield optimal results.\n",
    "\n",
    "However, the number of estimators does seem to be a debate within ML community:\n",
    "\n",
    "(Computational power vs Estimators)[https://www.researchgate.net/publication/230766603_How_Many_Trees_in_a_Random_Forest]\n",
    "\n",
    "In view of Condorcet's jury theorem, it seems that regardless of k class or n_features. Ultimately, in order to make a correct prediction, probability is more important. (P > 0.5)\n",
    "\n",
    "**Note**\n",
    "\n",
    "In AFML page 101, section 6.7. There is a short discussion on Support Vector Machine (SVM) scalability.\n",
    "\n",
    "This may not be important now, since we are focus on mean-revision strategy. \n",
    "\n",
    "However, for trend strategy. This might be useful.\n",
    "\n",
    "[SVM Trend Strategy](https://www.cs.princeton.edu/sites/default/files/uploads/saahil_madge.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Strat KFold accuracy with shuffle: 0.80450905\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "accuracy_array = np.zeros(5)\n",
    "\n",
    "skf = StratifiedKFold(n_splits=5, \n",
    "                      shuffle=True, #shuffle = True\n",
    "                      random_state = 42)\n",
    "\n",
    "i = 0\n",
    "for train_index, test_index in skf.split(X, y):\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n",
    "\n",
    "    rf_clf0.fit(X_train, y_train.values.ravel()) #use the original rf cls\n",
    "    y_pred_rf = rf_clf0.predict_proba(X_test)[:, 1] #True positive only\n",
    "    y_pred = rf_clf0.predict(X_test)\n",
    "    accuracy_array[i] = accuracy_score(y_test, y_pred)\n",
    "    i += 1\n",
    "\n",
    "print(\"Mean Strat KFold accuracy with shuffle: {:.8f}\".format(np.mean(accuracy_array)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OOB score vs Stratified KFold accuracy\n",
    "\n",
    "**Based on initial random forest input: OOB score: 0.8027444253859348**\n",
    "\n",
    "OOB accuracy is based on shuffled trained data (Train_test_split defaults) against ensemble data that was sampled (Which are part of instead of full ensemble, occasionally test data might be randomly picked) when fitting.\n",
    "\n",
    "Only subsampled of forest, which may introduce more randomness.\n",
    "\n",
    "Moreover, if observations were to be redundant (Non-IID). Repeated sampling on such data will inflate OOB score.\n",
    "\n",
    "**Based on Shuffled Stratified KFold: Mean KFold accuracy: 0.80450905**\n",
    "\n",
    "Stratified KFold will use the entire ensemble (full forest) to evaluate trained data, hence accuracy should be better. Stratification will balance the weight throughout the dataset based on class which makes it fair. \n",
    "\n",
    "With shuffle however, dataset will not be able to preserve it's order dependency.\n",
    "\n",
    "Observations were shuffled then splitted. It will still end up with the same problem with OOB methods, in fact even worst since they will use the entire ensemble data to evaluate. As such, their accuracy will also be more inflated.\n",
    "\n",
    "**Note**\n",
    "\n",
    "Without shuffle Kfold score is not inflated, kindly refer to [Notebook 4.1](http://localhost:8888/notebooks/AFML%204.1.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}