AFML 7.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-Validation in Finance\n",
    "\n",
    "This notebook will cover exercise answer.\n",
    "\n",
    "* Exercise 7.1\n",
    "* Exercise 7.2\n",
    "* Exercise 7.3\n",
    "* Exercise 7.4\n",
    "* Exercise 7.5\n",
    "\n",
    "As we go along, there will be some explanations.\n",
    "\n",
    "Most of the functions below can be found under Tool/cross_validate\n",
    "\n",
    "Contact: boyboi86@gmail.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num of CPU core:  4\n",
      "Machine info:  Windows-10-10.0.18362-SP0\n",
      "Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]\n",
      "Numpy 1.17.3\n",
      "Pandas 1.0.3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Anaconda3\\lib\\site-packages\\statsmodels\\tools\\_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
      "  import pandas.util.testing as tm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1500x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import research as rs\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kfold CV\n",
    "\n",
    "As demostrated in [notebook 6.1](https://github.com/boyboi86/AFML/blob/master/AFML%206.1.ipynb), when datasets are shuffled it is guaranteed to be overfit. Leakage issue will be much more prevailing than OOB method (subsampling), because the entire ensemble data will be used to evaluate.\n",
    "\n",
    "However, shuffling is not without merit. If datasets were truly IID, shuffling will add the necessary randomness to the training set. As such, outcome will be more reliable.\n",
    "\n",
    "The only problem is financial series are not considered IID. When financial datasets are shuffled before partition. Some test data will be used to evaluate, in fact when split is high there is the higher the proportion since only 1 of the set is reserved.\n",
    "\n",
    "During evaluation/fitting if split was 5, 80% of the shuffled data will be choosen to be train set. Assuming the shuffled test set (remaining 20%) is all part of the n_sample choosen to evaluate. As a result, CV score will be super inflated.\n",
    "\n",
    "However if dataset were not shuffled, the above will not be a problem. Because evaluation will be only based on n_sample in contingent. Unshuffled training set can only be evaluate against, n_samples (identical).\n",
    "\n",
    "**Conclusion**\n",
    "\n",
    "KFold can only be reliable as a method, when not non-IID samples are not shuffled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>close</th>\n",
       "      <th>ffd_series</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-01-04 23:20:12.567</th>\n",
       "      <td>2040.75</td>\n",
       "      <td>-0.003825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-07 10:55:23.194</th>\n",
       "      <td>2008.00</td>\n",
       "      <td>0.004166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-07 15:10:50.900</th>\n",
       "      <td>2014.50</td>\n",
       "      <td>-0.002360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 01:48:57.964</th>\n",
       "      <td>2037.25</td>\n",
       "      <td>0.006857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-08 05:47:32.006</th>\n",
       "      <td>2032.75</td>\n",
       "      <td>-0.003119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-11-28 01:31:48.252</th>\n",
       "      <td>2205.75</td>\n",
       "      <td>-0.099969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-04 23:32:49.403</th>\n",
       "      <td>2183.50</td>\n",
       "      <td>-0.102717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-05 02:06:52.025</th>\n",
       "      <td>2189.00</td>\n",
       "      <td>-0.096321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-08 12:46:05.346</th>\n",
       "      <td>2233.00</td>\n",
       "      <td>-0.101778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-14 20:09:50.768</th>\n",
       "      <td>2248.25</td>\n",
       "      <td>-0.103682</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>839 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                           close  ffd_series\n",
       "2015-01-04 23:20:12.567  2040.75   -0.003825\n",
       "2015-01-07 10:55:23.194  2008.00    0.004166\n",
       "2015-01-07 15:10:50.900  2014.50   -0.002360\n",
       "2015-01-08 01:48:57.964  2037.25    0.006857\n",
       "2015-01-08 05:47:32.006  2032.75   -0.003119\n",
       "...                          ...         ...\n",
       "2016-11-28 01:31:48.252  2205.75   -0.099969\n",
       "2016-12-04 23:32:49.403  2183.50   -0.102717\n",
       "2016-12-05 02:06:52.025  2189.00   -0.096321\n",
       "2016-12-08 12:46:05.346  2233.00   -0.101778\n",
       "2016-12-14 20:09:50.768  2248.25   -0.103682\n",
       "\n",
       "[839 rows x 2 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exercise 7.2\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import StratifiedKFold, KFold\n",
    "\n",
    "dollar = pd.read_csv('./research/Sample_data/dollar_bars.txt', \n",
    "                 sep=',', \n",
    "                 header=0, \n",
    "                 parse_dates = True, \n",
    "                 index_col=['date_time'])\n",
    "\n",
    "# For most part of the func we only use 'close'\n",
    "\n",
    "close = dollar['close'].to_frame()\n",
    "\n",
    "ffd_series = close.apply(np.log).cumsum()\n",
    "ffd_series = rs.fracDiff_FFD(ffd_series, \n",
    "                    d = 1.99999889, \n",
    "                    thres=1e-5\n",
    "                   ).dropna()\n",
    "\n",
    "cs_event = rs.cs_filter(data = ffd_series, limit=(ffd_series.std() * 0.2))\n",
    "\n",
    "df_mtx = pd.DataFrame(index = cs_event).assign(close = close,\n",
    "                                                ffd_series = ffd_series).drop_duplicates().dropna()\n",
    "df_mtx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Desktop\\Python\\research\\Labels\\triple_barrier_method.py:75: UserWarning: Data and events index shape must be same, reindex data to fit events\n",
      "  warnings.warn('Data and events index shape must be same, reindex data to fit events')\n",
      "C:\\Users\\Wei_X\\Desktop\\Python\\research\\Labels\\triple_barrier_method.py:112: UserWarning: Not Recommended: No side prediction provided\n",
      "  warnings.warn('Not Recommended: No side prediction provided')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[                                             t1                      sl  \\\n",
      "2015-01-07 15:10:50.900 2015-01-12 16:02:08.112                     NaT   \n",
      "2015-01-08 01:48:57.964 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-08 05:47:32.006 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-09 14:48:46.704 2015-01-14 19:14:20.771 2015-01-14 04:31:40.468   \n",
      "2015-01-12 14:36:34.243 2015-01-19 09:36:49.301 2015-01-14 19:14:20.771   \n",
      "...                                         ...                     ...   \n",
      "2015-09-01 19:46:17.742 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-01 22:05:09.069 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 00:20:04.277 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 02:09:17.333 2015-09-07 04:40:25.376                     NaT   \n",
      "2015-09-02 08:08:50.931 2015-09-08 01:27:51.915                     NaT   \n",
      "\n",
      "                                             pt  \n",
      "2015-01-07 15:10:50.900 2015-01-08 01:48:57.964  \n",
      "2015-01-08 01:48:57.964                     NaT  \n",
      "2015-01-08 05:47:32.006                     NaT  \n",
      "2015-01-09 14:48:46.704                     NaT  \n",
      "2015-01-12 14:36:34.243                     NaT  \n",
      "...                                         ...  \n",
      "2015-09-01 19:46:17.742                     NaT  \n",
      "2015-09-01 22:05:09.069                     NaT  \n",
      "2015-09-02 00:20:04.277                     NaT  \n",
      "2015-09-02 02:09:17.333                     NaT  \n",
      "2015-09-02 08:08:50.931                     NaT  \n",
      "\n",
      "[279 rows x 3 columns]] this out\n",
      "[                                             t1                      sl  \\\n",
      "2015-01-07 15:10:50.900 2015-01-12 16:02:08.112                     NaT   \n",
      "2015-01-08 01:48:57.964 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-08 05:47:32.006 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-09 14:48:46.704 2015-01-14 19:14:20.771 2015-01-14 04:31:40.468   \n",
      "2015-01-12 14:36:34.243 2015-01-19 09:36:49.301 2015-01-14 19:14:20.771   \n",
      "...                                         ...                     ...   \n",
      "2015-09-01 19:46:17.742 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-01 22:05:09.069 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 00:20:04.277 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 02:09:17.333 2015-09-07 04:40:25.376                     NaT   \n",
      "2015-09-02 08:08:50.931 2015-09-08 01:27:51.915                     NaT   \n",
      "\n",
      "                                             pt  \n",
      "2015-01-07 15:10:50.900 2015-01-08 01:48:57.964  \n",
      "2015-01-08 01:48:57.964                     NaT  \n",
      "2015-01-08 05:47:32.006                     NaT  \n",
      "2015-01-09 14:48:46.704                     NaT  \n",
      "2015-01-12 14:36:34.243                     NaT  \n",
      "...                                         ...  \n",
      "2015-09-01 19:46:17.742                     NaT  \n",
      "2015-09-01 22:05:09.069                     NaT  \n",
      "2015-09-02 00:20:04.277                     NaT  \n",
      "2015-09-02 02:09:17.333                     NaT  \n",
      "2015-09-02 08:08:50.931                     NaT  \n",
      "\n",
      "[279 rows x 3 columns],                                              t1  sl  pt\n",
      "2015-09-03 00:05:02.346 2015-09-08 01:27:51.915 NaT NaT\n",
      "2015-09-03 06:00:48.819 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 07:52:14.021 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 09:57:43.822 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 12:33:59.011 2015-09-09 03:06:59.837 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-01-21 16:00:11.522 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 18:20:55.061 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:14:12.535 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:36:37.429 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 20:07:23.306 2016-01-27 00:04:03.812 NaT NaT\n",
      "\n",
      "[279 rows x 3 columns]] this out\n",
      "[                                             t1                      sl  \\\n",
      "2015-01-07 15:10:50.900 2015-01-12 16:02:08.112                     NaT   \n",
      "2015-01-08 01:48:57.964 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-08 05:47:32.006 2015-01-13 09:38:58.103                     NaT   \n",
      "2015-01-09 14:48:46.704 2015-01-14 19:14:20.771 2015-01-14 04:31:40.468   \n",
      "2015-01-12 14:36:34.243 2015-01-19 09:36:49.301 2015-01-14 19:14:20.771   \n",
      "...                                         ...                     ...   \n",
      "2015-09-01 19:46:17.742 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-01 22:05:09.069 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 00:20:04.277 2015-09-07 01:34:00.944                     NaT   \n",
      "2015-09-02 02:09:17.333 2015-09-07 04:40:25.376                     NaT   \n",
      "2015-09-02 08:08:50.931 2015-09-08 01:27:51.915                     NaT   \n",
      "\n",
      "                                             pt  \n",
      "2015-01-07 15:10:50.900 2015-01-08 01:48:57.964  \n",
      "2015-01-08 01:48:57.964                     NaT  \n",
      "2015-01-08 05:47:32.006                     NaT  \n",
      "2015-01-09 14:48:46.704                     NaT  \n",
      "2015-01-12 14:36:34.243                     NaT  \n",
      "...                                         ...  \n",
      "2015-09-01 19:46:17.742                     NaT  \n",
      "2015-09-01 22:05:09.069                     NaT  \n",
      "2015-09-02 00:20:04.277                     NaT  \n",
      "2015-09-02 02:09:17.333                     NaT  \n",
      "2015-09-02 08:08:50.931                     NaT  \n",
      "\n",
      "[279 rows x 3 columns],                                              t1  sl  pt\n",
      "2015-09-03 00:05:02.346 2015-09-08 01:27:51.915 NaT NaT\n",
      "2015-09-03 06:00:48.819 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 07:52:14.021 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 09:57:43.822 2015-09-08 12:08:06.840 NaT NaT\n",
      "2015-09-03 12:33:59.011 2015-09-09 03:06:59.837 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-01-21 16:00:11.522 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 18:20:55.061 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:14:12.535 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 19:36:37.429 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-21 20:07:23.306 2016-01-27 00:04:03.812 NaT NaT\n",
      "\n",
      "[279 rows x 3 columns],                                              t1  sl  pt\n",
      "2016-01-21 20:15:15.382 2016-01-27 00:04:03.812 NaT NaT\n",
      "2016-01-22 01:19:34.937 2016-01-27 13:41:54.306 NaT NaT\n",
      "2016-01-22 04:21:15.807 2016-01-27 13:41:54.306 NaT NaT\n",
      "2016-01-22 06:19:21.200 2016-01-27 13:41:54.306 NaT NaT\n",
      "2016-01-22 08:48:09.889 2016-01-27 13:41:54.306 NaT NaT\n",
      "...                                         ...  ..  ..\n",
      "2016-11-28 01:31:48.252 2016-12-04 23:32:49.403 NaT NaT\n",
      "2016-12-04 23:32:49.403 2016-12-14 20:09:50.768 NaT NaT\n",
      "2016-12-05 02:06:52.025 2016-12-14 20:09:50.768 NaT NaT\n",
      "2016-12-08 12:46:05.346 2016-12-14 20:09:50.768 NaT NaT\n",
      "2016-12-14 20:09:50.768                     NaT NaT NaT\n",
      "\n",
      "[279 rows x 3 columns]] this out\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-05-31 21:42:02.557476 33.33% _pt_sl_t1 done after 0.11 minutes. Remaining 0.21 minutes.\r",
      "2020-05-31 21:42:02.588716 66.67% _pt_sl_t1 done after 0.11 minutes. Remaining 0.05 minutes.\r",
      "2020-05-31 21:42:02.666865 100.0% _pt_sl_t1 done after 0.11 minutes. Remaining 0.0 minutes.\n"
     ]
    }
   ],
   "source": [
    "df_mtx['volatility'] = rs.vol(df_mtx.close, span0 = 50) #one of our features, since we do not have a side\n",
    "\n",
    "df_mtx.dropna(inplace = True)\n",
    "\n",
    "vb = rs.vert_barrier(data = df_mtx.close, events = cs_event, period = 'days', freq = 5)\n",
    "\n",
    "# triple barrier events based on filter while data is also based on filtered index\n",
    "tb = rs.tri_barrier(data = df_mtx.close, \n",
    "                    events = cs_event, \n",
    "                    trgt = df_mtx['volatility'], \n",
    "                    min_req = 0.0002, \n",
    "                    num_threads = 3, \n",
    "                    ptSl= [2,2], #2x barriers\n",
    "                    t1 = vb, \n",
    "                    side = None)\n",
    "\n",
    "mlabel = rs.meta_label(data = df_mtx.close, \n",
    "                       events = tb, \n",
    "                       drop = 0.05) # because we do not have side, we need to drop rare labels\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 1.0    489\n",
       "-1.0    345\n",
       "Name: bin, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlabel['bin'].value_counts() #834"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ave Uniqueness of Observations 0.1201464235230114\n"
     ]
    }
   ],
   "source": [
    "X = df_mtx.reindex(mlabel.index)\n",
    "Z = tb.reindex(mlabel.index)\n",
    "y = mlabel['bin']\n",
    "\n",
    "idx_Mat0 = rs.mp_idx_matrix(data = X.close, events = Z)\n",
    "\n",
    "avgU = rs.av_unique(idx_Mat0).mean() #get ave uniqueness\n",
    "print(\"Ave Uniqueness of Observations\", avgU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rf_clf Mean CV score: -0.626236\n",
      "CV Variance: 0.011432\n"
     ]
    }
   ],
   "source": [
    "#based on book recommendation\n",
    "rf_clf = RandomForestClassifier(n_estimators = 1000,\n",
    "                                criterion = \"entropy\",\n",
    "                                max_samples=avgU, #note averge unique used\n",
    "                                bootstrap=True, \n",
    "                                n_jobs=1,\n",
    "                                random_state=42,\n",
    "                                class_weight=\"balanced_subsample\",\n",
    "                                oob_score=False) #use only one either OOB or CV\n",
    "\n",
    "cv_gen = KFold(n_splits=10, \n",
    "               #random_state=42, \n",
    "               shuffle=False)\n",
    "\n",
    "score = rs.cv_score(classifier = rf_clf,\n",
    "                     X = X,\n",
    "                     y = y,\n",
    "                     events = None,\n",
    "                     pct_embargo = .0,\n",
    "                     cv_gen = cv_gen,\n",
    "                     sample_weight = None,\n",
    "                     scoring = \"neg_log_loss\")\n",
    "\n",
    "print('rf_clf Mean CV score: {0:.6f}\\nCV Variance: {1:.6f}'.format(score.mean(), score.var()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rf_clf Mean CV score: -0.485730\n",
      "CV Variance: 0.002437\n"
     ]
    }
   ],
   "source": [
    "cv_gen0 = KFold(n_splits=10, \n",
    "                random_state=42, \n",
    "                shuffle=True) # shuffle is on!\n",
    "\n",
    "score = rs.cv_score(classifier = rf_clf,\n",
    "                     X = X,\n",
    "                     y = y,\n",
    "                     events = None,\n",
    "                     pct_embargo = .0,\n",
    "                     cv_gen = cv_gen0,\n",
    "                     sample_weight = None,\n",
    "                     scoring = \"neg_log_loss\")\n",
    "\n",
    "print('rf_clf Mean CV score: {0:.6f}\\nCV Variance: {1:.6f}'.format(score.mean(), score.var()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To shuffle or not.. That is the question\n",
    "\n",
    "Please note we are not using OOB score. All the scoring done previously was on OOB.\n",
    "\n",
    "**Based on KFold CV**\n",
    "    \n",
    "Shuffled:\n",
    "* Mean score: -0.485087\n",
    "* Variance: 0.002315\n",
    "\n",
    "Not Shuffled:\n",
    "* Mean score: -0.625331\n",
    "* CV Variance: 0.011186\n",
    "    \n",
    "**Based on Stratified KFold CV**\n",
    "    \n",
    "Shuffled:\n",
    "* Mean score: -0.485636\n",
    "* Variance: 0.001371\n",
    "\n",
    "Not Shuffled:\n",
    "* Mean score: -0.711593\n",
    "* CV Variance: 0.014060\n",
    "\n",
    "Based on the above, we can conclude that shuffle will give higher but more inflated score. But in both cases, shuffled datasets has lower variance.\n",
    "\n",
    "There is a presence of information leakage, kindly refer to [notebook 6.1](https://github.com/boyboi86/AFML/blob/master/AFML%206.1.ipynb).\n",
    "\n",
    "**Note**\n",
    "\n",
    "If your mean score is positive, please go back and review your own code. We are only using negative log loss.\n",
    "\n",
    "[sklearn.Metrics: log_loss](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html)\n",
    "\n",
    "[issue 9144](https://github.com/scikit-learn/scikit-learn/issues/9144)\n",
    "\n",
    "The issue stated /9144 about cross_val_score has not fixed yet (As of this writing).\n",
    "\n",
    "[negative_log_loss](https://stackoverflow.com/questions/43081251/sklearn-metrics-log-loss-is-positive-vs-scoring-neg-log-loss-is-negative)\n",
    "\n",
    "The utility value is correct just add negative sign will do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rf_clf Mean CV score: -0.651812\n",
      "CV Variance: 0.005689\n"
     ]
    }
   ],
   "source": [
    "#used in built PurgeKFold\n",
    "\n",
    "cv_gen = rs.PurgedKFold(n_splits = 10,\n",
    "                        events = Z, #included reindex tb\n",
    "                        pct_embargo = 0.01) #1% as a protection\n",
    "\n",
    "score = rs.cv_score(classifier = rf_clf,\n",
    "                     X = X,\n",
    "                     y = y,\n",
    "                     events = None,\n",
    "                     pct_embargo = None, # 1% of embargo\n",
    "                     cv_gen = cv_gen,\n",
    "                     sample_weight = None,\n",
    "                     scoring = \"neg_log_loss\",\n",
    "                     shuffle_after_split = False)\n",
    "\n",
    "print('rf_clf Mean CV score: {0:.6f}\\nCV Variance: {1:.6f}'.format(score.mean(), score.var()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rf_clf Mean CV score: -0.653224\n",
      "CV Variance: 0.006174\n"
     ]
    }
   ],
   "source": [
    "#shuffle after splitting using PurgedKFold\n",
    "\n",
    "score = rs.cv_score(classifier = rf_clf,\n",
    "                     X = X,\n",
    "                     y = y,\n",
    "                     events = None,\n",
    "                     pct_embargo = None, # 1% of embargo\n",
    "                     cv_gen = cv_gen,\n",
    "                     sample_weight = None,\n",
    "                     scoring = \"neg_log_loss\",\n",
    "                     shuffle_after_split = True) #added new features, not in the book\n",
    "\n",
    "print('rf_clf Mean CV score: {0:.6f}\\nCV Variance: {1:.6f}'.format(score.mean(), score.var()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Layer: Embargo\n",
    "\n",
    "Both Kfold method did not use shuffle.\n",
    "\n",
    "**Embargo:**\n",
    "\n",
    "* Mean CV score: -0.651812\n",
    "* CV Variance: 0.005689\n",
    "    \n",
    "**Without Embargo:**\n",
    "\n",
    "* Mean score: -0.625331\n",
    "* CV Variance: 0.011186\n",
    "    \n",
    "With embargo acting as an additional barrier to prevent leakage, the possiblity of leakage is further reduced.\n",
    "\n",
    "Since there will be an extra layer of seperation between training and testing sets.\n",
    "\n",
    "This is reflected in a lower mean score and variance (More relistic).\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "Model development is an iterative process from paper to reality, repeated testing and evetually built upon.\n",
    "\n",
    "No matter which method which you used to seperate train, test and validate samples (train_test_split, PurgedKFold, StratifiedKFold). \n",
    "\n",
    "Ultimately, as long as the 3 key samples do not cross-contaminate (leakage). Any method should be fine.\n",
    "\n",
    "For IID samples, having shuffled data will definitely provide that additional layer of randomness. \n",
    "\n",
    "For Non-IID however, consider splitting first then shuffle for trained data could be an alternative (if validation and test samples are not comtaminated). \n",
    "\n",
    "One of the data sample can be shuffled to add that layer of randomness. Preferably training set, for additional security consider using embargo.\n",
    "\n",
    "The below was tried with training set shuffled after split. No cross-contamination across test data/ validation \n",
    "\n",
    "**Embargo:**\n",
    "\n",
    "* Mean CV score: -0.653224\n",
    "* CV Variance: 0.006174\n",
    "\n",
    "Overall CV variance is slightly higher (Increased randomness), but mean CV score seems to be slightly lower (More realistic/ less bias).\n",
    "\n",
    "So from the above there is a variance vs bias trade-off.\n",
    "\n",
    "Given financial series are considered non-IID, a small change in sequential order of model development procedure should be used. To contain each datasets.\n",
    "\n",
    "With an additional layer of randomness, it may deter selection bias. (If you think about it, selection bias can only exist if you get to select. If one of the 3 samples are random, selection bias seems unlikely as random configurations are beyond control)\n",
    "\n",
    "**Note**\n",
    "\n",
    "This shuffle after split is not from the book, but something I hypothesized based on the question posted by Dr Marco.\n",
    "\n",
    "I believe if the datasets are not shuffled ML models will still be overfitted (with all precautions in place, but still marginally overfit). \n",
    "\n",
    "Therefore as long as the datasets do not suffer from leakage, some level of randomness should always be introduced.\n",
    "\n",
    "Update: In the later chapter, the author started to shuffle test set after split. So my guess is the correct answer should be to shuffle the test set to avoid selection bias."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}