AlphaWave Data Machine Learning to Predict Earnings for Stocks using Neural Networks example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning to Predict Earnings for Stocks: Neural Networks\n",
    "\n",
    "**Hugh Donnelly, CFA**<br> \n",
    "*AlphaWave Data*\n",
    "\n",
    "**September 2021**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "In this article, we are going to cover Neural Networks (NN). Let's begin by laying down the theoretical foundation of the algorithm.\n",
    "\n",
    "Jupyter Notebooks are available on [Google Colab](https://colab.research.google.com/drive/1_6X_AXuuRxOB1rwGZP_CZrbflDiEeJQQ?usp=sharing) and [Github](https://github.com/AlphaWaveData/Jupyter-Notebooks/blob/master/AlphaWave%20Data%20Machine%20Learning%20to%20Predict%20Earnings%20for%20Stocks%20using%20Neural%20Networks%20example.ipynb).\n",
    "\n",
    "For this project, we use several Python-based scientific computing technologies listed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import math\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have heard various buzz words associated with Neural Networks (NN) like Artificial Intelligence, [Machine Learning](https://hdonnelly6.medium.com/list/machine-learning-for-investing-7f2690bb1826), and Deep Learning.  For clarification, Deep Learning is a sub-field of Neural Networks which is a sub-field of Machine Learning and they all fall under the umbrella of Artificial Intelligence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>AI versus Machine Learning versus Neural Networks versus Deep Learning</h4>  \n",
    "<img src='ML Photos/1_NN_ML_Graph.PNG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural Networks (NN) is a prediction algorithm where you define a set of features to make predictions on a label.  These labels can be binary (e.g. Is this email spam?), multi-label classification (e.g. handwritten text), or [regression](https://hdonnelly6.medium.com/introduction-to-machine-learning-regression-fee4200132f0) (e.g. What is the price of an equity option?).  NN can also be used in adaptive control problems (e.g. autonomous driving).\n",
    "\n",
    "NN are ubiquitous.  Sophisticated market participants are using NN for alpha generation, hedging, and scenario analysis.  Technology companies are using NN to suggest which video you should watch next and what type of news you should be consuming.  So how can the NN algorithm be so flexible that it can be used to predict such a wide variety of labels?\n",
    "\n",
    "NN are loosely based on how the brain learns.  First, you set up a NN architecture that you believe is commiserate with the complexity of your problem.  The simplest NN architecture will include an input neuron layer, an output layer, and an activation function.  Next, you give the NN a set of features you believe are important when predicting an outcome.  The NN will then determine the relationship and patterns between each neuron to generate the most plausible outcome.  Before we use the NN to make predictions, we need to train the NN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Neural Network Equation</h4>  \n",
    "<img src='ML Photos/2_NN_Equation.PNG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Neural Network Layer Graph</h4>  \n",
    "<img src='ML Photos/2a_NN_Layers_Graph.PNG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the biggest misconceptions and misuses of NN is that people will feed more features or increase the complexity of the NN thinking this will allow the algorithm to detect more patterns and make better predictions.  This is not necessarily true as your NN is more likely to be overfit and therefore the predictions cannot be trusted.\n",
    "\n",
    "For NN to perform well, one must pre-process and normalize quality features, choose an appropriate architecture, and properly tune parameters.  Above all, domain knowledge is paramount.  There is no substitution for experience and intuition when working with data and making predictions.  Assuming one has domain experience and solid knowledge of how NN work, there are several advantages of using NN over other machine learning algorithms.\n",
    "\n",
    "NN can learn and adapt on their own which make them useful for real time applications.  NN also have fault tolerance meaning the algorithm will continue to operate if one or more neurons fail.  It also does a good job at handling nonlinear relationships.  NN can also handle multiple tasks in parallel lending itself to distributed computing.\n",
    "\n",
    "Let's look at how we can use NN to make binary predictions on earnings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initial Setup\n",
    "Now let's retrieve simulated quarterly fundamentals data over a ten year period for anonymized members of the S&P 500 from a saved pickle file for this analysis.  This pickle file contains more than 40 features that we will use to predict the direction of the next quarter's earnings based on the current quarter's fundamental data.\n",
    "\n",
    "If you wish, you can also use real financial data provided by [AlphaWave Data](https://www.alphawavedata.com/) in this analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Account Receivable Turnover</th>\n",
       "      <th>Current Ratio</th>\n",
       "      <th>Quick Ratio</th>\n",
       "      <th>Inventory Turnover</th>\n",
       "      <th>Total Debt To Equity</th>\n",
       "      <th>EBITDA Margin</th>\n",
       "      <th>ROA</th>\n",
       "      <th>ROE</th>\n",
       "      <th>...</th>\n",
       "      <th>Change in Equity to Fixed Assets</th>\n",
       "      <th>Change in Sales to Total Assets</th>\n",
       "      <th>Change in EBIT to revenue</th>\n",
       "      <th>Change in Profit margin</th>\n",
       "      <th>Change in Sales to Inventory</th>\n",
       "      <th>Change in Sales to Working capital</th>\n",
       "      <th>Change in R&amp;D to Revenue</th>\n",
       "      <th>Change in working cap to Assets</th>\n",
       "      <th>Change in Operating Income or Losses</th>\n",
       "      <th>Change in EBITDA Margin</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.394</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.745478</td>\n",
       "      <td>2.619692</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1282.212454</td>\n",
       "      <td>22.787274</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.644</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.995833</td>\n",
       "      <td>2.788106</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1277.191693</td>\n",
       "      <td>23.868634</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2.757370</td>\n",
       "      <td>6.592717</td>\n",
       "      <td>13.261200</td>\n",
       "      <td>11.661122</td>\n",
       "      <td>11.504201</td>\n",
       "      <td>-0.892583</td>\n",
       "      <td>56.254868</td>\n",
       "      <td>1.507437</td>\n",
       "      <td>20.728190</td>\n",
       "      <td>10.350641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.794</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.462660</td>\n",
       "      <td>3.048407</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1275.590018</td>\n",
       "      <td>26.162307</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>3.673544</td>\n",
       "      <td>-5.245019</td>\n",
       "      <td>21.163128</td>\n",
       "      <td>18.344738</td>\n",
       "      <td>-7.675397</td>\n",
       "      <td>-14.137102</td>\n",
       "      <td>0.616639</td>\n",
       "      <td>-1.588617</td>\n",
       "      <td>14.808099</td>\n",
       "      <td>19.895452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.854</td>\n",
       "      <td>-1.940000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.512796</td>\n",
       "      <td>2.160066</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1272.062872</td>\n",
       "      <td>18.660007</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-25.307362</td>\n",
       "      <td>8.587284</td>\n",
       "      <td>-66.485837</td>\n",
       "      <td>-127.380356</td>\n",
       "      <td>-9.344565</td>\n",
       "      <td>57.779330</td>\n",
       "      <td>16.129232</td>\n",
       "      <td>14.802637</td>\n",
       "      <td>-63.607881</td>\n",
       "      <td>-54.276757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.274</td>\n",
       "      <td>1.420000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.680017</td>\n",
       "      <td>2.374922</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1269.596817</td>\n",
       "      <td>22.370680</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>3.001286</td>\n",
       "      <td>3.372182</td>\n",
       "      <td>100.382200</td>\n",
       "      <td>-361.161166</td>\n",
       "      <td>12.828012</td>\n",
       "      <td>-4.124261</td>\n",
       "      <td>-32.412458</td>\n",
       "      <td>-2.905321</td>\n",
       "      <td>107.139453</td>\n",
       "      <td>58.713109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.914</td>\n",
       "      <td>0.306168</td>\n",
       "      <td>14.783556</td>\n",
       "      <td>2.844682</td>\n",
       "      <td>2.526065</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1399.524214</td>\n",
       "      <td>49.605136</td>\n",
       "      <td>4.393086</td>\n",
       "      <td>19.923301</td>\n",
       "      <td>...</td>\n",
       "      <td>-23.268399</td>\n",
       "      <td>4.423973</td>\n",
       "      <td>6.126118</td>\n",
       "      <td>69.289285</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-19.917751</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-8.792037</td>\n",
       "      <td>10.821108</td>\n",
       "      <td>6.053170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.784</td>\n",
       "      <td>-0.130000</td>\n",
       "      <td>14.864186</td>\n",
       "      <td>2.843043</td>\n",
       "      <td>2.631217</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1401.624076</td>\n",
       "      <td>53.525240</td>\n",
       "      <td>4.715484</td>\n",
       "      <td>20.690754</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.681214</td>\n",
       "      <td>18.362582</td>\n",
       "      <td>-15.160033</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.456669</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.223852</td>\n",
       "      <td>11.638151</td>\n",
       "      <td>10.519497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.634</td>\n",
       "      <td>-0.150000</td>\n",
       "      <td>15.180999</td>\n",
       "      <td>3.065134</td>\n",
       "      <td>2.847354</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1388.890037</td>\n",
       "      <td>53.701669</td>\n",
       "      <td>-0.439290</td>\n",
       "      <td>0.712332</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-4.431060</td>\n",
       "      <td>-2.815343</td>\n",
       "      <td>-24.099620</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-22.544284</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.832590</td>\n",
       "      <td>-7.121654</td>\n",
       "      <td>0.428377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.674</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>16.602697</td>\n",
       "      <td>3.221663</td>\n",
       "      <td>2.807657</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1385.848914</td>\n",
       "      <td>46.389200</td>\n",
       "      <td>4.210020</td>\n",
       "      <td>19.787080</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.264846</td>\n",
       "      <td>-26.519779</td>\n",
       "      <td>9.836592</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-20.725509</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-4.097929</td>\n",
       "      <td>-26.714389</td>\n",
       "      <td>-17.679338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>1.784</td>\n",
       "      <td>0.110000</td>\n",
       "      <td>16.641741</td>\n",
       "      <td>3.222968</td>\n",
       "      <td>2.735622</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1389.848624</td>\n",
       "      <td>41.653929</td>\n",
       "      <td>7.314623</td>\n",
       "      <td>27.510720</td>\n",
       "      <td>...</td>\n",
       "      <td>-11.098760</td>\n",
       "      <td>10.544785</td>\n",
       "      <td>-18.447261</td>\n",
       "      <td>-19.839339</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.637515</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.854259</td>\n",
       "      <td>-9.847699</td>\n",
       "      <td>-13.907141</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20200 rows × 52 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       EPS  change in EPS  Account Receivable Turnover  Current Ratio  \\\n",
       "ID                                                                      \n",
       "0    2.394            NaN                          NaN       3.745478   \n",
       "0    2.644       0.250000                          NaN       3.995833   \n",
       "0    2.794       0.150000                          NaN       4.462660   \n",
       "0    0.854      -1.940000                          NaN       3.512796   \n",
       "0    2.274       1.420000                          NaN       3.680017   \n",
       "..     ...            ...                          ...            ...   \n",
       "504  1.914       0.306168                    14.783556       2.844682   \n",
       "504  1.784      -0.130000                    14.864186       2.843043   \n",
       "504  1.634      -0.150000                    15.180999       3.065134   \n",
       "504  1.674       0.040000                    16.602697       3.221663   \n",
       "504  1.784       0.110000                    16.641741       3.222968   \n",
       "\n",
       "     Quick Ratio  Inventory Turnover  Total Debt To Equity  EBITDA Margin  \\\n",
       "ID                                                                          \n",
       "0       2.619692                 NaN           1282.212454      22.787274   \n",
       "0       2.788106                 NaN           1277.191693      23.868634   \n",
       "0       3.048407                 NaN           1275.590018      26.162307   \n",
       "0       2.160066                 NaN           1272.062872      18.660007   \n",
       "0       2.374922                 NaN           1269.596817      22.370680   \n",
       "..           ...                 ...                   ...            ...   \n",
       "504     2.526065                 NaN           1399.524214      49.605136   \n",
       "504     2.631217                 NaN           1401.624076      53.525240   \n",
       "504     2.847354                 NaN           1388.890037      53.701669   \n",
       "504     2.807657                 NaN           1385.848914      46.389200   \n",
       "504     2.735622                 NaN           1389.848624      41.653929   \n",
       "\n",
       "          ROA        ROE  ...  Change in Equity to Fixed Assets  \\\n",
       "ID                        ...                                     \n",
       "0         NaN        NaN  ...                               NaN   \n",
       "0         NaN        NaN  ...                          2.757370   \n",
       "0         NaN        NaN  ...                          3.673544   \n",
       "0         NaN        NaN  ...                        -25.307362   \n",
       "0         NaN        NaN  ...                          3.001286   \n",
       "..        ...        ...  ...                               ...   \n",
       "504  4.393086  19.923301  ...                        -23.268399   \n",
       "504  4.715484  20.690754  ...                               NaN   \n",
       "504 -0.439290   0.712332  ...                               NaN   \n",
       "504  4.210020  19.787080  ...                               NaN   \n",
       "504  7.314623  27.510720  ...                        -11.098760   \n",
       "\n",
       "     Change in Sales to Total Assets  Change in EBIT to revenue  \\\n",
       "ID                                                                \n",
       "0                                NaN                        NaN   \n",
       "0                           6.592717                  13.261200   \n",
       "0                          -5.245019                  21.163128   \n",
       "0                           8.587284                 -66.485837   \n",
       "0                           3.372182                 100.382200   \n",
       "..                               ...                        ...   \n",
       "504                         4.423973                   6.126118   \n",
       "504                        -5.681214                  18.362582   \n",
       "504                        -4.431060                  -2.815343   \n",
       "504                        -0.264846                 -26.519779   \n",
       "504                        10.544785                 -18.447261   \n",
       "\n",
       "     Change in Profit margin  Change in Sales to Inventory  \\\n",
       "ID                                                           \n",
       "0                        NaN                           NaN   \n",
       "0                  11.661122                     11.504201   \n",
       "0                  18.344738                     -7.675397   \n",
       "0                -127.380356                     -9.344565   \n",
       "0                -361.161166                     12.828012   \n",
       "..                       ...                           ...   \n",
       "504                69.289285                           NaN   \n",
       "504               -15.160033                           NaN   \n",
       "504               -24.099620                           NaN   \n",
       "504                 9.836592                           NaN   \n",
       "504               -19.839339                           NaN   \n",
       "\n",
       "     Change in Sales to Working capital  Change in R&D to Revenue  \\\n",
       "ID                                                                  \n",
       "0                                   NaN                       NaN   \n",
       "0                             -0.892583                 56.254868   \n",
       "0                            -14.137102                  0.616639   \n",
       "0                             57.779330                 16.129232   \n",
       "0                             -4.124261                -32.412458   \n",
       "..                                  ...                       ...   \n",
       "504                          -19.917751                       NaN   \n",
       "504                            5.456669                       NaN   \n",
       "504                          -22.544284                       NaN   \n",
       "504                          -20.725509                       NaN   \n",
       "504                            0.637515                       NaN   \n",
       "\n",
       "     Change in working cap to Assets  Change in Operating Income or Losses  \\\n",
       "ID                                                                           \n",
       "0                                NaN                                   NaN   \n",
       "0                           1.507437                             20.728190   \n",
       "0                          -1.588617                             14.808099   \n",
       "0                          14.802637                            -63.607881   \n",
       "0                          -2.905321                            107.139453   \n",
       "..                               ...                                   ...   \n",
       "504                        -8.792037                             10.821108   \n",
       "504                        -0.223852                             11.638151   \n",
       "504                        -0.832590                             -7.121654   \n",
       "504                        -4.097929                            -26.714389   \n",
       "504                         2.854259                             -9.847699   \n",
       "\n",
       "     Change in EBITDA Margin  \n",
       "ID                            \n",
       "0                        NaN  \n",
       "0                  10.350641  \n",
       "0                  19.895452  \n",
       "0                 -54.276757  \n",
       "0                  58.713109  \n",
       "..                       ...  \n",
       "504                 6.053170  \n",
       "504                10.519497  \n",
       "504                 0.428377  \n",
       "504               -17.679338  \n",
       "504               -13.907141  \n",
       "\n",
       "[20200 rows x 52 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load equity dataframe from the saved pickle file\n",
    "data = pd.read_pickle(\"./nn_data.pkl\")\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can put our data into the NN, we need to pre-process our data. Let's begin by outlining the steps we will take to make this prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Earnings movement prediction\n",
    "\n",
    "#### Forecast direction of next quarter earnings based on accounting information of the current quarter\n",
    "\n",
    "#### Steps:\n",
    "- Enhance data with additional information\n",
    "- Preprocess the data\n",
    "- Learn how to apply Neural Network algorithm on our dataset\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the first seven rows of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Account Receivable Turnover</th>\n",
       "      <th>Current Ratio</th>\n",
       "      <th>Quick Ratio</th>\n",
       "      <th>Inventory Turnover</th>\n",
       "      <th>Total Debt To Equity</th>\n",
       "      <th>EBITDA Margin</th>\n",
       "      <th>ROA</th>\n",
       "      <th>ROE</th>\n",
       "      <th>...</th>\n",
       "      <th>Change in Equity to Fixed Assets</th>\n",
       "      <th>Change in Sales to Total Assets</th>\n",
       "      <th>Change in EBIT to revenue</th>\n",
       "      <th>Change in Profit margin</th>\n",
       "      <th>Change in Sales to Inventory</th>\n",
       "      <th>Change in Sales to Working capital</th>\n",
       "      <th>Change in R&amp;D to Revenue</th>\n",
       "      <th>Change in working cap to Assets</th>\n",
       "      <th>Change in Operating Income or Losses</th>\n",
       "      <th>Change in EBITDA Margin</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.394</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.745478</td>\n",
       "      <td>2.619692</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1282.212454</td>\n",
       "      <td>22.787274</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.644</td>\n",
       "      <td>0.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.995833</td>\n",
       "      <td>2.788106</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1277.191693</td>\n",
       "      <td>23.868634</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2.757370</td>\n",
       "      <td>6.592717</td>\n",
       "      <td>13.261200</td>\n",
       "      <td>11.661122</td>\n",
       "      <td>11.504201</td>\n",
       "      <td>-0.892583</td>\n",
       "      <td>56.254868</td>\n",
       "      <td>1.507437</td>\n",
       "      <td>20.728190</td>\n",
       "      <td>10.350641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.794</td>\n",
       "      <td>0.15</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.462660</td>\n",
       "      <td>3.048407</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1275.590018</td>\n",
       "      <td>26.162307</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>3.673544</td>\n",
       "      <td>-5.245019</td>\n",
       "      <td>21.163128</td>\n",
       "      <td>18.344738</td>\n",
       "      <td>-7.675397</td>\n",
       "      <td>-14.137102</td>\n",
       "      <td>0.616639</td>\n",
       "      <td>-1.588617</td>\n",
       "      <td>14.808099</td>\n",
       "      <td>19.895452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.854</td>\n",
       "      <td>-1.94</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.512796</td>\n",
       "      <td>2.160066</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1272.062872</td>\n",
       "      <td>18.660007</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>-25.307362</td>\n",
       "      <td>8.587284</td>\n",
       "      <td>-66.485837</td>\n",
       "      <td>-127.380356</td>\n",
       "      <td>-9.344565</td>\n",
       "      <td>57.779330</td>\n",
       "      <td>16.129232</td>\n",
       "      <td>14.802637</td>\n",
       "      <td>-63.607881</td>\n",
       "      <td>-54.276757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.274</td>\n",
       "      <td>1.42</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.680017</td>\n",
       "      <td>2.374922</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1269.596817</td>\n",
       "      <td>22.370680</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>3.001286</td>\n",
       "      <td>3.372182</td>\n",
       "      <td>100.382200</td>\n",
       "      <td>-361.161166</td>\n",
       "      <td>12.828012</td>\n",
       "      <td>-4.124261</td>\n",
       "      <td>-32.412458</td>\n",
       "      <td>-2.905321</td>\n",
       "      <td>107.139453</td>\n",
       "      <td>58.713109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.574</td>\n",
       "      <td>0.30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.147761</td>\n",
       "      <td>2.557516</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1271.878524</td>\n",
       "      <td>27.391565</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0.764649</td>\n",
       "      <td>-3.881248</td>\n",
       "      <td>60.809354</td>\n",
       "      <td>33.878319</td>\n",
       "      <td>-13.313170</td>\n",
       "      <td>-15.590030</td>\n",
       "      <td>-1.029015</td>\n",
       "      <td>11.213908</td>\n",
       "      <td>54.567944</td>\n",
       "      <td>50.055277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.704</td>\n",
       "      <td>0.13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.010166</td>\n",
       "      <td>2.755451</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1269.347628</td>\n",
       "      <td>26.213858</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>4.291332</td>\n",
       "      <td>-6.001707</td>\n",
       "      <td>-8.553836</td>\n",
       "      <td>9.626472</td>\n",
       "      <td>10.275125</td>\n",
       "      <td>-5.016895</td>\n",
       "      <td>5.171649</td>\n",
       "      <td>-15.321621</td>\n",
       "      <td>-14.042167</td>\n",
       "      <td>-7.824481</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7 rows × 52 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      EPS  change in EPS  Account Receivable Turnover  Current Ratio  \\\n",
       "ID                                                                     \n",
       "0   2.394            NaN                          NaN       3.745478   \n",
       "0   2.644           0.25                          NaN       3.995833   \n",
       "0   2.794           0.15                          NaN       4.462660   \n",
       "0   0.854          -1.94                          NaN       3.512796   \n",
       "0   2.274           1.42                          NaN       3.680017   \n",
       "0   2.574           0.30                          NaN       4.147761   \n",
       "0   2.704           0.13                          NaN       4.010166   \n",
       "\n",
       "    Quick Ratio  Inventory Turnover  Total Debt To Equity  EBITDA Margin  ROA  \\\n",
       "ID                                                                              \n",
       "0      2.619692                 NaN           1282.212454      22.787274  NaN   \n",
       "0      2.788106                 NaN           1277.191693      23.868634  NaN   \n",
       "0      3.048407                 NaN           1275.590018      26.162307  NaN   \n",
       "0      2.160066                 NaN           1272.062872      18.660007  NaN   \n",
       "0      2.374922                 NaN           1269.596817      22.370680  NaN   \n",
       "0      2.557516                 NaN           1271.878524      27.391565  NaN   \n",
       "0      2.755451                 NaN           1269.347628      26.213858  NaN   \n",
       "\n",
       "    ROE  ...  Change in Equity to Fixed Assets  \\\n",
       "ID       ...                                     \n",
       "0   NaN  ...                               NaN   \n",
       "0   NaN  ...                          2.757370   \n",
       "0   NaN  ...                          3.673544   \n",
       "0   NaN  ...                        -25.307362   \n",
       "0   NaN  ...                          3.001286   \n",
       "0   NaN  ...                          0.764649   \n",
       "0   NaN  ...                          4.291332   \n",
       "\n",
       "    Change in Sales to Total Assets  Change in EBIT to revenue  \\\n",
       "ID                                                               \n",
       "0                               NaN                        NaN   \n",
       "0                          6.592717                  13.261200   \n",
       "0                         -5.245019                  21.163128   \n",
       "0                          8.587284                 -66.485837   \n",
       "0                          3.372182                 100.382200   \n",
       "0                         -3.881248                  60.809354   \n",
       "0                         -6.001707                  -8.553836   \n",
       "\n",
       "    Change in Profit margin  Change in Sales to Inventory  \\\n",
       "ID                                                          \n",
       "0                       NaN                           NaN   \n",
       "0                 11.661122                     11.504201   \n",
       "0                 18.344738                     -7.675397   \n",
       "0               -127.380356                     -9.344565   \n",
       "0               -361.161166                     12.828012   \n",
       "0                 33.878319                    -13.313170   \n",
       "0                  9.626472                     10.275125   \n",
       "\n",
       "    Change in Sales to Working capital  Change in R&D to Revenue  \\\n",
       "ID                                                                 \n",
       "0                                  NaN                       NaN   \n",
       "0                            -0.892583                 56.254868   \n",
       "0                           -14.137102                  0.616639   \n",
       "0                            57.779330                 16.129232   \n",
       "0                            -4.124261                -32.412458   \n",
       "0                           -15.590030                 -1.029015   \n",
       "0                            -5.016895                  5.171649   \n",
       "\n",
       "    Change in working cap to Assets  Change in Operating Income or Losses  \\\n",
       "ID                                                                          \n",
       "0                               NaN                                   NaN   \n",
       "0                          1.507437                             20.728190   \n",
       "0                         -1.588617                             14.808099   \n",
       "0                         14.802637                            -63.607881   \n",
       "0                         -2.905321                            107.139453   \n",
       "0                         11.213908                             54.567944   \n",
       "0                        -15.321621                            -14.042167   \n",
       "\n",
       "    Change in EBITDA Margin  \n",
       "ID                           \n",
       "0                       NaN  \n",
       "0                 10.350641  \n",
       "0                 19.895452  \n",
       "0                -54.276757  \n",
       "0                 58.713109  \n",
       "0                 50.055277  \n",
       "0                 -7.824481  \n",
       "\n",
       "[7 rows x 52 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by enriching our data with some additional columns. In a typical machine learning workflow, the majority of the effort is usually dedicated to data cleaning and data preparation. In order for us to run the NN successfully, we need to do a lot of the necessary work before we can actually feed the data into the model. To enhance the data, we follow the below steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Enhance data:\n",
    "- Change in Earnings per share : (Current Period EPS - Prior Period EPS)\n",
    "- Assign 1 to positive change in EPS and 0 to negative change\n",
    "- Shift data index by -1: we will be using current financial data to predict future change in earnings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create binary column of positive and negative earnings changes\n",
    "data['binary change'] = [1 if row['change in EPS'] > 0 else 0 for _,row in data.iterrows()]\n",
    "\n",
    "# Shift date index by -1 so we are predicting future changes: 1 or 0\n",
    "data['Future change'] = data['binary change'].shift(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Future change</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.394</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.644</td>\n",
       "      <td>0.25</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.794</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.854</td>\n",
       "      <td>-1.94</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.274</td>\n",
       "      <td>1.42</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.574</td>\n",
       "      <td>0.30</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      EPS  change in EPS  Future change\n",
       "ID                                     \n",
       "0   2.394            NaN            1.0\n",
       "0   2.644           0.25            1.0\n",
       "0   2.794           0.15            0.0\n",
       "0   0.854          -1.94            1.0\n",
       "0   2.274           1.42            1.0\n",
       "0   2.574           0.30            1.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Goal is to anticipate the sign of future earnings change from the financial data of the current quarter.\n",
    "# If the future earnings changes is + , we assign 1, otherwise 0, to Future change value of the current quarter\n",
    "data[['EPS','change in EPS','Future change']].head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using pandas describe function to examine our data, you can see there are a number of columns that have negative and positive infinity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPS</th>\n",
       "      <th>change in EPS</th>\n",
       "      <th>Account Receivable Turnover</th>\n",
       "      <th>Current Ratio</th>\n",
       "      <th>Quick Ratio</th>\n",
       "      <th>Inventory Turnover</th>\n",
       "      <th>Total Debt To Equity</th>\n",
       "      <th>EBITDA Margin</th>\n",
       "      <th>ROA</th>\n",
       "      <th>ROE</th>\n",
       "      <th>...</th>\n",
       "      <th>Change in EBIT to revenue</th>\n",
       "      <th>Change in Profit margin</th>\n",
       "      <th>Change in Sales to Inventory</th>\n",
       "      <th>Change in Sales to Working capital</th>\n",
       "      <th>Change in R&amp;D to Revenue</th>\n",
       "      <th>Change in working cap to Assets</th>\n",
       "      <th>Change in Operating Income or Losses</th>\n",
       "      <th>Change in EBITDA Margin</th>\n",
       "      <th>binary change</th>\n",
       "      <th>Future change</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>19404.00000</td>\n",
       "      <td>18904.000000</td>\n",
       "      <td>1.100700e+04</td>\n",
       "      <td>16477.000000</td>\n",
       "      <td>16475.000000</td>\n",
       "      <td>8737.000000</td>\n",
       "      <td>18739.000000</td>\n",
       "      <td>18047.000000</td>\n",
       "      <td>13282.000000</td>\n",
       "      <td>12508.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.833000e+04</td>\n",
       "      <td>1.910800e+04</td>\n",
       "      <td>12930.000000</td>\n",
       "      <td>1.604000e+04</td>\n",
       "      <td>5954.000000</td>\n",
       "      <td>15930.000000</td>\n",
       "      <td>1.897100e+04</td>\n",
       "      <td>17581.000000</td>\n",
       "      <td>20200.000000</td>\n",
       "      <td>20199.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.21570</td>\n",
       "      <td>0.026427</td>\n",
       "      <td>inf</td>\n",
       "      <td>3.158567</td>\n",
       "      <td>2.510007</td>\n",
       "      <td>20.707292</td>\n",
       "      <td>1433.474738</td>\n",
       "      <td>35.343390</td>\n",
       "      <td>4.086419</td>\n",
       "      <td>23.379214</td>\n",
       "      <td>...</td>\n",
       "      <td>inf</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.048570</td>\n",
       "      <td>inf</td>\n",
       "      <td>inf</td>\n",
       "      <td>-2.145640</td>\n",
       "      <td>inf</td>\n",
       "      <td>-5.566898</td>\n",
       "      <td>0.499802</td>\n",
       "      <td>0.499827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.52075</td>\n",
       "      <td>1.936901</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.548858</td>\n",
       "      <td>1.250579</td>\n",
       "      <td>25.147403</td>\n",
       "      <td>3678.680565</td>\n",
       "      <td>77.036993</td>\n",
       "      <td>7.077900</td>\n",
       "      <td>36.740955</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>112.057283</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>478.940556</td>\n",
       "      <td>NaN</td>\n",
       "      <td>861.516401</td>\n",
       "      <td>0.500012</td>\n",
       "      <td>0.500012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-37.79600</td>\n",
       "      <td>-46.530000</td>\n",
       "      <td>1.234000e+01</td>\n",
       "      <td>1.295010</td>\n",
       "      <td>1.251109</td>\n",
       "      <td>12.340000</td>\n",
       "      <td>1222.263535</td>\n",
       "      <td>-5571.530968</td>\n",
       "      <td>-459.673370</td>\n",
       "      <td>-606.632332</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.222895e+05</td>\n",
       "      <td>-inf</td>\n",
       "      <td>-1225.694465</td>\n",
       "      <td>-9.709747e+02</td>\n",
       "      <td>-100.000000</td>\n",
       "      <td>-18031.680774</td>\n",
       "      <td>-6.618496e+04</td>\n",
       "      <td>-102661.762624</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.56400</td>\n",
       "      <td>-0.190000</td>\n",
       "      <td>1.445305e+01</td>\n",
       "      <td>2.290449</td>\n",
       "      <td>1.806157</td>\n",
       "      <td>13.671417</td>\n",
       "      <td>1270.729889</td>\n",
       "      <td>26.360131</td>\n",
       "      <td>1.738212</td>\n",
       "      <td>14.048718</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.797601e+01</td>\n",
       "      <td>-3.124481e+01</td>\n",
       "      <td>-8.764146</td>\n",
       "      <td>-1.614403e+01</td>\n",
       "      <td>-6.673430</td>\n",
       "      <td>-9.133047</td>\n",
       "      <td>-2.219766e+01</td>\n",
       "      <td>-11.103207</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.93400</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>1.672261e+01</td>\n",
       "      <td>2.733112</td>\n",
       "      <td>2.158349</td>\n",
       "      <td>15.424271</td>\n",
       "      <td>1307.950369</td>\n",
       "      <td>34.270360</td>\n",
       "      <td>2.999536</td>\n",
       "      <td>17.972786</td>\n",
       "      <td>...</td>\n",
       "      <td>-8.867825e-01</td>\n",
       "      <td>-2.182519e+00</td>\n",
       "      <td>0.313884</td>\n",
       "      <td>8.187558e-02</td>\n",
       "      <td>-0.385609</td>\n",
       "      <td>-0.591811</td>\n",
       "      <td>-1.524765e+00</td>\n",
       "      <td>-0.022406</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.50400</td>\n",
       "      <td>0.240000</td>\n",
       "      <td>2.036282e+01</td>\n",
       "      <td>3.483186</td>\n",
       "      <td>2.733380</td>\n",
       "      <td>18.854123</td>\n",
       "      <td>1368.379941</td>\n",
       "      <td>46.431411</td>\n",
       "      <td>5.794245</td>\n",
       "      <td>24.998883</td>\n",
       "      <td>...</td>\n",
       "      <td>1.358098e+01</td>\n",
       "      <td>1.961293e+01</td>\n",
       "      <td>9.647077</td>\n",
       "      <td>1.937251e+01</td>\n",
       "      <td>6.781950</td>\n",
       "      <td>7.238105</td>\n",
       "      <td>1.703465e+01</td>\n",
       "      <td>10.406017</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>83.31400</td>\n",
       "      <td>56.758548</td>\n",
       "      <td>inf</td>\n",
       "      <td>24.116270</td>\n",
       "      <td>14.382505</td>\n",
       "      <td>1115.273333</td>\n",
       "      <td>482238.934464</td>\n",
       "      <td>1937.245011</td>\n",
       "      <td>71.886688</td>\n",
       "      <td>1474.380807</td>\n",
       "      <td>...</td>\n",
       "      <td>inf</td>\n",
       "      <td>inf</td>\n",
       "      <td>11920.297969</td>\n",
       "      <td>inf</td>\n",
       "      <td>inf</td>\n",
       "      <td>31651.342509</td>\n",
       "      <td>inf</td>\n",
       "      <td>29503.108808</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 54 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               EPS  change in EPS  Account Receivable Turnover  Current Ratio  \\\n",
       "count  19404.00000   18904.000000                 1.100700e+04   16477.000000   \n",
       "mean       2.21570       0.026427                          inf       3.158567   \n",
       "std        2.52075       1.936901                          NaN       1.548858   \n",
       "min      -37.79600     -46.530000                 1.234000e+01       1.295010   \n",
       "25%        1.56400      -0.190000                 1.445305e+01       2.290449   \n",
       "50%        1.93400       0.020000                 1.672261e+01       2.733112   \n",
       "75%        2.50400       0.240000                 2.036282e+01       3.483186   \n",
       "max       83.31400      56.758548                          inf      24.116270   \n",
       "\n",
       "        Quick Ratio  Inventory Turnover  Total Debt To Equity  EBITDA Margin  \\\n",
       "count  16475.000000         8737.000000          18739.000000   18047.000000   \n",
       "mean       2.510007           20.707292           1433.474738      35.343390   \n",
       "std        1.250579           25.147403           3678.680565      77.036993   \n",
       "min        1.251109           12.340000           1222.263535   -5571.530968   \n",
       "25%        1.806157           13.671417           1270.729889      26.360131   \n",
       "50%        2.158349           15.424271           1307.950369      34.270360   \n",
       "75%        2.733380           18.854123           1368.379941      46.431411   \n",
       "max       14.382505         1115.273333         482238.934464    1937.245011   \n",
       "\n",
       "                ROA           ROE  ...  Change in EBIT to revenue  \\\n",
       "count  13282.000000  12508.000000  ...               1.833000e+04   \n",
       "mean       4.086419     23.379214  ...                        inf   \n",
       "std        7.077900     36.740955  ...                        NaN   \n",
       "min     -459.673370   -606.632332  ...              -2.222895e+05   \n",
       "25%        1.738212     14.048718  ...              -1.797601e+01   \n",
       "50%        2.999536     17.972786  ...              -8.867825e-01   \n",
       "75%        5.794245     24.998883  ...               1.358098e+01   \n",
       "max       71.886688   1474.380807  ...                        inf   \n",
       "\n",
       "       Change in Profit margin  Change in Sales to Inventory  \\\n",
       "count             1.910800e+04                  12930.000000   \n",
       "mean                       NaN                      4.048570   \n",
       "std                        NaN                    112.057283   \n",
       "min                       -inf                  -1225.694465   \n",
       "25%              -3.124481e+01                     -8.764146   \n",
       "50%              -2.182519e+00                      0.313884   \n",
       "75%               1.961293e+01                      9.647077   \n",
       "max                        inf                  11920.297969   \n",
       "\n",
       "       Change in Sales to Working capital  Change in R&D to Revenue  \\\n",
       "count                        1.604000e+04               5954.000000   \n",
       "mean                                  inf                       inf   \n",
       "std                                   NaN                       NaN   \n",
       "min                         -9.709747e+02               -100.000000   \n",
       "25%                         -1.614403e+01                 -6.673430   \n",
       "50%                          8.187558e-02                 -0.385609   \n",
       "75%                          1.937251e+01                  6.781950   \n",
       "max                                   inf                       inf   \n",
       "\n",
       "       Change in working cap to Assets  Change in Operating Income or Losses  \\\n",
       "count                     15930.000000                          1.897100e+04   \n",
       "mean                         -2.145640                                   inf   \n",
       "std                         478.940556                                   NaN   \n",
       "min                      -18031.680774                         -6.618496e+04   \n",
       "25%                          -9.133047                         -2.219766e+01   \n",
       "50%                          -0.591811                         -1.524765e+00   \n",
       "75%                           7.238105                          1.703465e+01   \n",
       "max                       31651.342509                                   inf   \n",
       "\n",
       "       Change in EBITDA Margin  binary change  Future change  \n",
       "count             17581.000000   20200.000000   20199.000000  \n",
       "mean                 -5.566898       0.499802       0.499827  \n",
       "std                 861.516401       0.500012       0.500012  \n",
       "min             -102661.762624       0.000000       0.000000  \n",
       "25%                 -11.103207       0.000000       0.000000  \n",
       "50%                  -0.022406       0.000000       0.000000  \n",
       "75%                  10.406017       1.000000       1.000000  \n",
       "max               29503.108808       1.000000       1.000000  \n",
       "\n",
       "[8 rows x 54 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Examine data \n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will replace negative and positive infinity with NaN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace infinity with nan\n",
    "data = data.replace([np.inf, -np.inf], np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also drop the rows where the change in earnings per share is NaN. We do this because we are trying to predict the change in earnings, so rows with NaN, or missing values, would not be useful information in our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Drop rows where change in EPS is nan: they are no use to us \n",
    "data = data.dropna(subset = ['change in EPS', 'Future change'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are also going to drop three columns, EPS, change in EPS, and binary change. We no longer need these columns to continue examining the missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We no longer need these columns\n",
    "data = data.drop(columns = ['EPS','change in EPS','binary change'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see almost every column, other than future change, has some percentage of missing values and some columns have a substantial amount of missing values. We have to deal with these missing values before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percent of missing values per column:\n",
      " Account Receivable Turnover                43.4\n",
      "Current Ratio                              15.8\n",
      "Quick Ratio                                15.8\n",
      "Inventory Turnover                         54.2\n",
      "Total Debt To Equity                        4.1\n",
      "EBITDA Margin                               8.0\n",
      "ROA                                        30.3\n",
      "ROE                                        34.3\n",
      "Gross Profit Margin                        21.3\n",
      "Accounts Receivable Turnover               43.4\n",
      "Inventory to Sales                         42.0\n",
      "LT Debt to Total Equity                     3.8\n",
      "Sales to Total Assets                       0.3\n",
      "EBIT to revenue                             4.1\n",
      "Profit margin                               0.0\n",
      "Sales to Cash                               0.4\n",
      "Sales to Inventory                         32.2\n",
      "Sales to Working capital                   15.8\n",
      "Sales to Dep Fixed assets                  44.3\n",
      "Working capital to total Asset             15.8\n",
      "Operating Income to Total Assets            0.3\n",
      "Trailing 12M EBITDA Margin                  8.0\n",
      "Div as % of CF                              0.5\n",
      "change in Depreciation and Amortization     1.6\n",
      "change in Inventories                      32.2\n",
      "change in Inventory Turnover               56.0\n",
      "change in R&D Expense                      69.0\n",
      "change Total Assets                         0.4\n",
      "change in Long Term Debt                    5.3\n",
      "change in Short Term Debt                  16.0\n",
      "change in Revenue                           0.0\n",
      "change in Current Ratio                    15.9\n",
      "change in Quick Ratio                      15.9\n",
      "change in Tot Debt to Common Equity         8.5\n",
      "change in Gross Margin                     21.3\n",
      "Changes in Working Capital                 16.3\n",
      "Change in Inventory to Sales               53.6\n",
      "Change in Dep Amort Expense                 1.6\n",
      "Change in CAPAX to Assets                   4.0\n",
      "Change in LTD to Equity                    13.4\n",
      "Change in Equity to Fixed Assets           45.2\n",
      "Change in Sales to Total Assets             0.4\n",
      "Change in EBIT to revenue                   4.1\n",
      "Change in Profit margin                     0.0\n",
      "Change in Sales to Inventory               32.3\n",
      "Change in Sales to Working capital         16.0\n",
      "Change in R&D to Revenue                   69.0\n",
      "Change in working cap to Assets            16.5\n",
      "Change in Operating Income or Losses        0.5\n",
      "Change in EBITDA Margin                     8.0\n",
      "Future change                               0.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Examine missing data\n",
    "missing_column_data = 100*(data.isnull().sum() / data.shape[0]).round(3)\n",
    "print('Percent of missing values per column:\\n', missing_column_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Real world data often has missing values which require careful attention. The handling of missing values is very important during the preprocessing step because many machine learning algorithms do not work with missing data. There are two general ways of thinking about how to handle missing data. One way is to delete the rows with the missing data, but we risk losing valuable information doing this. The alternative is to try to compute the missing values using an array of different methods like mean or median imputation, neural networks, or Multiple Imputation by Chained Equations (MICE).\n",
    "\n",
    "In this exercise, we will drop columns that have more than 35% of data missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Account Receivable Turnover         43.4\n",
       "Inventory Turnover                  54.2\n",
       "Accounts Receivable Turnover        43.4\n",
       "Inventory to Sales                  42.0\n",
       "Sales to Dep Fixed assets           44.3\n",
       "change in Inventory Turnover        56.0\n",
       "change in R&D Expense               69.0\n",
       "Change in Inventory to Sales        53.6\n",
       "Change in Equity to Fixed Assets    45.2\n",
       "Change in R&D to Revenue            69.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop 10 columns that have more than 35% of data missing\n",
    "columns_to_drop = missing_column_data[missing_column_data > 35]\n",
    "columns_to_drop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will result in us dropping ten columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Dataframe shape : (18903, 41)\n"
     ]
    }
   ],
   "source": [
    "# Number of columns dropped, 10\n",
    "data = data.drop(columns = list(columns_to_drop.index))\n",
    "print( f'New Dataframe shape : {data.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's continue with preprocessing our data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess data:\n",
    "- Handle remaining missing values\n",
    "- Minimize influence of outliers by performing Winsorization\n",
    "- Standardize data \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Handle remaining missing data by replacing NaN by mean of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep in mind that this is a naive way to handle missing values. \n",
    "# This method can cause data leakage and does not factor the covariance between features.\n",
    "# For more robust methods, take a look at MICE or KNN\n",
    "\n",
    "for col in data.columns:\n",
    "    data[col].fillna(data[col].mean(), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percent of missing values per column:\n",
      " Current Ratio                              0.0\n",
      "Quick Ratio                                0.0\n",
      "Total Debt To Equity                       0.0\n",
      "EBITDA Margin                              0.0\n",
      "ROA                                        0.0\n",
      "ROE                                        0.0\n",
      "Gross Profit Margin                        0.0\n",
      "LT Debt to Total Equity                    0.0\n",
      "Sales to Total Assets                      0.0\n",
      "EBIT to revenue                            0.0\n",
      "Profit margin                              0.0\n",
      "Sales to Cash                              0.0\n",
      "Sales to Inventory                         0.0\n",
      "Sales to Working capital                   0.0\n",
      "Working capital to total Asset             0.0\n",
      "Operating Income to Total Assets           0.0\n",
      "Trailing 12M EBITDA Margin                 0.0\n",
      "Div as % of CF                             0.0\n",
      "change in Depreciation and Amortization    0.0\n",
      "change in Inventories                      0.0\n",
      "change Total Assets                        0.0\n",
      "change in Long Term Debt                   0.0\n",
      "change in Short Term Debt                  0.0\n",
      "change in Revenue                          0.0\n",
      "change in Current Ratio                    0.0\n",
      "change in Quick Ratio                      0.0\n",
      "change in Tot Debt to Common Equity        0.0\n",
      "change in Gross Margin                     0.0\n",
      "Changes in Working Capital                 0.0\n",
      "Change in Dep Amort Expense                0.0\n",
      "Change in CAPAX to Assets                  0.0\n",
      "Change in LTD to Equity                    0.0\n",
      "Change in Sales to Total Assets            0.0\n",
      "Change in EBIT to revenue                  0.0\n",
      "Change in Profit margin                    0.0\n",
      "Change in Sales to Inventory               0.0\n",
      "Change in Sales to Working capital         0.0\n",
      "Change in working cap to Assets            0.0\n",
      "Change in Operating Income or Losses       0.0\n",
      "Change in EBITDA Margin                    0.0\n",
      "Future change                              0.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Check for missing values\n",
    "missing_column_data = 100*(data.isnull().sum()/ data.shape[0]).round(3)\n",
    "print('Percent of missing values per column:\\n',missing_column_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we proceed further, we need to split the data into train and test. Splitting data into train and test is absolutely necessary in machine learning to avoid overfitting. It allows us to see how good our model really is and how well it performs on the new data we feed it. We train the model on the training data and then make a prediction using the model that we learned in the training phase. The prediction is made on the unlabeled test data.\n",
    "\n",
    "Here we split the data into train and test by allocating 80% of the data to train and 20% of the data to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we need to split our data into train and test. \n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Independent values/features\n",
    "X = data.iloc[:,:-1].values\n",
    "# Dependent values\n",
    "y = data.iloc[:,-1].values\n",
    "\n",
    "# Create test and train data sets, split data randomly into 20% test and 80% train\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to winsorize the data to limit the influence of the extreme values, typically by setting all outliers to a specified percentile of data. Notice how we are winsorizing train data and test data separately. If you winsorize all of your data together first and then partition it later into training and testing afterwards, you are allowing future data (i.e. test data) to influence your cutoff values. Since you won't know what the future is when you use your model, you cannot use data manipulation affected by your future test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import mstats\n",
    "# Winsorize top 1% and bottom 1% of points\n",
    "\n",
    "# Apply on X_train and X_test separately\n",
    "X_train = mstats.winsorize(X_train, limits = [0.01, 0.01])\n",
    "X_test = mstats.winsorize(X_test, limits = [0.01, 0.01])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one last thing that we have to do before we train the algorithm and that is to standardize the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$z=(x-mean) /  Standard Deviation$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standardization of a dataset is a common requirement for many machine learning estimators.  The reason for this is that these algorithms may not behave well if the individual features are not standard normally distributed data (e.g. Gaussian with 0 mean and unit variance).  This means there should be a mean of zero and unit variance.\n",
    "\n",
    "For instance many elements used in the objective function of a machine learning algorithm (such as the RBF kernel of Support-vector Machines (SVM) or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order.  If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPORTANT: During testing, it is important to construct the test feature vectors using the means and standard deviations saved from\n",
    "# the training data, rather than computing it from the test data. You must scale your test inputs using the saved means\n",
    "# and standard deviations, prior to sending them to your Neural Networks library for classification.\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "sc = StandardScaler()\n",
    "\n",
    "# Fit to training data and then transform it\n",
    "X_train = sc.fit_transform(X_train)\n",
    "# Perform standardization on testing data using mu and sigma from training data\n",
    "X_test = sc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a deeper dive into how NN work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Artificial Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most basic unit of a NN is called a perceptron.  A perceptron is a single layer NN used for binary classification.  It has an input layer, a bias, an activation function, and an output layer.  The input layer is made of neurons and each neuron has a weight.  NN that have more than three layers of neurons, including the input and output layers, are considered deep NN or deep learning.\n",
    "\n",
    "So how do we use a NN?  First, data is fed into the NN.  Input data is multiplied by the neuron's weight and is summed.  A bias is then added to the sum and that value is sent to the activation function.  The results from the activation function will determine your binary output.\n",
    "\n",
    "When we are training our NN, we are trying to determine optimal weights for each neuron and bias.  As you can see, the algorithm is computationally efficient since we are performing simple vector multiplication.  The flexibility, simplicity, and scalability of NN is the reason why it is the most active area of research in machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perceptron is a single layer neural network and a multi-layer perceptron is called Neural Networks.\n",
    "\n",
    "* First it sums values of each input x multiplied by weight w\n",
    "* Weighted sum is passed through an activation function \n",
    "* Activation function \"converts\" output to binary output of 0 or 1\n",
    "* Weights are a measure of influence that each input has on the final output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/perceptron.JPG'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is an Activation Function?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The activation function introduces non-linearity into a NN.  This is the key to turning a linear combination of inputs from neurons to a non-trivial output.  In a binary classification problem, we want an activation function that will act as a switch.  Given our inputs, will our activation function output an off or on result?  Depending on the type of problem, whether binary classification or regression, we need to choose an appropriate activation function.  Below are two commonly used activation functions for binary classification, a sigmoid function and a tangent function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid function\n",
    "* Activation function has \"switch on\" and \"switch off\" characteristic\n",
    "* Moves from 0 to 1 depending on the input values of x\n",
    "* Activation function adds non-linearity to the network\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyU9b328c93shMSEMJOICCbYEVtWFzaqmhd6tZNsdpqtXraU221Pj2PXY6tnqfn9LTHbke7eJS6oAIutZyWKlq1dWMVVHYjEBIQCARIIHvm+/wxAx1jgIC5c89krvfrlddsv5n7Ckzmmns3d0dERNJXJOwAIiISLhWBiEiaUxGIiKQ5FYGISJpTEYiIpDkVgYhImlMRSNIzsyvNbH6yTdfMXjKzrxzkMTOz35vZLjNbFFzKdqf9FzO7uiunKanNtB+BJAMzOx34CTABaAVWAze7++JQgx2Cmb0EzHT3+9p57GPAY8BYd98XYIYfAqPc/aqgpiHdX2bYAUTMrBD4E/A1YA6QDXwMaAwz14c0HNgYZAmIdBYtGpJkMAbA3R9z91Z3r3f3+e7+FoCZXWNmr+wfbGafNLO1ZrbHzH5tZn/bv4gmPvZVM/u5me02s/Vmdmr8/goz25642MTMepnZQ2ZWZWblZvZ9M4scZLrnmNma+HTvBqy9X8bMrgPuA04xs71mdkfb14qPczMbFb/+gJndY2Z/NrNaM1toZscmjJ1gZs+ZWbWZbTOz75rZecB3gcvj03kzPvbAIiszi8R/p/L47/6QmfWKP1YSz3C1mW0ysx1m9r2j/l+UlKUikGSwDmg1swfN7HwzO+ZgA82sCHgC+A7QF1gLnNpm2BTgrfjjjwKzgEnAKOAq4G4z6xkf+99AL2Ak8AngS8CXDzLdJ4HvA0XAu8Bp7WV09/uBrwKvu3tPd//B4f4B4q4A7gCOAcqAH8WnXQA8DzwDDI7/Hn9192eAfwdmx6czsZ3XvCb+c2b8d+wJ3N1mzOnAWGAacLuZHdfBvNJNqAgkdO5eQ+zDyIH/AarMbK6ZDWhn+AXASnd/yt1bgF8BW9uM2eDuv3f3VmA2UAzc6e6N7j4faAJGmVkGcDnwHXevdfeNwF3AFw8y3VXu/oS7NwO/aGe6H9ZT7r4o/ns9ApwYv/9CYKu73+XuDfGsCzv4mlcCP3P39e6+l1iBTjezxMXCd8Tnwt4E3gTaKxTpxlQEkhTcfbW7X+PuQ4HjiX3z/UU7QwcDFQnPc6CyzZhtCdfr4+Pa3teT2Df7bKA84bFyYEgHp1vRzrgPI7FY6uIZIVZk7x7law7mg79fJpBYsgebrqQJFYEkHXdfAzxArBDaeg8Yuv+GmVni7SO0A2gmtmJ3v2HA5oNMt7jNdIvbGXcw+4AeCc8feATPrQCOPchjh9vsbwsf/P1aeH9ZSppTEUjozGycmd1qZkPjt4uJLS9f0M7wPwMfMbNL44s3vg4cyYfqAfFFR3OAH5lZgZkNB74FzDzIdCeY2Wfi0/3GEU73zfjzTzSzXOCHR/DcPwEDzexmM8uJZ50Sf2wbULJ/BXc7HgNuMbMR8fUi+9cptBzB9KWbUxFIMqgltoJ3oZntI1YAK4Bb2w509x3A54ntc7ATGA8s4eg3Nb2J2Lf19cArxFYuzzjEdH8cn+5o4NWOTsTd1wF3Elvp+058Wh19bi1wDnARscU47xBb+QvwePxyp5m90c7TZwAPA38HNgANxH5nkQO0Q5mktPg34UrgSnd/Mew8IqlIcwSScszsXDPrbWY5xLajN9pfjCQiHaAikFR0CrGtaHYQW1xyqbvXhxtJJHVp0ZCISJrTHIGISJpLuYPOFRUVeUlJSdgxRERSytKlS3e4e7/2Hku5IigpKWHJkiVhxxARSSlmVn6wx7RoSEQkzakIRETSnIpARCTNqQhERNKcikBEJM0FVgRmNiN+arwVB3nczOxXZlZmZm+Z2clBZRERkYMLco7gAeC8Qzx+PrEjOI4GbgB+E2AWERE5iMD2I3D3v5tZySGGXAI8FD/T04L4QcQGuft7QWUSETkUd6e51WlqjdLY3EpTa5SmlvhPa5TWqH/wx52WqBONxi4PN8bdiXrsjELujjtE3XHil554Pzix8bgz7bgBTCzu3em/d5g7lA3h/af6q4zf94EiMLMbiM01MGzYsC4JJyKpob6plV11TVTva2J3XTPVdU3U1Dezr7GFfU2t7Gtsoa6phb2Nseux+1uoa2ylsSVKY0vscv+HfbIefs0MBvTK7XZFYO3c1+5/gbvfC9wLUFpamqT/TSLSmRqaW6mormPLnga27qln655GttbUs3VPA9tqGtlV18SuuiYamqOHfJ387Ax65GTSMyeT/JwMemRn0r8gl7y+GeRmZpCTFSE7I0JOZuwnO/6Tk5kRu54RIScrQmYkQmbEyMgwMszIjBiRSJtLMzIzEq5HIkQiHLjMsNj9ZmDxy4gZBgn3g2FE4mP2XwYpzCKo5P3nfB1K7PyqIpJGttc0sHJLDWXb97Jh5z427oj9vFfT8IFv50U9sxnYK5dBvXIZP7iQPvnZ9O6RRZ8e2fTukU2f/GyO6ZFFr7ws8nMyycvKIBIJ9kO0OwizCOYCN5rZLGKnKdyj9QMi3duW3fUs27SblVv2sHJLDSu31LBj7z/OMtq7RxYlffOZMrIvJX3zKSnqwZDeeQwozGVAYS7ZmdriPQiBFYGZPQacARSZWSXwAyALwN1/C8wDLgDKgDrgy0FlEZFwvLennpfX7WDhhmoWbthJ5a7Y+YMyI8boAQWcMbYfEwYXMmFwL8YM6EnvHtkhJ05PQW41dMVhHnfg60FNX0S6XjTqLK/czQurt/PXNdtZ/V4NAH3ys5lc0odrTxtBackxjB1YQE5mRshpZb+UOwy1iCSftVtreXr5ZuYu38Lm3fVkRIyPDj+G284fxxlj+zF2QEHgKzzl6KkIROSo1De18vTyzTz8ejmr3qshI2KcPqqIWz85hmnjBtCrR1bYEaWDVAQickQqqut4eEE5sxdXsKe+mXEDC/jhReP51AmD6VeQE3Y8OQoqAhHpkM2767n7hXd4fEklDpw3YSBXn1rCpJJjtNgnxakIROSQttc0cPeLZcxaFDsQwBemDOOrnziWwb3zQk4mnUVFICLtammN8uDr5fz8uXU0NLfy+dKh3HjWaIaoALodFYGIfMDS8mq+94cVrNlayyfG9OOOiydQUpQfdiwJiIpARA5oaG7lx39ZwwOvbWRQr1x+e9XJnDthoNYBdHMqAhEB4J1ttdz02DLWbK3lmlNL+Pa5Y8nP0UdEOtD/skiac3dmLa7gjv9dSX52Jr//8iTOHNs/7FjShVQEImmsqSXK9/7wNo8vreRjo4u467KJ9C/IDTuWdDEVgUia2lPXzFdnLuX19Tv5xrTR3DxttA7ZnKZUBCJpaNPOOr78wCI2Vdfxs8sm8pmTh4YdSUKkIhBJMys27+FLMxYRdWfmdVOYMrJv2JEkZCoCkTSycsserrp/IfnZmTx83WRG9usZdiRJAioCkTSxaksNV963kB5ZGTx2/VSG9e0RdiRJEjrvm0gaiJXAAvKyMnjsBpWAvJ+KQKSb27hjH1fdv5DcrAxm3TCV4X11qAh5PxWBSDe2u66Jax9YjLvz6PUqAWmf1hGIdFNNLVG+OnMplbvqmfmVKYzQQePkIFQEIt2Qu/Odp95mwfpqfn75RCaP6BN2JEliWjQk0g39+qV3efKNSr45bTSfPkk7i8mhqQhEuplXy3bwX/PXcvHEwdx89uiw40gKUBGIdCNVtY3cPHs5I4vy+fFnP6LzCEiHaB2BSDcRjTrfmrOcmvpmHrp2Mj2y9ectHaM5ApFu4rd/f5eX39nBDy6awHGDCsOOIylERSDSDSwtr+au+ev41AmDuGJycdhxJMWoCERSXH1TK7fMfpMhvfP4j89ovYAcOS1EFElxd81fy6bqOmbdMJXC3Kyw40gK0hyBSApbtmkXM17dwJVThjFV5xWQo6QiEElRTS1R/u+TbzGgMJfbzh8XdhxJYYEWgZmdZ2ZrzazMzG5r5/FhZvaimS0zs7fM7IIg84h0J79+qYx12/byo08fT4EWCcmHEFgRmFkGcA9wPjAeuMLMxrcZ9n1gjrufBEwHfh1UHpHuZO3WWu55sYxLTxzMWeMGhB1HUlyQcwSTgTJ3X+/uTcAs4JI2YxzYv8FzL2BLgHlEugV351//uIKeOZncftGEsONINxBkEQwBKhJuV8bvS/RD4CozqwTmATe190JmdoOZLTGzJVVVVUFkFUkZf1mxlUUbqrn1k2Ppk58ddhzpBoIsgvY2ZvY2t68AHnD3ocAFwMNm9oFM7n6vu5e6e2m/fv0CiCqSGhqaW/n3easZN7CA6ZO045h0jiCLoBJIfKcO5YOLfq4D5gC4++tALlAUYCaRlHb/Kxuo3FXP7ReOJzNDG/1J5wjynbQYGG1mI8wsm9jK4LltxmwCpgGY2XHEikDLfkTasa2mgXteLOPcCQM4dZS+L0nnCawI3L0FuBF4FlhNbOuglWZ2p5ldHB92K3C9mb0JPAZc4+5tFx+JCPCTZ9bS0up874K2G9+JfDiBHmLC3ecRWwmceN/tCddXAacFmUGkO3i7cg9PvlHJVz9xLMP69gg7jnQzWsgokgJ+8uwa+uRn8/Uzjw07inRDKgKRJLdw/U5efmcHX/vEsdqDWAKhIhBJYu7OXfPX0b8gh6umDg87jnRTKgKRJPZK2Q4WbazmxrNGkZedEXYc6aZUBCJJyt35r2fXMqR3Hpdr5zEJkIpAJEk9v3o7b1bu4ZvTRpOTqbkBCY6KQCQJRaPOXfPXMqIon8+c3PYQXSKdS0UgkoTmr9rGmq213Hz2aB1KQgKnd5hIknF3fvNSGcP79uBTHxkUdhxJAyoCkSTz2rs7ebNyD//08WM1NyBdQu8ykSTzm5fepV9BjtYNSJdREYgkkbcqd/NK2Q6+cvoIcrO0pZB0DRWBSBL59YvvUpibyRemDAs7iqQRFYFIkijbvpdnV23lS6eU6JhC0qVUBCJJ4nd/e5fsjAjXnFYSdhRJMyoCkSSwraaBp5dv5vJJxRT1zAk7jqQZFYFIEpi5oJyWqHPd6SPCjiJpSEUgErKG5lYeWbiJaeMGMLxvfthxJA2pCERC9sflm6ne18S1p5eEHUXSlIpAJETuzoxXNjJuYAGnjOwbdhxJUyoCkRC9/u5O1m6r5drTR2BmYceRNKUiEAnRjFc30Dc/m4snDg47iqQxFYFISDbs2Mdf12znyinDdDgJCZWKQCQkD762kcyI6aT0EjoVgUgI9ja28MTSSi48YTD9C3PDjiNpTkUgEoKnl21mb2MLXzxFcwMSPhWBSBdzd2YuKGf8oEJOKu4ddhwRFYFIV1tavos1W2v54inDtcmoJAUVgUgXm7mgnIKcTC45UZuMSnJQEYh0oZ17G5n39lY++9Gh9MjODDuOCBBwEZjZeWa21szKzOy2g4y5zMxWmdlKM3s0yDwiYZuzpJKm1ihX6gxkkkQC+0piZhnAPcA5QCWw2MzmuvuqhDGjge8Ap7n7LjPrH1QekbC1Rp1HF5UzdWQfRg8oCDuOyAFBzhFMBsrcfb27NwGzgEvajLkeuMfddwG4+/YA84iE6u/rqqiortcOZJJ0giyCIUBFwu3K+H2JxgBjzOxVM1tgZue190JmdoOZLTGzJVVVVQHFFQnWIwvLKeqZwyfHDww7isj7BFkE7W0X521uZwKjgTOAK4D7zOwDG1a7+73uXurupf369ev0oCJB27qngRfWbOey0qFkZ2obDUkuQb4jK4HihNtDgS3tjPmjuze7+wZgLbFiEOlWnnyjkqjDZaXFhx8s0sWCLILFwGgzG2Fm2cB0YG6bMU8DZwKYWRGxRUXrA8wk0uWiUWf24gpOGdmXkiKdilKST2BF4O4twI3As8BqYI67rzSzO83s4viwZ4GdZrYKeBH4trvvDCqTSBgWrN/Jpuo6pk/W3IAkp0D3aHH3ecC8NvfdnnDdgW/Ff0S6pVmLK+iVl8W5E7SSWJKT1lqJBGjXviaeWbGVT580RCefkaSlIhAJ0NPLN9PUGuXySVosJMlLRSASEHdn1qIKJhb35rhBhWHHETkoFYFIQJZX7Gbttlqma25AkpyKQCQgsxdX0CM7g4sm6nDTktxUBCIB2NvYwtw3t3DhCYPomaPDTUtyUxGIBODPb22hrqmVyyfpcNOS/FQEIgGYtbiC0f17cvIwnZNYkp+KQKSTrd1ay7JNu7l8UrHOSSwpoUMLL+MnjDkNGAzUAyuAJe4eDTCbSEqavbiCrAzjMycPDTuKSIccsgjM7EzgNqAPsAzYDuQClwLHmtkTwF3uXhN0UJFU0NjSylPLKvnkhIH0yc8OO45IhxxujuAC4Hp339T2ATPLBC4kdirKJwPIJpJy5q/cxu66Zu07ICnlkEXg7t8+xGMtxA4jLSJxsxdXMKR3HqcdWxR2FJEO69DKYjN72Mx6JdwuMbO/BhdLJPVs2lnHK2U7uHxSMZGIVhJL6ujoVkOvAAvN7AIzux6YD/wiuFgiqWfOkgoiBp8v1UpiSS0d2mrI3X9nZiuJnTxmB3CSu28NNJlICmlpjfL40grOGNufQb3ywo4jckQ6umjoi8AM4EvAA8A8M5sYYC6RlPK3dVVsq2nU4aYlJXX0ICifBU539+3AY2b2B2KFcFJQwURSyazFFRT1zOGscf3DjiJyxDo0R+Dul8ZLYP/tRcCUwFKJpJDtNQ28sGY7n/voULIytLO+pJ5DvmvN7Ptm1qe9x9y9yczOMrMLg4kmkhqeeKOS1qhrsZCkrMMtGnob+F8zawDeAKqI7Vk8GjgReB7490ATiiQxd2f24gqmjOjDiKL8sOOIHJXDzcd+zt1PA54FVgIZQA0wE5js7re4e1XAGUWS1oL11ZTvrGP6ZM0NSOo63BzBR81sOHAlcGabx/KIHYBOJG3NXryJgtxMzj9+UNhRRI7a4Yrgt8AzwEhgScL9Bnj8fpG0tLuuiXkrtjJ9UjG5WRlhxxE5aodcNOTuv3L344AZ7j4y4WeEu6sEJK099cZmmlqiWkksKa+jm49+LeggIqnE3Xls0SYmFvdmwuBeh3+CSBLTRs8iR2Fp+S7e2b6XKyfrnMSS+lQEIkfh0YWb6JmTyYUTtZJYUp+KQOQI7a5r4k9vv8elJw2mR3ZHj9IikrxUBCJH6A/LYiuJvzB5eNhRRDqFikDkCLg7jy6MrSQeP7gw7DginSLQIjCz88xsrZmVmdlthxj3OTNzMysNMo/Ih7V/JfEXtCexdCOBFYGZZQD3AOcD44ErzGx8O+MKgG8AC4PKItJZHl0UW0l80cTBYUcR6TRBzhFMBsrcfb27NwGzgEvaGfdvwE+AhgCziHxoe+qa+fNbWkks3U+QRTAEqEi4XRm/7wAzOwkodvc/HeqFzOwGM1tiZkuqqnSMOwnHU8sqaWyJcoX2HZBuJsgisHbu8wMPmkWAnwO3Hu6F3P1edy9199J+/fp1YkSRjtGexNKdBVkElUDiGrWhwJaE2wXA8cBLZrYRmArM1QpjSUZLy3exbptWEkv3FGQRLAZGm9kIM8sGpgNz9z/o7nvcvcjdS9y9BFgAXOzuS9p/OZHw7F9JfOEJWkks3U9gReDuLcCNxE5qsxqY4+4rzexOM7s4qOmKdLZd+5oOrCTOz9FKYul+An1Xu/s8YF6b+24/yNgzgswicrRmLa6gsSXKF6eWhB1FJBDas1jkEFqjzswF5Zwysi9jBxaEHUckECoCkUP46+ptbN5dz9Wn6rhC0n2pCEQO4cHXNzK4Vy5nHzcg7CgigVERiBxE2fZaXi3byZVTh5OZoT8V6b707hY5iAdfKyc7M8J0nZNYujkVgUg7ahqaefKNSi46YTB9e+aEHUckUCoCkXY8tbSSuqZWrSSWtKAiEGmjNeo88NpGTizuzQlDe4cdRyRwKgKRNp5btY2NO+u4/mMjw44i0iVUBCJt/M/L6ynuk8e5E7TJqKQHFYFIgqXlu1havovrThuhTUYlbeidLpLgvpfX0ysvi8+XapNRSR8qApG48p37eGblVq6cMkxHGZW0oiIQibv/lQ1kRoxrTi0JO4pIl1IRiBA758CcJRVceuIQ+hfmhh1HpEupCESAhxeU09Ac5fqPa5NRST8qAkl7extbuP+VDUwb158xA3TOAUk/KgJJew+9vpE99c18Y9rosKOIhEJFIGltX2ML9728gTPG9mNisQ4nIelJRSBp7ZGF5VTva+KmszQ3IOlLRSBpq76plXv/vp6PjS7io8OPCTuOSGhUBJK2Hl20iR17m7RuQNKeikDSUkNzK7/927ucMrIvk0r6hB1HJFQqAklLjyzcRFVtIzdNGxV2FJHQqQgk7dQ0NHP3C+9w+qgiTj22KOw4IqFTEUja+e1L77Krrpnbzh8XdhSRpKAikLTy3p567n9lA5eeOJjjh/QKO45IUlARSFr5+XPrcIdbPzk27CgiSUNFIGlj3bZanlhayZdOGU5xnx5hxxFJGioCSRv/+Zc15Odk8vUztaWQSKJAi8DMzjOztWZWZma3tfP4t8xslZm9ZWZ/NbPhQeaR9PXKOzv465rt/PMZozgmPzvsOCJJJbAiMLMM4B7gfGA8cIWZjW8zbBlQ6u4nAE8APwkqj6SvxpZWbv/jCkr69uDLp5WEHUck6QQ5RzAZKHP39e7eBMwCLkkc4O4vuntd/OYCYGiAeSRN/c/f17N+xz7uvOR4crMywo4jknSCLIIhQEXC7cr4fQdzHfCX9h4wsxvMbImZLamqqurEiNLdVVTX8d8vlPGpjwzi42P6hR1HJCkFWQTWzn3e7kCzq4BS4KftPe7u97p7qbuX9uunP2bpGHfnB3NXkhkx/vXCtkslRWS/IIugEihOuD0U2NJ2kJmdDXwPuNjdGwPMI2nmuVXbeGHNdm45ZwwDe+mE9CIHE2QRLAZGm9kIM8sGpgNzEweY2UnA74iVwPYAs0iaqW1o5o7/XcW4gQVcfWpJ2HFEklpgReDuLcCNwLPAamCOu680szvN7OL4sJ8CPYHHzWy5mc09yMuJHJF/+9Mq3ttTz48+/RGyMrS7jMihZAb54u4+D5jX5r7bE66fHeT0JT3NX7mVOUsq+fqZx+rMYyIdoK9K0q3s2NvId556mwmDC/nmtDFhxxFJCYHOEYh0JXfntiffpraxhccuP5HsTH3PEekI/aVIt/H4kkqeX72Nfzl3LGMGFIQdRyRlqAikW1iztYYfzF3J1JF9uPa0EWHHEUkpKgJJeXvqmvmnh5dSkJvJr6afRCTS3r6MInIwWkcgKa016nxz9jK27K5n1g1T6V+oHcdEjpTmCCSl/eL5dby0toofXDSBjw7vE3YckZSkIpCU9cyKrfz3C2VcVjqUK6cMCzuOSMpSEUhKWryxmm/OWsbE4t7cecnxmGm9gMjRUhFIyln9Xg3XPrCYIb3zmHF1qc4xIPIhqQgkpZTv3MeXZiyiZ04mD39lCn175oQdSSTlqQgkZWyvaeCL9y+iuTXKw9dNZkjvvLAjiXQLKgJJCRXVdVz2u9fZsbeR318ziVH9teewSGfRfgSS9N7ZVstV9y+kvqmVh6+bwknDdERRkc6kIpCktrxiN9f8fhFZGRHmfPUUxg0sDDuSSLejIpCk9cKabdz06DL69Mxm5nVTGN43P+xIIt2SikCSTjTq/OqFd/jF8+8wYXAhM66ZxAAdOkIkMCoCSSp76pu5ZfZyXliznc+ePJQfffp47ScgEjAVgSSNNzbt4pbZy9m8q55/u2QCV00drj2GRbqAikBC19Dcys+eW8d9L69nUK88Zt0wldISHUBOpKuoCCRUS8ur+fYTb7G+ah9fmDKM715wHD1z9LYU6Ur6i5NQbN5dz0+fWcPTy7cwpHceM6+bwumji8KOJZKWVATSpfY2tvCbl8q47+UNOPDPZxzLP585SnMBIiHSX590iep9TTz42kYefH0ju+uaufTEwXz7vHE6XpBIElARSKAqquu4/5UNzF5cQX1zK2cfN4CbzhrFxOLeYUcTkTgVgXS6huZW5q/axuzFm3i1bCeZEePSk4bwTx8fyegBOlicSLJREUinaG6N8tq7O3lmxXv8ZcVWdtc1M6R3Ht86ZwyfLx3KoF5aBCSSrFQEctR27G3ktXd38tLa7Ty/ahs1DS3kZ2cw7bgBXFZazKnH9iUS0Q5hIslORSAdtr2mgTc27WZpeTWvlu1k1Xs1APTKy+Kc8QM5//iBnD66SIeEEEkxKgL5gGjUqdhVx9qttazdWsuabbUs37SbzbvrAcjOiHDy8N58+9yxnD6qiOOH9CJD3/xFUpaKIE1Fo8622gYqquupqK5jU3UdFbvqeLdqH+9sq6WuqfXA2OI+eZxY3Jsvn1bCScOO4fghheRk6lu/SHcRaBGY2XnAL4EM4D53/3Gbx3OAh4CPAjuBy919Y5CZuit3p7ElSk19MzUNzeypb6aqtokdexupqm2kav9lwk9Ta/TA881gUGEuJUX5XD6pmLEDChg7sIAxAwrI185eIt1aYH/hZpYB3AOcA1QCi81srruvShh2HbDL3UeZ2XTgP4HLg8oUFHcn6hB1x+OXrVGnuTVKc2vssqXVaY5GD1xv2n9fa/TAuJbWKE3x6/XNrdQ3tVDfFP3H9eZW6pujB67vbWylNv7BX1Pf8r4P9kRm0KdHNv0KcuhXkMPIonz6FeZQfEwPivv0YFifHgzunatv+SJpKsivepOBMndfD2Bms4BLgMQiuAT4Yfz6E8DdZmbu7p0dZs7iCu59ef37Pqyj7kSj7/8gj/r+244TW4TiCY994PmdnvSDsjMi5GVnkJeV8b7LwtxMio/JozAvi8LcLArzMuOXWRTmZlLUM4f+BTn0yc8mMyMSfFARSUlBFsEQoCLhdiUw5WBj3L3FzPYAfYEdiYPM7AbgBoBhw4YdVZhj8rMZO6AAM4iYEYlfWsL1SIQDt434pdk/xkfskM83/jEmw4ysjAhZGbHLzITrsdtGdkaEzIiRlRkhKxIhK9PIjETIzoiQmx2hR3YmuZkRfYiLSFy7BP4AAAYCSURBVKCCLIL2NiNp+/25I2Nw93uBewFKS0uP6jv4OeMHcM74AUfzVBGRbi3Ir5qVQHHC7aHAloONMbNMoBdQHWAmERFpI8giWAyMNrMRZpYNTAfmthkzF7g6fv1zwAtBrB8QEZGDC2zRUHyZ/43As8Q2H53h7ivN7E5gibvPBe4HHjazMmJzAtODyiMiIu0LdANxd58HzGtz3+0J1xuAzweZQUREDk2bo4iIpDkVgYhImlMRiIikORWBiEias1TbWtPMqoDyo3x6EW32Wk4iyZpNuY5MsuaC5M2mXEfuaLINd/d+7T2QckXwYZjZEncvDTtHe5I1m3IdmWTNBcmbTbmOXGdn06IhEZE0pyIQEUlz6VYE94Yd4BCSNZtyHZlkzQXJm025jlynZkurdQQiIvJB6TZHICIibagIRETSXNoVgZmdaGYLzGy5mS0xs8lhZ9rPzG4ys7VmttLMfhJ2nrbM7P+YmZtZUdhZAMzsp2a2xszeMrM/mFnvkPOcF///KzOz28LMsp+ZFZvZi2a2Ov6++mbYmRKZWYaZLTOzP4WdJZGZ9TazJ+Lvr9VmdkrYmQDM7Jb4/+MKM3vMzHI743XTrgiAnwB3uPuJwO3x26EzszOJncP5BHefAPxXyJHex8yKgXOATWFnSfAccLy7nwCsA74TVhAzywDuAc4HxgNXmNn4sPIkaAFudffjgKnA15Mk137fBFaHHaIdvwSecfdxwESSIKOZDQG+AZS6+/HEDu/fKYfuT8cicKAwfr0XHzxrWli+BvzY3RsB3H17yHna+jnwL7RzKtGwuPt8d2+J31xA7Cx4YZkMlLn7endvAmYRK/ZQuft77v5G/HotsQ+0IeGmijGzocCngPvCzpLIzAqBjxM7Xwru3uTuu8NNdUAmkBc/o2MPOunzKx2L4Gbgp2ZWQexbd2jfItsYA3zMzBaa2d/MbFLYgfYzs4uBze7+ZthZDuFa4C8hTn8IUJFwu5Ik+cDdz8xKgJOAheEmOeAXxL5cRMMO0sZIoAr4fXyx1X1mlh92KHffTOwzaxPwHrDH3ed3xmsHemKasJjZ88DAdh76HjANuMXdnzSzy4i1/tlJkCsTOIbY7PskYI6ZjeyqU3ceJtt3gU92RY62DpXL3f8YH/M9YotAHunKbG1YO/clzdyTmfUEngRudveaJMhzIbDd3Zea2Rlh52kjEzgZuMndF5rZL4HbgH8NM5SZHUNsLnMEsBt43MyucveZH/a1u2URuPtBP9jN7CFiyyUBHqcLZ0sPk+trwFPxD/5FZhYldmCpqjCzmdlHiL3x3jQziC1+ecPMJrv71rByJeS7GrgQmBby+a4rgeKE20NJksWOZpZFrAQecfenws4TdxpwsZldAOQChWY2092vCjkXxP4vK919/5zTE8SKIGxnAxvcvQrAzJ4CTgU+dBGk46KhLcAn4tfPAt4JMUuip4nlwczGANkkwZEP3f1td+/v7iXuXkLsj+TkriiBwzGz84D/C1zs7nUhx1kMjDazEWaWTWwl3tyQM2Gx9r4fWO3uPws7z37u/h13Hxp/T00HXkiSEiD+3q4ws7Hxu6YBq0KMtN8mYKqZ9Yj/v06jk1Zid8s5gsO4HvhlfGVLA3BDyHn2mwHMMLMVQBNwdcjfcFPB3UAO8Fx8bmWBu381jCDu3mJmNwLPEtuaY4a7rwwjSxunAV8E3jaz5fH7vhs/n7gc3E3AI/FSXw98OeQ8xBdTPQG8QWxR6DI66VATOsSEiEiaS8dFQyIikkBFICKS5lQEIiJpTkUgIpLmVAQiImlORSAikuZUBCIiaU5FIPIhmdmk+DkRcs0sP368+OPDziXSUdqhTKQTmNn/I3bMnDxix6n5j5AjiXSYikCkE8QPRbCY2GFLTnX31pAjiXSYFg2JdI4+QE+ggNicgUjK0ByBSCcws7nEzko2Ahjk7jeGHEmkw9Lx6KMincrMvgS0uPuj8XMXv2ZmZ7n7C2FnE+kIzRGIiKQ5rSMQEUlzKgIRkTSnIhARSXMqAhGRNKciEBFJcyoCEZE0pyIQEUlz/x/NkaNzTWalWQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The main reason why we use sigmoid function is because it exists between (0 to 1). \n",
    "# Therefore, it is especially used for models where we have to predict the probability as an output.\n",
    "# Since probability of anything exists only between the range of 0 and 1, sigmoid is the right choice.\n",
    "# The function is differentiable. That means, we can find the slope of the sigmoid curve at any two points.\n",
    "# There are four commonly used and popular activation functions — sigmoid, hyperbolic tangent(tanh), ReLU and Softmax.\n",
    "\n",
    "x = np.arange(-8, 8, 0.1)\n",
    "f = 1 / (1 + np.exp(-x))\n",
    "plt.plot(x, f)\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('f(x)')\n",
    "plt.title('Sigmoid function')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tanh function\n",
    "* Maps values between -1 and 1\n",
    "* tanh is also sigmoidal (s - shaped)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxfdZ3v8dc7SZvu+0r3YkuhFFoIFcq4sFSRO0NRUdER64J1vKKjMzqCzlUvow7qnUG94x3pICMiAwy4dQQGWXUUwYZSulK7UJo0haZL0iVt0iSf+8fvhPkRkjZN8sv5/ZL38/H4PXLO92yfQJp3zvmec76KCMzMzE5WUdoFmJlZYXKAmJlZpzhAzMysUxwgZmbWKQ4QMzPrFAeImZl1igPErAsk/YWkR05i/TMlrZF0SNKyXNbW6rizJdX01PGsb3CAWK+X/LJu+TRLOpI1/+c9XM4NwP0RMSQilufqIJJekvQnLfMR8ceIGJGr41nfVJJ2AWa5FhFDWqYlbQeujYgOnzV0s2nAgykd26xb+QzE+jxJF0p6WlKtpCpJN0sqSZYNkBSSPippq6T9km5+7S70XUk1yTqXtnOcJ4ELgFuTs5+pkp6S9P6sdV65JNaRY0v6n5Kel3RQ0lpJ8yTdC4wDfpUc51OS5khqzNpuqqQHJO2T9EdJS7OW3STpTkl3JftdI2l+F/8zWy/kADGDY8B1wCjgDcCfAde2WudtwALgHOBDkt6cteyNQDkwGvgn4Na2DhIRi4CVZM6AhkTEjg7W1+axJV0DfB54LzAMuArYHxHvAnYDb0mO89029nkvsAmYCLwPuFnShVnL3w7cBowAHgW+3cFarQ9xgFifFxF/iIiVEdEUEVvJBMCbWq329Yg4EBEvAL8Bsv8i3xQRP4qIJuB2YJqk7uxvaO/Y1ybLno2MTRFReaKdSZoFnA18ISLqI6I8qfuarNUei4iHk+/pDl79/ZoBDhAzJJ0h6UFJL0s6AHwJGNNqtZeypuuAIcdZRqvlXdXesacAWzuxv1OA6og4ktX2IjCpA8c0e4UDxAz+BVgFnBoRw4AbAfXQsQ8Dg7LmJ5zEthXAqe0sO95rtquAsZIGZrVNBXaexLHNHCBmwFCgNiIOSZoLfLQHj70auCrpMJ8DfPAktr0VuF7S2cqYLWlysuxlYGY7220B1gBflVQq6RxgKXBn574F66scIGbwGeBaSYeA7wH39OCxv0nmdvpqYDnw445uGBF3AP8I3AccSL629L18DfhacmfYda22C+DdwBlkLlXdA3wuIv6ra9+K9TXygFJmZtYZPgMxM7NOcYCYmVmnOEDMzKxTHCBmZtYpfeplimPGjInp06enXYaZWUF55pln9kTE2NbtfSpApk+fTnl5edplmJkVFEkvttXuS1hmZtYpDhAzM+sUB4iZmXWKA8TMzDrFAWJmZp2SaoBIuk3Sbknr2lneMlTolmRYzXOyli2VtDn5LG1rezMzy520z0B+CFx2nOVvA2Yln2XAPwNIGgV8GXg9sBD4sqSROa3UzMxeJdXnQCLiN5KmH2eVJcCPktdPPyVphKSJwJuBhyNiH4Ckh8kE0V25rdjMuiIiqG9spq6hibqGRo40NFHX0MTRY000NQfHmoOm5mYam4Km5qCxOWhsNd/UHDRHEJEZNavljeItLxYPspdl5l+1PGtbstbJbuuh/xg9eTSWLprO6CGl3brPfH+QcBKZUddaVCZt7bW/hqRlZM5emDp1am6qNOvDIoLqQ/Xs2FvH9r11vHzgKHsPNbD3cD17DzWw51A9+w43cLi+kSPHmmj2CBKvUE+NewlcMX9SnwuQtv7zxnHaX9sYsZzMQD2UlZX5R9esC5qbgy3Vh3iuooa1O2t5rrKWzS8fpK6h6VXrDe5fzOghpYwe0p/JIwdy1uThDCntx6D+xQzsX8yg5DOwfwmD+mXaSopESbEoLip6Zbqk6NXzxUWiWJmvAEKgzC9iAVJLe0ubXvVLOrutzfV78jd6L5DvAVIJTMman0xmPOdKMpexstuf6LGqzPqQIw1N/G7LHh59/mUe2bib6oP1QCYkzpw0nHeXTWH66EFMGzOYaaMGccqIgQzoV5xy1dYT8j1AVgDXSbqbTId5bUTskvQQ8PWsjvO3ADekVaRZb7Rqx37ufGoH96+t4uixZoaUlvCm2WN582ljWTB1JDPHDKaoyH+x92WpBoiku8icSYyRVEnmzqp+ABHxfeAB4HJgC1AHfChZtk/S3wErk13d2NKhbmZd89vNe/juo5v5w/Z9DO5fzNsXTObyeRN4/YzR9C9J+8ZNyyd9akz0srKy8Nt4zdq2fc9hbvzlBh57fjcThg3gY2+aybvKpjCkNN8vVFiuSXomIspat/snw6yPiwhuf3I7X3/wefoViS9cPoeli6ZTWuJ+DDs+B4hZH3bg6DE+ffdqHnt+NxfPGcdN75jHuGED0i7LCoQDxKyPqthXx0duX8m26sPcuGQu15w/zbex2klxgJj1QS/sOcx7bvk9R4818aOPLGTRqWPSLskKkAPErI/ZsbeO9/3LUzQ1B/d9fBGzxw9NuyQrUA4Qsz5k/+EGrrntaY4ca+Kuj57v8LAu8U3dZn3EsaZmPvFvq9hVc5QfLD2P0ycOS7skK3A+AzHrI77x4PM8uXUv37rqLM6d5tEPrOt8BmLWB/xuyx5u/e0LXHP+NN5VNuXEG5h1gAPErJerPXKMz977HDPHDuYLl5+edjnWi/gSllkv9/cPbGT3wXp++vFFDOzvp8ut+/gMxKwXW1NZwz3lFXz4wumcPWVE2uVYL+MAMeulmpuDr6xYz+jBpXzqkllpl2O9kAPErJda8VwVq3bU8PnLTmPogH5pl2O9kAPErBdqbGrm5kf+yNxThvHOcyanXY71Ug4Qs17oF6ureHFvHX95ySyPGmg5k2qASLpM0iZJWyRd38bymyWtTj5/lFSTtawpa9mKnq3cLH81NjXzT49v4YyJw1h8xvi0y7FeLLXbeCUVA98DFgOVwEpJKyJiQ8s6EfGZrPU/CSzI2sWRiJjfU/WaFYr71+7ihT2HueWac/16dsupNM9AFgJbImJbRDQAdwNLjrP+e4G7eqQyswIVEfzgty9w6tjBLD7dZx+WW2kGyCSgImu+Mml7DUnTgBnAY1nNAySVS3pK0pXtHUTSsmS98urq6u6o2yxvPVtRw5rKWj64aLr7Pizn0gyQtn66o511rwbui4imrLapySDv7wO+LenUtjaMiOURURYRZWPHju1axWZ57vYntzO0tIR3+M4r6wFpBkglkP1Wt8lAVTvrXk2ry1cRUZV83QY8wav7R8z6nN0HjnL/ml28q2wKg0v9liLLvTQDZCUwS9IMSf3JhMRr7qaSdBowEvh9VttISaXJ9BjgQmBD623N+pJ7n6mksTm45oJpaZdifURqf6ZERKOk64CHgGLgtohYL+lGoDwiWsLkvcDdEZF9eet04BZJzWRC8Kbsu7fM+pqI4CerKlk4fRQzxgxOuxzrI1I9z42IB4AHWrV9qdX8V9rY7klgXk6LMysgqytq2FZ9mI+9cWbapVgf4ifRzXqBn67ayYB+RVw+b2LapVgf4gAxK3D1jU2seK6Kt86d4JcmWo9ygJgVuF9vqqb2yDHfums9zgFiVuAeXPcSIwb1Y9Gpo9MuxfoYB4hZAWtobOaRjS+z+PTx9Cv2P2frWf6JMytgv9u6h4NHG3nbvAlpl2J9kAPErIA9uHYXQ0tLuPB1Y9IuxfogB4hZgWpsaubhDS9z8enjKC0pTrsc64McIGYFqvzF/eyvO8Zb5/rylaXDAWJWoB7ftJuSIvGGWb58ZelwgJgVqF9vqua86aP88KClxgFiVoCqao7w/EsHuWiOx7ix9DhAzArQE5syo2u++bRxKVdifZkDxKwAPbFpN5NGDGTWuCFpl2J9mAPErMA0NDbzuy17ePNpY5E87rmlxwFiVmBWV9RwuKGJN852/4elK9UAkXSZpE2Stki6vo3lH5RULWl18rk2a9lSSZuTz9KerdwsPU9u3UOR4PyZfnmipSu1EQklFQPfAxYDlcBKSSvaGJr2noi4rtW2o4AvA2VAAM8k2+7vgdLNUvXk1r2cOWk4wwf69l1LV5pnIAuBLRGxLSIagLuBJR3c9q3AwxGxLwmNh4HLclSnWd6oa2jk2R37ucCvbrc8kGaATAIqsuYrk7bW3ilpjaT7JE05yW2RtExSuaTy6urq7qjbLDXl2/dzrClYdKqfPrf0pRkgbd0+Eq3m/wOYHhFnAY8At5/EtpnGiOURURYRZWPHutPRCtuTW/fSr1icN31k2qWYpRoglcCUrPnJQFX2ChGxNyLqk9l/Ac7t6LZmvdHvt+5hwZSRDOqfWvel2SvSDJCVwCxJMyT1B64GVmSvIGli1uwVwMZk+iHgLZJGShoJvCVpM+u1Dh49xtqdtZzv/g/LE6n9GRMRjZKuI/OLvxi4LSLWS7oRKI+IFcCnJF0BNAL7gA8m2+6T9HdkQgjgxojY1+PfhFkPeubF/TQHLJw+Ku1SzIAUAwQgIh4AHmjV9qWs6RuAG9rZ9jbgtpwWaJZHyrfvp7hILJg6Iu1SzAA/iW5WMFZu38fcU4YxuNT9H5YfHCBmBaC+sYnVFTWUTfPlK8sfDhCzArBu5wHqG5tZOMO371r+cICYFYDy7Zl7RM71GYjlEQeIWQFYuX0fM8YMZuzQ0rRLMXuFA8Qsz0UEz7y4n7Jpvnxl+cUBYpbnduyrY3/dMRZMdYBYfnGAmOW51RU1AMyf4uc/LL84QMzy3OqKGgb0K2L2eI9/bvnFAWKW556rqGHepOGUFPufq+UX/0Sa5bFjTc2sqzrA2ZN9+cryjwPELI89v+sgDY3NzPf7rywPOUDM8tjqykwHus9ALB85QMzy2HMVNYwe3J/JIwemXYrZazhAzPLYcxU1zJ8yAqmtUZzN0pVqgEi6TNImSVskXd/G8r+StEHSGkmPSpqWtaxJ0urks6L1tmaF7uDRY2ypPsTZfv7D8lRqAwtIKga+BywmM8b5SkkrImJD1mrPAmURUSfp48A3gfcky45ExPweLdqsB62trCUCB4jlrTTPQBYCWyJiW0Q0AHcDS7JXiIjHI6IumX0KmNzDNZql5r870IenXIlZ29IMkElARdZ8ZdLWno8AD2bND5BULukpSVe2t5GkZcl65dXV1V2r2KwHrd5Rw4wxgxkxqH/apZi1Kc2xMdvqFYw2V5TeD5QBb8pqnhoRVZJmAo9JWhsRW1+zw4jlwHKAsrKyNvdvlo+eq6zhgpmj0y7DrF1pnoFUAlOy5icDVa1XknQp8EXgioiob2mPiKrk6zbgCWBBLos160kv1R7l5QP17v+wvJZmgKwEZkmaIak/cDXwqrupJC0AbiETHruz2kdKKk2mxwAXAtmd72YFzW/gtUKQ2iWsiGiUdB3wEFAM3BYR6yXdCJRHxArgW8AQ4N7kPvgdEXEFcDpwi6RmMiF4U6u7t8wK2uqKGvoVi9MnDku7FLN2pdkHQkQ8ADzQqu1LWdOXtrPdk8C83FZnlp41lTXMmTCMAf2K0y7FrF1+Et0sz0QE63bWMs+371qec4CY5ZmKfUc4cLSRM09xgFh+c4CY5Zl1VbUAzJvkALH85gAxyzPrdtZSUiRmT/AQtpbfHCBmeWZd1QFmjx9KaYk70C2/OUDM8khEsH5nLWdO8u27lv8cIGZ55KUDR9l7uIEz3f9hBcABYpZH1u08AMBc34FlBcABYpZH1u2spUhw+sShaZdidkIOELM8sr6qllPHDmFQ/1RfEmHWIQ4Qszyydmet+z+sYDhAzPLE7oOZV7jPPcV3YFlhcICY5Yn1VZkOdJ+BWKFwgJjlifU7M68wOcNnIFYgOtRTJ2kcmUGbTgGOAOvIjNnRnMPazPqUdTsPMH30IIYN6Jd2KWYdctwAkXQRcD0wCngW2A0MAK4ETpV0H/APEXEg14Wa9Xbrqmo9AqEVlBNdwroc+GhEnBcRyyLibyPis8mogGeTCZXFnT24pMskbZK0RdL1bSwvlXRPsvxpSdOzlt2QtG+S9NbO1mCWD2rqGqjcf8T9H1ZQjnsGEhGfO86yRuDnnT2wpGLge2QCqBJYKWlFq6FpPwLsj4jXSboa+AbwHklnkBlDfS6Zy2qPSJodEU2drccsTa90oPsJdCsgHepEl3SHpOFZ89MlPdrFYy8EtkTEtohoAO4GlrRaZwlwezJ9H3CJMoOjLwHujoj6iHgB2JLsz6wgrUs60H0LrxWSjt6F9VvgaUmXS/oo8Cvg21089iSgImu+Mmlrc53kjKcWGN3BbQGQtExSuaTy6urqLpZslhvrqg4wacRARg7un3YpZh3WobuwIuIWSeuBx4E9wIKIeKmLx1Zbh+rgOh3ZNtMYsRxYDlBWVtbmOmZp8yvcrRB19BLWNcBtwAeAHwIPSDq7i8euBKZkzU8GqtpbR1IJMBzY18FtzQrCwaPH2LbnsPs/rOB09BLWO4E/iYi7IuIG4C/IBElXrARmSZohqT+ZTvEVrdZZASxNpq8CHouISNqvTu7SmgHMAv7QxXrMUrHBT6BbgeroJawrW83/QdLru3LgiGiUdB3wEFAM3BYR6yXdSOYhxRXAD4A7JG0hc+ZxdbLtekn/DmwAGoFP+A4sK1TrkgCZ60tYVmBO9CDh3wL/LyL2tV4WEQ2SLgYGRcQvO3PwiHgAeKBV25eypo8C72pn268BX+vMcc3yyfqdtYwbWsq4oQPSLsXspJzoDGQt8B+SjgKrgGoyT6LPAuYDjwBfz2mFZr3cuiq/wt0K04n6QK6KiAvJXGZaT+ZS0wHgx8DCiPhMRPjeWLNOOtLQxJbdhzjTz39YATrRGci5kqYBfw5c1GrZQDIvVjSzTtr40gGaA+b6DMQK0IkC5PvAfwIzgfKsdpF57mJmjuoy6xNaXuE+zwFiBei4l7Ai4rsRcTqZO6RmZn1mRITDw6yL1lTWMmpwfyYOdwe6FZ4OPQcSER/PdSFmfdHanbXMmzSczCvezAqLRyQ0S8mRhiY27z7EWZN9+coKkwPELCUbdh2gqTnc/2EFywFilpKWV7jP8xmIFSgHiFlK1lTWMmZIKROGuQPdCpMDxCwla3fWcNZkd6Bb4XKAmKWgrqGRLbsPuf/DCpoDxCwFG6oyT6A7QKyQOUDMUrCm0h3oVvgcIGYpWLuzlvHDShnvDnQrYA4QsxRknkAfkXYZZl2SSoBIGiXpYUmbk68j21hnvqTfS1ovaY2k92Qt+6GkFyStTj7ze/Y7MOu8Q/WNbK12B7oVvrTOQK4HHo2IWcCjyXxrdcAHImIucBnwbUnZf7J9LiLmJ5/VuS/ZrHus31lLBH6FiRW8tAJkCXB7Mn07cGXrFSLijxGxOZmuAnYDY3usQrMcWZs8ge5RCK3QpRUg4yNiF0DyddzxVpa0EOgPbM1q/lpyaetmSaXH2XaZpHJJ5dXVHjzR0rd2Zy2nDB/A2KHt/tiaFYScBYikRySta+Oz5CT3MxG4A/hQRDQnzTcAc4DzgFHA59vbPiKWR0RZRJSNHesTGEvf2spa375rvcKJRiTstIi4tL1lkl6WNDEidiUBsbud9YYB9wN/GxFPZe17VzJZL+lfgc92Y+lmOXPg6DG27TnMO86ZlHYpZl2W1iWsFcDSZHop8IvWK0jqD/wM+FFE3Ntq2cTkq8j0n6zLabVm3WT9zgMAzJvsW3it8KUVIDcBiyVtBhYn80gqk3Rrss67gTcCH2zjdt07Ja0F1gJjgK/2bPlmnbO6ogaAs9yBbr1Azi5hHU9E7AUuaaO9HLg2mf4x8ON2tr84pwWa5cizO/Yzc8xgRg7un3YpZl3mJ9HNekhE8GxFDfOn+vKV9Q4OELMesrPmCNUH61kwxQFivYMDxKyHPLsj0/+xYOpr3txjVpAcIGY95NkdNQzoV8RpE4amXYpZt3CAmPWQZyv2c9akEfQr9j876x38k2zWA+obm1i/8wAL3IFuvYgDxKwHrK86QENTM/PdgW69iAPErAeUb98HwLnT3YFuvYcDxKwHrNy+n+mjBzFuqIewtd7DAWKWY83NQfn2fZRNH5V2KWbdygFilmPb9hxif90xFjpArJdxgJjl2Mrt+wEoc/+H9TIOELMcW/nCPsYM6c+MMYPTLsWsWzlAzHJs5Yv7KJs2iszwNWa9hwPELIdeqj1Kxb4jvnxlvVIqASJplKSHJW1Ovrb5r0tSU9ZgUiuy2mdIejrZ/p5k9EKzvPP7bXsAOH/m6JQrMet+aZ2BXA88GhGzgEeT+bYciYj5yeeKrPZvADcn2+8HPpLbcs0658ktexk+sB9nTByWdilm3S6tAFkC3J5M305mXPMOScZBvxi4rzPbm/WUiODJrXu5YOZoiorc/2G9T1oBMj4idgEkX8e1s94ASeWSnpLUEhKjgZqIaEzmK4FJ7R1I0rJkH+XV1dXdVb/ZCVXsO8LOmiMsep0vX1nvlLMx0SU9AkxoY9EXT2I3UyOiStJM4DFJa4EDbawX7e0gIpYDywHKysraXc+su/1ua6b/Y9GpDhDrnXIWIBFxaXvLJL0saWJE7JI0Edjdzj6qkq/bJD0BLAB+AoyQVJKchUwGqrr9GzDroie37mXc0FJOHTsk7VLMciKtS1grgKXJ9FLgF61XkDRSUmkyPQa4ENgQEQE8Dlx1vO3N0hQR/H7rHhadOtrPf1ivlVaA3AQslrQZWJzMI6lM0q3JOqcD5ZKeIxMYN0XEhmTZ54G/krSFTJ/ID3q0erMT2LjrIHsONbDodWPSLsUsZ3J2Cet4ImIvcEkb7eXAtcn0k8C8drbfBizMZY1mXfH4psxV2TfPHptyJWa54yfRzXLgiU27mXvKMMYN8/gf1ns5QMy6WW3dMVbtqOGi09q7O92sd3CAmHWz/9pSTVNzcNEcX76y3s0BYtbNnthUzfCB/Zg/xS9QtN7NAWLWjZqbgyc2VfPG2WMp9utLrJdzgJh1o1U79rPnUD2Xnu7+D+v9HCBm3ejBdS/Rv7iIi+c4QKz3c4CYdZOI4D/XvcQbZo1h6IB+aZdjlnMOELNusqaylp01R7jszLbeIWrW+zhAzLrJg+teoqRILD5jfNqlmPUIB4hZN4gIHly3iwtOHc2IQR5h2foGB4hZN1i1Yz8v7q3jirNPSbsUsx7jADHrBvc9s5OB/Yp527yJaZdi1mMcIGZddPRYE79cU8VlZ05gSGkqL7g2S4UDxKyLHt7wMgePNvLOcyanXYpZj3KAmHXRfc9UMnH4AC7w2OfWx6QSIJJGSXpY0ubk62veOifpIkmrsz5HJV2ZLPuhpBeyls3v+e/CDLbvOcxvNlfzrrIpfveV9TlpnYFcDzwaEbOAR5P5V4mIxyNifkTMBy4G6oBfZa3yuZblEbG6R6o2a+VHv3+RYon3v35q2qWY9bi0AmQJcHsyfTtw5QnWvwp4MCLqclqV2Uk4XN/IveUVXD5vokcetD4prQAZHxG7AJKvJ3rz3NXAXa3aviZpjaSbJZW2t6GkZZLKJZVXV1d3rWqzLD9dVcnB+kaWLpqedilmqchZgEh6RNK6Nj5LTnI/E4F5wENZzTcAc4DzgFHA59vbPiKWR0RZRJSNHesR4qx7NDY1c+tvX+CsycM5Z+qItMsxS0XOblqPiEvbWybpZUkTI2JXEhC7j7OrdwM/i4hjWfvelUzWS/pX4LPdUrRZB/18dRUv7q1j+TXnIrnz3PqmtC5hrQCWJtNLgV8cZ9330uryVRI6KPMv90pgXQ5qNGtTY1Mz//exzZwxcZhfnGh9WloBchOwWNJmYHEyj6QySbe2rCRpOjAF+HWr7e+UtBZYC4wBvtoDNZsB8Ivk7OPTl87y2Yf1aam8dyEi9gKXtNFeDlybNb8dmNTGehfnsj6z9tQ1NPIPv9rEmZN89mHmJ9HNTsL3n9hKVe1Rvvxnc332YX2eA8Ssgyr21XHLb7ZxxdmncN70UWmXY5Y6B4hZB0QEX/z5Oookbrh8TtrlmOUFB4hZB/z46R385o/VfOHyOUwcPjDtcszyggPE7AS2VR/i6/dv5A2zxvD+86elXY5Z3nCAmB3HwaPH+Ngdz1Dar4hvXXW2O87Nsnj4NLN2NDcHn7lnNdv2HOaODy9kwnC/MNEsm89AzNoQEXx5xXoe2bibL/3pGSx63Zi0SzLLOw4Qs1Yight/uYE7nnqRj71xJh+4wP0eZm3xJSyzLA2NzXzhZ2u575lKPnzhDK5/2xz3e5i1wwFilqg+WM8n71rFU9v28alLZvEZv+vK7LgcIGbAoxtf5m/uW8Oh+kZufs/ZvH3B5LRLMst7DhDr0yr31/G1+zfy4LqXmDNhKHctO5/Z44emXZZZQXCAWJ+0s+YI339iK/eUV1Ak+OvFs1n2ppmUlhSnXZpZwXCAWJ9xpKGJ32yu5t7ySh57/mWKi8RV507muotnMWmEX09idrIcINZrRQQV+47w2y17eHTjy/x2yx7qG5sZM6Q/H3vTqbz//GkODrMuSCVAJL0L+ApwOrAwGUiqrfUuA74DFAO3RkTLyIUzgLuBUcAq4JqIaOiB0i0PRQT7646xfe9hduytY/Pug6yprGXtzlpq6o4BMHnkQN67cCqLzxjPwhmj6FfsR6DMuiqtM5B1wDuAW9pbQVIx8D0yQ95WAislrYiIDcA3gJsj4m5J3wc+Avxz7su27hARNDYHjU3BseZmGpuCxqZmjjUnX5uCo8eaOFTfyKGjjRxuaPzv6fpGDtY3sudQA9UHj1J9sJ7dB+o5WN/4yv6Li8Rp44dy2dwJnDV5BOdOG8ns8UN8S65ZN0trSNuNwIn+QS8EtkTEtmTdu4ElkjYCFwPvS9a7nczZTM4C5Is/W8vTL+wDMr/8WkT2StHm5KvWf+2y7PZou/3Vm7e53/b2edz9trtNR9bv4PfUqsbGpqCxORMQnSXBkP4ljB7Sn7FDSzltwlD+5HVjmDp6MNNGDWLa6EFMGTWIAf3cGW6Wa/ncBzIJqMiarwReD4wGaiKiMav9NeOmt5C0DFgGMHXq1E4VcsqIgZyWfWun2px8VSC+ur1VTR3Y5tXHyFqn3R3iHR0AAAZXSURBVGO3vf5rlrVzkJPdb0e/p2z9ikVJcRH9ijJfS4pF/+IiSpL5fsWipCjTPrBfMUNKSxhcWsKQASUMKc18BvYrpqjIZxJm+SBnASLpEWBCG4u+GBG/6Mgu2miL47S3KSKWA8sBysrKOvWn7ycuel1nNjMz69VyFiARcWkXd1EJTMmanwxUAXuAEZJKkrOQlnYzM+tB+XwrykpglqQZkvoDVwMrInNh/XHgqmS9pUBHzmjMzKwbpRIgkt4uqRK4ALhf0kNJ+ymSHgBIzi6uAx4CNgL/HhHrk118HvgrSVvI9In8oKe/BzOzvk6t76jpzcrKyqK8vM1HTszMrB2SnomIstbt+XwJy8zM8pgDxMzMOsUBYmZmneIAMTOzTulTneiSqoEXO7n5GDLPoOSbfK0L8rc213Vy8rUuyN/aeltd0yJibOvGPhUgXSGpvK27ENKWr3VB/tbmuk5OvtYF+VtbX6nLl7DMzKxTHCBmZtYpDpCOW552Ae3I17ogf2tzXScnX+uC/K2tT9TlPhAzM+sUn4GYmVmnOEDMzKxTHCAnQdJ8SU9JWi2pXNLCtGtqIemTkjZJWi/pm2nXk03SZyWFpDFp19JC0rckPS9pjaSfSRqRcj2XJf//tki6Ps1aWkiaIulxSRuTn6u/TLumbJKKJT0r6Zdp15JN0ghJ9yU/XxslXZB2TQCSPpP8f1wn6S5JA7q6TwfIyfkm8L8jYj7wpWQ+dZIuApYAZ0XEXOD/pFzSKyRNARYDO9KupZWHgTMj4izgj8ANaRUiqRj4HvA24AzgvZLOSKueLI3AX0fE6cD5wCfypK4Wf0lmqId88x3gPyNiDnA2eVCjpEnAp4CyiDgTKCYzxlKXOEBOTgDDkunh5M9IiB8HboqIeoCI2J1yPdluBv6G4ww7nIaI+FUy5gzAU2RGtkzLQmBLRGyLiAbgbjJ/EKQqInZFxKpk+iCZX4ST0q0qQ9Jk4H8At6ZdSzZJw4A3koxRFBENEVGTblWvKAEGSioBBtENv78cICfn08C3JFWQ+Ss/tb9aW5kNvEHS05J+Lem8tAsCkHQFsDMinku7lhP4MPBgisefBFRkzVeSJ7+oW0iaDiwAnk63kld8m8wfJs1pF9LKTKAa+Nfk8tqtkganXVRE7CTzO2sHsAuojYhfdXW/ORsTvVBJegSY0MaiLwKXAJ+JiJ9IejeZvzK6OvZ7d9RVAowkc5nhPODfJc2MHrhH+wR1fQF4S65raM/xaouIXyTrfJHMpZo7e7K2VtRGW96csUkaAvwE+HREHMiDev4U2B0Rz0h6c9r1tFICnAN8MiKelvQd4Hrgf6VZlKSRZM5qZwA1wL2S3h8RP+7Kfh0grUREu4Eg6UdkrrsC3EsPnj6foK6PAz9NAuMPkprJvDStOq26JM0j88P6nCTIXCJaJWlhRLyU67qOV1sLSUuBPwUu6YmwPY5KYErW/GTy5PKopH5kwuPOiPhp2vUkLgSukHQ5MAAYJunHEfH+lOuCzP/LyohoOVO7j0yApO1S4IWIqAaQ9FNgEdClAPElrJNTBbwpmb4Y2JxiLdl+TqYeJM0G+pPym0AjYm1EjIuI6RExncw/rHN6KjxORNJlwOeBKyKiLuVyVgKzJM2Q1J9M5+aKlGtCmeT/AbAxIv4x7XpaRMQNETE5+bm6GngsT8KD5Oe7QtJpSdMlwIYUS2qxAzhf0qDk/+sldEPnvs9ATs5Hge8knVBHgWUp19PiNuA2SeuABmBpyn9RF4J/AkqBh5MzpKci4i/SKCQiGiVdBzxE5u6Y2yJifRq1tHIhcA2wVtLqpO0LEfFAijUVgk8CdyZ/DGwDPpRyPSSX0+4DVpG5ZPss3fBaE7/KxMzMOsWXsMzMrFMcIGZm1ikOEDMz6xQHiJmZdYoDxMzMOsUBYmZmneIAMTOzTnGAmKVI0nnJmCQDJA1Oxms4M+26zDrCDxKapUzSV8m802kgmfco/X3KJZl1iAPELGXJKy9Wknk9zqKIaEq5JLMO8SUss/SNAoYAQ8mciZgVBJ+BmKVM0goyoxDOACZGxHUpl2TWIX4br1mKJH0AaIyIf0vGRn9S0sUR8VjatZmdiM9AzMysU9wHYmZmneIAMTOzTnGAmJlZpzhAzMysUxwgZmbWKQ4QMzPrFAeImZl1yv8HB27UEvcZEl0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-8, 8, 0.1)\n",
    "f = np.tanh(x)\n",
    "plt.plot(x, f)\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('f(x)')\n",
    "plt.title('Tanh function')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build sigmoid function for later use\n",
    "# sigmoid(w*x + b) = 1/(1+e^-(wTx+b))\n",
    "# z is (w*x+b), \n",
    "\n",
    "def sigmoid(z):\n",
    "    s = 1 / (1 + np.exp(-z))\n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Building blocks:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an understanding of the main components of a NN, let's build one from scratch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Structure of an Artificial Neural Network (ANN)\n",
    "* Input Layer is where data enters the network\n",
    "* Hidden Layers (on the picture there are two) is where function applies weights (w) to the inputs and directs them through the activation function like sigmoid or relu\n",
    "* Output Layer is where function returns the outputs from the last layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/nn_structure.jpg'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The general methodology to build a Neural Network is to:\n",
    "\n",
    "1. Define the neural network structure ( # of input units,  # of hidden layers, etc). \n",
    "2. Initialize the model's parameters\n",
    "3. Loop:\n",
    "    - Implement forward propagation\n",
    "    - Compute loss\n",
    "    - Implement backward propagation to get the gradients\n",
    "    - Update parameters (gradient descent)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1 & 2 Define and Initialize model's parameters\n",
    "\n",
    "- n_x : size of the input layer\n",
    "- n_h : size of the hidden layer\n",
    "- n_y : size of the output layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize weights (w) with random values and bias (b) as zeros.\n",
    "If we initialize weights with 0, the derivative with respect to a loss function will be the same for every w."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W1 = [[-0.00701855  0.00785847 -0.00459436]\n",
      " [ 0.01153013  0.01419068 -0.01077526]\n",
      " [-0.00384733 -0.01590927  0.00956441]]\n",
      "b1 = [[0.]\n",
      " [0.]\n",
      " [0.]]\n",
      "W2 = [[ 0.00965059  0.00204146 -0.0045111 ]]\n",
      "b2 = [[0.]]\n"
     ]
    }
   ],
   "source": [
    "# Start with a basic network initialization\n",
    "\n",
    "# Size of the input layer\n",
    "n_x = 3\n",
    "# Size of the hidden layer\n",
    "n_h = 3\n",
    "# Size of the output layer\n",
    "n_y = 1\n",
    "\n",
    "\n",
    "# W1 - weight matrix of shape (n_h, n_x)\n",
    "W1 = np.random.randn(n_h,n_x) * 0.01\n",
    "\n",
    "# b1 - bias vector of shape (n_h, 1)\n",
    "b1 = np.zeros((n_h,1))\n",
    "\n",
    "# W2 - weight matrix of shape (n_y, n_h)\n",
    "W2 = np.random.randn(n_y,n_h) * 0.01\n",
    "    \n",
    "# b2 - bias vector of shape (n_y, 1)\n",
    "b2 = np.zeros((n_y,1))\n",
    "\n",
    "print(\"W1 = \" + str(W1))\n",
    "print(\"b1 = \" + str(b1))\n",
    "print(\"W2 = \" + str(W2))\n",
    "print(\"b2 = \" + str(b2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build function to store parameters for later use\n",
    "\n",
    "def model_parameters(n_x, n_h, n_y): \n",
    "    W1 = np.random.randn(n_h, n_x) * 0.01\n",
    "    b1 = np.zeros((n_h, 1))\n",
    "    W2 = np.random.randn(n_y, n_h) * 0.01\n",
    "    b2 = np.zeros((n_y, 1))\n",
    "    # save to dictionary\n",
    "    parameters = {'W1' : W1,\n",
    "                  'b1' : b1,\n",
    "                  'W2' : W2,\n",
    "                  'b2' : b2}\n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forward propagation \n",
    "    \n",
    "* Calculations in the model that take us from an input layer all the way to the output (how NN make predictions)\n",
    "* Each independent feature x will be passed to the 1st hidden layer combined with some randomized weight\n",
    "* 1st hidden layer applies an activation function resulting in an output which then becomes an input for next hidden layer\n",
    "* Next hidden layer, repeats step above and progresses forward\n",
    "* The weights of a neuron can be thought of as weights between 2 layers "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/forward_nn.JPG'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Implement forward pass \n",
    "# parameters - dictionary of initial parameters\n",
    "# X - input data\n",
    "\n",
    "def forward_propagation(X, parameters):\n",
    "    W1 = parameters[\"W1\"]\n",
    "    b1 = parameters[\"b1\"]\n",
    "    W2 = parameters[\"W2\"]\n",
    "    b2 = parameters[\"b2\"]\n",
    "    \n",
    "    # Values from the picture above\n",
    "    Z1 = np.dot(W1,X) + b1\n",
    "    A1 = np.tanh(Z1)\n",
    "    Z2 = np.dot(W2,A1) + b2\n",
    "    # use previously built function sigmoid\n",
    "    A2 = sigmoid(Z2)\n",
    "    # save to dictionary\n",
    "    fwd_pass_values = {\"Z1\" : Z1,\n",
    "                       \"A1\" : A1,\n",
    "                       \"Z2\" : Z2,\n",
    "                       \"A2\" : A2}\n",
    "    return A2, fwd_pass_values\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the first forward pass has been completed and we have our prediction, how do we evaluate its accuracy? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "    \n",
    "* It measures cost associated with an incorrect prediction\n",
    "* Our goal is to find coefficients that minimize the loss function\n",
    "* Cross entropy loss is used in classification problems "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Implement loss function\n",
    "# cost = -(1/m) * Sum(y*log(a^[2](i)) + (1-y)*log(1-a^[2](i)))\n",
    "# A2 - output of sigmoid \n",
    "# Y is a true output against which we'll be measuring the loss\n",
    "\n",
    "def entropy_loss(A2, Y, parameters):\n",
    "    m = Y.shape[1]\n",
    "    log_prob = np.multiply(np.log(A2), Y) + np.multiply(np.log(1 - A2), 1 - Y)\n",
    "    cost = -(1 / m) * np.sum(log_prob)\n",
    "    # squeeze removes axes of length one from cost\n",
    "    cost = float(np.squeeze(cost))\n",
    "    return cost\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backpropagation\n",
    "* Mechanism for tuning the weights based on the loss function\n",
    "* During training we want to find weights and biases that minimize the error (loss function)\n",
    "* To measure change in the loss function, we need to take the derivative of a function with respect to all the weights and biases\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Implement function to measure derivatives\n",
    "# Pass dictionary of parameters, forward propagation values, input data and labeled data\n",
    "\n",
    "def backward_propagation(parameters, fwd_pass_values, X, Y):\n",
    "    m = X.shape[1]\n",
    "    \n",
    "    W1 = parameters[\"W1\"]\n",
    "    W2 = parameters[\"W2\"]\n",
    "    \n",
    "    A1 = fwd_pass_values[\"A1\"]\n",
    "    A2 = fwd_pass_values[\"A2\"]\n",
    "    \n",
    "    # Derivatives of loss func w.r.t parameters\n",
    "    dZ2 = fwd_pass_values[\"A2\"] - Y\n",
    "    dW2 = 1 / m*np.dot(dZ2, fwd_pass_values[\"A1\"].T)\n",
    "    db2 = 1 / m*np.sum(dZ2, axis=1, keepdims=True)\n",
    "    dZ1 = np.dot(W2.T, dZ2)*(1 - np.power(A1, 2))\n",
    "    dW1 = 1 / m*np.dot(dZ1, X.T)\n",
    "    db1 = 1 / m*np.sum(dZ1, axis=1, keepdims=True)\n",
    "    \n",
    "    gradients =       {\"dW1\" : dW1,\n",
    "                       \"db1\" : db1,\n",
    "                       \"dW2\" : dW2,\n",
    "                       \"db2\" : db2}\n",
    "    return gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have derivatives (i.e. sensitivity of the loss function to change in parameters) how do we use them to update our weights and biases in order to decrease our loss?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Descent\n",
    "* Optimization algorithm used to find the values of parameters that minimize a cost function\n",
    "* We can use it to recursively update the weights by iterating over all training samples\n",
    "* It takes into account learning rate and initial parameter values\n",
    "* Learning rate controls size of the step on each iteration\n",
    "* parameter = parameter - learning rate * (derivative of loss function with reference to parameter)\n",
    "* Derivative, slope of loss function, updates the change you want to make to the parameter \n",
    "* Ideally we want Gradient Descent converging to global optimum where derivative equals to zero"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/gradient_nn.JPG'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters - dictionary with randomly initialized parameters \n",
    "# gradients - derivatives from backward_propagation function\n",
    "# parameter = parameter - learning rate * (derivative of loss function w.r.t parameter)\n",
    "\n",
    "\n",
    "def update_parameters(parameters, gradients, learning_rate = 1.1):\n",
    "    W1 = parameters[\"W1\"]\n",
    "    b1 = parameters[\"b1\"]\n",
    "    W2 = parameters[\"W2\"]\n",
    "    b2 = parameters[\"b2\"]\n",
    "    \n",
    "    dW1 = gradients[\"dW1\"]\n",
    "    db1 = gradients[\"db1\"]\n",
    "    dW2 = gradients[\"dW2\"]\n",
    "    db2 = gradients[\"db2\"]\n",
    "    \n",
    "    W1 = W1 - learning_rate * dW1\n",
    "    b1 = b1 - learning_rate * db1\n",
    "    W2 = W2 - learning_rate * dW2\n",
    "    b2 = b2 - learning_rate * db2\n",
    "    \n",
    "    parameters = {\"W1\": W1,\n",
    "                  \"b1\": b1,\n",
    "                  \"W2\": W2,\n",
    "                  \"b2\": b2}\n",
    "    \n",
    "    return parameters\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine functions above and build your first Neural Network model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of X_train: (15122, 40)\n",
      "The shape of y_train: (15122,)\n",
      "The shape of X_test: (3781, 40)\n",
      "The shape of y_test: (3781,)\n"
     ]
    }
   ],
   "source": [
    "# Recall our dataset\n",
    "\n",
    "print ('The shape of X_train: ' + str(X_train.shape))\n",
    "print ('The shape of y_train: ' + str(y_train.shape))\n",
    "print ('The shape of X_test: ' + str(X_test.shape))\n",
    "print ('The shape of y_test: ' + str(y_test.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of X_train_new: (40, 15122)\n",
      "The shape of y_train_new: (1, 15122)\n",
      "The shape of X_test_new: (40, 3781)\n",
      "The shape of y_test_new: (1, 3781)\n"
     ]
    }
   ],
   "source": [
    "# Reshape the data \n",
    "\n",
    "X_train_new = X_train.T\n",
    "y_train_new = y_train.reshape(1, y_train.shape[0])\n",
    "X_test_new = X_test.T\n",
    "y_test_new = y_test.reshape(1, y_test.shape[0])\n",
    "\n",
    "print ('The shape of X_train_new: ' + str(X_train_new.shape))\n",
    "print ('The shape of y_train_new: ' + str(y_train_new.shape))\n",
    "print ('The shape of X_test_new: ' + str(X_test_new.shape))\n",
    "print ('The shape of y_test_new: ' + str(y_test_new.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The size of the input layer is: n_x = 40\n",
      "The size of the hidden layer is: n_h = 4\n",
      "The size of the output layer is: n_y = 1\n"
     ]
    }
   ],
   "source": [
    "# size of input layer\n",
    "n_x = X_train_new.shape[0] # size of input layer\n",
    "# size of hidden layer\n",
    "n_h = 4\n",
    "# size of output layer\n",
    "n_y = y_train_new.shape[0]\n",
    "\n",
    "print(\"The size of the input layer is: n_x = \" + str(n_x))\n",
    "print(\"The size of the hidden layer is: n_h = \" + str(n_h))\n",
    "print(\"The size of the output layer is: n_y = \" + str(n_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use model_parameter functions to initialize parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W1 = [[ 0.00851201 -0.00399024  0.00552071 -0.0086656  -0.00811778 -0.01744902\n",
      "  -0.00034416  0.00288709 -0.00312636  0.00893596 -0.00295572  0.00229517\n",
      "  -0.00195718  0.01885733  0.01297604 -0.00444813  0.00105632  0.00347064\n",
      "   0.00478134  0.00393837  0.00501915  0.01420705  0.01598184  0.00283492\n",
      "   0.00712364  0.01314206 -0.01263475 -0.00674642 -0.0164161   0.00811486\n",
      "   0.00027057 -0.00030224  0.0052736  -0.00429068  0.01988012 -0.00554774\n",
      "   0.01544936  0.0110721   0.02328301 -0.00283042]\n",
      " [ 0.00127137  0.00128385  0.0109509   0.00465384 -0.01742565  0.0059574\n",
      "   0.00892257 -0.00594419 -0.00868817  0.02679305  0.0058514   0.00137697\n",
      "  -0.01300018 -0.00455305 -0.00062089 -0.00811507  0.01661513 -0.00995547\n",
      "   0.00726189 -0.01420131  0.00339887  0.01053909  0.00760746 -0.00423589\n",
      "  -0.01311119 -0.0021165   0.00021947 -0.01339478 -0.00181273  0.01692416\n",
      "  -0.00038578  0.00341538  0.0025004   0.01293408  0.01202054  0.00378627\n",
      "  -0.00567488 -0.01966233 -0.01060219 -0.01320355]\n",
      " [-0.00801126  0.01195058  0.01382346  0.01303184 -0.00840089  0.01193994\n",
      "  -0.00503364 -0.00772795 -0.02266363  0.01172488 -0.01175854  0.0056938\n",
      "  -0.01343675  0.01082077  0.00425555  0.01171949  0.00506156 -0.00224699\n",
      "  -0.00206014 -0.01488327 -0.00625706  0.00212187  0.00489894  0.00893582\n",
      "   0.00031162 -0.00169434  0.00153609 -0.00162913  0.00489701  0.00727912\n",
      "  -0.00130325  0.01107833 -0.01387761  0.00087393  0.00398393 -0.00655454\n",
      "  -0.00647483  0.01169609  0.00683655 -0.00779694]\n",
      " [-0.01172355  0.00652858 -0.00304375 -0.00046671 -0.00179856 -0.02138792\n",
      "  -0.00154952 -0.00056409 -0.01063634  0.00136823  0.0195613   0.01427644\n",
      "   0.00611173  0.01266983 -0.00299923  0.0094449   0.00730629 -0.0124676\n",
      "  -0.00218278 -0.01423015  0.01611099 -0.01950608  0.01534728 -0.00189422\n",
      "  -0.00624407 -0.01054031  0.00541306  0.01095248  0.00387203 -0.0072574\n",
      "   0.00724837 -0.0117629  -0.00830568 -0.00110288 -0.01667909 -0.02430069\n",
      "  -0.00966345  0.00829467 -0.0107286  -0.0149911 ]]\n",
      "b1 = [[0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]]\n",
      "W2 = [[-0.01514068 -0.00358929 -0.00921589 -0.01249919]]\n",
      "b2 = [[0.]]\n"
     ]
    }
   ],
   "source": [
    "parameters = model_parameters(n_x, n_h, n_y)\n",
    "\n",
    "print(\"W1 = \" + str(parameters[\"W1\"]))\n",
    "print(\"b1 = \" + str(parameters[\"b1\"]))\n",
    "print(\"W2 = \" + str(parameters[\"W2\"]))\n",
    "print(\"b2 = \" + str(parameters[\"b2\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Neural Network model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0.6931421828913022\n",
      "1000 0.6284593986689866\n",
      "2000 0.6268111234810758\n",
      "3000 0.6258925304647401\n",
      "4000 0.6253483134733954\n",
      "5000 0.6251010719113644\n",
      "6000 0.6249450636634025\n",
      "7000 0.6247676545207661\n",
      "8000 0.6245918048170128\n",
      "9000 0.6244782967464007\n"
     ]
    }
   ],
   "source": [
    "# Number of iterations used in gradient descent for loop\n",
    "num_iterations = 10000\n",
    "\n",
    "for i in range(0, num_iterations):\n",
    "    \n",
    "    # Apply our forward propagation function\n",
    "    A2, fwd_pass_values = forward_propagation(X_train_new, parameters)\n",
    "    \n",
    "    # Calculate cost associated with an incorrect prediction\n",
    "    cost = entropy_loss(A2, y_train_new, parameters)\n",
    "    \n",
    "    # Apply backpropagation function to measure sensitivity of a loss function to parameters\n",
    "    gradients = backward_propagation(parameters, fwd_pass_values, X_train_new, y_train_new)\n",
    "    \n",
    "    # Update parameters using Gradient descent \n",
    "    parameters = update_parameters(parameters, gradients)\n",
    "    \n",
    "    # Print cost for every 1000th iteration\n",
    "    if i % 1000 == 0:\n",
    "        print(i,cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction  \n",
    "\n",
    "Now that we have our updated parameters that minimize the entropy loss, use forward propagation to make a prediction.\n",
    "\n",
    "A2 is a vector of probabilities, recall it is a sigmoid().\n",
    "\n",
    "if A2 > 0.5 then 1 and 0 otherwise.  A prediction of 1 indicates a predicted increase in earnings.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pass test data into forward_propagation function along with newly optimized parameters\n",
    "A2, fwd_pass_values = forward_propagation(X_test_new, parameters)\n",
    "\n",
    "predictions = (A2 > 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 61%\n"
     ]
    }
   ],
   "source": [
    "# Accuracy\n",
    "\n",
    "print ('Accuracy: %d' % float((np.dot(y_test_new , predictions.T) + np.dot(1 - y_test_new,1 - predictions.T))/float(y_test_new.size)*100) + '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Networks with scikit-learn\n",
    "\n",
    "Now that we have successfully built a NN from scratch, we have a better appreciation for scikit-learn's built-in support for NN models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.60\n"
     ]
    }
   ],
   "source": [
    "# Import accuracy score\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Multi-layer Perceptron classifier contains one or more hidden layers and can learn non-linear functions. \n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "# hidden_layer_sizes allows us to set the number of layers and the number of nodes we wish to have in the Neural Network Classifier\n",
    "# max_iter denotes the number of epochs.\n",
    "# activation function for the hidden layers.\n",
    "# solver specifies the algorithm for weight optimization across the nodes.\n",
    "\n",
    "mlp = MLPClassifier(hidden_layer_sizes = (150,100,50), max_iter=300,activation = 'relu',solver = 'adam', random_state = 0)\n",
    "\n",
    "# Train\n",
    "mlp.fit(X_train,y_train)\n",
    "# Predict \n",
    "y_pred = mlp.predict(X_test)\n",
    "# Accuracy\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print('Accuracy: {:.2f}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Resources\n",
    "\n",
    "#### Helpful Blog Posts\n",
    "Machine Learning for Investing: https://hdonnelly6.medium.com/list/machine-learning-for-investing-7f2690bb1826"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}