Learning the MNIST Database using Shallow Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Neural Network "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Neural Networks are pretty cool for a lot of applications. Here, I am trying to code it myself from my understanding from Professor Ng's course in Coursera and Stanford. The problem is a pretty standard one: classification of numbers. The numbers and labels are provided by NIST.\n",
    "\n",
    "It can be easily pulled from the TensorFlow Keras library. It is also availble on [Professor Yann LeCun's website](http://yann.lecun.com/exdb/mnist/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import struct as st\n",
    "import matplotlib.pyplot as plt\n",
    "import copy\n",
    "%matplotlib inline\n",
    "np.set_printoptions(threshold=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train,y_train), (x_test,y_test) = tf.keras.datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observe the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick any index number from 0 to 60000 to observe the output. y_train contains the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0xb4193b9e8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAN+UlEQVR4nO3dX4hcdZrG8edRZ4jGkeimjUFlMxtzI6tR6QRBGV0kQ/SmlagY4p8F2Ygm4GBAxYXohRdBdAaFVYz/JiOzjiOjmAvdnayOyIiMdkI0UXE10qIx6XSIMAmCWZN3L/pk6Ildv2qrTv1J3u8Hmqo6b506L4c8OVXnV6d+jggBOPod0+sGAHQHYQeSIOxAEoQdSIKwA0kc182NzZw5M+bMmdPNTQKpjIyMaPfu3Z6s1lbYbS+W9LCkYyU9GRFrSs+fM2eOhoeH29kkgILBwcGGtZbfxts+VtJ/SLpc0tmSlto+u9XXA9BZ7XxmXyjp04j4LCL2S/qdpKF62gJQt3bCfrqkLyY8/rJa9ndsL7c9bHt4bGysjc0BaEfHz8ZHxNqIGIyIwYGBgU5vDkAD7YR9u6QzJzw+o1oGoA+1E/Z3Jc2z/VPbP5Z0naT19bQFoG4tD71FxHe2V0r6b40PvT0dER/U1hmAWrU1zh4Rr0h6paZeAHQQX5cFkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgibZmcQV6af/+/cX64sWLG9a2bdtWXPe9994r1mfMmFGs96O2wm57RNJeSQckfRcRg3U0BaB+dRzZ/yUidtfwOgA6iM/sQBLthj0k/dH2RtvLJ3uC7eW2h20Pj42Ntbk5AK1qN+wXR8QFki6XtML2zw5/QkSsjYjBiBgcGBhoc3MAWtVW2CNie3W7S9JLkhbW0RSA+rUcdtvTbf/k0H1JP5e0ta7GANSrnbPxsyS9ZPvQ6/xnRPxXLV3hiLF379626iXTp08v1jdu3Fisv/HGGw1r8+fPL657/PHHF+tHopbDHhGfSSrvMQB9g6E3IAnCDiRB2IEkCDuQBGEHkuAS16PAjh07GtYeeeSR4rojIyNtbbvZ8FezS0lLHnrooWJ9y5YtxXpENKzNmzevuO7BgweL9SMRR3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9qPAW2+91bD2wAMPdHTb06ZNK9Zvv/32hrUXX3yxuO6qVata6umQ6vLrSa1YsaK47tF4iStHdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2I8Cjjz5arN95550tv/Ydd9xRrM+aNatYv+2224r1E044oWGt2Tj6ggULivXR0dFi/bTTTmtYu+iii4rrHo04sgNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyzHwH27dtXrH/zzTcNa2eddVZx3XvvvbdYP/HEE4v1Zvbs2dOwdv/99xfX3blzZ7HebErnxx57rGHtuOPy/dNvemS3/bTtXba3Tlh2iu0Ntj+pbk/ubJsA2jWVt/G/lrT4sGV3S3otIuZJeq16DKCPNQ17RLwp6fD3YkOS1lX310m6sua+ANSs1RN0syLi0ARjOyU1/AK17eW2h20Pj42Ntbg5AO1q+2x8jM+e13AGvYhYGxGDETE4MDDQ7uYAtKjVsI/ani1J1e2u+loC0Amthn29pJuq+zdJermedgB0StPBRtvPSbpU0kzbX0q6V9IaSb+3fbOkzyVd28kms7v22vLufeGFFxrWNm3aVFx39erVxfqaNWuK9W+//bZYL10v/+yzzxbXbfax7+GHHy7Wh4aGivVsmoY9IpY2KF1Wcy8AOoivywJJEHYgCcIOJEHYgSQIO5BEvuv8jkBnnHFGsX7ZZY0HRpoNvTWbNnnp0kaDMeOWLVtWrG/btq1YL2n2E9pLlixp+bUz4sgOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzn4EaPazxzNmzGj5tb/44oti/cILLyzWx3+oqDHbDWvNpppetGhRsY4fhiM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOPtRoNm0zL10/fXXN6ytWrWquO5JJ51UdzupcWQHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZz8CHDx4sFjfsGFDw1qz683bdcMNNxTr69at6+j2MXVNj+y2n7a9y/bWCcvus73d9ubq74rOtgmgXVN5G/9rSYsnWf6riDiv+nul3rYA1K1p2CPiTUl7utALgA5q5wTdStvvV2/zT270JNvLbQ/bHh4bG2tjcwDa0WrYH5M0V9J5knZIeqjREyNibUQMRsTgwMBAi5sD0K6Wwh4RoxFxICIOSnpC0sJ62wJQt5bCbnv2hIdXSdra6LkA+kPTcXbbz0m6VNJM219KulfSpbbPkxSSRiTd0sEe07v11luL9SeffLJhrfS77XXo9OujPk3DHhFLJ1n8VAd6AdBBfF0WSIKwA0kQdiAJwg4kQdiBJLjEtQv27t1brD///PPF+hNPPFGsl4a/LrnkkuK6CxYsKNYffPDBYv2rr74q1tE/OLIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs3fBxo0bi/VbbmnvCuHSOPyyZcuK67799tvFerNx9vnz5xfr6B8c2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcbZa/Dxxx8X60uWLGnr9ZuN059zzjkNa/v27Suuu2LFipZ6OmTu3LltrY/u4cgOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzl6DV199tVj/+uuvi/WrrrqqWD///POL9QMHDjSsvf7668V19+zZU6xHRLE+e/bsYh39o+mR3faZtv9k+0PbH9i+vVp+iu0Ntj+pbk/ufLsAWjWVt/HfSVoVEWdLulDSCttnS7pb0msRMU/Sa9VjAH2qadgjYkdEbKru75X0kaTTJQ1JWlc9bZ2kKzvVJID2/aATdLbnSDpf0l8kzYqIHVVpp6RZDdZZbnvY9vDY2FgbrQJox5TDbvtESX+Q9IuI+OvEWoyfxZn0TE5ErI2IwYgYHBgYaKtZAK2bUtht/0jjQf9tRLxYLR61Pbuqz5a0qzMtAqhD06E3j88H/JSkjyLilxNK6yXdJGlNdftyRzo8AhxzTPn/zNKUylOpl4bWJOmdd95pWLvmmmuK686cObNYv+uuu4r1oaGhYh39Yyrj7BdJukHSFtubq2X3aDzkv7d9s6TPJV3bmRYB1KFp2CPiz5IaHXouq7cdAJ3C12WBJAg7kARhB5Ig7EAShB1IgktcazA6OtrW+qeeemqxfvXVVxfr69evb3nbzS7PveCCC1p+bfQXjuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7DU499xz21r/8ccfL9ab/Zxz6ReAVq9eXVy3NN0zji4c2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcbZa9Dst9OfeeaZYn3lypXF+qJFi4r10m/DX3fddcV1kQdHdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IYirzs58p6TeSZkkKSWsj4mHb90n6N0lj1VPviYhXOtVoP5s2bVqxfuONN7ZVB+owlS/VfCdpVURssv0TSRttb6hqv4qIBzvXHoC6TGV+9h2SdlT399r+SNLpnW4MQL1+0Gd223MknS/pL9Wilbbft/207ZMbrLPc9rDt4bGxscmeAqALphx22ydK+oOkX0TEXyU9JmmupPM0fuR/aLL1ImJtRAxGxGDpt9IAdNaUwm77RxoP+m8j4kVJiojRiDgQEQclPSFpYefaBNCupmG3bUlPSfooIn45YfnsCU+7StLW+tsDUJepnI2/SNINkrbY3lwtu0fSUtvnaXw4bkTSLR3pEEAtpnI2/s+SPEkp5Zg6cKTiG3RAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkHBHd25g9JunzCYtmStrdtQZ+mH7trV/7kuitVXX29o8RMenvv3U17N/buD0cEYM9a6CgX3vr174kemtVt3rjbTyQBGEHkuh12Nf2ePsl/dpbv/Yl0VurutJbTz+zA+ieXh/ZAXQJYQeS6EnYbS+2/bHtT23f3YseGrE9YnuL7c22h3vcy9O2d9neOmHZKbY32P6kup10jr0e9Xaf7e3Vvtts+4oe9Xam7T/Z/tD2B7Zvr5b3dN8V+urKfuv6Z3bbx0r6X0mLJH0p6V1JSyPiw6420oDtEUmDEdHzL2DY/pmkfZJ+ExH/XC17QNKeiFhT/Ud5ckTc1Se93SdpX6+n8a5mK5o9cZpxSVdK+lf1cN8V+rpWXdhvvTiyL5T0aUR8FhH7Jf1O0lAP+uh7EfGmpD2HLR6StK66v07j/1i6rkFvfSEidkTEpur+XkmHphnv6b4r9NUVvQj76ZK+mPD4S/XXfO8h6Y+2N9pe3utmJjErInZU93dKmtXLZibRdBrvbjpsmvG+2XetTH/eLk7Qfd/FEXGBpMslrajervalGP8M1k9jp1OaxrtbJplm/G96ue9anf68Xb0I+3ZJZ054fEa1rC9ExPbqdpekl9R/U1GPHppBt7rd1eN+/qafpvGebJpx9cG+6+X0570I+7uS5tn+qe0fS7pO0voe9PE9tqdXJ05ke7qkn6v/pqJeL+mm6v5Nkl7uYS9/p1+m8W40zbh6vO96Pv15RHT9T9IVGj8jv03Sv/eihwZ9/ZOk96q/D3rdm6TnNP627v80fm7jZkn/IOk1SZ9I+h9Jp/RRb89K2iLpfY0Ha3aPertY42/R35e0ufq7otf7rtBXV/YbX5cFkuAEHZAEYQeSIOxAEoQdSIKwA0kQdiAJwg4k8f8/3S/CFebGuwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "image_index = 17\n",
    "print(y_train[image_index])\n",
    "plt.imshow(x_train[image_index],cmap = 'Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the shape of x_train and x_test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 28, 28)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first flatten the 3D shape of x_train and x_test to a 2D shape where the the first dimension tells us about the number of training images, and the second dimension contains the total number of pixels (stacked)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_flattened = np.reshape(x_train,(x_train.shape[0],-1))\n",
    "x_test_flattened = np.reshape(x_test,(x_test.shape[0],-1))\n",
    "y_train = y_train.reshape(y_train.shape[0],1)\n",
    "y_test = y_test.reshape(y_test.shape[0],1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the data is 'uint8' type, let us convert it to float32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_flattened = x_train_flattened.astype('float32')\n",
    "x_test_flattened = x_test_flattened.astype('float32')\n",
    "y_train = y_train.astype('float32')\n",
    "y_test = y_test.astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we are normalizing the data. The maximum value of the pixel will be 255 (darkest). So divide by 255.\n",
    "\n",
    "I also transpose the matrix because I want the second dimension to be the number of images. This dimnesion should be unaffected by the matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_flattened = x_train_flattened/255\n",
    "x_test_flattened = x_test_flattened/255\n",
    "X_train = x_train_flattened.T\n",
    "X_test = x_test_flattened.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to convert the labels into 1-hot encoded output so that its size matches with the output dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_labels = 10\n",
    "Y_true = np.zeros((n_labels,y_train.shape[0]))\n",
    "Y_test = np.zeros((n_labels,y_test.shape[0]))\n",
    "\n",
    "for i in range(len(y_train)):\n",
    "    Y_true[int(y_train[i])][i] = 1\n",
    "    \n",
    "for i in range(len(y_test)):\n",
    "    Y_test[int(y_test[i])][i] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_true[:,0].reshape(10,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_test[:,0].reshape(10,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us try to form the first nueron that links up the first layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "W1 = np.zeros((1,X_train.shape[0]))\n",
    "B1 = np.zeros((1,X_train.shape[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 60000)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z = np.matmul(W1,X_train)\n",
    "Z.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Professor Ng's notes on Coursera, he uses a row-vector for X and dot multiplies each row of the weights matrix. I prefer to think of the W and X as matrices. Thus, I will keep X as a column vector and use matmul."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to a non-linear activation function. There are various activation functions available like ReLU, tanh, sigmoid, etc. Let us define them below so that we can call them in future functions immediately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(z):\n",
    "    return np.maximum(z,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1.0/(1.0+np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tanh(z):\n",
    "    return np.tanh(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def atan(z):\n",
    "    return np.arctan(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also note that the last layer for multiple classification like our problem will be activated using a softmax function. Lets quickly create the softmax function and then get back to learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(z):\n",
    "    return np.divide(np.exp(z-np.max(z,axis=0)+1e-6),np.sum(np.exp(z-np.max(z,axis=0)+1e-6),axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, we have a (3,2) system. For the real output, we will have a (10,60000) output. This is similar to what we need. The difference here is that we are classifying the data into 3 categories instead of 10. And we have 2 such examples instead of 60000."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined some non-linear functions to activate our single neuron, let us now use one of these functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This concluded the construction of 1 neuron. \n",
    "\n",
    "To create more neurons, we just need to add more rows to the weight matrix. The bias vector will now reflect the number of neurons in the next line. Thus, W has the shape of (number of layers in the next layer, number of inputs), B has the shape of (number of layers in the next layer, number of images). Note here that the second dimension of B can be left as 1, as Python can broadcast the vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "next_layer = 10\n",
    "W1 = np.zeros((next_layer,X_train.shape[0]))\n",
    "B1 = np.zeros((1,X_train.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "Z = np.matmul(W1,X_train) + B1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 60000)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = relu(Z)\n",
    "A.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us try to put all of this in one function which we can call over and over if we want to create a nueral network.\n",
    "\n",
    "What are the things that go into making a layer? We need weights, bias, and activation. Further, we need to know what we got as input and what we need to give as output.\n",
    "\n",
    "Let us also keep a track of the network we create by creating a cache for weights and biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nn_layer(X,next_layer,activation):\n",
    "    W = np.random.rand(next_layer,X.shape[0])\n",
    "    B = np.random.rand(next_layer,1)\n",
    "    Z = np.matmul(W,X) + B\n",
    "    cache = {'W': W, 'B':B}\n",
    "    if activation == 'relu':\n",
    "        return relu(Z)\n",
    "    elif activation == 'tanh':\n",
    "        return tanh(Z)\n",
    "    elif activation == 'atan':\n",
    "        return atan(Z)\n",
    "    elif activation == 'sigmoid':\n",
    "        return sigmoid(Z)\n",
    "    elif activation == 'softmax':\n",
    "        return softmax(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "op = nn_layer(X_train,10,'softmax')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 60000)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "op.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a number of layers, it is not possible to set up a single function which is fast and flexible to accomodate for various neural network architectures. Thus, we have to incorporate a 'for' loop which will pass through the neural network architecture and create each layer and maintain a cache of all weights, biases, activations, etc.\n",
    "\n",
    "To create a full network, we will need the input (of course!), the neural network architecture (number of neurons in each layer, their activations), a final layer to convert the data into the classification labels.\n",
    "\n",
    "Just before we create the network, I want to create a quick function that reduces the if/else statements to one line in the main network. This is the 'activation_function'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def activation_function(z,activation):\n",
    "    if activation == 'relu':\n",
    "        return relu(z)\n",
    "    elif activation == 'tanh':\n",
    "        return tanh(z)\n",
    "    elif activation == 'atan':\n",
    "        return atan(z)\n",
    "    elif activation == 'sigmoid':\n",
    "        return sigmoid(z)\n",
    "    elif activation == 'softmax':\n",
    "        return softmax(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_parameters(X,nn_architecture,nn_activations,learning_rate):\n",
    "    cache = {}\n",
    "    h_params = {'nn_arch': nn_architecture,\n",
    "               'lr': learning_rate,\n",
    "               'nn_acti': nn_activations}\n",
    "    W = np.random.rand(nn_architecture[0],X.shape[0]) * 0.01\n",
    "    B = np.random.rand(nn_architecture[0],1)\n",
    "    cache.update({'W'+str(0): W, 'B'+str(0):B})\n",
    "    for i in range(1,len(nn_architecture)):\n",
    "        W = np.random.rand(nn_architecture[i],nn_architecture[i-1]) * 0.01\n",
    "        B = np.random.rand(nn_architecture[i],1)\n",
    "        cache.update({'W'+str(i): W, 'B'+str(i):B})\n",
    "        \n",
    "    return cache,h_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_nn_network(X,cache,h_params):\n",
    "    A = X\n",
    "    for i in range(len(h_params['nn_arch'])):\n",
    "        X = A\n",
    "        Z = np.matmul(cache['W'+str(i)],X) + cache['B'+str(i)]\n",
    "        A = activation_function(Z,h_params['nn_acti'][i])\n",
    "        cache.update({'A'+str(i): A,\n",
    "                     'Z'+str(i): Z})\n",
    "    return A,cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now, we have successfully completed a forward pass on the network. Now comes the tough part, the learning part. Just before we jump into learning, we need to know if what we obtained was accurate or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a variety of ways to calculate the cost(loss) of the calculation. Boiled down to the basics, what we are trying to do is to reduce this cost function. Thus, using a good cost function is very important. So what are some cost functions?\n",
    "\n",
    "Well, we already know some of them. Mean Absolute Error and Mean Squared Error are the most common ones. We have also seen Cross-Entropy function in most of the Deep Neural Network classes. I will define all of them in a function so that I don't have to think about them over and over again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost_function(Y_pred,Y_true,cost):\n",
    "    if cost == 'mae':\n",
    "        return 1.0/Y_pred.shape[1] * np.sum(np.abs(Y_pred-Y_true),axis = 1,keepdims = True)\n",
    "    elif cost == 'mse':\n",
    "        return 0.5/Y_pred.shape[1] * np.sum((Y_pred-Y_true)**2,axis = 1,keepdims = True)\n",
    "    elif cost == 'cross-entropy':\n",
    "        if Y_pred.shape[0] == 2:\n",
    "            return -1.0/Y_pred.shape[1] * np.sum(np.multiply(Y_true,np.log(Y_pred))+np.multiply((1.0-Y_true),np.log(1.0-Y_pred)),axis = 1, keepdims = True)\n",
    "        else:\n",
    "            return -1.0/Y_pred.shape[1] * np.sum(np.multiply(Y_true,np.log(Y_pred+1e-6)),axis = 1,keepdims = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now focus on the learning. Starting with the 2 neuron to 1 neuron network. Find the derivative of the cost function with respect to the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost_function_derivative(Y_pred,Y_train,cost):\n",
    "    if cost == 'mae':\n",
    "        return 1.0/Y_pred.shape[1]\n",
    "    elif cost == 'mse':\n",
    "        return 1.0/Y_pred.shape[1] * (Y_pred - Y_train)\n",
    "    elif cost == 'cross-entropy':\n",
    "        if Y_pred.shape[0] == 2:\n",
    "            return 1.0/Y_pred.shape[1] * np.divide((Y_pred-Y_train),((1-Y_pred)*(Y_pred)))\n",
    "        else:\n",
    "            return 1.0/Y_pred.shape[1] * np.divide(Y_train,Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def activation_derivative(Z,activation):\n",
    "    if activation == 'relu':\n",
    "        return (lambda X: X>=0)(Z) * 1\n",
    "    elif activation == 'atan':\n",
    "        return 1.0/(Z**2 +1.0)\n",
    "    elif activation == 'tanh':\n",
    "        return 1.0-(tanh(Z))**2\n",
    "    elif activation == 'sigmoid':\n",
    "        return np.multiply(sigmoid(Z),1.0-sigmoid(Z))\n",
    "    elif activation == 'softmax':\n",
    "        return np.multiply(softmax(Z),np.eye(Z.shape[0])-softmax(Z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined the derivative for the cost function, and the derivative for the activation, we need to know the derivative of the linear operation Z = WX + B with respect to W. Obviously, the derivative will be X. Thus, for the final layer, the value of W will be updated using the derivative of the cost function, activation function and the linear operation. For subsequent layers, this process will keep on multiplying to form the required matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward_nn_network(X,Y_pred,Y_true,cost,cache,h_params):\n",
    "    \n",
    "    # Obtain the pre-multiplier (also dB)\n",
    "    if h_params[\"nn_acti\"][-1] == 'softmax' and cost == 'cross-entropy':\n",
    "        pre_mul = Y_pred-Y_true\n",
    "    else:\n",
    "        # First find the derivatives for the cost function\n",
    "        dJ = cost_function_derivative(Y_pred,Y_true,cost)\n",
    "        # Then find the activation derivative for the last layer\n",
    "        dA_dZ = activation_derivative(cache['A'+str(len(h_params['nn_arch'])-1)],h_params['nn_acti'][len(h_params['nn_arch'])-1])\n",
    "        pre_mul = np.matmul(dJ.T,dA_dZ)\n",
    "        \n",
    "    # Obtain the dW and dB\n",
    "    dW = np.matmul(pre_mul,cache['A'+str(len(h_params['nn_arch'])-2)].T)\n",
    "    dB = np.sum(pre_mul,axis=1,keepdims=True)\n",
    "    # Store it in a cache\n",
    "    cache.update({'dW'+str(len(h_params['nn_arch'])-1): dW,\n",
    "                 'dB'+str(len(h_params['nn_arch'])-1): dB})\n",
    "\n",
    "    # Similarly for the middle layers\n",
    "    for i in range(len(h_params['nn_arch'])-2,0,-1):\n",
    "        pre_mul = np.matmul(cache['W'+str(i+1)].T,pre_mul)\n",
    "        dA_dZ = activation_derivative(cache['A'+str(i)],h_params['nn_acti'][i])\n",
    "        pre_mul = np.multiply(pre_mul,dA_dZ)\n",
    "        dW = np.matmul(pre_mul,cache['A'+str(i-1)].T)\n",
    "        dB = np.sum(pre_mul,axis=1,keepdims=True)\n",
    "        cache.update({'dW'+str(i): dW,\n",
    "                     'dB'+str(i): dB})\n",
    "\n",
    "    # Similarly for the first layer\n",
    "    pre_mul = np.matmul(cache['W'+str(1)].T,pre_mul)\n",
    "    dA_dZ = activation_derivative(cache['A'+str(0)],h_params['nn_acti'][0])\n",
    "    pre_mul = np.multiply(pre_mul,dA_dZ)\n",
    "    dW = np.matmul(pre_mul,X.T)\n",
    "    dB = np.sum(pre_mul,axis=1,keepdims=True)\n",
    "    cache.update({'dW'+str(0): dW,\n",
    "                 'dB'+str(0): dB})\n",
    "    \n",
    "    return cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_weights(cache,h_params):\n",
    "    for i in range(len(h_params['nn_arch'])):\n",
    "        cache['W'+str(i)] -= cache['dW'+str(i)] * h_params['lr']\n",
    "        cache['B'+str(i)] -= cache['dB'+str(i)] * h_params['lr']\n",
    "    return cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the tools for a Neural Network. Let us create a run which would compute what we need and see it run for one full iteration on its own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nn_network(X,Y_true,nn_arachitecture,nn_activations,cost,learning_rate,n_epoch):\n",
    "    cache,h_params = initialize_parameters(X,nn_architecture,nn_activations,learning_rate)\n",
    "    cost = []\n",
    "    for i in range(n_epoch):\n",
    "        Y_pred,cache = forward_nn_network(X,cache,h_params)\n",
    "        loss = cost_function(Y_pred,Y_true,cost)\n",
    "        print(np.sum(loss))\n",
    "        cache = backward_nn_network(X,Y_pred,Y_true,cost,cache,h_params)\n",
    "        cache = update_weights(cache,h_params)\n",
    "        cost.append(np.sum(loss))\n",
    "        \n",
    "    return cache,h_params,np.sum(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can try the run the above code as a script."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape of the network is given by 786-25-10. 786 is the input layer, 25 is the hidden one, 10 is the output layer. The size of the output layer is given by the size of Y. The output of each layer goes through a non-linear function to include the non-linearity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.360035293746627\n",
      "4.054977344529237\n",
      "2.7035065436068466\n",
      "2.4040635672401414\n",
      "2.3379261678581913\n",
      "2.3123018534179858\n",
      "2.303891183372144\n",
      "2.301680573881721\n",
      "2.3011533333312473\n",
      "2.301138644182368\n"
     ]
    }
   ],
   "source": [
    "cache1,h_params1 = initialize_parameters(X_train,[25,10],['sigmoid','softmax'],1e-4)\n",
    "cost1 = []\n",
    "for i in range(10):\n",
    "    Y_pred1,cache1 = forward_nn_network(X_train,cache1,h_params1)\n",
    "    loss1 = cost_function(Y_pred1,Y_true,'cross-entropy')\n",
    "    if i % 1 == 0:\n",
    "        print(np.sum(loss1))\n",
    "    cache1 = backward_nn_network(X_train,Y_pred1,Y_true,'cross-entropy',cache1,h_params1)\n",
    "    cache1 = update_weights(cache1,h_params1)\n",
    "    cost1.append(np.sum(loss1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_params1[\"lr\"] = 1e-5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2024218229650654\n",
      "1.1895021298082096\n",
      "1.1768336265588977\n",
      "1.164460110963645\n",
      "1.152432857326249\n",
      "1.1407172183617944\n",
      "1.1292144453059016\n",
      "1.1180074013358532\n",
      "1.107274194755957\n",
      "1.0970087625819676\n",
      "1.0870916405893503\n",
      "1.0774034927843654\n",
      "1.0679130238007575\n",
      "1.0586885934999442\n",
      "1.0497976060448833\n",
      "1.041238150602744\n",
      "1.0329695585842047\n",
      "1.0249231790920428\n",
      "1.0171064135714694\n",
      "1.0094619916688032\n",
      "1.0020070489676645\n",
      "0.9947702230915085\n",
      "0.9876718123503082\n",
      "0.9807602110743803\n",
      "0.9740809412601484\n",
      "0.9676337086050344\n",
      "0.9613969825827418\n",
      "0.955311536368718\n",
      "0.9493306488808592\n",
      "0.9434355423109904\n",
      "0.9376258930230777\n",
      "0.9319032973096005\n",
      "0.926317710956761\n",
      "0.9208755202023449\n",
      "0.9155444231821296\n",
      "0.9103470861129247\n",
      "0.9053730135847858\n",
      "0.900631648159126\n",
      "0.8960637511940569\n",
      "0.8916085125142439\n",
      "0.8872423375412688\n",
      "0.8830208892561872\n",
      "0.8788822650490562\n",
      "0.8748403874485782\n",
      "0.8708317288087327\n",
      "0.8668297681513194\n",
      "0.8628552224020951\n",
      "0.8588636943818276\n",
      "0.8548671642684833\n",
      "0.8508315835612402\n"
     ]
    }
   ],
   "source": [
    "for i in range(1000):\n",
    "    Y_pred1,cache = forward_nn_network(X_train,cache1,h_params1)\n",
    "    loss1 = cost_function(Y_pred1,Y_true,'cross-entropy')\n",
    "    if i % 10 == 0:\n",
    "        print(np.sum(loss1))\n",
    "    cache1 = backward_nn_network(X_train,Y_pred1,Y_true,'cross-entropy',cache1,h_params1)\n",
    "    cache1 = update_weights(cache1,h_params1)\n",
    "    cost1.append(np.sum(loss1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xb4d50d9e8>]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfBElEQVR4nO3deXRc5Znn8e+j3dqtxbIsyZblBTAEvLHZEEwChAaapUNOJ2ESIDB00sk06cmcdJKeTk7nTM/pnOnTCemwhCwdwjAhHZbEDUlYbUzs4CCv4AVb8ipbu2xrsSVb0jN/1JWRRMmSbMmlW/59zqmjuve+VD031/nprfe+b8ncHRERCb+EWBcgIiJjQ4EuIhInFOgiInFCgS4iEicU6CIicSIpVm9cUFDg5eXlsXp7EZFQWrduXZO7F0Y7FrNALy8vp7KyMlZvLyISSma2d6hjGnIREYkTCnQRkTihQBcRiRMKdBGROKFAFxGJEwp0EZE4oUAXEYkToQz0375TS0vH8ViXISIyoYw40M0s0cw2mNkLUY6lmtkvzazKzNaaWflYFtlfQ1snf/3Uej7/5LrxegsRkVAaTQ/9QWDbEMfuAw65+2zgu8B3zrSwoRzv7gXgwOFj4/UWIiKhNKJAN7NS4Gbgx0M0uQ14Inj+DPBRM7MzL29o+ktLIiIDjbSH/j3gq0DvEMdLgP0A7t4NHAHyz7i6KMb594SISGgNG+hmdgvQ4O5nPGhtZg+YWaWZVTY2Np7py4mISD8j6aEvBW41sz3A08BHzOz/DmpzACgDMLMkIAdoHvxC7v64uy9298WFhVG//VFERE7TsIHu7l9391J3Lwc+Cbzu7v9lULPlwN3B8zuDNhrkFhE5i077+9DN7NtApbsvB34CPGlmVUALkeAXEZGzaFSB7u4rgZXB82/2298JfGIsCxu2lrP5ZiIiIRC6laKa4yIiEl3oAl1ERKILbaDrlquIyEChDXQRERkotIGuBaMiIgOFNtA15CIiMlDoAl09cxGR6EIX6CIiEl1oA921tEhEZIDQBrqIiAwUukDvuxlqWjMqIjJA6AK9j4ZcREQGCl2gK8ZFRKILX6BrArqISFShC/Q+ynURkYFCF+gKchGR6EIX6CIiEp0CXUQkToQu0DXkIiISXfgCXRMXRUSiCl2g91Gsi4gMFLpA15CLiEh0wwa6maWZ2Z/MbJOZbTGzf4zS5h4zazSzjcHj/vEpVz1zEZGhJI2gTRfwEXdvN7Nk4A9m9jt3f2tQu1+6+5fGvkQRERmJYQPdI2vt24PN5OARs46ylv6LiEQ3ojF0M0s0s41AA/CKu6+N0uzjZrbZzJ4xs7IhXucBM6s0s8rGxsbTKlhxLiIS3YgC3d173H0+UApcZmYXDWryn0C5u18MvAI8McTrPO7ui919cWFh4ZnUrZujIiKDjGqWi7sfBlYANw7a3+zuXcHmj4FFY1NetBrG65VFRMJtJLNcCs0sN3g+Cbge2D6oTXG/zVuBbWNZ5EBKdBGRaEYyy6UYeMLMEon8AvgPd3/BzL4NVLr7cuBvzOxWoBtoAe4Zr4Lfp2AXEelvJLNcNgMLouz/Zr/nXwe+PralDVXP2XgXEZHwCd9K0ZPP9EeiRUT6C12gv09ddRGR/kIX6BpyERGJLnyBrp65iEhUoQv0Puqpi4gMFLpAV5CLiESnQBcRiROhC3QREYkudIGum6IiItGFL9CV5yIiUYUu0Pso10VEBgptoIuIyEChC3QNuYiIRBe6QBcRkehCF+ia5SIiEl34Al15LiISVegCvY8r2UVEBghdoCvGRUSiC1+gq2cuIhJV6AJdRESiC12gq38uIhJd+AJdiS4iEtWwgW5maWb2JzPbZGZbzOwfo7RJNbNfmlmVma01s/LxKLY/5bqIyEAj6aF3AR9x90uA+cCNZnbFoDb3AYfcfTbwXeA7Y1tmf4pyEZFohg10j2gPNpODx+BUvQ14Inj+DPBRM7Mxq3JAPePxqiIi4TeiMXQzSzSzjUAD8Iq7rx3UpATYD+Du3cARID/K6zxgZpVmVtnY2HhGhSvYRUQGGlGgu3uPu88HSoHLzOyi03kzd3/c3Re7++LCwsLTeQkNuIiIDGFUs1zc/TCwArhx0KEDQBmAmSUBOUDzWBT4wRoiP8dnQEdEJLxGMsul0Mxyg+eTgOuB7YOaLQfuDp7fCbzu47ykU0MuIiIDJY2gTTHwhJklEvkF8B/u/oKZfRuodPflwE+AJ82sCmgBPjleBWvpv4hIdMMGurtvBhZE2f/Nfs87gU+MbWlD1HM23kREJIRCt1K0j3rqIiIDhS7QleMiItGFL9A16CIiElXoAl1ERKILX6Crgy4iElXoAl15LiISXegCvY+CXURkoNAFuma5iIhEF75AV99cRCSq0AW6iIhEF7pA15CLiEh04Qv0WBcgIjJBhS7QT1Kyi4gMELpA7/tSrh6NvYiIDBC+QA9+9vQq0EVE+gtdoPfpVQ9dRGSA8AV6kOPd6qGLiAwQukDvW1ikDrqIyEChC3QREYkudIGunrmISHQKdBGROBG6QBcRkeiGDXQzKzOzFWa21cy2mNmDUdosM7MjZrYxeHxzfMrVAlERkaEkjaBNN/AVd19vZlnAOjN7xd23Dmr3prvfMvYlDuQacxERiWrYHrq717r7+uB5G7ANKBnvwkREZHRGNYZuZuXAAmBtlMNXmtkmM/udmV04xH//gJlVmlllY2PjqIsFDbmIiAxlxIFuZpnAs8CX3b110OH1wAx3vwT4N+DX0V7D3R9398XuvriwsPC0CtaIi4hIdCMKdDNLJhLmT7n7c4OPu3uru7cHz38LJJtZwZhWKiIipzSSWS4G/ATY5u7/OkSbqUE7zOyy4HWbx7LQ96mLLiISzUhmuSwFPgO8Y2Ybg33fAKYDuPtjwJ3AF8ysGzgGfNLHaTqKhlxERKIbNtDd/Q+ADdPmB8APxqooEREZvdCtFO3fQe880ROzOkREJprwBXq/RG89diJ2hYiITDChC/T+jijQRUROCl2gX1CcxTVzI3PYhwr0nl7nRE/v2SxLRCTmRjLLZUKpKMzk72++gDd2NFJz6BiLyyP79zUf5Y2djby5o5E/VjfT1tXNzIIMPre0nLsun0FCwinv64qIhF7oAh1gZkEGKYkJPPnWXt7Y0Ujl3hb2txwDoCR3ErdcMo0pWamsrmriH36zhbd2t/C9v5xPcmLoPpCIiIxYKAM9OTGBv7qmgodXVLGnqYNLy/O4b+lMPjy3kJkFGQRrnPjydXP44apd/PPvtjM1O41/uGVejCsXERk/oQx0gK/ccB5fWDaLtKTEIYdTzIzPXzOLuiOd/OQPu5lflsufXzLtLFcqInJ2hHoMIj0laURj439/8wUsmjGZrz27merG9rNQmYjI2RfqQB+p5MQEfvDpBaQmJ/JXT66j9sixWJckIjLmzolAByjOmcTDn15I7eFjfOy7q/jRql0cPd4d67JERMbMORPoAFfOyuc3X7qKBdMn80+/3caSf36d//PSdhpaO2NdmojIGbNY/Y3OxYsXe2VlZUzeG2Dd3hZ+tGo3L22tIzkhgVvnT+P+q2dy/tTsmNUkIjIcM1vn7oujHQvtLJcztWhGHos+k8eepg5+uno3v6qs4Zl1NVw9p4D7rprJNXMLT05/FBEJg3O2hz7Y4aPH+X9/2scTa/ZQ39rFnCmZ3HfVTG5fUEJacmKsyxMRAU7dQ1egD3K8u5cX3znIj1btZmttK/kZKXzuqpncvaSczNRz9gONiEwQCvTT4O68tauFH66qZuV7jeSmJ3N/EOxZacmxLk9EzlEK9DO0cf9hvv/aTl7f3kDOpGTuu2om9ywtJ1vBLiJnmQJ9jGyuiQT7q9sayE5L4nNXzeTepTPJmaRgF5GzQ4E+xt49cISHXtvJK1vryUpL4t6lM7lv6Uxy0hXsIjK+FOjjZMvBI/zba1X8fksdmalJ3LOknPuumsnkjJRYlyYicUqBPs621bbyb6/v5Lfv1JGRksjdS8q5/+oK8hTsIjLGThXowy79N7MyM1thZlvNbIuZPRiljZnZ982sysw2m9nCsSg8LC4ozuaRuxbx0pc/zLXnT+HRN6q56juv879e2Eq9vlZARM6SYXvoZlYMFLv7ejPLAtYBt7v71n5tbgL+G3ATcDnwkLtffqrXjace+mA769t4ZGU1yzcdJNGMjy8q5fPXVDAjPyPWpYlIyJ1RD93da919ffC8DdgGlAxqdhvwc494C8gNfhGck+YUZfHdv5zPiq8s4xOLS3l2XQ3X/stKHnx6A9vrWmNdnojEqVF926KZlQMLgLWDDpUA+/tt1/DB0MfMHjCzSjOrbGxsHF2lITQ9P51/uuND/OHvruX+qyt4dWs9N37vTe5/opIN+w7FujwRiTMjDnQzywSeBb7s7qfVzXT3x919sbsvLiwsPJ2XCKUp2Wl846YLWP21j/C3182lcm8Ldzyyhk//6C1WVzURqxvTIhJfRhToZpZMJMyfcvfnojQ5AJT12y4N9kk/uekpPHjdHFb/3Uf4nzdfQFVDO3f9eC23P7KGl7fU0durYBeR0zeSWS4G/ATY5u7/OkSz5cBng9kuVwBH3L12DOuMKxmpSdx/dQWrvnot//uOD3Go4zgPPLmOGx9axa83HKC7pzfWJYpICI1klstVwJvAO0Bf0nwDmA7g7o8Fof8D4EbgKHCvu59yCks8z3IZre6eXl58p5aHV1Sxo76dsrxJfP6aWXx8Yam+uldEBtDCopDo7XVe297AD1ZUsWn/YaZkpfJfr67g05dPJ0Nf3SsiKNBDx935Y3UzD6+sYnVVMzmTkrl3aTn3LCknN12rT0XOZQr0ENuw7xCPrKzmla31pKckctfl07n/6gqKstNiXZqIxIACPQ68V9fGoyurWL7pIEkJCdy5uJTPf3gW0/PTY12aiJxFCvQ4sq/5KI+tquaZyhp63Pnzi4v5wrLZnDc1K9alichZoECPQ/Wtnfz4zV08tXYfR4/3cP28Ir547Wzml+XGujQRGUcK9Dh2qOM4P1uzh5+t2cORYydYOjufLy6bzZWz8onMJhWReKJAPwe0d3Xzi7X7ePzNXTS2dTG/LJcvXjubj54/hYQEBbtIvFCgn0M6T/Tw7PoaHnujmv0txzivKIu/vnYWN3+omKTEUX0Xm4hMQAr0c1B3Ty8vbK7lkZWR1afT89Ijq08XlZCapNWnImGlQD+H9fY6r26r5+EVVWyqOUJRdmT16acu0+pTkTBSoAvuzprqZh5eUcWa6mZy05O5d8lM7l4yQ6tPRUJEgS4DrN93iEdWVPPqtnoyUhK564oZ3H/VTKZo9anIhKdAl6i217Xy6Mpq/nPTQZISE/jEolI+f80syvK0+lRkolKgyyntbe7gsTd28ey6yOrTWy+ZxheWzWJukVafikw0CnQZkboj768+PXaih2vmFnLPknKumVuouewiE4QCXUblUMdxfv7HvTy1di8NbV2U56fzmSvL+cTiUrLTkmNdnsg5TYEup+V4dy+/31LHE2v2sG7vIdJTErljQQn3LClnjoZjRGJCgS5n7N0DR/jZmj0s33SQ4929LJmVz12Xz+D6eUWkJGkFqsjZokCXMdPc3sXTb+/nqbf2cvBIJ/kZKdy5qJS/vLSMisLMWJcnEvcU6DLmenqdVTsa+cWf9vHa9gZ6ep0rKvL41GXT+diFU/XHrUXGiQJdxlVDaye/WlfD02/vY3/LMXLTk/mLBaV86rIyjbWLjDEFupwVvb3O6uomnv7Tfl7eWseJHmfRjMl8YlEpN19cTJZmyIicsTMKdDP7KXAL0ODuF0U5vgz4DbA72PWcu397uKIU6PGtqb2LZ9fV8MvK/exq7CAtOYGPXTiVjy8sZensAhI1r13ktJxpoH8YaAd+fopA/x/ufstoilKgnxvcnY37D/PMuhr+c9NBWju7mZqdxh0LS/j4wlJmT9GNVJHROFWgD/v9qe6+yszKx7ooOTeYGQumT2bB9Mn8wy3zeG1bA8+ur+HxVbt4dGU188ty+fiiUm69eBo56RqSETkTIxpDDwL9hVP00J8FaoCDRHrrW4Z4nQeABwCmT5++aO/evadbt4RcQ1snyzce5Jl1NWyvayMlMYHr5k3hjgWlXDO3UHPbRYZwxjdFhwn0bKDX3dvN7CbgIXefM9xrashFIDIks+VgK8+ur2H5xoM0dxxncnoyt1w8jdsXlLBweq7+2LVIP+Ma6FHa7gEWu3vTqdop0GWwEz29/GFnE89tOMDLW+ro6u5lRn46t88v4fYFJcwsyIh1iSIxd0Zj6CN48alAvbu7mV0GJADNZ/q6cu5JTkzg2vOncO35U2jrPMFLW+p5fkMN3399Jw+9tpP5Zbn8xcISbv5QMfmZqbEuV2TCGcksl18Ay4ACoB74FpAM4O6PmdmXgC8A3cAx4L+7+5rh3lg9dBmpuiOdLN90gOc3HGRbbStJCcY1cwu5fUEJ188r0qpUOadoYZHEje11rTy/4QC/2XCQutZOMlOT+LOLpnLHghIur8jX/HaJewp0iTs9vc7aXc08v+EAv3u3jvauyPz22xZM444FJZw/NTvWJYqMCwW6xLXOEz28srWeX284wBs7Gunudc6fmsUdC0q4bX4JU3P0x68lfijQ5ZzR3N7FC5treX7DATbuP4wZLJmVz+3zS7jxoqn6PhkJPQW6nJN2N3Xw/IYD/HrDAfa1HCUtOYGPXlDEn19czLLzpuhmqoSSAl3Oae7O+n2HeX5DDb99p46WjuNkpCRy3bwibrl4Gh+eW0BqksJdwkGBLhLo7unlj7uaeXFzLb/fUsfhoyfISk3i+guLuOXiYq6ara8dkIlNgS4SxYmeXlZXNfHi5lpe2lJHa2c3OZOS+diFRdx88TSWzMonOVHhLhOLAl1kGMe7e3lzZyMvbq7l5a31tHd1Mzk9mRsvmsqNFxVzZUW+eu4yISjQRUah80QPq3Y08uI7tby6tZ6O4z1kpSZx7flTuOHCIq6ZW6jZMhIz4/pdLiLxJi05kRsunMoNF06l80QPq6uaeHlLPa9uq2f5poOkJCawZHY+N8ybynXzpjAlS/PcZWJQD11khHp6nfX7DvHyljpe2lLPvpajmMHC6ZO5YV4R180roqIgQ1/3K+NKQy4iY8zdea++jZe31PPy1jrePdAKwPS8dJadV8iy8wq5sqKASSmaDiljS4EuMs5qDh1lxfYGVr7XyJrqZo6d6CElKYHLZ+ax7LwpLDuvUL13GRMKdJGzqPNED2/vaWHle42sfK+B6sYOAMryJrFsbiTcL6/IJzNVt7Bk9BToIjG0v+UoK3c08sZ7DayuivTeExOMS0pzWDq7gCWzClg4I1erVWVEFOgiE0RXdw/r9hxidXUTq6ua2VxzmF6HtOQELi3PY8msApbOzufCaTn6bneJSoEuMkG1dp5g7a4WVlc1saa6iR317QBkpyVxRUU+l5bncenMPC6clq1VqwJoHrrIhJWdlsz184q4fl4RAA1tnfyxupnVVU28tauFl7fWA5Ee/IKyyVxaPpnF5XksnDFZY/DyAeqhi0xgDa2dvL3nEG/vaaFybwtbD7bS65BgcN7UbOaX5TC/LJdLynKZMyVLwzTnAA25iMSJ9q5uNuw7xNu7W9iw/zCb9h+mtbMbgPSURD5UEgn4vpAvzknTVMk4oyEXkTiRmZrE1XMKuXpOIRBZ4LS7qYNNNYfZuO8wG2uO8O+r93C8pxeAwqxU5pfl8qGSHOYVZzNvWrZCPo4NG+hm9lPgFqDB3S+KctyAh4CbgKPAPe6+fqwLFZEPMjMqCjOpKMzkjgWlQGQmzbbaNjbtP8zGoBf/SjAWD5CbnhwJ9+JsLizJZl5xDhWFGbrpGgdG0kP/GfAD4OdDHP8zYE7wuBx4NPgpIjGQmpR4ctjl7mBfe1c379W1svVgK1trIz+ffGsvXd2RnnxKUgJzizKZW5TF3KIszivKYk5RJiW5k9SbD5FhA93dV5lZ+Sma3Ab83COD8W+ZWa6ZFbt77RjVKCJnKDM1iUUz8lg0I+/kvu6eXnY3dZwM+K21rayuauK59QdOtslISWR2URZzp2Ry3tQs5hRlMbcok6nZGraZiMZiDL0E2N9vuybYp0AXmcCSEhOYUxQJ6dvml5zcf+ToCXY0tLGjvo2d9e3sqG9jxXuN/Gpdzck2WWlJzJ6SSUVBJhWFGVQUZFBRmMmM/HT98e0YOqs3Rc3sAeABgOnTp5/NtxaREcpJT44saCrPG7C/peN4EPJt7KhvZ2dDG3+oauTZ9e8HvRmU5E5iZkEGswozmVmQQUVhBjMLMpiWM4kETascV2MR6AeAsn7bpcG+D3D3x4HHITJtcQzeW0TOkryMFK6oyOeKivwB+9u7utnT1EF1Yzu7mzrY1djB7qYOflW5n47jPSfbpSQmUDp5EmV56UwPHief56drodQYGIv/BZcDXzKzp4ncDD2i8XORc0dmahIXleRwUUnOgP3uTkNbF7saO9jV1M6+lqPsbznKvpajrN93iLZg/nyfvIwUyvLSmTE48PPTmZqdpkVTIzCSaYu/AJYBBWZWA3wLSAZw98eA3xKZslhFZNriveNVrIiEh5lRlJ1GUXYaV87K/8DxI0dPsC8I+L7H/pajbNx/mBffqaWn9/0P8cmJxtScNKblTKJk8iRKcicxLXfgT/0xEa0UFZEJqLunl9ojnQPC/uDhYxw8fIwDh45R19pJ76DoystIYVruwNAvzplEUXYqU7LSmJKdGhc3bLVSVERCJSkxgbJgyGVplOPdPb3Ut3WdDPgDfWF/+Bh7mjtYXdU0YPy+T3ZaEkXZkXAvykpjSnYaU7JSB+0Lb/Ar0EUkdJISEygJhlouLf/gcXentbOb2iPHaGjtor61k4a2LhpaO6lv7aKhrZO1u1tobOs6+TUJ/WWlJlGQlUpBZgoFmakUZqVSkNn3SKEgK5XCYHsiDfUo0EUk7pgZOZOSyZmUzPlTh27n7hw+eoL6ts4Bwd/Y1kVTe+Tnjvo21lQ3c+TYiaivkZGSODDws1L6hX8qhf22M8Z5Jo8CXUTOWWbG5IwUJmeknDL4AY5399Lc0UVT2/FI2Le/H/pN7cdpauuiurGdtbu7OHQ0evinpyRSkJnKZ6+cwf1XV4z5+SjQRURGICUpgeKcyI3W4Zzo6aWl4/jJnn5Te//nXRRmpY5LjQp0EZExlpyYcHLK5tmk78sUEYkTCnQRkTihQBcRiRMKdBGROKFAFxGJEwp0EZE4oUAXEYkTCnQRkTgRs6/PNbNGYO9p/ucFQNMYljOR6NzCSecWTmE8txnuXhjtQMwC/UyYWeVQ3wccdjq3cNK5hVO8nZuGXERE4oQCXUQkToQ10B+PdQHjSOcWTjq3cIqrcwvlGLqIiHxQWHvoIiIyiAJdRCROhC7QzexGM3vPzKrM7Guxrme0zKzMzFaY2VYz22JmDwb788zsFTPbGfycHOw3M/t+cL6bzWxhbM/g1Mws0cw2mNkLwfZMM1sb1P9LM0sJ9qcG21XB8fJY1j0cM8s1s2fMbLuZbTOzK+Pomv1t8G/xXTP7hZmlhfW6mdlPzazBzN7tt2/U18nM7g7a7zSzu2NxLqcjVIFuZonAw8CfAfOAT5nZvNhWNWrdwFfcfR5wBfDF4By+Brzm7nOA14JtiJzrnODxAPDo2S95VB4EtvXb/g7wXXefDRwC7gv23wccCvZ/N2g3kT0E/N7dzwcuIXKOob9mZlYC/A2w2N0vAhKBTxLe6/Yz4MZB+0Z1ncwsD/gWcDlwGfCtvl8CE567h+YBXAm81G/768DXY13XGZ7Tb4DrgfeA4mBfMfBe8PyHwKf6tT/ZbqI9gFIi/4f5CPACYERW4SUNvn7AS8CVwfOkoJ3F+hyGOK8cYPfg+uLkmpUA+4G84Dq8AHwszNcNKAfePd3rBHwK+GG//QPaTeRHqHrovP+Pr09NsC+Ugo+rC4C1QJG71waH6oCi4HmYzvl7wFeB3mA7Hzjs7t3Bdv/aT55XcPxI0H4imgk0Av8eDCf92MwyiINr5u4HgH8B9gG1RK7DOuLjuvUZ7XUKzfUbLGyBHjfMLBN4Fviyu7f2P+aRbkGo5pOa2S1Ag7uvi3Ut4yAJWAg86u4LgA7e/9gOhPOaAQRDCbcR+aU1Dcjgg0MWcSOs12mkwhboB4Cyftulwb5QMbNkImH+lLs/F+yuN7Pi4Hgx0BDsD8s5LwVuNbM9wNNEhl0eAnLNLClo07/2k+cVHM8Bms9mwaNQA9S4+9pg+xkiAR/2awZwHbDb3Rvd/QTwHJFrGQ/Xrc9or1OYrt8AYQv0t4E5wR34FCI3b5bHuKZRMTMDfgJsc/d/7XdoOdB3N/1uImPrffs/G9yRvwI40u/j44Th7l9391J3LydyXV5397uAFcCdQbPB59V3vncG7Sdkz8nd64D9ZnZesOujwFZCfs0C+4ArzCw9+LfZd26hv279jPY6vQTcYGaTg08wNwT7Jr5YD+Kfxg2Pm4AdQDXw97Gu5zTqv4rIR77NwMbgcRORccjXgJ3Aq0Be0N6IzOypBt4hMhsh5ucxzDkuA14InlcAfwKqgF8BqcH+tGC7KjheEeu6hzmn+UBlcN1+DUyOl2sG/COwHXgXeBJIDet1A35B5F7ACSKfrO47nesEfC44xyrg3lif10gfWvovIhInwjbkIiIiQ1Cgi4jECQW6iEicUKCLiMQJBbqISJxQoIuIxAkFuohInPj/Sxns8tMdVs0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.]\n",
      "[0 0 0 1 0 0 0 0 0 0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0xb4d636ac8>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAO3klEQVR4nO3df4xV9ZnH8c8jSqIWI+6MSKhKbUwMWSNtTsiqqF3JNuo/A8ZoialIGsFEkzY2ZpGVwB/+AZvF6h/YhK6mgF1NtUUnwdQqNiFNTOOFsPLD7KJmEAgwY0TBGOIyPPvHHM0U5nzvcM+591x43q9kcu+c5557nrmZz5w753vP+Zq7C8C577y6GwDQGYQdCIKwA0EQdiAIwg4EcX4nN9bT0+PTp0/v5CaBUAYGBvTpp5/aWLVSYTezOyQ9K2mCpP9095Wpx0+fPl2NRqPMJgEkZFlWWGv5bbyZTZC0RtKdkmZImm9mM1p9PgDtVeZ/9lmSPnT3j939a0kvS+qrpi0AVSsT9mmS9o36fn++7O+Y2SIza5hZY2hoqMTmAJTR9qPx7r7W3TN3z3p7e9u9OQAFyoT9gKQrR33/3XwZgC5UJuzvSbrWzL5nZhMl/URSfzVtAahay0Nv7n7CzB6V9KZGht5ecPddlXUGoFKlxtnd/Q1Jb1TUC4A24uOyQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBFFqFldU48svvyxVX79+fWFtw4YNyXV3796drLt7sm5myfprr71WWLvllluS61566aXJOs5MqbCb2YCkY5KGJZ1w96yKpgBUr4o9+z+7+6cVPA+ANuJ/diCIsmF3SX82s61mtmisB5jZIjNrmFljaGio5OYAtKps2Ge7+w8l3SnpETO79dQHuPtad8/cPevt7S25OQCtKhV2dz+Q3w5K2ihpVhVNAahey2E3s4vNbNI39yX9WNLOqhoDUK0yR+OnSNqYj7OeL+m/3P1PlXTVhT755JPC2mOPPVbqubdu3Zqs79u3r9TzpzQbJy+7/rx58wprkydPTq572223JevLly9P1q+//vpkPZqWw+7uH0u6ocJeALQRQ29AEIQdCIKwA0EQdiAIwg4EwSmuuWbDX3PmzCmsNTsFtZmyp5GW0dPTk6w3+9mOHz/e8raPHDmSrKdOj5Wk/v7+ZP3xxx8vrD355JPJdS+88MJk/WzEnh0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgmCcPff1118n62XH0ttp4sSJhbXFixcn1125cmWy3uxSYm+//XayvnHjxsLapk2bkus2Mzw8nKyvWrWqsNbsswtPPfVUSz11M/bsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxCENTuXukpZlnmj0ejY9s7Eu+++m6w3m164jL6+vmR94cKFyfoll1xSWLv11tMm6emokydPFtZWr16dXPeJJ55I1stcB+CKK65Irrt///5kvVtlWaZGozHmD86eHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeC4Hz23I033pisL126tLD25ptvJtdds2ZNsp5lWbJ+rmo2Tl62nvLss8+2vO7Zqume3cxeMLNBM9s5atllZvaWme3Jb9MTbQOo3Xjexv9W0h2nLFsiabO7Xytpc/49gC7WNOzuvkXSZ6cs7pO0Lr+/TtLcivsCULFWD9BNcfeD+f1DkqYUPdDMFplZw8waza5nBqB9Sh+N95GjJIVHStx9rbtn7p719vaW3RyAFrUa9sNmNlWS8tvB6loC0A6thr1f0oL8/gJJr1fTDoB2aXo+u5m9JOlHknokHZa0XNJrkn4v6SpJeyXd6+6nHsQ7TTefz47WHD16NFlPfcZg2bJlpbbd7Hf37rvvLqy9/PLLyXUnTJjQUk91S53P3vRDNe4+v6A0p1RXADqKj8sCQRB2IAjCDgRB2IEgCDsQBKe4ngNSQ1BfffVVqed++umnk/VnnnkmWf/iiy9KbT/lhhtuSNafe+65wtrZOrRWBnt2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcfYKHD9+PFnfu3dvsl72ssap7a9fv77Uc5eZFrndtm3bVtu2z0bs2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMbZK3Deeem/mffcc0+yvnv37mS9zrHsbtbX15esv/jii4W1SZMmVd1O12PPDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM5egYkTJybr11xzTbK+a9euKtup1LRp05L1qVOnJuv79+8vrB06dKilnr7R39+frD/88MOFtQ0bNiTXbfbZibNR05/IzF4ws0Ez2zlq2QozO2Bm2/Ovu9rbJoCyxvPn67eS7hhj+a/cfWb+9Ua1bQGoWtOwu/sWSZ91oBcAbVTmH5NHzez9/G3+5KIHmdkiM2uYWWNoaKjE5gCU0WrYfy3p+5JmSjooaXXRA919rbtn7p719va2uDkAZbUUdnc/7O7D7n5S0m8kzaq2LQBVaynsZjZ6vGWepJ1FjwXQHWwc1wV/SdKPJPVIOixpef79TEkuaUDSYnc/2GxjWZZ5o9Eo1fDZ6NixY8n6kiVL2rbtuXPnJutXX311st5sHL3ZeeGDg4OFtZtuuim57sDAQLJe5pr2R48eTa570UUXJevdKssyNRqNMX/wph+qcff5Yyx+vnRXADrq3PuYEIAxEXYgCMIOBEHYgSAIOxAEp7h2QLPhqTVr1nSok867/PLLC2vLly9Prrtw4cKq2/nWq6++mqw/8MADbdt2XdizA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQjLOjrYaHhwtrzS7B3U6p8f9zFXt2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcXa01caNGwtr999/f1u3PWPGjMLa7bff3tZtdyP27EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBOPsZ4HPP/88WT9+/HjLz93T05Osn39++ldk3759yfqDDz5YWGs25XIzJ0+eTNZT14av81z6ujTds5vZlWb2FzPbbWa7zOzn+fLLzOwtM9uT305uf7sAWjWet/EnJP3S3WdI+idJj5jZDElLJG1292slbc6/B9Clmobd3Q+6+7b8/jFJH0iaJqlP0rr8YeskzW1XkwDKO6MDdGY2XdIPJP1N0hR3P5iXDkmaUrDOIjNrmFljaGioRKsAyhh32M3sO5L+IOkX7n50dM1HjrSMebTF3de6e+buWW9vb6lmAbRuXGE3sws0EvTfufsf88WHzWxqXp8qabA9LQKoQtOhNzMzSc9L+sDdnx5V6pe0QNLK/Pb1tnR4DtixY0eyvmLFimR9y5YtyfqRI0fOtKVvNTvVc/bs2cn6pk2bkvXUsODIr1br7rvvvmT9qquuKvX855rxjLPfLOmnknaY2fZ82VKNhPz3ZvYzSXsl3dueFgFUoWnY3f2vkor+BM+pth0A7cLHZYEgCDsQBGEHgiDsQBCEHQiCU1wr8NFHHyXrWZYl6ydOnEjWy45Hp7zzzjvJ+ubNm5P1dvY2d276dIsNGzYk6xdccEGV7Zz12LMDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCMs1egv78/WR8eHi71/DfffHOyPjhYfN2QPXv2lNp2WalLNi9evDi57sqVK5N1xtHPDHt2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcfYKzJs3L1lftWpVsr5s2bJk/aGHHkrWU+fDDwwMJNd95ZVXkvVmrrvuumR9zpziCxA3my4a1WLPDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBmLunH2B2paT1kqZIcklr3f1ZM1sh6SFJQ/lDl7r7G6nnyrLMG41G6aYBjC3LMjUajTEv5j+eD9WckPRLd99mZpMkbTWzt/Lar9z9P6pqFED7jGd+9oOSDub3j5nZB5KmtbsxANU6o//ZzWy6pB9I+lu+6FEze9/MXjCzyQXrLDKzhpk1hoaGxnoIgA4Yd9jN7DuS/iDpF+5+VNKvJX1f0kyN7PlXj7Weu69198zds97e3gpaBtCKcYXdzC7QSNB/5+5/lCR3P+zuw+5+UtJvJM1qX5sAymoadhuZpvN5SR+4+9Ojlk8d9bB5knZW3x6AqoznaPzNkn4qaYeZbc+XLZU038xmamQ4bkBS+rrAAGo1nqPxf5U01rhdckwdQHfhE3RAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgml5KutKNmQ1J2jtqUY+kTzvWwJnp1t66tS+J3lpVZW9Xu/uY13/raNhP27hZw92z2hpI6NbeurUvid5a1aneeBsPBEHYgSDqDvvamref0q29dWtfEr21qiO91fo/O4DOqXvPDqBDCDsQRC1hN7M7zOx/zOxDM1tSRw9FzGzAzHaY2XYzq3V+6XwOvUEz2zlq2WVm9paZ7clvx5xjr6beVpjZgfy1225md9XU25Vm9hcz221mu8zs5/nyWl+7RF8ded06/j+7mU2Q9L+S/kXSfknvSZrv7rs72kgBMxuQlLl77R/AMLNbJX0pab27/2O+7N8lfebuK/M/lJPd/V+7pLcVkr6sexrvfLaiqaOnGZc0V9KDqvG1S/R1rzrwutWxZ58l6UN3/9jdv5b0sqS+Gvroeu6+RdJnpyzuk7Quv79OI78sHVfQW1dw94Puvi2/f0zSN9OM1/raJfrqiDrCPk3SvlHf71d3zffukv5sZlvNbFHdzYxhirsfzO8fkjSlzmbG0HQa7046ZZrxrnntWpn+vCwO0J1utrv/UNKdkh7J3652JR/5H6ybxk7HNY13p4wxzfi36nztWp3+vKw6wn5A0pWjvv9uvqwruPuB/HZQ0kZ131TUh7+ZQTe/Hay5n2910zTeY00zri547eqc/ryOsL8n6Voz+56ZTZT0E0n9NfRxGjO7OD9wIjO7WNKP1X1TUfdLWpDfXyDp9Rp7+TvdMo130TTjqvm1q336c3fv+JekuzRyRP4jSf9WRw8FfV0j6b/zr1119ybpJY28rfs/jRzb+Jmkf5C0WdIeSW9LuqyLetsgaYek9zUSrKk19TZbI2/R35e0Pf+6q+7XLtFXR143Pi4LBMEBOiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0I4v8BC6igx9tnXtQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "image_index = 25377\n",
    "print(y_train[image_index])\n",
    "print((Y_pred1[:,image_index] >= np.max(Y_pred1[:,image_index])) * 1)\n",
    "plt.imshow(x_train[image_index],cmap = 'Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the accuracy of the network, we can take the values of the output and assign 1 to the highest output and make all others 0. Now, if we subtract that from the true values, we should be left with 0s whenever the answer is correct. If the prediction is incorrect, we will be left with 2 ones (+1 and -1) in that column. But that does not mean we are twice wrong. So to counter this problem, we take the absolute and divide by 2. Now, we can add all the entries in one column and do that for the size of the dataset. Thus, if the prediciton is correct, we will have a sum of 0, if not, we will sum to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$ output =\n",
    "\\left\\{\n",
    "\t\\begin{array}{ll}\n",
    "\t\t0  & \\mbox{if } prediction = true value \\\\\n",
    "\t\t1 & \\mbox{if } prediction \\neq true value\n",
    "\t\\end{array}\n",
    "\\right.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if the answers are correct, the particular value we described should be 0 or close to it. Accuracy, however, is measured as the correctness of the solution and generally, higher the accuracy, the better the solution. In our case, we do not have that (answer tends to 0 when we are correct and tends to 1 when we are not). To flip the thing around, subtract the particular value from 1 (100%). Now, if we are correct, the answer will be 1 or close to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72.37\n"
     ]
    }
   ],
   "source": [
    "accuracy_train = (1-np.sum(np.abs((Y_pred1 >= np.max(Y_pred1,axis=0)) * 1 - Y_true))/2/Y_true.shape[1]) * 100\n",
    "print(accuracy_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, a simple Neural Network with one hidden layer of 25 neurons, which I ran for about 1000 passes gave me an accuracy of 72.37% on the training set. That is pretty good. Can we do better? Surely. What can we do to make it better? There are too many hyperparameters to play with.\n",
    "\n",
    "But first, let us see if this network does any good on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we are going to take the [786,10000] test set and run it through the Neural Network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred_test,cache_test = forward_nn_network(X_test,cache1,h_params1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2.]\n",
      "[0 0 1 0 0 0 0 0 0 0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0xb4d732be0>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAANuElEQVR4nO3df6jUdb7H8dfbWvullV0HOWSsmwURF68rg1xY2Yxll0zINigyWLwouJCFgsSNvZb9Ef247br0Ry2d3WTdZWsT3Egw9m5XhJBgOaNYWta14pTaUUf6sW6/tqPv+8f5uhz1zGfmzPc78x19Px8wzMz3Pd/5vhnO63xnvp+Z78fcXQDOfRPKbgBAdxB2IAjCDgRB2IEgCDsQxPnd3NjUqVN9xowZ3dwkEMrg4KCOHj1qY9Vyhd3MbpL0pKTzJP3G3R9LPX7GjBmq1Wp5NgkgoVqtNqy1/TbezM6T9JSkBZKul7TYzK5v9/kAdFaez+xzJb3r7u+7+z8k/VHSomLaAlC0PGG/UtL+UfcPZMtOYWbLzaxmZrV6vZ5jcwDy6PjReHfvd/equ1crlUqnNweggTxhPyjpqlH3p2fLAPSgPGEfkHStmX3HzCZKulPS5mLaAlC0tofe3H3YzO6R9D8aGXpb7+5vFtYZgELlGmd395clvVxQLwA6iK/LAkEQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EESuWVxx9vv6669z1SdMSO8vJk2aNO6e0Bm5wm5mg5KOSTouadjdq0U0BaB4RezZb3T3owU8D4AO4jM7EETesLukv5jZDjNbPtYDzGy5mdXMrFav13NuDkC78oZ9nrvPkbRA0goz+/7pD3D3fnevunu1Uqnk3ByAduUKu7sfzK6PSHpR0twimgJQvLbDbmaXmNnkk7cl/UjSnqIaA1CsPEfjp0l60cxOPs9z7v7nQrrCKQ4dOpSsv/POOw1rzzzzTHLdPXvS/5+b1S+77LJkfeHChQ1rTz31VK7nxvi0HXZ3f1/SvxXYC4AOYugNCIKwA0EQdiAIwg4EQdiBIPiJawG+/PLLZP3hhx9O1tetW5esf/PNN8n6iRMnkvVO+uyzz5L15557rmFtYGAgue7u3buT9YkTJybrOBV7diAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgnH2ArzwwgvJ+qOPPtqlTs40ZcqUZL3ZqZ5vu+22ZP3uu+9O1m+88caGtX379iXXffzxx5P1Bx54IFnHqdizA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQjLMXoNl4cKetXr26YW3NmjXJdTt9uuarr766Ye2jjz5Krrtly5ZknXH28WHPDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM5egKVLlybr69evT9anT5+erD/xxBPJ+qxZsxrWJkw4e/+fp34Lj/Fr+pdgZuvN7IiZ7Rm17Aoze8XM9mXX6TMkAChdK//2fyvpptOW3S9pq7tfK2lrdh9AD2sadnd/VdLHpy1eJGlDdnuDpFsL7gtAwdr9QDfN3Yey24ckTWv0QDNbbmY1M6vV6/U2Nwcgr9xHb9zdJXmi3u/uVXevViqVvJsD0KZ2w37YzPokKbs+UlxLADqh3bBvlrQku71E0kvFtAOgU5qOs5vZ85LmS5pqZgckrZX0mKSNZrZM0geS7uhkk73uvvvuS9bvvffeZP3CCy8ssp2uev3115P1Dz/8sO3nvuuuu9peF2dqGnZ3X9yg9IOCewHQQWfv16sAjAthB4Ig7EAQhB0IgrADQfAT1y44m4fWjh07lqzPmzcvWf/888/b3vY111zT9ro4E3t2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcXYk9ff3J+t5xtFXrlyZrJ/N30/oRezZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIxtmDa/Z79UceeaRj2168uNGJi0eYWce2HRF7diAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgnH2s8Dbb7+drG/cuLFhbf/+/cl1N23alKx/+umnyXoea9euTdZnzZqVrK9atSpZ7+vrG3dP57Kme3YzW29mR8xsz6hlD5nZQTPblV1u7mybAPJq5W38byXdNMbyX7r77OzycrFtASha07C7+6uSPu5CLwA6KM8BunvM7I3sbf6URg8ys+VmVjOzWr1ez7E5AHm0G/ZfSZopabakIUm/aPRAd+9396q7VyuVSpubA5BXW2F398PuftzdT0j6taS5xbYFoGhthd3MRo9p/FjSnkaPBdAbzN3TDzB7XtJ8SVMlHZa0Nrs/W5JLGpT0U3cfaraxarXqtVotV8Nno6+++ipZX7NmTbL+5JNPJuvHjx8fd0/ngosuuihZnz17dsPakiVLkuvecsstyfrll1+erJd1zvtqtaparTbmiQCafqnG3cc6w8CzubsC0FV8XRYIgrADQRB2IAjCDgRB2IEgmg69FelsHno7ceJEw9rg4GBy3WZTE2/ZsqWdlrqi2RDSwoULk/UdO3Y0rB04cCC57vDwcLJepuuuuy5Zz/Pz2q1bt7a9bmrojT07EARhB4Ig7EAQhB0IgrADQRB2IAjCDgTBqaRbtH379oa1+fPnd6+Rgs2ZMydZ37ZtW7I+efLktrf93nvvJet79+5N1pv9TPWTTz4Zd0+tanZ672b1MrBnB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgGGdv0YIFC8puoaHUaY2XLl2aXPfBBx9M1vOMozczc+bMXPVmv4ffuXNnw9rAwEBy3Q0bNiTreTWbjroT2LMDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCcN75FZmOeirtprQjNpgd+7bXXGtaand8c55Zc5403s6vMbJuZvWVmb5rZymz5FWb2ipnty66nFN04gOK08jZ+WNJqd79e0r9LWmFm10u6X9JWd79W0tbsPoAe1TTs7j7k7juz28ck7ZV0paRFkk5+p3CDpFs71SSA/MZ1gM7MZkj6rqS/Sprm7kNZ6ZCkaQ3WWW5mNTOr1ev1HK0CyKPlsJvZJEmbJK1y97+NrvnIUb4xj/S5e7+7V929WqlUcjULoH0thd3MvqWRoP/B3f+ULT5sZn1ZvU/Skc60CKAITX/iaiPjSs9K2uvu60aVNktaIumx7PqljnTYI1asWNGw9vTTTyfXveGGG5L122+/PVlftmxZsn7BBRck64DU2u/ZvyfpJ5J2m9mubNnPNBLyjWa2TNIHku7oTIsAitA07O6+XVKjb438oNh2AHQKX5cFgiDsQBCEHQiCsANBEHYgCH7i2qLh4eGGtS+++CK57sUXX5ysn38+Z/RGMXL9xBXAuYGwA0EQdiAIwg4EQdiBIAg7EARhB4JggLdFqbHwSy+9tIudAO1hzw4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBNA27mV1lZtvM7C0ze9PMVmbLHzKzg2a2K7vc3Pl2AbSrlZNXDEta7e47zWyypB1m9kpW+6W7/7xz7QEoSivzsw9JGspuHzOzvZKu7HRjAIo1rs/sZjZD0ncl/TVbdI+ZvWFm681sSoN1lptZzcxq9Xo9V7MA2tdy2M1skqRNkla5+98k/UrSTEmzNbLn/8VY67l7v7tX3b1aqVQKaBlAO1oKu5l9SyNB/4O7/0mS3P2wux939xOSfi1pbufaBJBXK0fjTdKzkva6+7pRy/tGPezHkvYU3x6AorRyNP57kn4iabeZ7cqW/UzSYjObLcklDUr6aUc6BFCIVo7Gb5c01nzPLxffDoBO4Rt0QBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIMzdu7cxs7qkD0YtmirpaNcaGJ9e7a1X+5LorV1F9vZtdx/z/G9dDfsZGzeruXu1tAYSerW3Xu1Lord2das33sYDQRB2IIiyw95f8vZTerW3Xu1Lord2daW3Uj+zA+iesvfsALqEsANBlBJ2M7vJzN4xs3fN7P4yemjEzAbNbHc2DXWt5F7Wm9kRM9szatkVZvaKme3LrsecY6+k3npiGu/ENOOlvnZlT3/e9c/sZnaepP+T9ENJByQNSFrs7m91tZEGzGxQUtXdS/8Chpl9X9LfJf3O3f81W/bfkj5298eyf5RT3P0/e6S3hyT9vexpvLPZivpGTzMu6VZJ/6ESX7tEX3eoC69bGXv2uZLedff33f0fkv4oaVEJffQ8d39V0senLV4kaUN2e4NG/li6rkFvPcHdh9x9Z3b7mKST04yX+tol+uqKMsJ+paT9o+4fUG/N9+6S/mJmO8xsednNjGGauw9ltw9JmlZmM2NoOo13N502zXjPvHbtTH+eFwfozjTP3edIWiBpRfZ2tSf5yGewXho7bWka724ZY5rxfyrztWt3+vO8ygj7QUlXjbo/PVvWE9z9YHZ9RNKL6r2pqA+fnEE3uz5Scj//1EvTeI81zbh64LUrc/rzMsI+IOlaM/uOmU2UdKekzSX0cQYzuyQ7cCIzu0TSj9R7U1FvlrQku71E0ksl9nKKXpnGu9E04yr5tSt9+nN37/pF0s0aOSL/nqT/KqOHBn1dLen17PJm2b1Jel4jb+u+0cixjWWS/kXSVkn7JP2vpCt6qLffS9ot6Q2NBKuvpN7maeQt+huSdmWXm8t+7RJ9deV14+uyQBAcoAOCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIP4fQeA8LaSSYU0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "image_index = 6754\n",
    "print(y_test[image_index])\n",
    "print((Y_pred_test[:,image_index] >= np.max(Y_pred_test[:,image_index])) * 1)\n",
    "plt.imshow(x_test[image_index],cmap = 'Greys')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72.88\n"
     ]
    }
   ],
   "source": [
    "accuracy_test = (1-np.sum(np.abs((Y_pred_test >= np.max(Y_pred_test,axis=0)) * 1 - Y_test))/2/Y_test.shape[1]) * 100\n",
    "print(accuracy_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, our Neural Network is predicting the numbers with 72.88% accuracy for the test set. That is incredible!\n",
    "\n",
    "Note here that I did play around with the learning rate. I started off with $lr = 10^{-4}$ and then changed it to $lr=10^{-5}$ as I felt the weights were going in the right direction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next challenge is to improve this model to get higher accuracy."
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}