explain-02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "53dc3931-6869-47c0-90b1-570d6afe020b",
   "metadata": {},
   "source": [
    "# Let's think of a very simple classification problem\n",
    "## i.e. you have a set of images (tiny ones, say 25x20)\n",
    "## The images are split into 10 different categories\n",
    "## So your NN takes in 500 inputs (25*20) and has 10 outputs\n",
    "## Feed in each pixel value of the image to its corresponding \"row\"\n",
    "## You want the NN to output a \"1.0\" in exactly one of the outputs depending on which category the image belongs to\n",
    "## So let's use tanh(x) as an activation function. Why? because it clamps the inputs down between -1.0 and 1.0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a72f6b3c-1d6a-4251-a1f6-726033a5d0f6",
   "metadata": {},
   "source": [
    "### Create a matrix (10,500) and fill it with random values between 0 and 1.0 (sometimes -1.0 to 1.0 works too!)\n",
    "### for each image, simply do a tanh(matrix*vector) and see what you get.\n",
    "### Now you can create a \"LOSS\" function which is how far your NN is from the KNOWN value.\n",
    "#### i.e. for each test image, you know exactly which category it should be.\n",
    "#### Thus you can calculate how far off the answer that you get is, from the answer that you want,\n",
    "#### you sum that error value up for each test image\n",
    "#### and that is the \"LOSS\" function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4d35675-b9dc-42e1-902b-48e889507870",
   "metadata": {},
   "source": [
    "## So are we done?\n",
    "### Do we just plug in the matrix * vector into a symbolic math package and have it calculate the differentiation???"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bc6d47e-74bf-4c6f-9ce1-fc63a96a0d58",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import math\n",
    "import sympy as yp\n",
    "import scipy as sp\n",
    "from sympy.abc import x,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94500da5-9528-4278-b763-2f7c3641f6b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = yp.Symbol('x')\n",
    "x, y, c = yp.symbols('x y c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f066fb45-2b95-4bf7-8a18-99c18c03fde2",
   "metadata": {},
   "outputs": [],
   "source": [
    "(x+y)*(x+y)*(x+y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4d3c2df-d5c0-4b51-aa03-7e31217bdcb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "c = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b83ea35b-d166-45fa-b780-a5669befec8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "z = (yp.exp(x) - yp.exp(-x))/(yp.exp(x)+yp.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "386c7bae-2755-45cc-9afe-9911413daa46",
   "metadata": {},
   "outputs": [],
   "source": [
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95c1fe70-55ad-4b2d-aa6c-383ed0b1743f",
   "metadata": {},
   "outputs": [],
   "source": [
    "yp.diff(y,x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "767cc71f-a848-4220-9b7d-048b11f4183b",
   "metadata": {},
   "outputs": [],
   "source": [
    "z2 = yp.diff(z, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0eef0a8-2214-4863-85de-83b2e446e001",
   "metadata": {},
   "outputs": [],
   "source": [
    "z2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a78ead4a-94bf-4b19-9cd6-9f2dc0188f67",
   "metadata": {},
   "outputs": [],
   "source": [
    "yp.simplify(z2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3377b3f5-19f1-4c3f-bbd6-843ae3520618",
   "metadata": {},
   "outputs": [],
   "source": [
    "q = 1 - z**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59a94d1f-0c42-4cd9-aeae-1b6932e7575e",
   "metadata": {},
   "outputs": [],
   "source": [
    "q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "328b390a-22a2-4058-a848-c45562ba6930",
   "metadata": {},
   "outputs": [],
   "source": [
    "yp.simplify(q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb49c4a1-2e68-45d6-90d2-72ef18496563",
   "metadata": {},
   "outputs": [],
   "source": [
    "i = yp.Symbol('i')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7167614-1ea1-4c6b-8257-1afb528ba916",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = yp.MatrixSymbol('A', 3,3)\n",
    "y = yp.MatrixSymbol('y', 3,1)\n",
    "B = A*y                    \n",
    "C = yp.tanh(B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da3e3b2f-194c-439b-a7cd-f2e0c017543b",
   "metadata": {},
   "outputs": [],
   "source": [
    "C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "274327f8-7dcb-48de-9366-6c7da7291e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "yp.diff(B,A)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6248342e-a78f-449d-8e8c-19e25fd06dbc",
   "metadata": {},
   "source": [
    "###  just trust me on this one:\n",
    "#### plugging a matrix*vector into a symbolic computation package and asking it to give you a solution is likely NOT going to help you."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f40db1-e3de-40dd-8bec-4eb64dd9d013",
   "metadata": {},
   "source": [
    "# THE KEY INVENTION (in the last 20 years or so)\n",
    "## THIS BIZARRE TRICK makes neural networks tractable:\n",
    "## YOU CAN CALCULATE a piece-wise derivative of an expression\n",
    "## COMBINE each piece-wise derivative\n",
    "## and chain it backward, towards the input variables\n",
    "## i.e. use the CHAIN RULE!\n",
    "## NOW YOU're DONE!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a23f38f-8b74-4627-adf3-cf5f24f56572",
   "metadata": {},
   "source": [
    "# So where do you start?\n",
    "## Assume you managed find a good equation for your loss function.\n",
    "## You have the output of the loss function, which is going to be some number\n",
    "### You want to figure out how to change the values in the matrix so that your resulting loss is minimized."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "271c1f48-c5db-4559-be03-dca6c2256f86",
   "metadata": {},
   "source": [
    "# Key thing is that the d(x)/d(x) == 1 (ALWAYS)\n",
    "# so now, you start with the d(LOSS)/d(LOSS) is 1.0\n",
    "# now calculate d(LOSS) w.r.t. to the NEXT piece of your equation graph (i.e. just one operation at a time!)\n",
    "## next step is to calculate d (LOSS[i])/d(LOSS(i+1]) for each stage [i] of your expression graph. \n",
    "## i.e. calculate the partial differential at stage [i+1] with respect to each source variable in stage [i]\n",
    "# What is a \"stage\"?\n",
    "## It's simply all of the subexpressions that make up the destination expression at some point\n",
    "## i.e. y = tanh(X @ V + b)\n",
    "### \n",
    "### LOSS = (y - yANS)**2\n",
    "### y = tanh(v[0])\n",
    "### v[0] = v[1] + b\n",
    "### v[1] = X @ V\n",
    "## In this case, only the matrix $X$ and the bias vector $b$ are the parameters of your NN\n",
    "## $V$ is the input to the matrix\n",
    "### So ultimately we want to figure out how to change $X$ and $b$ to minimize the LOSS w.r.t. the training set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8858d842-f370-409c-b364-8cf300723bab",
   "metadata": {},
   "source": [
    "# So what is the derivative of a matrix?\n",
    "## simple (!) (BUT ANNOYING!)\n",
    "# We will cover it later!\n",
    "### Remember that we ultimately want to calculate the new .grad of both the left hand matrix and the right hand matrix\n",
    "### i.e. we need to calculate the outgoing gradient w.r.t. each $LH[i,j]$, i.e each value in  the left hand matrix and also $RH[i,j]$ for each value in the right hand matrix.\n",
    "\n",
    "### Assume `A` is (3,4) and `B` is (4,2). \n",
    "### if we do `C = A @ B`, the result of the matrix multiply `C` is a (3,2) matrix\n",
    "\n",
    "        C        = A                @ B\n",
    "        (3,2)      (3,4)              (4,2)  )\n",
    "        c00 c01    a00 a01 a02 a03    b00 b01\n",
    "        c10 c11  = a10 a11 a12 a13  @ b10 b11\n",
    "        c20 c21    a20 a21 a22 a23    b20 b21\n",
    "                                      b30 b31\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}