kumin and lecun uniform added

Author: MVRonkin

Diff for: DL_In_ImageProcessing_4_Activation_and_Weights_Init.ipynb

@@ -624,8 +624,13 @@
     "&= \\sum_i^n \\text{Var}(x_i)\\text{Var}(w_i) = \\left( n \\text{Var}(w) \\right) \\text{Var}(x) \\\\\\\\\n",
     "&\\text{Thus, due to }\\text{Var}(aX) = a^2\\text{Var}(X),\\text{ we need to have }a = \\frac{1}{\\sqrt{n}}\n",
     "\\end{align}$$\n",
+    "For uniform distribution\n",
+    "$$w_i\\sim U\\big[-\\frac{\\sqrt{3}}{\\sqrt{n_{i}}},\\frac{\\sqrt{3}}{\\sqrt{n_{i} }}\\big]$$\n",
+    "Kumar initialization\n",
+    "    $w_i \\sim N(0,\\frac{16}{n_{i+1}}$  \n",
     "  </ul>   \n",
-    "<!-- https://cs231n.github.io/neural-networks-2/#batchnorm -->"
+    "<!-- https://cs231n.github.io/neural-networks-2/#batchnorm \n",
+    "https://towardsdatascience.com/weight-initialization-in-deep-neural-networks-268a306540c0 -->"
    ]
   },
   {
@@ -652,12 +657,14 @@
     "where $n_i$ - is the number of weights in layer.<br>\n",
     "\n",
     "</ul>\n",
-    "<li><b>Xavier uniform initialization are recommended for sigmoid layer.</b><ul>\n",
+    "<li><b>Xavier uniform initialization are recommended for sigmoid layer.</b><ul>  \n",
     "<li>    in some cases of <b>tanh layer</b> it is recommended to use $w_i\\sim N\\big[0,\\frac{1}{{n_{i}}}\\big]$\n",
+    "<ul>\n",
     "<!-- \n",
     "https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79\n",
     "https://pouannes.github.io/blog/initialization/\n",
-    "https://www.deeplearning.ai/ai-notes/initialization/ -->"
+    "https://www.deeplearning.ai/ai-notes/initialization/ \n",
+    "https://towardsdatascience.com/weight-initialization-in-deep-neural-networks-268a306540c0-->"
    ]
   },
   {
@@ -773,7 +780,7 @@
     "width": "165px"
    },
    "toc_section_display": true,
-   "toc_window_display": false
+   "toc_window_display": true
   }
  },
  "nbformat": 4,