Keras_Tuner.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Keras_Tuner.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bIaY909U7dl3",
        "colab_type": "text"
      },
      "source": [
        "# **Keras Tuner**\n",
        "\n",
        "### Basically a really great library for deciding number of units and kernels We need for optimizing the neural network. In simple words we can say that with help of this library we can find all those important hyperparameters that helps us.\n",
        "\n",
        "### Starting with you need install the \"keras-tuner\" first(for deep learning always use google colab with GPU)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oDuZLrSqnTcZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 532
        },
        "outputId": "b36a0974-a00a-4572-a5e6-8fb29a512796"
      },
      "source": [
        "!pip install keras-tuner"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting keras-tuner\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a7/f7/4b41b6832abf4c9bef71a664dc563adb25afc5812831667c6db572b1a261/keras-tuner-1.0.1.tar.gz (54kB)\n",
            "\r\u001b[K     |██████                          | 10kB 19.8MB/s eta 0:00:01\r\u001b[K     |████████████                    | 20kB 1.7MB/s eta 0:00:01\r\u001b[K     |██████████████████              | 30kB 2.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 40kB 2.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 51kB 1.9MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 61kB 1.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.16.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (1.18.4)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.8.7)\n",
            "Collecting terminaltables\n",
            "  Downloading https://files.pythonhosted.org/packages/9b/c4/4a21174f32f8a7e1104798c445dacdc1d4df86f2f26722767034e4de4bff/terminaltables-3.1.0.tar.gz\n",
            "Collecting colorama\n",
            "  Downloading https://files.pythonhosted.org/packages/c9/dc/45cdef1b4d119eb96316b3117e6d5708a08029992b2fee2c143c7a0a5cc5/colorama-0.4.3-py2.py3-none-any.whl\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (4.41.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (2.23.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (1.4.1)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.22.2.post1)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (2.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (2020.4.5.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->keras-tuner) (1.24.3)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->keras-tuner) (0.15.0)\n",
            "Building wheels for collected packages: keras-tuner, terminaltables\n",
            "  Building wheel for keras-tuner (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras-tuner: filename=keras_tuner-1.0.1-cp36-none-any.whl size=73200 sha256=0ddfc528a1a6debe3182ae7b364f156aaf616bc4ccf8388d93bcdee77e4b46ad\n",
            "  Stored in directory: /root/.cache/pip/wheels/b9/cc/62/52716b70dd90f3db12519233c3a93a5360bc672da1a10ded43\n",
            "  Building wheel for terminaltables (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for terminaltables: filename=terminaltables-3.1.0-cp36-none-any.whl size=15356 sha256=d1f2a73e188503479976510721800e763983b247c90dda26316e49aa14db24e9\n",
            "  Stored in directory: /root/.cache/pip/wheels/30/6b/50/6c75775b681fb36cdfac7f19799888ef9d8813aff9e379663e\n",
            "Successfully built keras-tuner terminaltables\n",
            "Installing collected packages: terminaltables, colorama, keras-tuner\n",
            "Successfully installed colorama-0.4.3 keras-tuner-1.0.1 terminaltables-3.1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "961Rmiv78ojh",
        "colab_type": "text"
      },
      "source": [
        "### Then, Import the tensorflow and keras with necessary librarires with it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MnNclfFS2OtX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "from tensorflow.keras import datasets, layers, models"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5eG6VvCS81V1",
        "colab_type": "text"
      },
      "source": [
        "### Here I'm taking mnist hand-written dataset for train and test set with their labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BLqS8JwU3i8B",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "d3ee2adf-d814-4945-aaa8-f2dc77650822"
      },
      "source": [
        "(train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ID-3RZfA37ln",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "cdc0ae6c-09c6-432d-d658-542e720d1f38"
      },
      "source": [
        "# size of image\n",
        "train_images[0].shape"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(28, 28)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fKztSzr93_aA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# normalization for images\n",
        "train_images=train_images/255.0\n",
        "test_images=test_images/255.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CnW6xZcX4DIQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# converting 3d tensors to 4d tensors\n",
        "train_images=train_images.reshape(len(train_images),28,28,1)\n",
        "test_images=test_images.reshape(len(test_images),28,28,1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DUN5GzH64G8d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# function for model building\n",
        "def build_model(hp):\n",
        "    # here sequential model is taken with convolution layer, number of kernels and kernel size is decided  \n",
        "  model = keras.Sequential([\n",
        "    keras.layers.Conv2D(\n",
        "        filters=hp.Int('conv_1_filter', min_value=32, max_value=128, step=16),\n",
        "        kernel_size=hp.Choice('conv_1_kernel', values = [3,5]),\n",
        "        activation='relu',\n",
        "        input_shape=(28,28,1)\n",
        "    ),\n",
        "    # pooling layer\n",
        "    keras.layers.MaxPooling2D( pool_size=(2, 2), \n",
        "                              strides=None, \n",
        "                              padding=\"valid\"),\n",
        "    keras.layers.Conv2D(\n",
        "        filters=hp.Int('conv_2_filter', min_value=32, max_value=64, step=16),\n",
        "        kernel_size=hp.Choice('conv_2_kernel', values = [3,5]),\n",
        "        activation='relu'\n",
        "    ),\n",
        "    keras.layers.MaxPooling2D( pool_size=(2, 2), \n",
        "                              strides=None, \n",
        "                              padding=\"valid\"),\n",
        "    # flatten layer                        \n",
        "    keras.layers.Flatten(),\n",
        "    keras.layers.Dense(\n",
        "        units=hp.Int('dense_1_units', min_value=32, max_value=128, step=16),\n",
        "        activation='relu'\n",
        "    ),\n",
        "    keras.layers.Dense(10, activation='softmax')\n",
        "  ])\n",
        "  \n",
        "  model.compile(optimizer=keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-3])),\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "  \n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IBJesLbZ_HkB",
        "colab_type": "text"
      },
      "source": [
        "### After building model, Now import RandomSearch which does all trail and error for you. Later list all models with their accuracy and even hyperparameter values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TQeaIdnj5aID",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from kerastuner import RandomSearch\n",
        "from kerastuner.engine.hyperparameters import HyperParameters"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9IyDvZd55d83",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner_search=RandomSearch(build_model,\n",
        "                          objective='val_accuracy',\n",
        "                          max_trials=5, # max trails would 5\n",
        "                          directory='/content/', # directory where all logs should be saved\n",
        "                          project_name=\"log\") # folder name where all logs will be saved"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "07gNBv7NAMcU",
        "colab_type": "text"
      },
      "source": [
        "### After building RandomSearch, Fit the model to dataset. The **.search** does the same job which **fit** does for the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NzGYjzqM5lPx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "342305ee-aa13-4339-c1c9-fe238e316572"
      },
      "source": [
        "tuner_search.search(train_images,train_labels,epochs=3,validation_split=0.1)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "1688/1688 [==============================] - 74s 44ms/step - loss: 0.1500 - accuracy: 0.9545 - val_loss: 0.0719 - val_accuracy: 0.9795\n",
            "Epoch 2/3\n",
            "1688/1688 [==============================] - 73s 43ms/step - loss: 0.0862 - accuracy: 0.9750 - val_loss: 0.0725 - val_accuracy: 0.9812\n",
            "Epoch 3/3\n",
            "1688/1688 [==============================] - 73s 43ms/step - loss: 0.0741 - accuracy: 0.9790 - val_loss: 0.1073 - val_accuracy: 0.9782\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 5d99661ce0065c939522c8e3b413e9cc</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.981166660785675</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_1_filter: 80</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_1_kernel: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_2_filter: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_2_kernel: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-dense_1_units: 112</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "1688/1688 [==============================] - 53s 31ms/step - loss: 0.1375 - accuracy: 0.9588 - val_loss: 0.0800 - val_accuracy: 0.9742\n",
            "Epoch 2/3\n",
            "1688/1688 [==============================] - 53s 32ms/step - loss: 0.0814 - accuracy: 0.9761 - val_loss: 0.0650 - val_accuracy: 0.9818\n",
            "Epoch 3/3\n",
            "1688/1688 [==============================] - 53s 31ms/step - loss: 0.0753 - accuracy: 0.9783 - val_loss: 0.0614 - val_accuracy: 0.9827\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 325b914e879412717c245c99815c7744</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9826666712760925</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_1_filter: 48</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_1_kernel: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_2_filter: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_2_kernel: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-dense_1_units: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "1688/1688 [==============================] - 108s 64ms/step - loss: 0.1300 - accuracy: 0.9599 - val_loss: 0.0576 - val_accuracy: 0.9842\n",
            "Epoch 2/3\n",
            "1688/1688 [==============================] - 107s 64ms/step - loss: 0.0435 - accuracy: 0.9866 - val_loss: 0.0399 - val_accuracy: 0.9887\n",
            "Epoch 3/3\n",
            "1688/1688 [==============================] - 108s 64ms/step - loss: 0.0310 - accuracy: 0.9903 - val_loss: 0.0405 - val_accuracy: 0.9890\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 1ed8371468e6e7b58e4adc16e93d5207</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9890000224113464</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_1_filter: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_1_kernel: 5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_2_filter: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_2_kernel: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-dense_1_units: 80</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "1688/1688 [==============================] - 97s 57ms/step - loss: 0.1559 - accuracy: 0.9514 - val_loss: 0.0890 - val_accuracy: 0.9713\n",
            "Epoch 2/3\n",
            "1688/1688 [==============================] - 96s 57ms/step - loss: 0.0964 - accuracy: 0.9721 - val_loss: 0.0726 - val_accuracy: 0.9813\n",
            "Epoch 3/3\n",
            "1688/1688 [==============================] - 95s 56ms/step - loss: 0.0880 - accuracy: 0.9751 - val_loss: 0.0836 - val_accuracy: 0.9805\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: b6baabb2e7d33bb147b67d4706a87b00</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.981333315372467</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_1_filter: 112</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_1_kernel: 5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_2_filter: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_2_kernel: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-dense_1_units: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "1688/1688 [==============================] - 118s 70ms/step - loss: 0.1465 - accuracy: 0.9566 - val_loss: 0.1100 - val_accuracy: 0.9698\n",
            "Epoch 2/3\n",
            "1688/1688 [==============================] - 117s 70ms/step - loss: 0.0950 - accuracy: 0.9729 - val_loss: 0.0530 - val_accuracy: 0.9862\n",
            "Epoch 3/3\n",
            "1688/1688 [==============================] - 118s 70ms/step - loss: 0.0878 - accuracy: 0.9764 - val_loss: 0.0868 - val_accuracy: 0.9765\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: fb05e7593b71af4f8c394926bf476430</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9861666560173035</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_1_filter: 112</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_1_kernel: 5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-conv_2_filter: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-conv_2_kernel: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-dense_1_units: 112</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OawYGQzDA0PD",
        "colab_type": "text"
      },
      "source": [
        "### As you saw the number of filters and kernel size is so odd that we would normally never think of but if you see **score**, it doing great job in it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JvU7toTO9rAR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# get the best model out of it\n",
        "model=tuner_search.get_best_models(num_models=1)[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBlRWhyd9sFf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 407
        },
        "outputId": "5ccc102c-99f8-4a74-dd3d-4d2ddfa7eff2"
      },
      "source": [
        "# model summary\n",
        "model.summary()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d (Conv2D)              (None, 24, 24, 96)        2496      \n",
            "_________________________________________________________________\n",
            "max_pooling2d (MaxPooling2D) (None, 12, 12, 96)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_1 (Conv2D)            (None, 10, 10, 64)        55360     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_1 (MaxPooling2 (None, 5, 5, 64)          0         \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 1600)              0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 80)                128080    \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 10)                810       \n",
            "=================================================================\n",
            "Total params: 186,746\n",
            "Trainable params: 186,746\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h7OJzh7F9vje",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "outputId": "71aaf403-dde5-4e95-ccef-1ff68362ef4e"
      },
      "source": [
        "# now fit the model, start training from 4th epoch because model is already trained till 3rd epoch.\n",
        "model.fit(train_images, train_labels, epochs=10, validation_split=0.1, initial_epoch=3)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 4/10\n",
            "1688/1688 [==============================] - 108s 64ms/step - loss: 0.0236 - accuracy: 0.9922 - val_loss: 0.0330 - val_accuracy: 0.9903\n",
            "Epoch 5/10\n",
            "1688/1688 [==============================] - 107s 63ms/step - loss: 0.0178 - accuracy: 0.9944 - val_loss: 0.0351 - val_accuracy: 0.9908\n",
            "Epoch 6/10\n",
            "1688/1688 [==============================] - 107s 63ms/step - loss: 0.0145 - accuracy: 0.9950 - val_loss: 0.0291 - val_accuracy: 0.9910\n",
            "Epoch 7/10\n",
            "1688/1688 [==============================] - 107s 63ms/step - loss: 0.0115 - accuracy: 0.9961 - val_loss: 0.0380 - val_accuracy: 0.9913\n",
            "Epoch 8/10\n",
            "1688/1688 [==============================] - 107s 63ms/step - loss: 0.0091 - accuracy: 0.9968 - val_loss: 0.0564 - val_accuracy: 0.9873\n",
            "Epoch 9/10\n",
            "1688/1688 [==============================] - 106s 63ms/step - loss: 0.0080 - accuracy: 0.9974 - val_loss: 0.0393 - val_accuracy: 0.9922\n",
            "Epoch 10/10\n",
            "1688/1688 [==============================] - 106s 63ms/step - loss: 0.0081 - accuracy: 0.9973 - val_loss: 0.0464 - val_accuracy: 0.9908\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7ff2d5149e80>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KqZTv6XtBqxw",
        "colab_type": "text"
      },
      "source": [
        "### As you saw its taking from 99% validation accuarcy itself which means model is doing its best and the **Kerar Tuner** helped alot with only few lines of code.\n",
        "\n",
        "### You can keras tuner even with standard neural network(ANN) and also for sklearn model(classifiers)\n",
        "\n",
        "### Find the below link for more information.\n",
        "### https://blog.tensorflow.org/2020/01/hyperparameter-tuning-with-keras-tuner.html"
      ]
    }
  ]
}