dogsbreeds.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "dogsbreeds.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/vishbin/tensorflow-google-colab/blob/master/dogsbreeds.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "4iwUIQLU6zlM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Using Deep Learning Model to Classify Dog Breeds\n",
        "\n",
        "In this notebook I am going to show results of applying pre-trained deep learning models to dogs breeds classification. The main goal of the notebook is to see how simple it could be to achive good classification results using \"out-of-the-box\" models and common training strategies without applying too fancy data transformations or using sophisticated fine-tuning methods.\n",
        "\n",
        "The dataset is taken from [Dog Breed Identification](https://www.kaggle.com/c/dog-breed-identification/data) and contains around ~10K training images (in various resolution)."
      ]
    },
    {
      "metadata": {
        "id": "gKjCfI8m6zlR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![Image](assets/header.png)"
      ]
    },
    {
      "metadata": {
        "id": "N_VRuC7F6zlU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As a small spoiler, pretrained models shows quite good results even without. And, though it is not enough to achieve state of art results, it allows to classify images with quite good accuracy."
      ]
    },
    {
      "metadata": {
        "id": "PzfRI91q6zlX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Imports and Extensions"
      ]
    },
    {
      "metadata": {
        "id": "3TPWk4XN6zle",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "8d0e0a10-bb1a-4ab3-ee99-9cf0349f7948"
      },
      "cell_type": "code",
      "source": [
        "%load_ext autoreload"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The autoreload extension is already loaded. To reload it, use:\n",
            "  %reload_ext autoreload\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "jp5Sajbt6zln",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%matplotlib inline\n",
        "%autoreload 2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4d-xT_Up6zlw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore', category=FutureWarning)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Dg2bcrUe6zl2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 373
        },
        "outputId": "0fbdf335-3eb6-466d-d8c8-7f21bcdfd7ca"
      },
      "cell_type": "code",
      "source": [
        "import os\n",
        "import csv\n",
        "import tempfile\n",
        "from os import listdir\n",
        "from io import StringIO\n",
        "from pathlib import Path\n",
        "from itertools import islice\n",
        "from os.path import join, abspath\n",
        "from collections import namedtuple\n",
        "\n",
        "import tqdm\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from PIL import Image\n",
        "from keras.models import load_model\n",
        "\n",
        "from sklearn.metrics import log_loss\n",
        "from sklearn.externals import joblib\n",
        "from sklearn.ensemble import BaggingClassifier\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.feature_selection import VarianceThreshold\n",
        "\n",
        "from swissknife.config import notebook_logger\n",
        "from swissknife.utils import calculate_layout\n",
        "from swissknife.files import FilesStream, SavingFolder\n",
        "from swissknife.transform import GeneratorPipeline\n",
        "from swissknife.kaggle.api import ClassifierSubmission\n",
        "from swissknife.kaggle.datasets import KaggleClassifiedImagesSource, KaggleTestImagesIterator\n",
        "\n",
        "# File paths from local folder\n",
        "from basedir import TRAIN_IMAGES, VALID_IMAGES, TEST_IMAGES, LABELS_FILE"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ImportError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m\u001b[0m",
            "\u001b[0;31mImportError\u001b[0mTraceback (most recent call last)",
            "\u001b[0;32m<ipython-input-11-a0a850166579>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mos\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mlistdir\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mio\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mStringIO\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mpathlib\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mPath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mitertools\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mislice\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mjoin\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mabspath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mImportError\u001b[0m: No module named pathlib",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "V0uVGNlj6zl7",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "log = notebook_logger()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eO-jNwAd6zmB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Dataset Overview"
      ]
    },
    {
      "metadata": {
        "id": "0k9v8uuh6zmH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Though the main purpose of this post to show power of pretrained models on ImageNet like datasets, it is a good idea to anaylize dataset a bit before starting to build any machine learning pipeline.\n",
        "\n",
        "> Note that testing data _shouldn't be touched_ during this process. All statistics computations, validation and plots should be performed only on training subset.\n",
        "\n",
        "Analysed dataset consists of two folders with images, `train` and `test`, and file with labels which maps file name to class, represeted by that file. Let's traverse dataset directories to get lists of available images and parse dataset labels into dictionary which allow to convert image ID into its class:"
      ]
    },
    {
      "metadata": {
        "id": "FgLOAlzg6zmK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def list_files(folder):\n",
        "    return [join(folder, filename) for filename in listdir(folder)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "B2Q59c2j6zmP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def read_labels(labels_path):\n",
        "    \"\"\"Reads dataset labels from the labels file.\"\"\"\n",
        "    with open(labels_path) as fp:\n",
        "        reader = csv.DictReader(fp)\n",
        "        labels = {row['id']: row['breed'] for row in reader}\n",
        "        log.info('Number of unique classes: %d', len(set(labels.values())))\n",
        "    return labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0-ySguXp6zmW",
        "colab_type": "code",
        "colab": {},
        "outputId": "7e160f57-96fe-49dd-eef1-0e5381dba06e"
      },
      "cell_type": "code",
      "source": [
        "train_files = list_files(TRAIN_IMAGES)\n",
        "valid_files = list_files(VALID_IMAGES)\n",
        "labels = read_labels(LABELS_FILE)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of unique classes: 120\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "cYpkORkI6zmm",
        "colab_type": "code",
        "colab": {},
        "outputId": "a1b3700a-7ba7-4593-ebe3-4f8f23677f8c"
      },
      "cell_type": "code",
      "source": [
        "len(train_files)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "9200"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "pDEcq-T66zmt",
        "colab_type": "code",
        "colab": {},
        "outputId": "846c41ba-4d28-4b29-8b96-a1cad141d1fb"
      },
      "cell_type": "code",
      "source": [
        "len(valid_files)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1022"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "hSv8Iny76zm1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, let's create a couple of utilities to present images in convenient format. Each image file should be converted into array of bytes to be plotted with `matplotlib` library. Also, that would be helpful to take small preview of dataset in compact form. The following functions help to achive these goals:"
      ]
    },
    {
      "metadata": {
        "id": "iB6cZHiw6zm3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def read_image(filename, labels, dtype=np.uint8):\n",
        "    \"\"\"Returns an image and it's class using provided filename.\"\"\"\n",
        "    \n",
        "    path = Path(filename)\n",
        "    image = plt.imread(path).astype(dtype)\n",
        "    return image, labels[path.stem]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nrcyT9XZ6zm8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def read_images(files, labels):    \n",
        "    \"\"\"Reads a list of images alongside with their labels using provided ID numbers.\"\"\"\n",
        "    \n",
        "    image_label_pairs = [read_image(file, labels) for file in files]\n",
        "    images, labels = list(zip(*image_label_pairs))\n",
        "    return images, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "arBM2ZNb6znE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def show_images(dataset, labels, preview_size=16, \n",
        "                plot_title='Images Grid', thumbnail_size=None,\n",
        "                tight=False, figsize=(10, 10), n_cols=4):\n",
        "    \"\"\"Plots a group of dataset images alongside with their labels\n",
        "    organized into grid,\n",
        "    \n",
        "    Args:\n",
        "        dataset: List of images (numpy arrays).\n",
        "        labels: List of labels (strings).\n",
        "        preview_size: Number of images taken from dataset to be plotted.\n",
        "        title: Title of generated grid.\n",
        "        figsize: Tuple with canvas size (n, m).\n",
        "        n_cols: Number of columns in grid.\n",
        "    \n",
        "    \"\"\"\n",
        "    grid_size = calculate_layout(preview_size, n_cols=n_cols)\n",
        "    f, axes = plt.subplots(*grid_size, figsize=figsize)\n",
        "    \n",
        "    axes = axes.flatten()            \n",
        "    sample_subset = islice(zip(dataset, labels), preview_size)\n",
        "    \n",
        "    for ax, (image, title) in zip(axes, sample_subset):\n",
        "        if thumbnail_size is not None:\n",
        "            pil_image = Image.fromarray(image).resize(thumbnail_size)        \n",
        "            image = np.array(pil_image)\n",
        "        ax.imshow(image.astype(np.uint8))        \n",
        "        title = title.replace('_', ' ').title()\n",
        "        ax.set_title(title, fontsize=16)\n",
        "        ax.axis('off')\n",
        "        \n",
        "    f.suptitle(plot_title, fontsize=20)\n",
        "    f.tight_layout(rect=[0, 0.03, 1, 0.95])  # or: f.subplots_adjust(top=0.90)\n",
        "    return f    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SmsrSeVZ6znH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "First, plotting a preview of training samples and labels:"
      ]
    },
    {
      "metadata": {
        "id": "H2BHBTPp6znJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Nubmer of images to be shown\n",
        "preview_size = 20"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tHv3RFOq6znT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Take a sample from train data\n",
        "_ = show_images(*read_images(train_files[:preview_size], labels),\n",
        "                preview_size=preview_size,\n",
        "                plot_title='Train Subset Preview',\n",
        "                thumbnail_size=(256, 256),\n",
        "                figsize=(12, 12))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uYhnDYiz6zna",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, doing the same for validation samples:"
      ]
    },
    {
      "metadata": {
        "id": "JIZa9ZCu6zne",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "_ = show_images(*read_images(valid_files[:preview_size], labels),\n",
        "                preview_size=preview_size,\n",
        "                plot_title='Validation Subset Preview',\n",
        "                thumbnail_size=(256, 256),\n",
        "                figsize=(12, 12))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wp_J6mwd6znq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "It seems that, generally speaking, images have different resolution, zoom, could have more then one dog depicted on them and were taken in various lighting conditions. \n",
        "\n",
        "And what about images classes? How are they distributed? For this purpose a histogram of classes could be created to see if there is any bias:"
      ]
    },
    {
      "metadata": {
        "id": "daXHEmhf6zns",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "encoder = LabelEncoder()\n",
        "breeds_encoded = encoder.fit_transform(list(labels.values()))\n",
        "n_classes = len(encoder.classes_)\n",
        "mean = np.mean(breeds_encoded)\n",
        "title = 'Distribution of Dog Breeds in Dataset\\n (%2.0f samples per class on average)' % mean\n",
        "\n",
        "f, ax = plt.subplots(1, 1, figsize=(10, 6))\n",
        "ax.set_xticks([])\n",
        "ax.hlines(mean, 0, n_classes - 1, color='white')\n",
        "ax.set_title(title, fontsize=18)\n",
        "_ = ax.hist(breeds_encoded, bins=n_classes)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nOlAz-ic6zn2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Classes are more or less balanced, as historgram shows, with `59` sampler per class on average."
      ]
    },
    {
      "metadata": {
        "id": "LmJC_7cp6zn3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Bootstrapped SGD using Bottleneck Features\n",
        "\n",
        "One of the most straightforward strategies to apply pretrained deep learning models is to use them as **feature extractors**. Before modern neural architectures were developed, image features were extracted using manually crafted filters. Nowadays it is possible to derive features automatically, from data.\n",
        "\n",
        "Schematically, a deep learning classifier could be represented as a sequence of blocks, transforming image representation from raw pixels into more and more abstract features like edges, contours, an so on. \n",
        "\n",
        "![Deep Model](assets/deep.png)\n",
        "_Picture 1. Bottleneck Features_\n",
        "\n",
        "As it is shown on the picture, deep learning model without top layers generates a set of high level abstract features for each shown image. These high level representations are called **bottleneck features**, i.e. a hidden layer activations taken from the model right before feeding them into dense layers. These abstract features, automatically infered by model, could be used then with any \"classical\" machine learning algorithm to predict targets.\n",
        "\n",
        "In this section we're going to extract image features using following pretrained deep learning models:\n",
        "* InceptionV3\n",
        "* InceptionResNetV2\n",
        "* Xception\n",
        "\n",
        "Then, extractred features will be used to train a boosted version SGD classifier and to validate its performance.\n",
        "\n",
        "\n",
        "### Feature Extractor\n",
        "The following class called `FeaturesExtractor` is a simple wrapper build on top of functions taken from `keras.applications` module. It creates deep model without top layer, loads pretrained weights and feeds training images into model, saving generated bottleneck features onto disk:"
      ]
    },
    {
      "metadata": {
        "id": "Jx7X9i8j6zn4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class FeaturesExtractor:\n",
        "    \"\"\"Runs pretrained model without top layers on dataset and saves generated\n",
        "    bottleneck features onto disk.\n",
        "    \"\"\"\n",
        "    def __init__(self, build_fn, preprocess_fn, source,\n",
        "                 target_size=(299, 299, 3), batch_size=128):        \n",
        "        \n",
        "        self.build_fn = build_fn\n",
        "        self.preprocess_fn = preprocess_fn\n",
        "        self.source = source\n",
        "        self.target_size = target_size\n",
        "        self.batch_size = batch_size\n",
        "        \n",
        "    def __call__(self, folder, filename, pool='avg'):\n",
        "        model = self.build_fn(weights='imagenet', include_top=False, pooling=pool)\n",
        "        stream = self.source(\n",
        "            folder=folder, target_size=self.target_size,\n",
        "            batch_size=self.batch_size, infinite=False)\n",
        "        \n",
        "        batches = []\n",
        "        with tqdm.tqdm_notebook(total=stream.steps_per_epoch) as bar:\n",
        "            for x_batch, y_batch in stream:\n",
        "                x_preprocessed = self.preprocess_fn(x_batch)\n",
        "                batch = model.predict_on_batch(x_preprocessed)\n",
        "                batches.append(batch)\n",
        "                bar.update(1)\n",
        "                \n",
        "        all_features = np.vstack(batches)\n",
        "        np.save(filename, all_features)\n",
        "        return filename"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lnFNzcXj6zoB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "A group of helper functions defined below will help to run feature extractor, validate number of generatated samples and save results onto disk. Then these features will be used as predictors to train SGD classfier:"
      ]
    },
    {
      "metadata": {
        "id": "HoMvBaJy6zoC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def extract_features(extractor, folder, filename, size):\n",
        "    path = extractor(folder, filename)\n",
        "    features = np.load(path)\n",
        "    assert len(features) == size\n",
        "    return path\n",
        "\n",
        "\n",
        "def extract_train_features(features_extractor, model_name):\n",
        "    filename = '%s_train_features.npy' % model_name\n",
        "    return extract_features(features_extractor, TRAIN_IMAGES, filename, size=9200)\n",
        "\n",
        "\n",
        "def extract_valid_features(features_extractor, model_name):\n",
        "    filename = '%s_valid_features.npy' % model_name\n",
        "    return extract_features(features_extractor, VALID_IMAGES, filename, size=1022)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EbmooN_H6zoE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Also, as soon as the dataset is too large to be uploaded into memory at once, a special class called `KaggleClassifiedImagesSource` from `swissknife` package (which was written to clean up my notebooks code and factor out common functionality) is used to build training samples generator. This class accepts path to labels file and a name of column in this file which contains verbose labels names:"
      ]
    },
    {
      "metadata": {
        "id": "HYoDbd746zoE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "source = KaggleClassifiedImagesSource(LABELS_FILE, 'breed')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zgnGMrlY6zoH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next we're going to use this instance to create a generator accepted by Keras model's `train_generator` method like it's shown below:\n",
        "```Python\n",
        "generator = source(folder=folder,\n",
        "                   target_size=target_size,\n",
        "                   batch_size=batch_size\n",
        "                   infinite=False)\n",
        "                   \n",
        "model.train_generator(generator, steps=generator.steps_per_epoch)\n",
        "```\n",
        "\n",
        "I.e. we provide a path to folder with images and the generator will gradually read images from disk on demand when samples asked by training method.\n",
        "\n",
        "The reason why `KaggleClassifiedImagesSource` is used as factory instead of being used as generator itself and why folder name and samples yielding parameters should be provided in additional call instead of using `__init__` is that we are going to use two different generators during training process: one for traning and one for validation. Therefore, despite of having same set of target labels, we need two different streams of images.\n",
        "\n",
        "Also, we need targets to train bottleneck classfier. As soon as targets do not occupy too much space, they could be directly loaded into memory:"
      ]
    },
    {
      "metadata": {
        "id": "7mThBvLD6zoH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def create_targets(source, folder):\n",
        "    \"\"\"Converts file paths into target labels.\"\"\"\n",
        "    labels = [\n",
        "        one_hot.argmax()\n",
        "        for one_hot in (\n",
        "            source.identifier_to_label[Path(filename).stem]\n",
        "            for filename in os.listdir(folder))]\n",
        "    return np.asarray(labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "k-B1rnBL6zob",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train_labels = create_targets(source, TRAIN_IMAGES)\n",
        "valid_labels = create_targets(source, VALID_IMAGES)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2O0d9m8_6zof",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Boosted SGD Classifier\n",
        "\n",
        "[Stochastic Gradient Descent](http://scikit-learn.org/stable/modules/sgd.html#sgd) (SGD) is a simple yet very efficient approach to discriminative learning of linear classifiers under convex loss functions. In our case we would like to choose a **logistic regression** classifier to predict dogs breeds.\n",
        "\n",
        "To make training process more stable and repeatble, we're going to extend SGD with **bagging**, an approach that trains an _ensemble_ of SGD classifiers on different subsets of training data and gives a final prediction by averaging responses from separate estimators. The following picture schematically shows the idea. (Note that in our case we not only splitting original dataset into subsets, but also taking different subsets of features to train each classifier):\n",
        "\n",
        "![voting](assets/voting.png)\n",
        "\n",
        "Also, we're going to apply _varience threshold_ to features extract by deep learning networks. The reason why we need this preprocessing step is that generated feature vectors are usually quite sparse - many of their elements are close or equal to zero.\n",
        "\n",
        "> **Note:** Parameters for classifier were taken after a few runs with different values. Ones that have shown the best results were kept. More \"educated\" approach would be to run a [grid](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) or [random](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) search."
      ]
    },
    {
      "metadata": {
        "id": "HfQ1NMv36zog",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def sgd(x_train, y_train, x_valid, y_valid, variance_threshold=0.1):    \n",
        "    threshold = VarianceThreshold(variance_threshold)\n",
        "        \n",
        "    sgd_classifier = SGDClassifier(\n",
        "        alpha=1./len(x_train),\n",
        "        class_weight='balanced',\n",
        "        loss='log', penalty='elasticnet', \n",
        "        fit_intercept=False, tol=0.001, n_jobs=-1)\n",
        "    \n",
        "    bagging = BaggingClassifier(\n",
        "        base_estimator=sgd_classifier,\n",
        "        bootstrap_features=True,\n",
        "        n_jobs=-1, max_samples=0.5, max_features=0.5)    \n",
        "    \n",
        "    x_thresh = threshold.fit_transform(x_train)\n",
        "    bagging.fit(x_thresh, y_train)\n",
        "    train_metrics = build_metrics(bagging, x_thresh, y_train)\n",
        "\n",
        "    x_thresh = threshold.transform(x_valid)\n",
        "    valid_metrics = build_metrics(bagging, x_thresh, y_valid)\n",
        "\n",
        "    return bagging, train_metrics, valid_metrics    \n",
        "\n",
        "\n",
        "def build_metrics(model, X, y):\n",
        "    probs = model.predict_proba(X)\n",
        "    preds = np.argmax(probs, axis=1)\n",
        "    metrics = dict(\n",
        "        probs=probs,\n",
        "        preds=preds,\n",
        "        loss=log_loss(y, probs),\n",
        "        accuracy=np.mean(preds == y))\n",
        "    return namedtuple('Predictions', metrics.keys())(**metrics)    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5tcQzP006zoj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now when all preparations have done, we can start classifier training on features extracted with pretrained models.\n",
        "\n",
        "### InceptionV3 Features\n",
        "\n",
        "The first model selected to generate bottleneck features is [InceptionV3](https://github.com/keras-team/keras/blob/master/keras/applications/inception_v3.py) architecture."
      ]
    },
    {
      "metadata": {
        "id": "GqLQQYD86zoj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from keras.applications.inception_v3 import InceptionV3, preprocess_input\n",
        "\n",
        "extractor = FeaturesExtractor(\n",
        "    build_fn=InceptionV3,\n",
        "    preprocess_fn=preprocess_input,\n",
        "    source=source)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BovnjsP-6zom",
        "colab_type": "code",
        "colab": {},
        "outputId": "896114af-c44d-4d2d-b621-559797613b7a"
      },
      "cell_type": "code",
      "source": [
        "extract_train_features(extractor, 'inception')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "443d954086f547ed9977a3530f9a5471",
              "version_major": 2,
              "version_minor": 0
            },
            "text/html": [
              "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
              "<p>\n",
              "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
              "  that the widgets JavaScript is still loading. If this message persists, it\n",
              "  likely means that the widgets JavaScript library is either not installed or\n",
              "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
              "  Widgets Documentation</a> for setup instructions.\n",
              "</p>\n",
              "<p>\n",
              "  If you're reading this message in another frontend (for example, a static\n",
              "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
              "  it may mean that your frontend doesn't currently support widgets.\n",
              "</p>\n"
            ],
            "text/plain": [
              "HBox(children=(IntProgress(value=0, max=72), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'inception_train_features.npy'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "metadata": {
        "id": "xZtdXZae6zor",
        "colab_type": "code",
        "colab": {},
        "outputId": "bccdf6c1-28d4-429c-9638-f62838f2906c"
      },
      "cell_type": "code",
      "source": [
        "extract_valid_features(extractor, 'inception')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ce0ac0253fe04f3b989042eb26def595",
              "version_major": 2,
              "version_minor": 0
            },
            "text/html": [
              "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
              "<p>\n",
              "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
              "  that the widgets JavaScript is still loading. If this message persists, it\n",
              "  likely means that the widgets JavaScript library is either not installed or\n",
              "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
              "  Widgets Documentation</a> for setup instructions.\n",
              "</p>\n",
              "<p>\n",
              "  If you're reading this message in another frontend (for example, a static\n",
              "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
              "  it may mean that your frontend doesn't currently support widgets.\n",
              "</p>\n"
            ],
            "text/plain": [
              "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'inception_valid_features.npy'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "metadata": {
        "id": "reKUjIf76zov",
        "colab_type": "code",
        "colab": {},
        "outputId": "10f5001c-eedb-4379-aa38-faa6ecf1c18b"
      },
      "cell_type": "code",
      "source": [
        "train_features = np.load('inception_train_features.npy')\n",
        "valid_features = np.load('inception_valid_features.npy')\n",
        "\n",
        "model, train, valid = sgd(\n",
        "    x_train=train_features, \n",
        "    y_train=train_labels,\n",
        "    x_valid=valid_features, \n",
        "    y_valid=valid_labels,\n",
        "    variance_threshold=0.1)\n",
        "\n",
        "print(f'Train: loss {train.loss:2.6f} - acc: {train.accuracy:2.2%}')\n",
        "print(f'Valid: loss {valid.loss:2.6f} - acc: {valid.accuracy:2.2%}')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train: loss 0.317071 - acc: 94.02%\n",
            "Valid: loss 0.471370 - acc: 88.55%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8EjCXlnc6zoy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Not bad! Pretrained model shows quite good validation accuracy out of the box, taking into account that training dataset contains 120 classes. Let's try another architecture.\n",
        "\n",
        "### Xception Features\n",
        "\n",
        "Now it is turn for [Xception](https://github.com/keras-team/keras/blob/master/keras/applications/xception.py) model:"
      ]
    },
    {
      "metadata": {
        "id": "1p0p2xWk6zoy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from keras.applications.xception import Xception, preprocess_input\n",
        "\n",
        "extractor = FeaturesExtractor(\n",
        "    build_fn=Xception,\n",
        "    preprocess_fn=preprocess_input,\n",
        "    stream_factory=source)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yTZP1Mwn6zo1",
        "colab_type": "code",
        "colab": {},
        "outputId": "edc036ad-11f1-4806-e2fd-c586a6ffea7b"
      },
      "cell_type": "code",
      "source": [
        "extract_train_features(extractor, 'xception')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "72/72 [==============================] - 74s 1s/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'xception_train_features.npy'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "metadata": {
        "id": "Vj-IdvE36zo6",
        "colab_type": "code",
        "colab": {},
        "outputId": "4430d624-493a-4b32-a6ca-da70a999c720"
      },
      "cell_type": "code",
      "source": [
        "extract_valid_features(extractor, 'xception')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "8/8 [==============================] - 9s 1s/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'xception_valid_features.npy'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "metadata": {
        "id": "6SOLDMuq6zpC",
        "colab_type": "code",
        "colab": {},
        "outputId": "7fa0b063-a0fe-4569-8d29-33043f70c15a"
      },
      "cell_type": "code",
      "source": [
        "train_features = np.load('xception_train_features.npy')\n",
        "valid_features = np.load('xception_valid_features.npy')\n",
        "\n",
        "model, train, valid = sgd(\n",
        "    x_train=train_features, \n",
        "    y_train=train_labels,\n",
        "    x_valid=valid_features, \n",
        "    y_valid=valid_labels,\n",
        "    variance_threshold=0.01)\n",
        "\n",
        "print(f'Train: loss {train.loss:2.6f} - acc: {train.accuracy:2.2%}')\n",
        "print(f'Valid: loss {valid.loss:2.6f} - acc: {valid.accuracy:2.2%}')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train: loss 0.284703 - acc: 95.40%\n",
            "Valid: loss 0.410348 - acc: 90.80%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Jpk0EZc26zpL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The model shows even better results on both validation and training data. Note that `varience_threshold` parameter was chagned because with value of `0.1` selected for the previous model **all generatered features** are excluded from training. Therefore we can say that the magnitude of variance of Xception features is small then in previous model.\n",
        "\n",
        "### InceptionResNetV2\n",
        "\n",
        "The last model we are going to try is [InceptionResNetV2](https://github.com/keras-team/keras/blob/master/keras/applications/inception_resnet_v2.py)."
      ]
    },
    {
      "metadata": {
        "id": "piv6K0HE6zpM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input\n",
        "\n",
        "extractor = FeaturesExtractor(\n",
        "    build_fn=InceptionResNetV2,\n",
        "    preprocess_fn=preprocess_input,\n",
        "    stream_factory=source)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Lfh70By-6zpP",
        "colab_type": "code",
        "colab": {},
        "outputId": "5e8f6d22-2289-426f-feca-71067de75a75"
      },
      "cell_type": "code",
      "source": [
        "extract_train_features(extractor, 'inception_resnet')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "72/72 [==============================] - 82s 1s/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'inception_resnet_train_features.npy'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "metadata": {
        "id": "QdRrQB_W6zpR",
        "colab_type": "code",
        "colab": {},
        "outputId": "ea5aab02-3146-41f4-fcd3-5bbc91778b48"
      },
      "cell_type": "code",
      "source": [
        "extract_valid_features(extractor, 'inception_resnet')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "8/8 [==============================] - 12s 1s/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'inception_resnet_valid_features.npy'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "metadata": {
        "id": "TUw8TXrU6zpW",
        "colab_type": "code",
        "colab": {},
        "outputId": "e40527ed-1a39-4373-98af-d1dae857c6b4"
      },
      "cell_type": "code",
      "source": [
        "train_features = np.load('inception_resnet_train_features.npy')\n",
        "valid_features = np.load('inception_resnet_valid_features.npy')\n",
        "\n",
        "model, train, valid = sgd(\n",
        "    x_train=train_features, \n",
        "    y_train=train_labels,\n",
        "    x_valid=valid_features, \n",
        "    y_valid=valid_labels,\n",
        "    variance_threshold=0.001)\n",
        "\n",
        "print(f'Train: loss {train.loss:2.6f} - acc: {train.accuracy:2.2%}')\n",
        "print(f'Valid: loss {valid.loss:2.6f} - acc: {valid.accuracy:2.2%}')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train: loss 0.198887 - acc: 94.53%\n",
            "Valid: loss 0.302672 - acc: 92.47%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_-MMSqZg6zpa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The architecture shows even better results in comparison with previous ones, achieving validation accuracy of `92.47%`. But how good our validation accuracy reflects results we could get on Kaggle's test dataset? Let's check it in next section.\n",
        "\n",
        "\n",
        "### Submitting Best Architecture Results\n",
        "\n",
        "As we've seen above, the best results were shown with **InceptionResNetV2**. We're going to reload extracted features and train model again, but with train and valid features concatenated together."
      ]
    },
    {
      "metadata": {
        "id": "WO5QW56n6zpa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input   "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "R2xqygAZ6zpc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train_features = np.load('inception_resnet_train_features.npy')\n",
        "valid_features = np.load('inception_resnet_valid_features.npy')\n",
        "\n",
        "source = KaggleClassifiedImagesSource(LABELS_FILE, 'breed')\n",
        "train_labels = create_targets(source, TRAIN_IMAGES)\n",
        "valid_labels = create_targets(source, VALID_IMAGES)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "obJrokz96zpd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "X = np.vstack([train_features, valid_features])\n",
        "y = np.hstack([train_labels, valid_labels])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oF_4PWxZ6zpf",
        "colab_type": "code",
        "colab": {},
        "outputId": "8d8e7754-3c4c-4183-9828-2b888ce91977"
      },
      "cell_type": "code",
      "source": [
        "X.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10222, 1536)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "metadata": {
        "id": "9PTLkLpY6zph",
        "colab_type": "code",
        "colab": {},
        "outputId": "9ef84bfc-2fdd-4daa-ca12-150c9b7e3594"
      },
      "cell_type": "code",
      "source": [
        "y.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10222,)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "metadata": {
        "id": "-7TWCSxG6zpk",
        "colab_type": "code",
        "colab": {},
        "outputId": "a5f037b4-63c7-4689-88b2-4d63bcbf54c1"
      },
      "cell_type": "code",
      "source": [
        "threshold = VarianceThreshold(0.001)\n",
        "        \n",
        "sgd_classifier = SGDClassifier(\n",
        "    alpha=1./X.shape[0],\n",
        "    class_weight='balanced',\n",
        "    loss='log', penalty='elasticnet', \n",
        "    fit_intercept=False, tol=0.001, n_jobs=-1)\n",
        "    \n",
        "bagging = BaggingClassifier(\n",
        "    base_estimator=sgd_classifier,\n",
        "    n_estimators=100,\n",
        "    bootstrap_features=True,\n",
        "    n_jobs=-1, max_samples=0.5, max_features=0.5)\n",
        "\n",
        "bagging.fit(threshold.fit_transform(X), y)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BaggingClassifier(base_estimator=SGDClassifier(alpha=9.782821365681862e-05, average=False,\n",
              "       class_weight='balanced', epsilon=0.1, eta0=0.0, fit_intercept=False,\n",
              "       l1_ratio=0.15, learning_rate='optimal', loss='log', max_iter=None,\n",
              "       n_iter=None, n_jobs=-1, penalty='elasticnet', power_t=0.5,\n",
              "       random_state=None, shuffle=True, tol=0.001, verbose=0,\n",
              "       warm_start=False),\n",
              "         bootstrap=True, bootstrap_features=True, max_features=0.5,\n",
              "         max_samples=0.5, n_estimators=100, n_jobs=-1, oob_score=False,\n",
              "         random_state=None, verbose=0, warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "metadata": {
        "id": "qrXxz4qY6zpn",
        "colab_type": "code",
        "colab": {},
        "outputId": "5bfc2179-cd9e-493b-a59b-c162eb9f02c8"
      },
      "cell_type": "code",
      "source": [
        "joblib.dump(bagging, 'sgd_bagging.pickle')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['sgd_bagging.pickle']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "bk80Aidz6zpo",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "bagging = joblib.load('sgd_bagging.pickle')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5-n4Mwc16zpq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "extractor = InceptionResNetV2(weights='imagenet', include_top=False, pooling='avg')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t2znOuog6zpz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "threshold = VarianceThreshold(0.001).fit(X)\n",
        "\n",
        "test_images = KaggleTestImagesIterator(\n",
        "    test_folder=TEST_IMAGES,\n",
        "    target_size=(299, 299, 3), \n",
        "    batch_size=128,\n",
        "    with_identifiers=True)\n",
        "\n",
        "file_preds = {}\n",
        "for images, identifiers in test_images:\n",
        "    features = extractor.predict(preprocess_input(images))\n",
        "    features_selected = threshold.transform(features)\n",
        "    preds = bagging.predict_proba(features_selected)\n",
        "    file_preds.update(dict(zip(identifiers, preds)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wqpgOIVF6zp1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "classes = source.binarizer.classes_.tolist()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UEucWS0I6zp5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "submission = ClassifierSubmission(log=log)\n",
        "_, filename = tempfile.mkstemp(suffix='.csv')\n",
        "submission.create(classes, file_preds, output=filename)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6OAgmfJs6zp9",
        "colab_type": "code",
        "colab": {},
        "outputId": "7651339e-105d-4426-8bfc-f6ce011bb2d7"
      },
      "cell_type": "code",
      "source": [
        "result = submission.submit(filename, 'dog-breed-identification', message='Bagging with SGD')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sending submission...\n",
            "Successfully submitted to Dog Breed Identification\n",
            "Evaluating results...\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "17_v0wtZ6zqB",
        "colab_type": "code",
        "colab": {},
        "outputId": "b6405444-0e0b-4b39-ecf3-bb1514fe5f23"
      },
      "cell_type": "code",
      "source": [
        "print(f'Public score: {result[\"publicScore\"]:2.6f}')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Public score: 0.307820\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "l21-aM1n6zqK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can see that loss value achieved on validation data approximates loss on public test dataset very well.\n",
        "\n",
        "Boosted SGD classfier has shown quite good results on dogs breeds dataset. But can we do better? In next section we're going to check performance of deep models with re-trained top layer and data augmentation."
      ]
    },
    {
      "metadata": {
        "id": "BdH7jIix6zqM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Pretrained Models Fine-tuning"
      ]
    },
    {
      "metadata": {
        "id": "iDKUqbsz6zqN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Bottleneck features training showed that extracted features allows to achieve good prediction results. But could we improve result with **fine-tunining** orignal models re-training top layers from scratch and applying data augmentation techniques?\n",
        "\n",
        "Briefly speaking, the purpose of fine-tuning process is to adjust pretrained model to your data. Because in most cases, the model you're going to re-use was trained on dataset with different number of classes. Therefore, you need at least replace a top classifying layer. Other layers could be \"locked\", or _frozen_, i.e. during traning new top layer their weights do not receive updates. "
      ]
    },
    {
      "metadata": {
        "id": "9NrCURdQ6zqN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<p align=\"center\"><img src=\"assets/finetuning.png\"/></p>"
      ]
    },
    {
      "metadata": {
        "id": "KhrSUkxk6zqR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Actully, the process of training new top layer is not too different from previous approach of using network as a feature extractor. We just use a different type of classifier (single layer perceptron). The only difference is aforementioned **data augmentation**. Each fine-tuned network was trained using slightly modified copies of images from original dataset (a bit rotated, zoomed in, etc.).\n",
        "\n",
        "### Chosen Models and Training Process\n",
        "\n",
        "A group of models taken from Keras library was trained on augmented dataset during fixed number of training epochs. Training process takes time and was performed outside of this notebook. So in this section we're going to restore trained models and their training history to see how well they perform on analysed dataset.\n",
        "\n",
        "The following models were selected to be fine-tuned on dog breeds datasets:\n",
        "* ResNet50\n",
        "* InceptionV3\n",
        "* Xception\n",
        "* InceptionResNetV2\n",
        "\n",
        "Each model was trained during **100 epochs** with early stopping and with **128 samples per bach** using same opmitizer - Stochastic Gradient Descent with Nesterov momentum enabled:\n",
        "\n",
        "```python\n",
        "from keras.optimizers import SGD\n",
        "sgd = SGD(lr=0.001, momentum=0.99, nesterov=True)\n",
        "```\n",
        "\n",
        "The following augmentation paramers were chosen:\n",
        "```python\n",
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "transformer = ImageDataGenerator(\n",
        "    width_shift_range=0.2,\n",
        "    height_shift_range=0.2,\n",
        "    zoom_range=0.2,\n",
        "    rotation_range=30,\n",
        "    vertical_flip=False,\n",
        "    horizontal_flip=True)\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "N2rA8ajt6zqR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Function `run_model` is used to perform the following sequence of steps:\n",
        "1. Restores model from disk\n",
        "2. Applies model to validation data and reports results\n",
        "3. Applies model to testing data and builds predictions dictionary"
      ]
    },
    {
      "metadata": {
        "id": "QL4bzPVt6zqT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def run_model(path, preprocess, target_size):\n",
        "    \"\"\"Runs fine-tuned model with pre-trained weights on validation\n",
        "    and test data, reporting validation results.\n",
        "    \n",
        "    Args:\n",
        "        path: Path to folder with cached model.\n",
        "        preprocess: Function applied to images before feeding into model. \n",
        "            Should be same preprocessing function which was applied to samples\n",
        "            during model's training.\n",
        "        target_size: Size to rescale validation and testing images. \n",
        "            Should be same value which was used during training process.\n",
        "    \n",
        "    Returns:\n",
        "        classes, file_preds: The list of classes in dataset and dictionary \n",
        "            with predictions arrays (one array per image). \n",
        "    \n",
        "    \"\"\"\n",
        "    \n",
        "    \n",
        "    def as_generator(func):\n",
        "    \"\"\"\n",
        "    Wraps function with infinite generator loop.\n",
        "    \"\"\"\n",
        "    while True:\n",
        "        x, y = yield\n",
        "        transformed = func(x)\n",
        "        yield transformed, y\n",
        "    \n",
        "    \n",
        "    log.info('Restoring model: %s', path)\n",
        "    saved = SavingFolder(path)\n",
        "    model = load_model(saved.best_checkpoint())\n",
        "    \n",
        "    source = KaggleClassifiedImagesSource(LABELS_FILE, 'breed')\n",
        "    train_labels = create_targets(source, TRAIN_IMAGES)\n",
        "    valid_labels = create_targets(source, VALID_IMAGES)\n",
        "    \n",
        "    validation_data = source(\n",
        "        VALID_IMAGES, \n",
        "        target_size=target_size,\n",
        "        batch_size=128,\n",
        "        infinite=False)\n",
        "\n",
        "    pipeline = GeneratorPipeline(validation_data, as_generator(preprocess))\n",
        "    log.info('Running restored model on validation data...')\n",
        "    probs = model.predict_generator(\n",
        "        generator=pipeline,\n",
        "        steps=validation_data.steps_per_epoch)\n",
        "    \n",
        "    preds = probs.argmax(axis=1)\n",
        "    val_loss = log_loss(valid_labels, probs)\n",
        "    accuracy = np.mean(preds == valid_labels)\n",
        "    log.info(f'Validation: loss {val_loss:2.6f} - accuracy {accuracy:2.2%}')\n",
        "    \n",
        "    test_images = KaggleTestImagesIterator(\n",
        "        test_folder=TEST_IMAGES,\n",
        "        target_size=target_size, \n",
        "        batch_size=128,\n",
        "        with_identifiers=True)\n",
        "\n",
        "    file_preds = {}\n",
        "    log.info('Running restored model on test set...')\n",
        "    for images, identifiers in test_images:\n",
        "        preprocessed = preprocess(images)\n",
        "        preds = model.predict_on_batch(preprocessed)\n",
        "        file_preds.update(dict(zip(identifiers, preds)))\n",
        "    \n",
        "    classes = source.binarizer.classes_.tolist()\n",
        "    return classes, file_preds"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vgP5AKrw6zqW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Evaluation and Submissions"
      ]
    },
    {
      "metadata": {
        "id": "Ze_K0epj6zqX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "On my local machine, the following folders with trained models were created:"
      ]
    },
    {
      "metadata": {
        "id": "bFECNB9y6zqX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "MODELS = !ls -1 $TF_OUTPUT_DIR/models/ | grep -E \"*e100$\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rWITDgtq6zqZ",
        "colab_type": "code",
        "colab": {},
        "outputId": "1dc63e52-128c-4378-8878-db959e741eb2"
      },
      "cell_type": "code",
      "source": [
        "MODELS"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['densenet_sgd_m099_avg_bs128_e100',\n",
              " 'inception3_sgd_m099_bs128_e100',\n",
              " 'inception_resnet_sgd_m099_bs128_e100',\n",
              " 'nasnet_sgd_m099_avg_bs128_e100',\n",
              " 'resnet50_top_only_sgd_m099_bs128_e100',\n",
              " 'xception_sgd_m099_avg_bs128_e100']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "metadata": {
        "id": "65S1kaK36zqd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's restore each of models and see validate their performance. Next helper function runs model and submits testing results to Kaggle:"
      ]
    },
    {
      "metadata": {
        "id": "azd1a8r66zqd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def make_kaggle_submission(\n",
        "    path_to_model,\n",
        "    preprocess_input,\n",
        "    target_size,\n",
        "    message):\n",
        "     \n",
        "    classes, predictions = run_model(\n",
        "        path=path_to_model,\n",
        "        preprocess=preprocess_input,\n",
        "        target_size=target_size)\n",
        "        \n",
        "    submission = ClassifierSubmission(log=log)\n",
        "    _, filename = tempfile.mkstemp(suffix='.csv')\n",
        "    submission.create(classes, predictions, output=filename)\n",
        "    \n",
        "    result = submission.submit(filename, 'dog-breed-identification', message=message)\n",
        "    log.info('Kaggle public score: %2.6f', result['publicScore'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mcOT3dXW6zqp",
        "colab_type": "code",
        "colab": {},
        "outputId": "1e4fb06f-9dd0-4823-bc66-128c26618ced"
      },
      "cell_type": "code",
      "source": [
        "from keras.applications import resnet50\n",
        "\n",
        "make_kaggle_submission(\n",
        "    path_to_model='resnet50_top_only_sgd_m099_bs128_e100',\n",
        "    preprocess_input=resnet50.preprocess_input,\n",
        "    target_size=(224, 224, 3),\n",
        "    message='ResNet50 Fine-Tuning')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Restoring model: resnet50_top_only_sgd_m099_bs128_e100\n",
            "Running restored model on validation data...\n",
            "Validation: loss 0.923906 - accuracy 73.39%\n",
            "Running restored model on test set...\n",
            "Sending submission...\n",
            "Successfully submitted to Dog Breed Identification\n",
            "Evaluating results...\n",
            "Kaggle public score: 0.940900\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "lqTzPwLe6zqs",
        "colab_type": "code",
        "colab": {},
        "outputId": "0dbdbfdc-5e96-4517-e372-672a26e7fd75"
      },
      "cell_type": "code",
      "source": [
        "from keras.applications import inception_v3\n",
        "\n",
        "make_kaggle_submission(\n",
        "    path_to_model='inception3_sgd_m099_bs128_e100',\n",
        "    preprocess_input=inception_v3.preprocess_input,\n",
        "    target_size=(299, 299, 3),\n",
        "    message='InceptionV3 Fine-Tuning')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Restoring model: inception3_sgd_m099_bs128_e100\n",
            "Running restored model on validation data...\n",
            "Validation: loss 0.344614 - accuracy 88.26%\n",
            "Running restored model on test set...\n",
            "Sending submission...\n",
            "Successfully submitted to Dog Breed Identification\n",
            "Evaluating results...\n",
            "Kaggle public score: 0.343280\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "7P09PYeM6zqv",
        "colab_type": "code",
        "colab": {},
        "outputId": "defde5c6-fe7e-44bd-c845-02644d1e711a"
      },
      "cell_type": "code",
      "source": [
        "from keras.applications import xception\n",
        "\n",
        "make_kaggle_submission(\n",
        "    path_to_model='xception_sgd_m099_avg_bs128_e100',\n",
        "    preprocess_input=xception.preprocess_input,\n",
        "    target_size=(299, 299, 3),\n",
        "    message='Xception Fine-Tuning')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Restoring model: xception_sgd_m099_avg_bs128_e100\n",
            "Running restored model on validation data...\n",
            "Validation: loss 0.313156 - accuracy 90.31%\n",
            "Running restored model on test set...\n",
            "Sending submission...\n",
            "Successfully submitted to Dog Breed Identification\n",
            "Evaluating results...\n",
            "Kaggle public score: 0.341680\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "ZkQJmkbE6zqx",
        "colab_type": "code",
        "colab": {},
        "outputId": "8b45015c-b75a-4183-ddfc-5687f18be9e9"
      },
      "cell_type": "code",
      "source": [
        "from keras.applications import inception_resnet_v2\n",
        "\n",
        "make_kaggle_submission(\n",
        "    path_to_model='inception_resnet_sgd_m099_bs128_e100',\n",
        "    preprocess_input=inception_resnet_v2.preprocess_input,\n",
        "    target_size=(299, 299, 3),\n",
        "    message='InceptionResNetV2 Fine-Tuning')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Restoring model: inception_resnet_sgd_m099_bs128_e100\n",
            "Running restored model on validation data...\n",
            "Validation: loss 0.256078 - accuracy 91.59%\n",
            "Running restored model on test set...\n",
            "Sending submission...\n",
            "Successfully submitted to Dog Breed Identification\n",
            "Evaluating results...\n",
            "Kaggle public score: 0.280520\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "OofHbMcZ6zq0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Appendix: Visualizing Predictions"
      ]
    },
    {
      "metadata": {
        "id": "wFhgcolH6zq1",
        "colab_type": "code",
        "colab": {},
        "outputId": "6fff1036-9664-4b95-a5b2-3d454df867be"
      },
      "cell_type": "code",
      "source": [
        "import os\n",
        "from os.path import join\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from keras.models import load_model\n",
        "from keras.applications import inception_resnet_v2\n",
        "\n",
        "from swissknife.plotting import plot_predictions\n",
        "from swissknife.images import FallbackImageLoader\n",
        "from swissknife.files import SavingFolder\n",
        "from swissknife.kaggle.datasets import KaggleClassifiedImagesSource\n",
        "\n",
        "# File paths from local folder\n",
        "from basedir import LABELS_FILE"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/home/ck/anaconda3/envs/deep/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
            "  from ._conv import register_converters as _register_converters\n",
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "awCHJg0Z6zrN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "loader = FallbackImageLoader()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_yJNS-Ir6zrQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "IMAGES_ROOT = join(os.getcwd(), 'test_images/')\n",
        "images = [join(IMAGES_ROOT, name) for name in (\n",
        "    'dog1.png',\n",
        "    'dog2.png',\n",
        "    'dog3.png',\n",
        "    'cat1.png'\n",
        ")]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SQ9yZ2PJ6zrS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def load_to_predict(path):\n",
        "    return loader(path, target_size=(299, 299, 3))\n",
        "\n",
        "def load_to_display(path):\n",
        "    return loader(path).astype(np.uint8)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eTBzaK3x6zri",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "saved = SavingFolder('inception_resnet_sgd_m099_bs128_e100')\n",
        "model = load_model(saved.best_checkpoint())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DrX11AdB6zrk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "source = KaggleClassifiedImagesSource(LABELS_FILE, 'breed')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uzhvcdL26zrn",
        "colab_type": "code",
        "colab": {},
        "outputId": "e4fdd996-3259-439d-83da-61b8de891bb9"
      },
      "cell_type": "code",
      "source": [
        "batch = np.asarray([load_to_predict(path) for path in images])\n",
        "batch = inception_resnet_v2.preprocess_input(batch)\n",
        "batch.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(4, 299, 299, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "metadata": {
        "id": "3zRGqHFA6zrq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "preds = model.predict_on_batch(batch)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wL6_oe9r6zrs",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "indexes = np.argsort(-preds, axis=1)[:, :5]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TO6pXI-E6zrx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "labels = [source.integer_to_verbose(i) for i in indexes]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Fp9eBkZ36zrz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "f, axes = plt.subplots(4, 1, figsize=(16, 20))\n",
        "axes = axes.flatten()\n",
        "for i, (path, labels, ax) in enumerate(zip(images, indexes, axes)):\n",
        "    probs = preds[i][labels].tolist()\n",
        "    verbose = [source.integer_to_verbose(i).replace('_', ' ').title() for i in labels]\n",
        "    weights = dict(zip(verbose, probs))\n",
        "    image = load_to_display(path)\n",
        "    plot_predictions(image, weights, labels_font_size=16, sort_by_probability=True, ax=ax)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2cJOeOip6zr1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}