lab2/Part2_FaceDetection.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Ag_e7xtTzT1W"
      },
      "source": [
        "<table align=\"center\">\n",
        "  <td align=\"center\"><a target=\"_blank\" href=\"http://introtodeeplearning.com\">\n",
        "        <img src=\"https://i.ibb.co/Jr88sn2/mit.png\" style=\"padding-bottom:5px;\" />\n",
        "      Visit MIT Deep Learning</a></td>\n",
        "  <td align=\"center\"><a target=\"_blank\" href=\"https://colab.research.google.com/github/aamini/introtodeeplearning/blob/2023/lab2/Part2_FaceDetection.ipynb\">\n",
        "        <img src=\"https://i.ibb.co/2P3SLwK/colab.png\"  style=\"padding-bottom:5px;\" />Run in Google Colab</a></td>\n",
        "  <td align=\"center\"><a target=\"_blank\" href=\"https://github.com/aamini/introtodeeplearning/blob/2023/lab2/Part2_FaceDetection.ipynb\">\n",
        "        <img src=\"https://i.ibb.co/xfJbPmL/github.png\"  height=\"70px\" style=\"padding-bottom:5px;\"  />View Source on GitHub</a></td>\n",
        "</table>\n",
        "\n",
        "# Copyright Information"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rNbf1pRlSDby"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 MIT Introduction to Deep Learning. All Rights Reserved.\n",
        "# \n",
        "# Licensed under the MIT License. You may not use this file except in compliance\n",
        "# with the License. Use and/or modification of this code outside of MIT Introduction\n",
        "# to Deep Learning must reference:\n",
        "#\n",
        "# © MIT Introduction to Deep Learning\n",
        "# http://introtodeeplearning.com\n",
        "#"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOpPUH3FR179"
      },
      "source": [
        "# Laboratory 2: Computer Vision\n",
        "\n",
        "# Part 2: Diagnosing Bias in Facial Detection Systems\n",
        "\n",
        "In this lab, we'll explore a prominent aspect of applied deep learning for computer vision: facial detection. \n",
        "\n",
        "Consider the task of facial detection: given an image, is it an image of a face?  This seemingly simple -- but extremely important and pervasive -- task is subject to significant amounts of algorithmic bias among select demographics, as [seminal studies](https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf) have shown.\n",
        "\n",
        "Deploying fair, unbiased AI systems is critical to their long-term acceptance. In this lab, we will build computer vision models for facial detection. We will extend beyond that to build a model to **uncover and diagnose** the biases and issues that exist with standard facial detection models. To do this, we will build a semi-supervised variational autoencoder (SS-VAE) that learns the *latent distribution* of features underlying face image datasets in order to [uncover hidden biases](http://introtodeeplearning.com/AAAI_MitigatingAlgorithmicBias.pdf).\n",
        "\n",
        "Our work here will set the foundation for the next lab, where we'll build automated tools to mitigate the underlying issues of bias and uncertainty in facial detection."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Ezfc6Yv6IhI"
      },
      "source": [
        "Let's get started by installing the relevant dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E46sWVKK6LP9"
      },
      "outputs": [],
      "source": [
        "# Import Tensorflow 2.0\n",
        "%tensorflow_version 2.x\n",
        "import tensorflow as tf\n",
        "\n",
        "import IPython\n",
        "import functools\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from tqdm import tqdm\n",
        "\n",
        "\n",
        "# Download and import the MIT Introduction to Deep Learning package\n",
        "!pip install mitdeeplearning\n",
        "import mitdeeplearning as mdl"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0e77oOM3udR"
      },
      "source": [
        "## 2.1 Datasets\n",
        "\n",
        "In order to train our facial detection models, we will need a dataset of positive examples (i.e., of faces) and a dataset of negative examples (i.e., of things that are not faces). We will use these data to train our models to classify images as either faces or not faces.\n",
        "\n",
        "1.   **Positive training data**: [CelebA Dataset](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). A large-scale dataset (over 200K images) of celebrity faces.   \n",
        "2.   **Negative training data**: [ImageNet](http://www.image-net.org/). A large-scale dataset with many images across many different categories. We will take negative examples from a variety of non-human categories.\n",
        "\n",
        "We will evaluate trained models on an independent test dataset of face images to diagnose potential issues with *bias, fairness, and confidence*.\n",
        "\n",
        "Let's begin by importing these datasets. We've written a class that does a bit of data pre-processing to import the training data in a usable format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RWXaaIWy6jVw"
      },
      "outputs": [],
      "source": [
        "# Get the training data: both images from CelebA and ImageNet\n",
        "path_to_training_data = tf.keras.utils.get_file('train_face.h5', 'https://www.dropbox.com/s/hlz8atheyozp1yx/train_face.h5?dl=1')\n",
        "# Instantiate a TrainingDatasetLoader using the downloaded dataset\n",
        "loader = mdl.lab2.TrainingDatasetLoader(path_to_training_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yIE321rxa_b3"
      },
      "source": [
        "We can look at the size of the training dataset and grab a batch of size 100:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DjPSjZZ_bGqe"
      },
      "outputs": [],
      "source": [
        "number_of_training_examples = loader.get_train_size()\n",
        "(images, labels) = loader.get_batch(100)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxtkJoqF6oH1"
      },
      "source": [
        "Play around with displaying images to get a sense of what the training data actually looks like!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "Jg17jzwtbxDA"
      },
      "outputs": [],
      "source": [
        "#@title Change the sliders to look at positive and negative training examples! { run: \"auto\" }\n",
        "\n",
        "### Examining the CelebA training dataset ###\n",
        "\n",
        "face_images = images[np.where(labels==1)[0]]\n",
        "not_face_images = images[np.where(labels==0)[0]]\n",
        "\n",
        "idx_face = 19 #@param {type:\"slider\", min:0, max:50, step:1}\n",
        "idx_not_face = 8 #@param {type:\"slider\", min:0, max:50, step:1}\n",
        "\n",
        "plt.figure(figsize=(8,4))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.imshow(face_images[idx_face])\n",
        "plt.title(\"Face\"); plt.grid(False)\n",
        "\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.imshow(not_face_images[idx_not_face])\n",
        "plt.title(\"Not Face\"); plt.grid(False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NDj7KBaW8Asz"
      },
      "source": [
        "### Thinking about bias\n",
        "\n",
        "We will be training our facial detection classifiers on the large, well-curated CelebA dataset (and ImageNet), and then evaluate their accuracy as well as inspect and diagnose their hidden flaws. Our goal is to identify any potential issues and biases that may exist with the trained facial detection classifiers, and then diagnose what those issues and biases are.\n",
        "\n",
        "What exactly do we mean when we say a classifier is biased? In order to formalize this, we'll need to think about [*latent variables*](https://en.wikipedia.org/wiki/Latent_variable), variables that define a dataset but are not strictly observed. As defined in the generative modeling lecture, we use the term *latent space* to refer to the probability distributions of the aforementioned latent variables. Putting these ideas together, we consider a classifier *biased* if its classification decision changes after it sees some additional latent features or variables. This definition of bias will be helpful to keep in mind throughout the rest of the lab."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AIFDvU4w8OIH"
      },
      "source": [
        "## 2.2 CNN for facial detection \n",
        "\n",
        "First, we will define and train a baseline CNN on the facial detection task of classifying whether a given image is a face, or is not a face. We will then evaluate its accuracy. The CNN model has a relatively standard architecture consisting of a series of convolutional layers with batch normalization followed by two fully connected layers to flatten the convolution output and generate a class prediction. \n",
        "\n",
        "### Define and train the CNN model\n",
        "\n",
        "Like we did in the first part of the lab, we will define our CNN model, and then train on the CelebA and ImageNet datasets using the `tf.GradientTape` class and the `tf.GradientTape.gradient` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "82EVTAAW7B_X"
      },
      "outputs": [],
      "source": [
        "### Define the CNN model ###\n",
        "\n",
        "n_filters = 12 # base number of convolutional filters\n",
        "\n",
        "'''Function to define a standard CNN model'''\n",
        "def make_standard_classifier(n_outputs=1):\n",
        "  Conv2D = functools.partial(tf.keras.layers.Conv2D, padding='same', activation='relu')\n",
        "  BatchNormalization = tf.keras.layers.BatchNormalization\n",
        "  Flatten = tf.keras.layers.Flatten\n",
        "  Dense = functools.partial(tf.keras.layers.Dense, activation='relu')\n",
        "\n",
        "  model = tf.keras.Sequential([ \n",
        "    Conv2D(filters=1*n_filters, kernel_size=5,  strides=2),\n",
        "    BatchNormalization(),\n",
        "    \n",
        "    Conv2D(filters=2*n_filters, kernel_size=5,  strides=2),\n",
        "    BatchNormalization(),\n",
        "\n",
        "    Conv2D(filters=4*n_filters, kernel_size=3,  strides=2),\n",
        "    BatchNormalization(),\n",
        "\n",
        "    Conv2D(filters=6*n_filters, kernel_size=3,  strides=2),\n",
        "    BatchNormalization(),\n",
        "\n",
        "    Flatten(),\n",
        "    Dense(512),\n",
        "    Dense(n_outputs, activation=None),\n",
        "  ])\n",
        "  return model\n",
        "\n",
        "standard_classifier = make_standard_classifier()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-eWf3l_lCri"
      },
      "source": [
        "Now let's train the standard CNN!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eJlDGh1o31G1"
      },
      "outputs": [],
      "source": [
        "### Train the standard CNN ###\n",
        "\n",
        "# Training hyperparameters\n",
        "batch_size = 32\n",
        "num_epochs = 2  # keep small to run faster\n",
        "learning_rate = 5e-4\n",
        "\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate) # define our optimizer\n",
        "loss_history = mdl.util.LossHistory(smoothing_factor=0.99) # to record loss evolution\n",
        "plotter = mdl.util.PeriodicPlotter(sec=2, scale='semilogy')\n",
        "if hasattr(tqdm, '_instances'): tqdm._instances.clear() # clear if it exists\n",
        "\n",
        "@tf.function\n",
        "def standard_train_step(x, y):\n",
        "  with tf.GradientTape() as tape:\n",
        "    # feed the images into the model\n",
        "    logits = standard_classifier(x) \n",
        "    # Compute the loss\n",
        "    loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=logits)\n",
        "\n",
        "  # Backpropagation\n",
        "  grads = tape.gradient(loss, standard_classifier.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(grads, standard_classifier.trainable_variables))\n",
        "  return loss\n",
        "\n",
        "# The training loop!\n",
        "for epoch in range(num_epochs):\n",
        "  for idx in tqdm(range(loader.get_train_size()//batch_size)):\n",
        "    # Grab a batch of training data and propagate through the network\n",
        "    x, y = loader.get_batch(batch_size)\n",
        "    loss = standard_train_step(x, y)\n",
        "\n",
        "    # Record the loss and plot the evolution of the loss as a function of training\n",
        "    loss_history.append(loss.numpy().mean())\n",
        "    plotter.plot(loss_history.get())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AKMdWVHeCxj8"
      },
      "source": [
        "### Evaluate performance of the standard CNN\n",
        "\n",
        "Next, let's evaluate the classification performance of our CelebA-trained standard CNN on the training dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "35-PDgjdWk6_"
      },
      "outputs": [],
      "source": [
        "### Evaluation of standard CNN ###\n",
        "\n",
        "# TRAINING DATA\n",
        "# Evaluate on a subset of CelebA+Imagenet\n",
        "(batch_x, batch_y) = loader.get_batch(5000)\n",
        "y_pred_standard = tf.round(tf.nn.sigmoid(standard_classifier.predict(batch_x)))\n",
        "acc_standard = tf.reduce_mean(tf.cast(tf.equal(batch_y, y_pred_standard), tf.float32))\n",
        "\n",
        "print(\"Standard CNN accuracy on (potentially biased) training set: {:.4f}\".format(acc_standard.numpy()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qu7R14KaEEvU"
      },
      "source": [
        "## 2.3 Diagnosing algorithmic bias\n",
        "\n",
        "CNNs like the one we just built are pervasive as the standard solution for facial detection pipelines implemented throughout society. Despite their pervasiveness, these models -- including those implemented by top tech companies -- suffer from tremendous amounts of algorithmic bias. The seminal work of [Buolamwini and Gebru](https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf) provided an approach and benchmark dataset to evaluate facial analysis algorithms, revealing startling accuracy discrepancies across skin tone and gender demographics.\n",
        "\n",
        "In order to solve this problem and build fair and robust models, the first step is to determine the source of the problem. How can we determine the ***source*** of these accuracy discrepancies to identify and diagnose biases?\n",
        "\n",
        "### Naive approach\n",
        "\n",
        "A naive approach -- and one that is being adopted by many companies and organizations -- would be to annotate different subclasses (i.e., light-skinned females, males with hats, etc.) within the training data, and then evaluate classifier performance with respect to these groups.\n",
        "\n",
        "But this approach has two major disadvantages. First, it requires annotating massive amounts of data, which is not scalable. Second, it requires that we know what potential biases (e.g., race, gender, pose, occlusion, hats, glasses, etc.) to look for in the data. As a result, manual annotation may not capture all the different sources of bias and uncertainty that may exist.\n",
        "\n",
        "### Automatically uncovering hidden biases\n",
        "\n",
        "Imbalances in the training data can result in unwanted algorithmic bias. For example, the majority of faces in CelebA (our training set) are those of light-skinned females. As a result, a classifier trained on CelebA will be better suited at recognizing and classifying faces with features similar to these, and will thus be biased.\n",
        "\n",
        "What if we actually ***learned*** the distribution of data features in an unbiased, unsupervised manner, without the need for any annotation? What could such an approach tell us about hidden biases that may exist in the data, or regions of the data in which the model is less confident in its predictions?\n",
        "\n",
        "In the rest of this lab, we will tackle exactly these questions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLemS7dqECsI"
      },
      "source": [
        "## 2.4 Variational autoencoder (VAE) for learning latent structure\n",
        "\n",
        "The accuracy of facial detection classifiers can vary significantly across different demographics. Consider the dataset the CNN model was trained on, CelebA. If certain features, such as dark skin or hats, are *rare* in CelebA, the model may end up biased against these as a result of training with a biased dataset. That is to say, its classification accuracy will be worse on faces that have under-represented features, such as dark-skinned faces or faces with hats, relevative to faces with features well-represented in the training data! This is a problem.\n",
        "\n",
        "Our goal is to train a model that **learns a representation of the underlying latent space** to the face training data. Such a learned representation will provide information on what features are under-represented or over-represented in the data. The key design requirement for our model is that it can learn an *encoding* of the latent features in the face data in an entirely *unsupervised* way, without any supervised annotation by us humans. To achieve this, we turn to variational autoencoders (VAEs).\n",
        "\n",
        "![The concept of a VAE](https://i.ibb.co/3s4S6Gc/vae.jpg)\n",
        "\n",
        "As shown in the schematic above and in Lecture 4, VAEs rely on an encoder-decoder structure to learn a latent representation of the input data. In the context of computer vision, the encoder network takes in input images, encodes them into a series of variables defined by a mean and standard deviation, and then draws from the distributions defined by these parameters to generate a set of sampled latent variables. The decoder network then \"decodes\" these variables to generate a reconstruction of the original image, which is used during training to help the model identify which latent variables are important to learn. \n",
        "\n",
        "Let's formalize two key aspects of the VAE model and define relevant functions for each."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KmbXKtcPkTXA"
      },
      "source": [
        "### Understanding VAEs: loss function\n",
        "\n",
        "In practice, how can we train a VAE? In learning the latent space, we constrain the means and standard deviations to approximately follow a unit Gaussian. Recall that these are learned parameters, and therefore must factor into the loss computation, and that the decoder portion of the VAE is using these parameters to output a reconstruction that should closely match the input image, which also must factor into the loss. What this means is that we'll have two terms in our VAE loss function:\n",
        "\n",
        "1.  **Latent loss ($L_{KL}$)**: measures how closely the learned latent variables match a unit Gaussian and is defined by the Kullback-Leibler (KL) divergence.\n",
        "2.   **Reconstruction loss ($L_{x}{(x,\\hat{x})}$)**: measures how accurately the reconstructed outputs match the input and is given by the $L^1$ norm of the input image and its reconstructed output."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ux3jK2wc153s"
      },
      "source": [
        "The equation for the latent loss is provided by:\n",
        "\n",
        "$$L_{KL}(\\mu, \\sigma) = \\frac{1}{2}\\sum_{j=0}^{k-1} (\\sigma_j + \\mu_j^2 - 1 - \\log{\\sigma_j})$$\n",
        "\n",
        "The equation for the reconstruction loss is provided by:\n",
        "\n",
        "$$L_{x}{(x,\\hat{x})} = ||x-\\hat{x}||_1$$\n",
        "\n",
        "Thus for the VAE loss we have:\n",
        "\n",
        "$$L_{VAE} = c\\cdot L_{KL} + L_{x}{(x,\\hat{x})}$$\n",
        "\n",
        "where $c$ is a weighting coefficient used for regularization. Now we're ready to define our VAE loss function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S00ASo1ImSuh"
      },
      "outputs": [],
      "source": [
        "### Defining the VAE loss function ###\n",
        "\n",
        "''' Function to calculate VAE loss given:\n",
        "      an input x, \n",
        "      reconstructed output x_recon, \n",
        "      encoded means mu, \n",
        "      encoded log of standard deviation logsigma, \n",
        "      weight parameter for the latent loss kl_weight\n",
        "'''\n",
        "def vae_loss_function(x, x_recon, mu, logsigma, kl_weight=0.0005):\n",
        "  # TODO: Define the latent loss. Note this is given in the equation for L_{KL}\n",
        "  # in the text block directly above\n",
        "  latent_loss = # TODO\n",
        "\n",
        "  # TODO: Define the reconstruction loss as the mean absolute pixel-wise \n",
        "  # difference between the input and reconstruction. Hint: you'll need to \n",
        "  # use tf.reduce_mean, and supply an axis argument which specifies which \n",
        "  # dimensions to reduce over. For example, reconstruction loss needs to average \n",
        "  # over the height, width, and channel image dimensions.\n",
        "  # https://www.tensorflow.org/api_docs/python/tf/math/reduce_mean\n",
        "  reconstruction_loss = # TODO\n",
        "\n",
        "  # TODO: Define the VAE loss. Note this is given in the equation for L_{VAE}\n",
        "  # in the text block directly above\n",
        "  vae_loss = # TODO\n",
        "  \n",
        "  return vae_loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqtQH4S5fO8F"
      },
      "source": [
        "### Understanding VAEs: sampling and reparameterization \n",
        "\n",
        "As you may recall from lecture, VAEs use a \"reparameterization  trick\" for sampling learned latent variables. Instead of the VAE encoder generating a single vector of real numbers for each latent variable, it generates a vector of means and a vector of standard deviations that are constrained to roughly follow Gaussian distributions. We then sample a noise value $\\epsilon$ from a Gaussian distribution, and then scale it by the standard deviation and add back the mean to output the result as our sampled latent vector. Formalizing this for a latent variable $z$ where we sample $\\epsilon \\sim N(0,(I))$ we have:\n",
        "\n",
        "$$z = \\mu + e^{\\left(\\frac{1}{2} \\cdot \\log{\\Sigma}\\right)}\\circ \\epsilon$$\n",
        "\n",
        "where $\\mu$ is the mean and $\\Sigma$ is the covariance matrix. This is useful because it will let us neatly define the loss function for the VAE, generate randomly sampled latent variables, achieve improved network generalization, **and** make our complete VAE network differentiable so that it can be trained via backpropagation. Quite powerful!\n",
        "\n",
        "Let's define a function to implement the VAE sampling operation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cT6PGdNajl3K"
      },
      "outputs": [],
      "source": [
        "### VAE Sampling ###\n",
        "\n",
        "\"\"\"Sample latent variables via reparameterization with an isotropic unit Gaussian.\n",
        "# Arguments\n",
        "    z_mean, z_logsigma (tensor): mean and log of standard deviation of latent distribution (Q(z|X))\n",
        "# Returns\n",
        "    z (tensor): sampled latent vector\n",
        "\"\"\"\n",
        "def sampling(z_mean, z_logsigma):\n",
        "  # By default, random.normal is \"standard\" (ie. mean=0 and std=1.0)\n",
        "  batch, latent_dim = z_mean.shape\n",
        "  epsilon = tf.random.normal(shape=(batch, latent_dim))\n",
        "\n",
        "  # TODO: Define the reparameterization computation!\n",
        "  # Note the equation is given in the text block immediately above.\n",
        "  z = # TODO\n",
        "  \n",
        "  return z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bcpznUHHuR6I"
      },
      "source": [
        "Great! Now that we have a more concrete sense of how VAEs work, let's explore how we can leverage this network structure to diagnoses hidden biases in facial detection classifiers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qtHEYI9KNn0A"
      },
      "source": [
        "## 2.5 Semi-supervised variational autoencoder (SS-VAE)\n",
        "\n",
        "Now, we will use the general idea behind the VAE architecture to build a model to automatically uncover (potentially) unknown biases present within the training data, while simultaneously learning the facial detection task. This draws direct inspiration from [a recent paper](http://introtodeeplearning.com/AAAI_MitigatingAlgorithmicBias.pdf) proposing this as a general approach for automatic bias detection and mitigation.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3IOB3d61WSN"
      },
      "source": [
        "### Semi-supervised VAE architecture\n",
        "\n",
        "We will develop a VAE that has a supervised component in order to both output a classification decision for the facial detection task and analyze where the biases in our model may be resulting from. While previous works like that of  Buolamwini and Gebru have focused on skin tone and gender as two categories where facial detection models may be experiencing bias, there may be other unlabeled features that also are biased, resulting in poorer classification performance. We will build our semi-supervised VAE (SS-VAE) to learn these underlying latent features.\n",
        "\n",
        "A general schematic of the SS-VAE architecture is shown here.\n",
        "\n",
        "![SS-VAE](https://raw.githubusercontent.com/aamini/introtodeeplearning/master/lab2/img/SS-VAE.png)\n",
        "\n",
        "We will apply our SS-VAE to a *supervised classification* problem -- the facial detection task. Importantly, note how the encoder portion in the SS-VAE architecture also outputs a single supervised variable, $z_o$, corresponding to the class prediction -- face or not face. Usually, VAEs are not trained to output any supervised variables (such as a class prediction)! This is the key distinction between the SS-VAE and a traditional VAE. \n",
        "\n",
        "Keep in mind that we only want to learn the latent representation of *faces*, as that is where we are interested in uncovering potential biases, even though we are training a model on a binary classification problem. So, we will need to ensure that, **for faces**, our SS-VAE model both learns a representation of the unsupervised latent variables, captured by the distribution $q_\\phi(z|x)$, and outputs a supervised class prediction $z_o$, but that, **for negative examples**, it only outputs a class prediction $z_o$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XggIKYPRtOZR"
      },
      "source": [
        "### Defining the SS-VAE loss function\n",
        "\n",
        "This means we'll need to be a bit clever about the loss function for the SS-VAE. The form of the loss will depend on whether it's a face image or a non-face image that's being considered. \n",
        "\n",
        "For **face images**, our loss function will have two components:\n",
        "\n",
        "1.   **VAE loss ($L_{VAE}$)**: consists of the latent loss and the reconstruction loss.\n",
        "2.   **Classification loss ($L_y(y,\\hat{y})$)**: standard cross-entropy loss for a binary classification problem. \n",
        "\n",
        "In contrast, for images of **non-faces**, our loss function is solely the classification loss. \n",
        "\n",
        "We can write a single expression for the loss by defining an indicator variable ${I}_f$which reflects which training data are images of faces (${I}_f(y) = 1$ ) and which are images of non-faces (${I}_f(y) = 0$). Using this, we obtain:\n",
        "\n",
        "$$L_{total} = L_y(y,\\hat{y}) + {I}_f(y)\\Big[L_{VAE}\\Big]$$\n",
        "\n",
        "Let's write a function to define the SS-VAE loss function:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VjieDs8Ovcqs"
      },
      "outputs": [],
      "source": [
        "### Loss function for SS-VAE ###\n",
        "\n",
        "\"\"\"Loss function for SS-VAE.\n",
        "# Arguments\n",
        "    x: true input x\n",
        "    x_pred: reconstructed x\n",
        "    y: true label (face or not face)\n",
        "    y_logit: predicted labels\n",
        "    mu: mean of latent distribution (Q(z|X))\n",
        "    logsigma: log of standard deviation of latent distribution (Q(z|X))\n",
        "# Returns\n",
        "    total_loss: SS-VAE total loss\n",
        "    classification_loss: SS-VAE classification loss\n",
        "\"\"\"\n",
        "def ss_vae_loss_function(x, x_pred, y, y_logit, mu, logsigma):\n",
        "\n",
        "  # TODO: call the relevant function to obtain VAE loss, defined earlier in the lab\n",
        "  vae_loss = vae_loss_function('''TODO''') # TODO\n",
        "\n",
        "  # TODO: define the classification loss using sigmoid_cross_entropy\n",
        "  # https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits\n",
        "  classification_loss = # TODO\n",
        "\n",
        "  # Use the training data labels to create variable face_indicator:\n",
        "  #   indicator that reflects which training data are images of faces\n",
        "  face_indicator = tf.cast(tf.equal(y, 1), tf.float32)\n",
        "\n",
        "  # TODO: define the SS-VAE total loss! Use tf.reduce_mean to average over all\n",
        "  # samples\n",
        "  total_loss = # TODO\n",
        "\n",
        "  return total_loss, classification_loss, vae_loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YIu_2LzNWwWY"
      },
      "source": [
        "### Defining the SS-VAE architecture\n",
        "\n",
        "Now we're ready to define the SS-VAE architecture. To build the SS-VAE, we will use the standard CNN classifier from above as our encoder, and then define a decoder network. We will create and initialize the encoder and decoder networks, and then construct the end-to-end VAE. We will use a latent space with 32 latent variables.\n",
        "\n",
        "The decoder network will take as input the sampled latent variables, run them through a series of deconvolutional layers, and output a reconstruction of the original input image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JfWPHGrmyE7R"
      },
      "outputs": [],
      "source": [
        "### Define the decoder portion of the SS-VAE ###\n",
        "\n",
        "def make_face_decoder_network(n_filters=12):\n",
        "\n",
        "  # Functionally define the different layer types we will use\n",
        "  Conv2DTranspose = functools.partial(tf.keras.layers.Conv2DTranspose, padding='same', activation='relu')\n",
        "  BatchNormalization = tf.keras.layers.BatchNormalization\n",
        "  Flatten = tf.keras.layers.Flatten\n",
        "  Dense = functools.partial(tf.keras.layers.Dense, activation='relu')\n",
        "  Reshape = tf.keras.layers.Reshape\n",
        "\n",
        "  # Build the decoder network using the Sequential API\n",
        "  decoder = tf.keras.Sequential([\n",
        "    # Transform to pre-convolutional generation\n",
        "    Dense(units=4*4*6*n_filters),  # 4x4 feature maps (with 6N occurences)\n",
        "    Reshape(target_shape=(4, 4, 6*n_filters)),\n",
        "\n",
        "    # Upscaling convolutions (inverse of encoder)\n",
        "    Conv2DTranspose(filters=4*n_filters, kernel_size=3,  strides=2),\n",
        "    Conv2DTranspose(filters=2*n_filters, kernel_size=3,  strides=2),\n",
        "    Conv2DTranspose(filters=1*n_filters, kernel_size=5,  strides=2),\n",
        "    Conv2DTranspose(filters=3, kernel_size=5,  strides=2),\n",
        "  ])\n",
        "\n",
        "  return decoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yWCMu12w1BuD"
      },
      "source": [
        "Now, we will put this decoder together with the standard CNN classifier as our encoder to define the SS-VAE. Here, we will define the core VAE architecture by sublassing the `Model` class; defining encoding, sampling, and decoding operations; and calling the network end-to-end."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dSFDcFBL13c3"
      },
      "outputs": [],
      "source": [
        "### Defining and creating the SS-VAE ###\n",
        "\n",
        "class SS_VAE(tf.keras.Model):\n",
        "  def __init__(self, latent_dim):\n",
        "    super(SS_VAE, self).__init__()\n",
        "    self.latent_dim = latent_dim\n",
        "\n",
        "    # Define the number of outputs for the encoder. Recall that we have \n",
        "    # `latent_dim` latent variables, as well as a supervised output for the \n",
        "    # classification.\n",
        "    num_encoder_dims = 2*self.latent_dim + 1\n",
        "\n",
        "    self.encoder = make_standard_classifier(num_encoder_dims)\n",
        "    self.decoder = make_face_decoder_network()\n",
        "\n",
        "  # function to feed images into encoder, encode the latent space, and output\n",
        "  #   classification probability \n",
        "  def encode(self, x):\n",
        "    # encoder output\n",
        "    encoder_output = self.encoder(x)\n",
        "\n",
        "    # classification prediction\n",
        "    y_logit = tf.expand_dims(encoder_output[:, 0], -1)\n",
        "    # latent variable distribution parameters\n",
        "    z_mean = encoder_output[:, 1:self.latent_dim+1] \n",
        "    z_logsigma = encoder_output[:, self.latent_dim+1:]\n",
        "\n",
        "    return y_logit, z_mean, z_logsigma\n",
        "\n",
        "  # Decode the latent space and output reconstruction\n",
        "  def decode(self, z):\n",
        "    # TODO: use the decoder (self.decoder) to output the reconstruction\n",
        "    reconstruction = # TODO\n",
        "    return reconstruction\n",
        "\n",
        "  # The call function will be used to pass inputs x through the core VAE\n",
        "  def call(self, x): \n",
        "    # Encode input to a prediction and latent space\n",
        "    y_logit, z_mean, z_logsigma = self.encode(x)\n",
        "\n",
        "    # TODO: call the sampling function that you created above using \n",
        "    #       z_mean and z_logsigma\n",
        "    z = # TODO\n",
        "\n",
        "    # TODO: reconstruction\n",
        "    recon = # TODO\n",
        "    \n",
        "    return y_logit, z_mean, z_logsigma, recon\n",
        "\n",
        "  # Predict face or not face logit for given input x\n",
        "  def predict(self, x):\n",
        "    y_logit, z_mean, z_logsigma = self.encode(x)\n",
        "    return y_logit\n",
        "\n",
        "ss_vae = SS_VAE(latent_dim=32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M-clbYAj2waY"
      },
      "source": [
        "As stated, the encoder architecture is identical to the CNN from earlier in this lab. Note the outputs of our constructed SS-VAE model in the `call` function: `y_logit, z_mean, z_logsigma, z`. Think carefully about why each of these are outputted and their significance to the problem at hand.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nbDNlslgQc5A"
      },
      "source": [
        "### Training the SS-VAE\n",
        "\n",
        "We are ready to train our SS-VAE model! Complete the `TODO`s in the following training loop to train the SS-VAE with face classification output."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xwQs-Gu5bKEK"
      },
      "outputs": [],
      "source": [
        "### Training the SS-VAE ###\n",
        "\n",
        "# Hyperparameters\n",
        "batch_size = 32\n",
        "learning_rate = 5e-4\n",
        "latent_dim = 32\n",
        "\n",
        "# SS-VAE needs slightly more epochs to train since its more complex than \n",
        "# the standard classifier so we use 6 instead of 2\n",
        "num_epochs = 6\n",
        "\n",
        "# instantiate a new SS-VAE model and optimizer\n",
        "ss_vae = SS_VAE(latent_dim)\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate)\n",
        "\n",
        "# To define the training operation, we will use tf.function which is a powerful tool \n",
        "#   that lets us turn a Python function into a TensorFlow computation graph.\n",
        "@tf.function\n",
        "def ss_vae_train_step(x, y):\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "    # Feed input x into ss_vae. Note that this is using the SS_VAE call function!\n",
        "    y_logit, z_mean, z_logsigma, x_recon = ss_vae(x)\n",
        "\n",
        "    '''TODO: call the SS_VAE loss function to compute the loss'''\n",
        "    loss, class_loss = ss_vae_loss_function('''TODO arguments''') # TODO\n",
        "  \n",
        "  '''TODO: use the GradientTape.gradient method to compute the gradients.\n",
        "     Hint: this is with respect to the trainable_variables of the SS_VAE.'''\n",
        "  grads = tape.gradient('''TODO''', '''TODO''') # TODO\n",
        "\n",
        "  # apply gradients to variables\n",
        "  optimizer.apply_gradients(zip(grads, ss_vae.trainable_variables))\n",
        "  return loss\n",
        "\n",
        "# get training faces from data loader\n",
        "all_faces = loader.get_all_train_faces()\n",
        "\n",
        "if hasattr(tqdm, '_instances'): tqdm._instances.clear() # clear if it exists\n",
        "\n",
        "# The training loop -- outer loop iterates over the number of epochs\n",
        "for i in range(num_epochs):\n",
        "\n",
        "  IPython.display.clear_output(wait=True)\n",
        "  print(\"Starting epoch {}/{}\".format(i+1, num_epochs))\n",
        "  \n",
        "  # get a batch of training data and compute the training step\n",
        "  for j in tqdm(range(loader.get_train_size() // batch_size)):\n",
        "    # load a batch of data\n",
        "    (x, y) = loader.get_batch(batch_size)\n",
        "    # loss optimization\n",
        "    loss = ss_vae_train_step(x, y)\n",
        "    \n",
        "    # plot the progress every 200 steps\n",
        "    if j % 500 == 0: \n",
        "      mdl.util.plot_sample(x, y, ss_vae)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uZBlWDPOVcHg"
      },
      "source": [
        "Wonderful! Now we should have a trained SS-VAE facial classification model, ready for evaluation!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eo34xC7MbaiQ"
      },
      "source": [
        "## 2.6 Using the SS-VAE to uncover and diagnose biases\n",
        "\n",
        "With the SS-VAE model trained, we are ready to use it to uncover and diagnose hidden biases that exist within the dataset.\n",
        "\n",
        "Recall that our goal with the SS-VAE was to learn the underlying ***latent distribution*** of features in the training dataset, in order to uncover potential feature representation disparities that exist within the data.\n",
        "\n",
        "Additionally, training the SS-VAE required both a VAE reconstruction loss as well as a supervised classification loss. The VAE reconstruction loss direclty reflects how well the model is able to handle particular input data -- the higher the reconstruction loss, the harder that particular example is for the model to learn.\n",
        "\n",
        "We consider both these aspects to understand sources of uncertainty and bias within the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QfVngr5J6sj3"
      },
      "source": [
        "### Linking model performance to uncertainty and bias\n",
        "\n",
        "We begin by considering the examples in the dataset with the highest loss. What can you tell about which features seemed harder to learn for the VAE? What might this tell us about where the model struggles, and what predictions it may be more biased or uncertain about?\n",
        "\n",
        "#### **TODO: Analysis and reflection**\n",
        "\n",
        "Complete the analysis in the code block below. Write short (~1 sentence) answers to the following questions and include them in your Debiasing Faces Lab submission to complete the `TODO`s!\n",
        "\n",
        "1. What, if any, trends do you observe comparing the samples with the highest and lowest reconstruction loss?\n",
        "2. Based on these observations, which features seemed harder to learn for the VAE?\n",
        "3. How does reconstruction loss relate to uncertainty? Think back to our lecture on Robust & Trustworthy Deep Learning! What can you say about examples on which the model may be more or less uncertain?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BRAwm0Jlrktc"
      },
      "outputs": [],
      "source": [
        "### Linking model performance to uncertainty and bias\n",
        "\n",
        "# Load a random sample of 5000 faces from our dataset and compute the model performance on them\n",
        "(x, y) = loader.get_batch(5000, only_faces=True)\n",
        "y_logit, z_mean, z_logsigma, x_recon = ss_vae(x)\n",
        "loss, class_loss, vae_loss = ss_vae_loss_function(x, x_recon, y, y_logit, z_mean, z_logsigma)\n",
        "\n",
        "# Sort the results by the vae loss scores\n",
        "vae_loss = vae_loss.numpy()\n",
        "ind = np.argsort(vae_loss, axis=None)\n",
        "\n",
        "# Plot the 25 samples with the highest and lowest reconstruction losses\n",
        "fig, ax = plt.subplots(1, 2, figsize=(16, 8))\n",
        "ax[0].imshow(mdl.util.create_grid_of_images(x[ind[:25]]))\n",
        "ax[0].set_title(\"Samples with the lowest reconstruction loss \\n\" + \n",
        "                f\"Average recon loss: {np.mean(vae_loss[ind[:25]]):.2f}\")\n",
        "\n",
        "ax[1].imshow(mdl.util.create_grid_of_images(x[ind[-25:]]))\n",
        "ax[1].set_title(\"Samples with the highest reconstruction loss \\n\" + \n",
        "                f\"Average recon loss: {np.mean(vae_loss[ind[-25:]]):.2f}\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8SQSszTFjstZ"
      },
      "source": [
        "### Uncovering hidden biases through learned latent features \n",
        "\n",
        "As we've seen above, loss is a powerful way to visualize which samples in our dataset the model has had trouble learning -- these examples are those that have high *model uncertainty*. However, this is not necessarily the same as bias!\n",
        "\n",
        "How can we determine the relative frequencies and distributions of different latent features learned by the model? How may these metrics reveal underlying biases?\n",
        "\n",
        "Let's investigate how well the SS-VAE actually learned the latent features of the faces. To do this, we will inspect individual latent features -- holding all others constant -- and look at the distribution of these features in the data and their corresponding examples. We can examine the shape and probability density of the learned latent features. Further we directly compare different values of individual latent variables to corresponding relative classification accuracies (marginalizing out the effects of the other latent variables).\n",
        "\n",
        "#### **TODO: Analysis and reflection**\n",
        "\n",
        "Complete the analysis in the code blocks below. Carefully inspect the different latent variables and their corresponding frequency distributions. Write short (~1 sentence) answers to the following questions and include them in your Debiasing Faces Lab submission to complete the `TODO`s!\n",
        "\n",
        "1. Pick two latent variables and describe what semantic meaning they reflect. Include screenshots of the realizations and probability distribution for the latent variables you select.\n",
        "2. For the latent variables selected, what can you tell about which features are under- or over-represented in the data? What might this tell us about how the model is biased?\n",
        "3. For the latent variables selected, how do these feature distribution differences affect classification performance? What, if any, general trends do you observe across the latent variables?\n",
        "4. Based on these observations, please describe your understanding of the bias of the facial detection classifier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "8qcR9uvfCJku"
      },
      "outputs": [],
      "source": [
        "### Inspect different latent features\n",
        "\n",
        "#@title Change the sliders to inspect different latent features! { run: \"auto\" }\n",
        "idx_latent = 8 #@param {type:\"slider\", min:0, max:31, step:1}\n",
        "num_steps = 15\n",
        "\n",
        "# Extract all latent samples from the desired dimension\n",
        "latent_samples = z_mean[:, idx_latent]\n",
        "\n",
        "# Compute their density and plot\n",
        "density, latent_bins = np.histogram(latent_samples, num_steps, density=True)\n",
        "fig, ax = plt.subplots(2, 1, figsize=(15, 4))\n",
        "ax[0].bar(latent_bins[1:], density)\n",
        "ax[0].set_ylabel(\"Data density\")\n",
        "\n",
        "# Visualize reconstructions as we walk across the latent space\n",
        "latent_steps = np.linspace(np.min(latent_samples), np.max(latent_samples), num_steps)\n",
        "baseline_latent = tf.reduce_mean(z_mean, 0, keepdims=True)\n",
        "\n",
        "recons = []\n",
        "for step in latent_steps: \n",
        "  # Adjust the latent vector according to our step\n",
        "  latent = baseline_latent.numpy()\n",
        "  latent[0, idx_latent] = step\n",
        "  # Decode the reconstruction and store\n",
        "  recons.append(ss_vae.decode(latent)[0])\n",
        "\n",
        "# Visualize all of the reconstructions!\n",
        "ax[1].imshow(mdl.util.create_grid_of_images(recons, (1, num_steps)))\n",
        "ax[1].set_xlabel(\"Latent step\")\n",
        "ax[1].set_ylabel(\"Visualization\");\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3ExRRPO2z27z"
      },
      "source": [
        "\n",
        "### Inspect how the accuracy changes as a function of density in the latent space\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PnmPXmkGLBVU"
      },
      "outputs": [],
      "source": [
        "### Accuracy vs. density in latent space\n",
        "\n",
        "# Loop through every latent dimension\n",
        "avg_logit_per_bin = []\n",
        "for idx_latent in range(latent_dim): \n",
        "  latent_samples = z_mean[:, idx_latent]\n",
        "  start = np.percentile(latent_samples, 5)\n",
        "  end = np.percentile(latent_samples, 95)\n",
        "  latent_steps = np.linspace(start, end, num_steps)\n",
        "\n",
        "  # Find which samples fall in which bin of the latent dimension\n",
        "  which_latent_bin = np.digitize(latent_samples, latent_steps)\n",
        "  \n",
        "  # For each latent bin, compute the accuracy (average logit score)\n",
        "  avg_logit = []\n",
        "  for j in range(0, num_steps+1): \n",
        "    inds_in_bin = np.where(which_latent_bin == j)\n",
        "    avg_logit.append(y_logit.numpy()[inds_in_bin].mean())\n",
        "\n",
        "  avg_logit_per_bin.append(avg_logit)\n",
        "  \n",
        "# Average the results across all latent dimensions and all samples\n",
        "accuracy_per_latent = np.mean(avg_logit_per_bin, 0)\n",
        "accuracy_per_latent = (accuracy_per_latent - accuracy_per_latent.min()) / np.ptp(accuracy_per_latent)\n",
        "\n",
        "# Plot the results\n",
        "plt.plot(np.linspace(np.min(z_mean), np.max(z_mean), num_steps+1), accuracy_per_latent,'-o')\n",
        "plt.xlabel(\"Latent step\")\n",
        "plt.ylabel(\"Relative accuracy\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rESoXRPQo_mq"
      },
      "source": [
        "## 2.8 Conclusion and submission information\n",
        "\n",
        "**To be eligible for the Debiasing Faces Lab prize, you must submit a document of your answers to the short-answer `TODO`s with your complete lab submission.** Please see the short-answer `TODO`s replicated again here:\n",
        "\n",
        "#### **TODO: Linking model performance to uncertainty and bias**\n",
        "\n",
        "1. What, if any, trends do you observe comparing the samples with the highest and lowest reconstruction loss?\n",
        "2. Based on these observations, which features seemed harder to learn for the VAE?\n",
        "3. How does reconstruction loss relate to uncertainty? Think back to our lecture on Robust & Trustworthy Deep Learning! What can you say about examples on which the model may be more or less uncertain?\n",
        "\n",
        "#### **TODO: Uncovering hidden biases through learned latent features**\n",
        "\n",
        "1. Pick two latent variables and describe what semantic meaning they reflect. Include screenshots of the realizations and probability distribution for the latent variables you select.\n",
        "2. For the latent variables selected, what can you tell about which features are under- or over-represented in the data? What might this tell us about how the model is biased?\n",
        "3. For the latent variables selected, how do these feature distribution differences affect classification performance? What, if any, general trends do you observe across the latent variables?\n",
        "4. Based on these observations, please describe your understanding of the bias of the facial detection classifier.\n",
        "\n",
        "**To enter the competition, you must upload a document write-up as part of your complete lab submission for the Debiasing Faces Lab ([submission upload link](https://www.dropbox.com/request/TTYz3Ikx5wIgOITmm5i2)).** Complete details on submission are included in the next lab!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mPRZReq4p68k"
      },
      "source": [
        "## 2.9 Thinking ahead\n",
        "\n",
        "Beyond this, we encourage you to think about the following questions as you prepare for the next lab, which will focus on mitigating the issues of bias and uncertainty that you just uncovered. Consider:\n",
        "* How do the samples with highest reconstruction loss and samples with highest bias compare? Which features is each one highlighting? Why do you think this is?\n",
        "* In what ways is the dataset biased so far? Can you imagine other features that the dataset is biased against that we have not uncovered yet?\n",
        "*  How can the performance of the supervised VAE classifier be improved?\n",
        "* Do you have ideas for other ways to address issues of bias, particularly in terms of the training data?\n",
        "\n",
        "Hopefully this lab has shed some light on a few concepts, from vision based tasks, to VAEs, to algorithmic bias. We like to think it has, but we're biased ;).\n",
        "\n",
        "<img src=\"https://i.ibb.co/BjLSRMM/ezgif-2-253dfd3f9097.gif\" />"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "Ag_e7xtTzT1W",
        "NDj7KBaW8Asz"
      ],
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.0 (v3.9.0:9cf6752276, Oct  5 2020, 11:29:23) \n[Clang 6.0 (clang-600.0.57)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "7812ea015bdcee6f23a998adcdd2ef97c151c0c241b7b7070987d9313e41299d"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}