tutorial-bias-word-embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NOL9zUq9C3up"
   },
   "source": [
    "# Tutorial // Exploring Gender Bias in Word Embedding\n",
    "\n",
    "## https://learn.responsibly.ai/word-embedding\n",
    "\n",
    "Powered by [`responsibly`](https://docs.responsibly.ai/) - Toolkit for auditing and mitigating bias and fairness of machine learning systems 🔎🤖🧰\n",
    "\n",
    "# Overview\n",
    "\n",
    "## Learning Objectives:\n",
    "\n",
    "1. Gaining an intuitive technical understanding of bias in machine learning systems.\n",
    "\n",
    "2. Exploring the interplay between data, algorithms, application, workflow, and human context when considering responsible AI.\n",
    "\n",
    "## Audience\n",
    "Everyone, really. No previous knowledge is assumed. If you have a background, you will be able to understand the topic more deeply.\n",
    "\n",
    "\n",
    "## Method\n",
    "\n",
    "1. Dive into one family of machine learning models/building-block as a scaffolder.\n",
    "\n",
    "2. Focus on one example of bias: by gender.\n",
    "\n",
    "## Disclaimers\n",
    "\n",
    "1. Word embeddings are not very important by themselves in the context of responsible AI, but bias can be demonstrated with them in an intuitive way.\n",
    "\n",
    "2. We focus on gender bias, and treat it as binary for the simplicity in this work. Nevertheless, gender is a complex social construct, and we should keep it in mind when we go back from a learning context to the real-world.\n",
    "\n",
    "2. We don't aim to give a comprehensive overview of neither bias in machine learning nor fairness nor responsible AI.\n",
    "\n",
    "3. If you need to work on one of these topics, this workshop is far from being enough, but it can serve as a starting point for your learning path.\n",
    "\n",
    "4. On top of that, it is an active area of research.\n",
    "\n",
    "5. And there is much more to say about this topic, especially from Science and Technology Studies (STS) point of view (but not only).\n",
    "\n",
    "5. Therefore, we will provide good learning resources at the end.\n",
    "\n",
    "\n",
    "## Legend\n",
    "💎 Important\n",
    "\n",
    "⚡ Be Aware - Debated issue / interpret carefully / simplicity over precision\n",
    "\n",
    "🛠️ Setup/Technical (a.k.a \"the code is not important, just run it!\")\n",
    "\n",
    "🧪 Methodological Issue\n",
    "\n",
    "💻 Hands-On - Your turn! NO programming background\n",
    "\n",
    "⌨️ ... Some programming background (in Python) is required\n",
    "\n",
    "🦄 Out of Scope\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qs4pH9HEEq4h"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "idhysORlC3uy"
   },
   "source": [
    "# Part One: Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vn-4xIh6C3uz"
   },
   "source": [
    "## 1.1 - 🛠️ Install `responsibly`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_nQ508U3C3uz"
   },
   "outputs": [],
   "source": [
    "%pip install --user responsibly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7KVoylBYC3u0"
   },
   "source": [
    "---\n",
    "\n",
    "### You might need to restart youe notebook\n",
    "<big>⚠️</big>\n",
    "\n",
    "If you get an error of **`ModuleNotFoundError: No module named 'responsibly'`** after the `import responsibly` in the next cell, and you work on either **Colab** or **Binder** - this is **normal**.\n",
    "<br/> <br/>\n",
    "**Restart** the Kernel/Runtime (use the menu on top or the botton in the notebook), **skip** the installation cell (`!pip install --user responsibly`) and **run** the previous cell again (`import responsibly`).\n",
    "\n",
    "Now it should all work fine!\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rU1uolhsC3u0"
   },
   "source": [
    "## 1.2 - Validate Installation of `responsibly`\n",
    "<big>🛠️</big>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E63RXd26C3u0"
   },
   "outputs": [],
   "source": [
    "import responsibly\n",
    "\n",
    "# You should get '0.1.3'\n",
    "responsibly.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7D44KcxlEs2A"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ND0y48N0C3u1"
   },
   "source": [
    "# Part Two: Examples of Bias in Language Technology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7bC6uNO_DRAm"
   },
   "source": [
    "## 2.1 - Translation\n",
    "\n",
    "<img src=\"./images/example-translate.jpg\"/>\n",
    "\n",
    "Source: [Google Blog](https://www.blog.google/products/translate/reducing-gender-bias-google-translate/), [Google AI Blog](https://ai.googleblog.com/2018/12/providing-gender-specific-translations.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "87IEqEWwDRK3"
   },
   "source": [
    "## 2.2 - Automated Speech Recognition (ASR) \n",
    "\n",
    "<img src=\"./images/asr-wer.jpg\" />\n",
    "\n",
    "WER = Average Word Error Rate\n",
    "\n",
    "`(substitutions + deletions + insertions) / total number of words`\n",
    "\n",
    "Koenecke, Allison, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. \"[Racial disparities in automated speech recognition](https://www.pnas.org/content/117/14/7684).\" Proceedings of the National Academy of Sciences 117, no. 14 (2020): 7684-7689.\n",
    "\n",
    "[Stanford News](https://news.stanford.edu/2020/03/23/automated-speech-recognition-less-accurate-blacks/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zkXSkI3SC3u1"
   },
   "source": [
    "## 2.3 - Recruiting tool\n",
    "\n",
    "\"Amazon scraps secret AI recruiting tool that showed bias against women\" ([Reuters](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G))\n",
    "\n",
    "\"But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6112Hp2dDaYG"
   },
   "source": [
    "\n",
    "## 2.4 - Natural Language Generation  (based on language models)\n",
    "\n",
    "\n",
    "### WARNING: The following demonstration contains examples which are offensive in nature.\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "[Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) (OpenAI GPT-2)\n",
    "\n",
    "<img src=\"./images/nlg-prompt.png\" width=\"400\"/>\n",
    "\n",
    "1. Sheng, E., Chang, K. W., Natarajan, P., & Peng, N. (2019). [The woman worked as a babysitter: On biases in language generation](https://arxiv.org/pdf/1909.01326.pdf). arXiv preprint arXiv:1909.01326.\n",
    "2. [StereoSet](https://stereoset.mit.edu/)\n",
    "Nadeem, M., Bethke, A., & Reddy, S. (2020). [StereoSet: Measuring stereotypical bias in pretrained language models](https://arxiv.org/pdf/2004.09456.pdf). arXiv preprint arXiv:2004.09456."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OP-aP2sZEu6o"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j580hhNJC3u2"
   },
   "source": [
    "# Part Three: Motivation - Why to use Word Embeddings?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_jN4gKKRDgBl"
   },
   "source": [
    "## 3.1 - [NLP (Natural Language Processing)](https://en.wikipedia.org/wiki/Natural_language_processing)\n",
    "**Very partial** list of tasks\n",
    "\n",
    "\n",
    "### 1. Classification\n",
    "- Fake news classification\n",
    "- Toxic comment classification\n",
    "- Review raiting (sentiment analysis)\n",
    "- Hiring decision making by CV\n",
    "- Automated essay scoring\n",
    "\n",
    "### 3. Machine Translation\n",
    "\n",
    "### 2. Information Retrieval\n",
    "- Search engine\n",
    "- Plagiarism detection\n",
    "\n",
    "### 3. Conversation Chatbot\n",
    "\n",
    "### 4. Coreference Resolution\n",
    "<img src=\"./images/corefexample.png\" />\n",
    "\n",
    "<small>Source: [Stanford Natural Language Processing Group](https://nlp.stanford.edu/projects/coref.shtml)</small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DyseaT6ADi5U"
   },
   "source": [
    "## 3.2 - Machine Learning (NLP) Pipeline\n",
    "\n",
    "<img src=\"./images/nlp-pipeline.png\" />\n",
    "\n",
    "<small>Source: [Kai-Wei Chang (UCLA) - What It Takes to Control Societal Bias in Natural Language Processing](https://www.youtube.com/watch?v=RgcXD_1Cu18)</small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0pCZrVekC3u2"
   },
   "source": [
    "## 3.3 - Esessional Question - How to represent language to machine?\n",
    "\n",
    "We need some kind of *dictionary* 📖 to transform/encode\n",
    "\n",
    "→ from a human representation (words) 🗣 🔡\n",
    "\n",
    "→ to a machine representation (numbers) 🤖 🔢\n",
    "\n",
    "### First Atempt\n",
    "\n",
    "### Idea: Bag of Words (for a document)\n",
    "<img src=\"./images/bow.png\" width=\"300\" />\n",
    "<small>Source: Zheng, A.& Casari, A. (2018). Feature Engineering for Machine Learning. O'Reilly Media.</small>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7SIe0KUQC3u2"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vocabulary = ['it', 'they', 'puppy', 'and', 'cat', 'aardvark', 'cute', 'extremely', 'not']\n",
    "\n",
    "vectorizer = CountVectorizer(vocabulary=vocabulary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nkDdYeLzC3u3"
   },
   "outputs": [],
   "source": [
    "sentence = 'it is a puppy and it is extremely cute'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fuqtsTSpC3u3"
   },
   "source": [
    "### Bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9euJwWBnC3u4"
   },
   "outputs": [],
   "source": [
    "vectorizer.fit_transform([sentence]).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lPd-yhuJC3u4"
   },
   "outputs": [],
   "source": [
    "vectorizer.fit_transform(['it is not a puppy and it is extremely cute']).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pT80g0l7C3u4"
   },
   "outputs": [],
   "source": [
    "vectorizer.fit_transform(['it is a puppy and it is extremely not cute']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t723YhaCC3u4"
   },
   "source": [
    "🦄 Read more about scikit-learn's text feature extraction [here](https://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OYLHZnIrC3u5"
   },
   "source": [
    "### One-hot representation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Z6iVvcLzC3u5"
   },
   "outputs": [],
   "source": [
    "[vectorizer.fit_transform([word]).toarray()\n",
    " for word in sentence.split()\n",
    " if word in vocabulary]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mptgh5VAC3u5"
   },
   "source": [
    "### The problem with one-hot representation\n",
    "\n",
    "<img src=\"./images/audio-image-text.png\" />\n",
    "\n",
    "<small>Source: [Tensorflow Documentation](https://www.tensorflow.org/tutorials/representation/word2vec)</small>\n",
    "\n",
    "[Color Picker](https://www.google.com/search?q=color+picker)\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tAxMVpIcC3u5"
   },
   "source": [
    "## 3.4 - Idea: Embedding a word in a n-dimensional space\n",
    "<big>💎</big>\n",
    "\n",
    "### Distributional Hypothesis\n",
    "> \"a word is characterized by the company it keeps\" - [John Rupert Firth](https://en.wikipedia.org/wiki/John_Rupert_Firth)\n",
    "\n",
    "**Distance ~ Meaning Similarity**\n",
    "\n",
    "\n",
    "### Examples (algorithms and pre-trained models)\n",
    "<big>🦄</big>\n",
    "- [Word2Vec](https://code.google.com/archive/p/word2vec/)\n",
    "- [GloVe](https://nlp.stanford.edu/projects/glove/)\n",
    "- [fastText](https://fasttext.cc/)\n",
    "- [ELMo](https://allennlp.org/elmo) (contextualized)\n",
    "\n",
    "#### Training: using *word-context* relationships from a corpus.\n",
    "<big>🦄</big>\n",
    "\n",
    "See: [The Illustrated Word2vec by Jay Alammar](http://jalammar.github.io/illustrated-word2vec/)\n",
    "\n",
    "#### State of the Art - Contextual Word Embedding → Language Models\n",
    "<big>🦄</big>\n",
    "- [The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) by Jay Alammar](http://jalammar.github.io/illustrated-bert/)\n",
    "- Microsoft - [NLP Best Practices](https://github.com/microsoft/nlp-recipes)\n",
    "- [Tracking Progress in Natural Language Processing](https://nlpprogress.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-DY34WMFExJo"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MxWD54mFC3u6"
   },
   "source": [
    "# Part Four: Playing with Word2Vec word embedding!\n",
    "\n",
    "[Word2Vec](https://code.google.com/archive/p/word2vec/) - Google News - 100B tokens, 3M vocab, cased, 300d vectors - only lowercase vocab extracted\n",
    "\n",
    "Loaded using [`responsibly`](http://docs.responsibly.ai) package, the function [`responsibly.we.load_w2v_small`]() returns a [`gensim`](https://radimrehurek.com/gensim/)'s [`KeyedVectors`](https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.KeyedVectors) object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AZmUr2o3Dwai"
   },
   "source": [
    "## 4.1 - Basic Properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VafEGudyC3u6"
   },
   "outputs": [],
   "source": [
    "# 🛠️⚡ ignore warnings\n",
    "# generally, you shouldn't do that, but for this tutorial we'll do so for the sake of simplicity\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_AKj8lzLC3u6"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import load_w2v_small\n",
    "\n",
    "w2v_small = load_w2v_small()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_5SjqSjtC3u6"
   },
   "outputs": [],
   "source": [
    "# vocabulary size\n",
    "\n",
    "len(w2v_small.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-vYyVOfnC3u7"
   },
   "outputs": [],
   "source": [
    "# get the vector of the word \"home\"\n",
    "\n",
    "print('home =', w2v_small['home'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WCuUyFuIC3u7"
   },
   "outputs": [],
   "source": [
    "# the word embedding dimension, in this case, is 300\n",
    "\n",
    "len(w2v_small['home'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ADMLat0gC3u7"
   },
   "outputs": [],
   "source": [
    "# all the words are normalized (=have norm equal to one as vectors)\n",
    "\n",
    "from numpy.linalg import norm\n",
    "\n",
    "norm(w2v_small['home'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "an6l7AKwC3u7"
   },
   "outputs": [],
   "source": [
    "# 🛠️ make sure that all the vectors are normalized!\n",
    "\n",
    "from numpy.testing import assert_almost_equal\n",
    "\n",
    "length_vectors = norm(w2v_small.vectors, axis=1)\n",
    "\n",
    "assert_almost_equal(actual=length_vectors,\n",
    "                    desired=1,\n",
    "                    decimal=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kuWUyD8MC3u7"
   },
   "source": [
    "## 4.2 - Mesuring Distance between Words\n",
    "<big>💎</big>\n",
    "\n",
    "<img src=\"./images/sphere.png\" width=\"300\"/>\n",
    "\n",
    "<small>Source: [Wikipedia](https://en.wikipedia.org/wiki/File:Sphere_wireframe_10deg_6r.svg)</small>\n",
    "\n",
    "### Mesure of Similiarty: [Cosine Similariy](https://en.wikipedia.org/wiki/Cosine_similarity)\n",
    "- Measures the cosine of the angle between two vecotrs.\n",
    "- Ranges between 1 (same vector) to -1 (opposite/antipode vector)\n",
    "- In Python, for normalized vectors (Numpy's array), use the `@`(at) operator!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GdXNWnCRC3u8"
   },
   "outputs": [],
   "source": [
    "w2v_small['cat'] @ w2v_small['cat']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lqqZKLNlC3u8"
   },
   "outputs": [],
   "source": [
    "w2v_small['cat'] @ w2v_small['cats']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "u7XLxEr3C3u8"
   },
   "outputs": [],
   "source": [
    "from math import acos, degrees\n",
    "\n",
    "degrees(acos(w2v_small['cat'] @ w2v_small['cats']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fgLdUfefC3u8"
   },
   "outputs": [],
   "source": [
    "w2v_small['cat'] @ w2v_small['dog']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2gVuthONC3u8"
   },
   "outputs": [],
   "source": [
    "degrees(acos(w2v_small['cat'] @ w2v_small['dog']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8066mq5iC3u8"
   },
   "outputs": [],
   "source": [
    "w2v_small['cat'] @ w2v_small['cow']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1axuP3UmC3u9"
   },
   "outputs": [],
   "source": [
    "degrees(acos(w2v_small['cat'] @ w2v_small['cow']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1I1d9F1-C3u9"
   },
   "outputs": [],
   "source": [
    "w2v_small['cat'] @ w2v_small['graduated']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rNWC7LF4C3u9"
   },
   "outputs": [],
   "source": [
    "degrees(acos(w2v_small['cat'] @ w2v_small['graduated']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DZ7AGgicC3u9"
   },
   "source": [
    "💎 In general, the use of Word Embedding to encode words, as an input for NLP systems (*), improve their performance compared to one-hot representation.\n",
    "\n",
    "\\* Sometimes the embedding is learned as part of the NLP system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SUJV2eErC3u9"
   },
   "source": [
    "## 4.3 - Visualizing Word Embedding in 2D using T-SNE \n",
    "<big>🛠️</big>\n",
    "\n",
    "<small>Source: [Google's Seedbank](https://research.google.com/seedbank/seed/pretrained_word_embeddings)</small>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lk71tJP_C3u9"
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "from matplotlib import pylab as plt\n",
    "\n",
    "# take the most common words in the corpus between 200 and 600\n",
    "words = [word for word in w2v_small.index2word[200:600]]\n",
    "\n",
    "# convert the words to vectors\n",
    "embeddings = [w2v_small[word] for word in words]\n",
    "\n",
    "# perform T-SNE\n",
    "words_embedded = TSNE(n_components=2).fit_transform(embeddings)\n",
    "\n",
    "# ... and visualize!\n",
    "plt.figure(figsize=(20, 20))\n",
    "for i, label in enumerate(words):\n",
    "    x, y = words_embedded[i, :]\n",
    "    plt.scatter(x, y)\n",
    "    plt.annotate(label, xy=(x, y), xytext=(5, 2), textcoords='offset points',\n",
    "                 ha='right', va='bottom', size=11)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZQpxDNE1C3u9"
   },
   "source": [
    "### Extra: [Tensorflow Embedding Projector](http://projector.tensorflow.org)\n",
    "⚡ Be cautious: It is easy to see \"patterns\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7EJaiKzYC3u9"
   },
   "source": [
    "## 4.4 - Most Similar\n",
    "\n",
    "What are the most simlar words (=closer) to a given word?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "saL5WQkuC3u-"
   },
   "outputs": [],
   "source": [
    "w2v_small.most_similar('cat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6FNLPsbxC3u-"
   },
   "source": [
    "### EXTRA: Doesn't Match\n",
    "\n",
    "Given a list of words, which one doesn't match?\n",
    "\n",
    "The word further away from the mean of all words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mQ4JrRpIC3u-"
   },
   "outputs": [],
   "source": [
    "w2v_small.doesnt_match('breakfast cereal dinner lunch'.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cEIaww5bC3u-"
   },
   "source": [
    "## 4.5 - Vector Arithmetic\n",
    "\n",
    "<img src=\"./images/vector-addition.png\" width=\"400\"/>\n",
    "\n",
    "<small>Source: [Wikipedia](https://commons.wikimedia.org/wiki/File:Vector_add_scale.svg)</small>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "h9y5YEcdC3u-"
   },
   "outputs": [],
   "source": [
    "# nature + science = ?\n",
    "\n",
    "w2v_small.most_similar(positive=['nature', 'science'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O9Q-3tQdC3u_"
   },
   "source": [
    "## 4.6 - Vector Analogy\n",
    "<big>💎</big>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vTl21bZiC3u_"
   },
   "source": [
    "<img src=\"./images/linear-relationships.png\" />\n",
    "\n",
    "<small>Source: [Tensorflow Documentation](https://www.tensorflow.org/tutorials/representation/word2vec)</small>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GRgutcU5C3u_"
   },
   "outputs": [],
   "source": [
    "# man:king :: woman:?\n",
    "# king - man + woman = ?\n",
    "\n",
    "w2v_small.most_similar(positive=['king', 'woman'],\n",
    "                       negative=['man'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "02Rm6_wuC3u_"
   },
   "outputs": [],
   "source": [
    "w2v_small.most_similar(positive=['big', 'smaller'],\n",
    "                       negative=['small'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nJMNz3cqC3u_"
   },
   "source": [
    "## 4.10 - Think about a DIRECTION in word embedding as a RELATION\n",
    "\n",
    "## $\\overrightarrow{she} - \\overrightarrow{he}$\n",
    "## $\\overrightarrow{smaller} - \\overrightarrow{small}$\n",
    "## $\\overrightarrow{Spain} - \\overrightarrow{Madrid}$\n",
    "\n",
    "\n",
    "**⚡ Direction is not a word vector by itself!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PUblfTSiC3u_"
   },
   "source": [
    "### ⚡ But it doesn't work all the time..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "j95uaEETC3vA"
   },
   "outputs": [],
   "source": [
    "w2v_small.most_similar(positive=['forward', 'up'],\n",
    "                       negative=['down'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wE7kHZHwC3vA"
   },
   "source": [
    "It might be because we have the phrase \"looking forward\" which is acossiated with \"excitement\" in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XCnE-hnMC3vA"
   },
   "source": [
    "⚡🦄 Keep in mind the word embedding was generated by learning the co-occurrence of words, so the fact that it *empirically* exhibit \"concept arithmetic\", it doesn't necessarily mean it learned it! In fact, it seems it didn't.\n",
    "See: [king - man + woman is queen; but why? by Piotr Migdał](https://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html)\n",
    "\n",
    "🦄 EXTRA: [Demo - Word Analogies Visualizer by Julia Bazińska](https://lamyiowce.github.io/word2viz/)\n",
    "\n",
    "⚡🦄 In fact, `w2v_small.most_similar` find the most closest word which *is not one* of the given ones. This is a real methodological issue. Nowadays, it is not a common practice anymore to evaluate word embedding with analogies.\n",
    "\n",
    "You can use [`responsibly.we.most_similar`](https://docs.responsibly.ai/word-embedding-bias.html#responsibly.we.utils.most_similar) for the unrestricted version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8R3W9HsgE11S"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uJx1tKxrC3vA"
   },
   "source": [
    "# Part Five: Gender Bias\n",
    "\n",
    "**⚡ We use the word *bias* merely as a technical term, without jugement of \"good\" or \"bad\". Later on we will put the bias into *human contextes* to evaluate it.**\n",
    "\n",
    "Keep in mind, the data is from Google News, the writers are professional journalists.\n",
    "\n",
    "Bolukbasi Tolga, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. [Man is to computer programmer as woman is to homemaker? debiasing word embeddings](https://arxiv.org/abs/1607.06520). NIPS 2016."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "P8T1Vu6mC3vA"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import load_w2v_small\n",
    "\n",
    "w2v_small = load_w2v_small()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VFoKGcr2C3vA"
   },
   "source": [
    "## 5.1 - Gender appropriate he-she analogies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mzHt7krQC3vB"
   },
   "outputs": [],
   "source": [
    "# she:sister :: he:?\n",
    "# sister - she + he = ?\n",
    "\n",
    "w2v_small.most_similar(positive=['sister', 'he'],\n",
    "                       negative=['she'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HctdMqv8C3vB"
   },
   "source": [
    "```\n",
    "queen-king\n",
    "waitress-waiter\n",
    "sister-brother\n",
    "mother-father\n",
    "ovarian_cancer-prostate_cancer\n",
    "convent-monastery\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JmdyMQbBC3vB"
   },
   "source": [
    "## 5.2 - Gender stereotype he-she analogies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1Lq26hcCC3vB"
   },
   "outputs": [],
   "source": [
    "w2v_small.most_similar(positive=['nurse', 'he'],\n",
    "                       negative=['she'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KVDAu5JbC3vB"
   },
   "source": [
    "```\n",
    "sewing-carpentry\n",
    "nurse-doctor\n",
    "blond-burly\n",
    "giggle-chuckle\n",
    "sassy-snappy\n",
    "volleyball-football\n",
    "register_nurse-physician\n",
    "interior_designer-architect\n",
    "feminism-conservatism\n",
    "vocalist-guitarist\n",
    "diva-superstar\n",
    "cupcakes-pizzas\n",
    "housewife-shopkeeper\n",
    "softball-baseball\n",
    "cosmetics-pharmaceuticals\n",
    "petite-lanky\n",
    "charming-affable\n",
    "hairdresser-barber\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-p30UdbhC3vB"
   },
   "source": [
    "### Methodological Issue: The unrestricted version of analogy generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jr2eqeFJC3vC"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import most_similar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iq1gAfdhC3vC"
   },
   "outputs": [],
   "source": [
    "most_similar(w2v_small,\n",
    "             positive=['nurse', 'he'],\n",
    "             negative=['she'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OkxadIKoC3vC"
   },
   "source": [
    "⚡ Be Aware: According to a recent paper, it seems that the method of generating analogies enforce producing gender sterotype ones!\n",
    "\n",
    "Nissim, M., van Noord, R., van der Goot, R. (2019). [Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor](https://arxiv.org/abs/1905.09866).\n",
    "\n",
    "... and a [Twitter thread](https://twitter.com/adamfungi/status/1133865428663635968) between the authors of the two papares.\n",
    "\n",
    "My takeaway (and as well as of other researchers): Analogies are not approriate method to observe bias in word embedding.\n",
    "\n",
    "🧪 What if our methodology introduce a bias?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KDLn6n64C3vC"
   },
   "source": [
    "## 5.3 - What can we take from analogies? Gender Direction!\n",
    "<big>💎</big>\n",
    "\n",
    "### $\\overrightarrow{she} - \\overrightarrow{he}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6Fl65I1vC3vC"
   },
   "outputs": [],
   "source": [
    "from numpy.linalg import norm\n",
    "\n",
    "gender_direction = w2v_small['she'] - w2v_small['he']\n",
    "\n",
    "gender_direction /= norm(gender_direction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-6uAKJvTC3vC"
   },
   "outputs": [],
   "source": [
    "gender_direction @ w2v_small['architect']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "55MCmDhWC3vC"
   },
   "outputs": [],
   "source": [
    "gender_direction @ w2v_small['interior_designer']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KXoDjjFSC3vD"
   },
   "source": [
    "**⚡Interprete carefully: The word *architect* appears in more contexts with *he* than with *she*, and vice versa for *interior designer*.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j6_SqDfEC3vD"
   },
   "source": [
    "🦄 In practice, we calculate the gender direction using multiple definitional pair of words for better estimation (words may have more than one meaning):\n",
    "\n",
    "- woman - man\n",
    "- girl - boy\n",
    "- she - he\n",
    "- mother - father\n",
    "- daughter - son\n",
    "- gal - guy\n",
    "- female - male\n",
    "- her - his\n",
    "- herself - himself\n",
    "- Mary - John"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HiqVsyXiC3vD"
   },
   "source": [
    "## 5.4 - Try some words by yourself\n",
    "<big>💻</big>\n",
    "⚡ Keep in mind: You are performing exploratory data analysis, and not evaluate systematically!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bvnuDrC7C3vD"
   },
   "outputs": [],
   "source": [
    "gender_direction @ w2v_small['house']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3uLorhpmC3vD"
   },
   "source": [
    "## 5.5 - So What?\n",
    "<big>💎</big>\n",
    "\n",
    "Downstream Application - Putting a system into a human context\n",
    "\n",
    "### Toy Example - Search Engine Ranking\n",
    "\n",
    "- \"BU computer science PhD student\"\n",
    "- \"doctoral candidate\" ~ \"PhD student\"\n",
    "- John:computer programmer :: Mary:homemaker\n",
    "\n",
    "### Universal Embeddings\n",
    "- Pre-trained on a large corpus\n",
    "- Plugged in downstream task models (sentimental analysis, classification, translation …)\n",
    "- Improvement of performances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y_aqSxNLC3vD"
   },
   "source": [
    "## 5.6 - Measuring Bias in Word Embedding\n",
    "\n",
    "# Think-Pair-Share"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qISdexCIC3vD"
   },
   "source": [
    "```\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "**Basic Ideas: Use neutral-gender words!**\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "**Neutral Professions!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9DuWu9yVC3vE"
   },
   "source": [
    "## 5.7 - Projections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ucDbQ13uC3vE"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import GenderBiasWE\n",
    "\n",
    "w2v_small_gender_bias = GenderBiasWE(w2v_small, only_lower=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wOy5EDmRC3vE"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_bias.positive_end, w2v_small_gender_bias.negative_end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NzlmJWFdC3vE"
   },
   "outputs": [],
   "source": [
    "# gender direction\n",
    "w2v_small_gender_bias.direction[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KI8U2QhCC3vE"
   },
   "outputs": [],
   "source": [
    "from responsibly.we.data import BOLUKBASI_DATA\n",
    "\n",
    "neutral_profession_names = BOLUKBASI_DATA['gender']['neutral_profession_names']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tAHD1RdXC3vF"
   },
   "outputs": [],
   "source": [
    "neutral_profession_names[:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "loqvR0qUC3vF"
   },
   "source": [
    "Note: Why `actor` is in the neutral profession names list while `actress` is not there?\n",
    "1. Due to the statistical nature of the method that is used to find the gender- specific and natural words\n",
    "2. That might be because `actor` nowadays is much more gender-neutral, compared to waiter-waitress (see [Wikipedia - The term Actress](https://en.wikipedia.org/wiki/Actor#The_term_actress))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fLgkcOuGC3vF"
   },
   "outputs": [],
   "source": [
    "len(neutral_profession_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qby3XDrfC3vF"
   },
   "outputs": [],
   "source": [
    "# the same of using the @ operator on the bias direction\n",
    "\n",
    "w2v_small_gender_bias.project_on_direction(neutral_profession_names[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "th7UryxdC3vF"
   },
   "source": [
    "**Let's visualize the projections of professions (neutral and specific by the orthography) on the gender direction**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "juiycs76C3vF"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pylab as plt\n",
    "\n",
    "f, ax = plt.subplots(1, figsize=(10, 10))\n",
    "\n",
    "w2v_small_gender_bias.plot_projection_scores(n_extreme=20, ax=ax);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K1MgiofyC3vF"
   },
   "source": [
    "EXTRA: Demo - Visualizing gender bias with [Word Clouds](http://wordbias.umiacs.umd.edu/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1IlRykZjC3vG"
   },
   "source": [
    "## 5.8 - Are the projections of occupation words on the gender direction related to the real world?\n",
    "\n",
    "Let's take the percentage of female in various occupations from the Labor Force Statistics of 2017 Population Survey.\n",
    "\n",
    "Taken from: https://arxiv.org/abs/1804.06876"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-xrMYFBXC3vG"
   },
   "outputs": [],
   "source": [
    "from operator import itemgetter  # 🛠️ For idiomatic sorting in Python\n",
    "\n",
    "from responsibly.we.data import OCCUPATION_FEMALE_PRECENTAGE\n",
    "\n",
    "sorted(OCCUPATION_FEMALE_PRECENTAGE.items(), key=itemgetter(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "i2j4_uCFC3vG"
   },
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(1, figsize=(10, 8))\n",
    "\n",
    "w2v_small_gender_bias.plot_factual_association(ax=ax);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vrfs4qf5C3vG"
   },
   "source": [
    "### Also: Word embeddings quantify 100 years of gender stereotypes\n",
    "\n",
    "Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). [Word embeddings quantify 100 years of gender and ethnic stereotypes](https://www.pnas.org/content/pnas/115/16/E3635.full.pdf). Proceedings of the National Academy of Sciences, 115(16), E3635-E3644.\n",
    "\n",
    "<img src=\"./images/gender-bias-over-decades.png\" />\n",
    "\n",
    "<small>Data: Google Books/Corpus of Historical American English (COHA)</small>\n",
    "\n",
    "Word embedding is sometimes used to analyze a collection of text in **digital humanities** - putting a system into a human context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yaisUYOxC3vG"
   },
   "source": [
    "🧪 Quite strong and interesting observation! We used \"external\" data which wan't used directly to create the word embedding.\n",
    "\n",
    "It takes us to think about the *data generation process* - in both cases it is the \"world\", but it will be difficult to argue for causality only in one direction:\n",
    "1. Text in newspapers\n",
    "2. Employment by gender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KwZmL19LC3vG"
   },
   "source": [
    "## 5.9 - Direct Bias Measure\n",
    "\n",
    "1. Project each **neutral profession names** on the gender direction\n",
    "2. Calculate the absolute value of each projection\n",
    "3. Average it all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MdXFI9EqC3vG"
   },
   "outputs": [],
   "source": [
    "# using responsibly\n",
    "\n",
    "w2v_small_gender_bias.calc_direct_bias()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MbB7U8IFC3vH"
   },
   "outputs": [],
   "source": [
    "# what responsibly does:\n",
    "\n",
    "neutral_profession_projections = [w2v_small[word] @ w2v_small_gender_bias.direction\n",
    "                                  for word in neutral_profession_names]\n",
    "\n",
    "abs_neutral_profession_projections = [abs(proj) for proj in neutral_profession_projections]\n",
    "\n",
    "sum(abs_neutral_profession_projections) / len(abs_neutral_profession_projections)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dv_Zav4DC3vH"
   },
   "source": [
    "🧪 What are the assumptions of the direct bias measure? How the choice of neutral word effect on the definition of the bias?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z3CzXF0iC3vH"
   },
   "source": [
    "## 5.10 - [EXTRA] Indirect Bias Measure\n",
    "Similarity due to shared \"gender direction\" projection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hmHvybUKC3vH"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_bias.generate_closest_words_indirect_bias('softball',\n",
    "                                                           'football')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b0eYowtwFBrH"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x2AxxNoXC3vH"
   },
   "source": [
    "# Part Six: Mitigating Bias\n",
    "\n",
    "> We intentionally do not reference the resulting embeddings as \"debiased\" or free from all gender bias, and\n",
    "prefer the term \"mitigating bias\" rather that \"debiasing,\" to guard against the misconception that the resulting\n",
    "embeddings are entirely \"safe\" and need not be critically evaluated for bias in downstream tasks. <small>James-Sorenson, H., & Alvarez-Melis, D. (2019). [Probabilistic Bias Mitigation in Word Embeddings](https://arxiv.org/pdf/1910.14497.pdf). arXiv preprint arXiv:1910.14497.</small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TlW6-I0vECj5"
   },
   "source": [
    "## 6.1 - Neutralize\n",
    "\n",
    "In this case, we will remove the gender projection from all the words, except the neutral-gender ones, and then normalize.\n",
    "\n",
    "🦄 We need to \"learn\" what are the gender-specific words in the vocabulary for a seed set of gender-specific words (by semi-automatic use of [WordNet](https://en.wikipedia.org/wiki/WordNet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RzIzwzcXC3vH"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias = w2v_small_gender_bias.debias(method='neutralize', inplace=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "J3hgtHyxC3vH"
   },
   "outputs": [],
   "source": [
    "print('home:',\n",
    "      'before =', w2v_small_gender_bias.model['home'] @ w2v_small_gender_bias.direction,\n",
    "      'after = ', w2v_small_gender_debias.model['home'] @ w2v_small_gender_debias.direction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IZC7y2I9C3vI"
   },
   "outputs": [],
   "source": [
    "print('man:',\n",
    "      'before =', w2v_small_gender_bias.model['man'] @ w2v_small_gender_bias.direction,\n",
    "      'after = ', w2v_small_gender_debias.model['man'] @ w2v_small_gender_debias.direction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mPh777kyC3vI"
   },
   "outputs": [],
   "source": [
    "print('woman:',\n",
    "      'before =', w2v_small_gender_bias.model['woman'] @ w2v_small_gender_bias.direction,\n",
    "      'after = ', w2v_small_gender_debias.model['woman'] @ w2v_small_gender_debias.direction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hFf08ZQmC3vI"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.calc_direct_bias()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G3sd0hLUC3vI"
   },
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(1, figsize=(10, 10))\n",
    "\n",
    "w2v_small_gender_debias.plot_projection_scores(n_extreme=20, ax=ax);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aMChX6EtC3vI"
   },
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(1, figsize=(10, 8))\n",
    "\n",
    "w2v_small_gender_debias.plot_factual_association(ax=ax);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MKh54qxZC3vI"
   },
   "source": [
    "## 6.2 [EXTRA] Equalize\n",
    "\n",
    "- Do you see that `man` and `woman` have a different projection on the gender direction? \n",
    "\n",
    "- It might cause to different similarity (distance) to neutral words, such as to `babysitter`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ulo550zHC3vJ"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.model['grandfather'] @ w2v_small_gender_debias.model['babysitter']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "W_bY_M7_C3vJ"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.model['grandmother'] @ w2v_small_gender_debias.model['babysitter']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2BcJxNYWC3vJ"
   },
   "outputs": [],
   "source": [
    "BOLUKBASI_DATA['gender']['equalize_pairs'][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jZrAc0y-C3vJ"
   },
   "source": [
    "## 6.3 - Hard Debias = Neutralize + Equalize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gB5VkrfeC3vJ"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias = w2v_small_gender_bias.debias(method='hard', inplace=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "R4UL6HBOC3vJ"
   },
   "outputs": [],
   "source": [
    "print('home:',\n",
    "      'before =', w2v_small_gender_bias.model['home'] @ w2v_small_gender_bias.direction,\n",
    "      'after = ', w2v_small_gender_debias.model['home'] @ w2v_small_gender_debias.direction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LnTmE2L5C3vJ"
   },
   "outputs": [],
   "source": [
    "print('man:',\n",
    "      'before =', w2v_small_gender_bias.model['man'] @ w2v_small_gender_bias.direction,\n",
    "      'after = ', w2v_small_gender_debias.model['man'] @ w2v_small_gender_debias.direction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zD8SJLt3C3vJ"
   },
   "outputs": [],
   "source": [
    "print('woman:',\n",
    "      'before =', w2v_small_gender_bias.model['woman'] @ w2v_small_gender_bias.direction,\n",
    "      'after = ', w2v_small_gender_debias.model['woman'] @ w2v_small_gender_debias.direction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8ZVur5jOC3vK"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.calc_direct_bias()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IPslsCZJC3vK"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.model['grandfather'] @ w2v_small_gender_debias.model['babysitter']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0uqdHLMDC3vK"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.model['grandmother'] @ w2v_small_gender_debias.model['babysitter']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZGLeaGqLC3vK"
   },
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(1, figsize=(10, 10))\n",
    "\n",
    "w2v_small_gender_debias.plot_projection_scores(n_extreme=20, ax=ax);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The disadvantage of equalization is that it might remove meaningful association, such as the verb meaning of \"grandfather\", e.g. \"to grandfather a regulation\". Equalization removes this distinction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FEBDBsB-C3vK"
   },
   "source": [
    "## 6.4 - Compare Preformances\n",
    "\n",
    "After debiasing, the performance of the word embedding, using standard benchmarks, get only slightly worse!\n",
    "\n",
    "**⚠️ It might take few minutes to run!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iFNefinzC3vK"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_bias.evaluate_word_embedding()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8pnxA6vzC3vK"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.evaluate_word_embedding()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8gOQoBKoFG8m"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IF3mIYJQC3vL"
   },
   "source": [
    "# Part Seven: So What?\n",
    "<big>💎</big>\n",
    "\n",
    "We removed the gender bias, **as we defined it**, in a word embedding - Is there any impact on a downstream application?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First example: coreference resolution\n",
    "\n",
    "Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K. W. (2018). [Gender bias in coreference resolution: Evaluation and debiasing methods](https://par.nsf.gov/servlets/purl/10084252). NAACL-HLT 2018.\n",
    "\n",
    "\n",
    "### WinoBias Dataset\n",
    "<img src=\"./images/coref-example.png\" width=\"400\"/>\n",
    "\n",
    "\n",
    "### Stereotypical Occupations (the source of `responsibly.we.data.OCCUPATION_FEMALE_PRECENTAGE`)\n",
    "<img src=\"./images/coref-occupations.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RfjVADJZC3vL"
   },
   "source": [
    "### Results on *UW End-to-end Neural Coreference Resolution System*\n",
    "\n",
    "#### No Intervention - Baseline\n",
    "\n",
    "| Word Embedding | OnoNotes | Type 1 - Pro-stereotypical | Type 1 - Anti-stereotypical |  Avg |  Diff |\n",
    "|:--------------:|:--------:|:--------------------------:|:---------------------------:|:----:|:-----:|\n",
    "|    Original    |   67.7   |            76.0            |             49.4            | 62.7 | 26.6* |\n",
    "\n",
    "#### Intervention: Named-entity anonymization\n",
    "\n",
    "| Word Embedding | OnoNotes | Type 1 - Pro-stereotypical | Type 1 - Anti-stereotypical |  Avg |  Diff |\n",
    "|:--------------:|:--------:|:--------------------------:|:---------------------------:|:----:|:-----:|\n",
    "|    Original    |   66.4   |            73.5            |             51.2            | 62.6 | 21.3* |\n",
    "|  Hard Debiased |   66.5   |            67.2            |             59.3            | 63.2 |  7.9* |\n",
    "\n",
    "#### Interventions: Named-entity anonymization + Gender swapping\n",
    "\n",
    "| Word Embedding | OnoNotes | Type 1 - Pro-stereotypical | Type 1 - Anti-stereotypical |  Avg |  Diff |\n",
    "|:--------------:|:--------:|:--------------------------:|:---------------------------:|:----:|:-----:|\n",
    "|    Original    |   66.2   |            65.1            |             59.2            | 62.2 |  5.9* |\n",
    "|  Hard Debiased |   66.3   |            63.9            |             62.8            | 63.4 |  1.1  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZZT_xomjC3vL"
   },
   "source": [
    "## Second example: another bias mitigation method\n",
    "\n",
    "Zhao, J., Zhou, Y., Li, Z., Wang, W., & Chang, K. W. (2018). [Learning gender-neutral word embeddings](https://arxiv.org/pdf/1809.01496.pdf). EMNLP 2018.\n",
    "\n",
    "The mitigation method is tailor-made for GloVe training process.\n",
    "\n",
    "<img src=\"./images/gn-glove-results.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y12-l0UHFIHQ"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3paASfPLC3vL"
   },
   "source": [
    "# Part Eight: Meta \"So What?\" - I\n",
    "<big>💎💎</big>\n",
    "\n",
    "## How should we definition of \"bias\" in word embedding?\n",
    "\n",
    "### 1. Intrinsic (e.g., direct bias)\n",
    "\n",
    "### 2. External - Downstream application (e.g., coreference resolution, classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SGFwBIuBFI6x"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YacvyFIIC3vL"
   },
   "source": [
    "# Part Nine: Have we really removed the bias?\n",
    "<big>💎</big>\n",
    "\n",
    "Let's look on another metric, called **WEAT** (Word Embedding Association Test) which is inspired by **IAT** (Implicit-Association Test) from Pyschology.\n",
    "\n",
    "Try IAT by yourself: https://implicit.harvard.edu/implicit/\n",
    "\n",
    "**Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). [Semantics derived automatically from language corpora contain human-like biases.](http://www.cs.bath.ac.uk/~jjb/ftp/CaliskanEtAl-authors-full.pdf) Science, 356(6334), 183-186.**\n",
    "\n",
    "\n",
    "## 9.1 - Ingredients\n",
    "\n",
    "1. Attribute words (e.g., Male ve. Female)\n",
    "\n",
    "2. Target words (e.g., Math vs. Arts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iGP_L_7RC3vL"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import load_w2v_small\n",
    "\n",
    "w2v_small = load_w2v_small()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "j79iz-KBC3vM"
   },
   "outputs": [],
   "source": [
    "# 🛠️ For copying a nested data structure in Python\n",
    "from copy import deepcopy\n",
    "\n",
    "from responsibly.we.weat import WEAT_DATA\n",
    "\n",
    "# B. A. Nosek, M. R. Banaji, A. G. Greenwald, Math=male, me=female, therefore math≠me.,\n",
    "# Journal of Personality and Social Psychology 83, 44 (2002).\n",
    "weat_gender_science_arts = deepcopy(WEAT_DATA[7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QyTkAjlUC3vM"
   },
   "outputs": [],
   "source": [
    "# 🛠️ filter words from the original IAT experiment that are not presend in the reduced Word2Vec model\n",
    "\n",
    "from responsibly.we.weat import _filter_by_model_weat_stimuli\n",
    "\n",
    "_filter_by_model_weat_stimuli(weat_gender_science_arts, w2v_small)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g_Rjk5lYC3vM"
   },
   "outputs": [],
   "source": [
    "weat_gender_science_arts['first_attribute']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "p3Ix3XZfC3vM"
   },
   "outputs": [],
   "source": [
    "weat_gender_science_arts['second_attribute']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g6agv1aFC3vN"
   },
   "outputs": [],
   "source": [
    "weat_gender_science_arts['first_target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XFDSnSirC3vN"
   },
   "outputs": [],
   "source": [
    "weat_gender_science_arts['second_target']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvdGg3BxC3vN"
   },
   "source": [
    "## 9.2 - Recipe\n",
    "\n",
    "➕ Male x Science\n",
    "\n",
    "➖ Male x Arts\n",
    "\n",
    "➖ Female x Science\n",
    "\n",
    "➕ Female x Arts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PwyKfv_5C3vN"
   },
   "outputs": [],
   "source": [
    "def calc_combination_similiarity(model, attribute, target):\n",
    "    score = 0\n",
    "\n",
    "    for attribute_word in attribute['words']:\n",
    "\n",
    "        for target_word in target['words']:\n",
    "\n",
    "            score += w2v_small.similarity(attribute_word,\n",
    "                                          target_word)\n",
    "\n",
    "    return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YenqrQ3qC3vN"
   },
   "outputs": [],
   "source": [
    "male_science_score = calc_combination_similiarity(w2v_small,\n",
    "                                                  weat_gender_science_arts['first_attribute'],\n",
    "                                                  weat_gender_science_arts['first_target'])\n",
    "\n",
    "male_science_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IHpOPD76C3vN"
   },
   "outputs": [],
   "source": [
    "male_arts_score = calc_combination_similiarity(w2v_small,\n",
    "                                               weat_gender_science_arts['first_attribute'],\n",
    "                                               weat_gender_science_arts['second_target'])\n",
    "\n",
    "male_arts_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BBgm_DLuC3vN"
   },
   "outputs": [],
   "source": [
    "female_science_score = calc_combination_similiarity(w2v_small,\n",
    "                                                    weat_gender_science_arts['second_attribute'],\n",
    "                                                    weat_gender_science_arts['first_target'])\n",
    "\n",
    "female_science_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vhuoitgoC3vN"
   },
   "outputs": [],
   "source": [
    "female_arts_score = calc_combination_similiarity(w2v_small,\n",
    "                                                 weat_gender_science_arts['second_attribute'],\n",
    "                                                 weat_gender_science_arts['second_target'])\n",
    "\n",
    "female_arts_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2dfCEEI9C3vO"
   },
   "outputs": [],
   "source": [
    "male_science_score - male_arts_score - female_science_score + female_arts_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "umaznjxuC3vO"
   },
   "outputs": [],
   "source": [
    "len(weat_gender_science_arts['first_attribute']['words'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tCuSX-_bC3vO"
   },
   "outputs": [],
   "source": [
    "(male_science_score - male_arts_score - female_science_score + female_arts_score) / 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zMJQggN3C3vO"
   },
   "source": [
    "## 9.3 - All WEAT Tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6YK12PWDC3vO"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import calc_all_weat\n",
    "\n",
    "calc_all_weat(w2v_small, [weat_gender_science_arts])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r3QNqiT9C3vO"
   },
   "source": [
    "### ⚡ Important Note: Our results are a bit different because we use a reduced Word2Vec.\n",
    "\n",
    "\n",
    "### Results from the Paper (computed on the complete Word2Vec):\n",
    "\n",
    "<img src=\"./images/weat-w2v.png\" width=\"600\"/>\n",
    "\n",
    "\n",
    "### ⚡Caveats\n",
    "\n",
    "#### Comparing WEAT to the IAT\n",
    "\n",
    "- Individuals (IAT) vs. Words (WEAT)\n",
    "- Therefore, the meanings of the effect size and p-value are totally different!\n",
    "\n",
    "#### ⚡🦄 WEAT score definition\n",
    "\n",
    "The definition of the WEAT score is structured differently (but it is computationally equivalent). The original formulation matters to compute the p-value. Refer to the paper for details.\n",
    "\n",
    "### 🧪 Effect size comparision between human and machine bias\n",
    "\n",
    "With the effect size, we can \"compare\" a human bias to a machine one. It raises the question whether the baseline for meauring bias/fairness of a machine should be human bias? Then a well-performing machine shouldn't be necessarily not biased, but only less biased than human (think about autonomous cars or semi-structured vs. unstructured interview)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b9beH0kaC3vP"
   },
   "source": [
    "## 9.4 - WEAT vs. IAT\n",
    "\n",
    "Lewis, M., & Lupyan, G. [What are we learning from language? Associations between gender biases and distributional semantics in 25 languages](https://mollylewis.shinyapps.io/iatlang_SI/).\n",
    "\n",
    "<img src=\"./images/iat-weat.png\" />\n",
    "\n",
    "\n",
    "1. Implicit male-career association (adjusted for participant age, gender, and congruent/incongruent block order) as a function of the linguistic male-career association derived from word-embeddings *r*(23) = 0.48 [0.11, 0.74]; *p* = 0.01; *n* = 25; Study 1b). Each point corresponds to a language. The size of the point is proportional to the number of participants who come from the country in which the language is dominant (total = 656,636 participants). Linguistic associations are estimated from models trained on text in each language from the Wikipedia corpus. Larger values indicate a greater tendency to associate men with the concept of career and women with the concept of family.\n",
    "\n",
    "2. Difference (UK minus US) in implicit association versus linguistic association for 31 IAT types (*N* = 27,045 participants). Error bands indicate standard error of the linear model estimate.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9A6lJbNeC3vP"
   },
   "source": [
    "## 9.5 - Let's go back to our question - did we removed the bias?\n",
    "\n",
    "\n",
    "**Gonen, H., & Goldberg, Y. (2019, June). [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://arxiv.org/pdf/1903.03862.pdf). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 609-614).**\n",
    "\n",
    "They used multiple methods, we'll show only two:\n",
    "1. WEAT\n",
    "2. Neutral words clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sWW9t22qC3vP"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import GenderBiasWE\n",
    "\n",
    "w2v_small_gender_bias = GenderBiasWE(w2v_small, only_lower=True)\n",
    "\n",
    "w2v_small_gender_debias = w2v_small_gender_bias.debias(method='neutralize', inplace=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LYPKzZ1NC3vP"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_bias.calc_direct_bias()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X6jAJ6c7C3vQ"
   },
   "outputs": [],
   "source": [
    "w2v_small_gender_debias.calc_direct_bias()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f7qS1ouIC3vQ"
   },
   "source": [
    "### 9.5.1 -  WEAT - before and after"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nlar5X1_C3vQ"
   },
   "source": [
    "<img src=\"./images/weat-experiment.png\" />\n",
    "\n",
    "See `responsibly` [demo page on word embedding](https://docs.responsibly.ail/notebooks/demo-word-embedding-bias.html#first-experiment-weat-before-and-after-debias) for a complete example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_khszwn_C3vQ"
   },
   "source": [
    "### 9.5.2 - Clustering Neutral Gender Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7RxiEzz8C3vR"
   },
   "outputs": [],
   "source": [
    "from responsibly.we.data import BOLUKBASI_DATA\n",
    "\n",
    "w2v_vocab = {word for word in w2v_small_gender_bias.model.vocab.keys()}\n",
    "\n",
    "# 🦄 how we got these words - read the Bolukbasi's paper for details\n",
    "all_gender_specific_words = set(BOLUKBASI_DATA['gender']['specific_full_with_definitional_equalize'])\n",
    "\n",
    "all_gender_neutral_words = w2v_vocab - all_gender_specific_words\n",
    "\n",
    "print('#vocab =', len(w2v_vocab),\n",
    "      '#specific =', len(all_gender_specific_words),\n",
    "      '#neutral =', len(all_gender_neutral_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uXli_Lg6C3vR"
   },
   "outputs": [],
   "source": [
    "neutral_words_gender_projections = [(w2v_small_gender_bias.project_on_direction(word), word)\n",
    "                                    for word in all_gender_neutral_words]\n",
    "\n",
    "neutral_words_gender_projections.sort()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cg7VOp1dC3vR"
   },
   "outputs": [],
   "source": [
    "neutral_words_gender_projections[:-20:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oCvcPhU8C3vR"
   },
   "outputs": [],
   "source": [
    "neutral_words_gender_projections[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jK2csWf1C3vR"
   },
   "outputs": [],
   "source": [
    "# Neutral words: top 500 male-biased and top 500 female-biased words\n",
    "\n",
    "GenderBiasWE.plot_most_biased_clustering(w2v_small_gender_bias, w2v_small_gender_debias);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q4qouxfoC3vR"
   },
   "source": [
    "Note: In the paper, they got a stronger result, 92.5% accuracy for the debiased model.\n",
    "However, they perform clustering on all the words from the reduced word embedding, both gender- neutral and specific words, and applied slightly different pre-processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "akeu6HDcC3vS"
   },
   "source": [
    "### 9.5.3 - Strong words form the paper (emphasis mine):\n",
    "<big>💎</big>\n",
    "\n",
    "> The experiments ...\n",
    "reveal a **systematic bias** found in the embeddings,\n",
    "which is **independent of the gender direction**.\n",
    "\n",
    "\n",
    "> The implications are alarming: while suggested\n",
    "debiasing methods work well at removing the gender direction, the **debiasing is mostly superficial**.\n",
    "The bias stemming from world stereotypes and\n",
    "learned from the corpus is **ingrained much more\n",
    "deeply** in the embeddings space.\n",
    "\n",
    "\n",
    "> .. real concern from biased representations is **not the association** of a concept with\n",
    "words such as “he”, “she”, “boy”, “girl” **nor** being\n",
    "able to perform **gender-stereotypical word analogies**... algorithmic discrimination is more likely to happen by associating one **implicitly gendered** term with\n",
    "other implicitly gendered terms, or picking up on\n",
    "**gender-specific regularities** in the corpus by learning to condition on gender-biased words, and generalizing to other gender-biased words.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cO2b1tZXFLBV"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Hx0Z4tXPC3vS"
   },
   "source": [
    "# Part Ten: Meta \"So What?\" - II\n",
    "<big>💎💎</big>\n",
    "\n",
    "## Can we debias at all a word embedding?\n",
    "\n",
    "## Under some downstream applications, maybe the bias in the word embedding is desirable?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qGt-7q1RFNFx"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EVMLVzKaTM5-"
   },
   "source": [
    "# Part Eleven: Your Turn!\n",
    "<big>⌨️</big>\n",
    "\n",
    "Note: The first two tasks require a basic background in Python programming. For the last task, you need some experience with Machine Learning and Natural Langauge Processing (NLP) as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BgsASXY2TM5_"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import load_w2v_small\n",
    "\n",
    "w2v_small = load_w2v_small()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gPVjrwpwTM6G"
   },
   "source": [
    "## Task 1 - Racial bias\n",
    "\n",
    "Let's explor racial bias usint Tolga's approche. Will use the [`responsibly.we.BiasWordEmbedding`](http://docs.responsibly.ai/word-embedding-bias.html#ethically.we.bias.BiasWordEmbedding) class. `GenderBiasWE` is a sub-class of `BiasWordEmbedding`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8gB82y7hTM6H"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import BiasWordEmbedding\n",
    "\n",
    "w2v_small_racial_bias = BiasWordEmbedding(w2v_small, only_lower=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V5Pa-_OOTM6J"
   },
   "source": [
    "💎💎💎 Identify the racial direction using the `sum` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "d8NjJ3fPTM6K"
   },
   "outputs": [],
   "source": [
    "white_common_names = ['Emily', 'Anne', 'Jill', 'Allison', 'Laurie', 'Sarah', 'Meredith', 'Carrie',\n",
    "                      'Kristen', 'Todd', 'Neil', 'Geoffrey', 'Brett', 'Brendan', 'Greg', 'Matthew',\n",
    "                      'Jay', 'Brad']\n",
    "\n",
    "black_common_names = ['Aisha', 'Keisha', 'Tamika', 'Lakisha', 'Tanisha', 'Latoya', 'Kenya', 'Latonya',\n",
    "                      'Ebony', 'Rasheed', 'Tremayne', 'Kareem', 'Darnell', 'Tyrone', 'Hakim', 'Jamal',\n",
    "                      'Leroy', 'Jermaine']\n",
    "\n",
    "w2v_small_racial_bias._identify_direction('Whites', 'Blacks',\n",
    "                                          definitional=(white_common_names, black_common_names),\n",
    "                                          method='sum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BQzNVMgITM6N"
   },
   "source": [
    "Use the neutral profession names to measure the racial bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6VsIaNeTTM6N"
   },
   "outputs": [],
   "source": [
    "from responsibly.we.data import BOLUKBASI_DATA\n",
    "\n",
    "neutral_profession_names = BOLUKBASI_DATA['gender']['neutral_profession_names']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iQMUSBpKTM6Q"
   },
   "outputs": [],
   "source": [
    "neutral_profession_names[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E3appKv5TM6S"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pylab as plt\n",
    "\n",
    "f, ax = plt.subplots(1, figsize=(10, 10))\n",
    "\n",
    "w2v_small_racial_bias.plot_projection_scores(neutral_profession_names, n_extreme=20, ax=ax);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QvmZXAdKTM6Z"
   },
   "source": [
    "Calculate the direct bias measure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "80S8S-3hTM6a"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MSFdc0NWTM6d"
   },
   "source": [
    "Keep exploring the racial bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GDjRDxz_TM6d"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g9mOVcXWTM6g"
   },
   "source": [
    "## Task 2 - Your WEAT test\n",
    "\n",
    "Open the [word embedding demo page in `responsibly` documentation](http://docs.responsibly.ai/notebooks/demo-word-embedding-bias.html#it-is-possible-also-to-expirements-with-new-target-word-sets-as-in-this-example-citizen-immigrant), and look on the use of the function `calc_weat_pleasant_unpleasant_attribute`. What was the attempt in that experiment? What was the result? Can you come up with other experiments?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hvlhI95-TM6h"
   },
   "outputs": [],
   "source": [
    "from responsibly.we import calc_weat_pleasant_unpleasant_attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VMmc9vNwTM6j"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Fud0_MaTM6o"
   },
   "source": [
    "## Task 3 - Sentiment Analysis\n",
    "\n",
    "#### Notes:\n",
    "1. This task requires some background with NLP, particularly with training a text classifier in Python.\n",
    "2. Our goal is to learn how word embeddings might affect downstream application from a gender bias perspective, focusing on learning. So we won't follow the best practices in NLP or use the most advanced techniques.\n",
    "\n",
    "One way to examine bias in word embeddings is through a downstream application. Here we will use a sentiment analysis classifier of tweets; given a tweet, the system would infer the *valence* of the sentiment expressed in a tweet. The valence is expressed as a real number between 0 and 1, where 0 represents the negative and 1 is for the positive end.\n",
    "\n",
    "The system is going to be rather simple and consists of three components:\n",
    "\n",
    "1. Preprocessing (e.g., removing stopwords and punctuation, [tockenization](https://en.wikipedia.org/wiki/Text_segmentation#Word_segmentation))\n",
    "2. Transforming the tweets' tokens  into a single 300-dimensional vector.\n",
    "3. Applying logistic regression to predict the valence.\n",
    "\n",
    "Our goal is to assess the word embedding's impact in its original version and the neutralize-\"debiased\" one on the system bias. We are going to build two versions of that system, each using one version of the two word embedding, and compare its performance on the [Equity Evaluation Corpus (EEC)](http://saifmohammad.com/WebPages/Biases-SA.html), which is designed to assess gender bias in sentiment analysis systems.\n",
    "\n",
    "**Reference:**\n",
    "Kiritchenko, S., & Mohammad, S. M. (2018). [Examining gender and race bias in two hundred sentiment analysis systems](https://arxiv.org/pdf/1805.04508.pdf). arXiv preprint arXiv:1805.04508."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iZY369yFKD91"
   },
   "source": [
    "### Data\n",
    "\n",
    "First, let's dowload and load the datasets \"Affect in Tweets\" taken from the [SemEval 2018](https://competitions.codalab.org/competitions/17751#learn_the_details-datasets) competition. We have training, development, and test datasets. We will only use the first and the last datasets, but feel free to use the development dataset to tune select models and hyperparameters with cross-validation.\n",
    "\n",
    "There are three columns:\n",
    "\n",
    "1. `Tweet` - The tweet itself as a string, the input.\n",
    "2. `Intensity Score` - The sentiment's valence of the tweet in the range [0, 1], the output\n",
    "3. `Affect Dimension` - You can ignore it. It is `'valence'` for all of the data points.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bCSmY4EZqn4Y"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "wget https://learn.responsibly.ai/word-embedding/data/SemEval2018-Task1-all-data.zip \\\n",
    "     -O SemEval2018-Task1-all-data.zip -q\n",
    "\n",
    "unzip -qq -o SemEval2018-Task1-all-data.zip -d ./data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-w6q4eEPKCu4"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "train_df = pd.read_csv('./data/SemEval2018-Task1-all-data/English/V-reg/2018-Valence-reg-En-train.txt',\n",
    "                       sep='\\t', index_col=0)\n",
    "dev_df = pd.read_csv('./data/SemEval2018-Task1-all-data/English/V-reg/2018-Valence-reg-En-dev.txt',\n",
    "                       sep='\\t', index_col=0)\n",
    "test_df = pd.read_csv('./data/SemEval2018-Task1-all-data/English/V-reg/2018-Valence-reg-En-test-gold.txt',\n",
    "                       sep='\\t', index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rrtR0AaLKC1f"
   },
   "outputs": [],
   "source": [
    "# A few examples\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vuw2NxOJKCyR"
   },
   "outputs": [],
   "source": [
    "# Convert all the labels from real numbers into boolean values,\n",
    "# setting the threshold at 0.5, and creating a new column named\n",
    "# `label`\n",
    "\n",
    "train_df['label'] = train_df['Intensity Score'] > 0.5\n",
    "dev_df['label'] = dev_df['Intensity Score'] > 0.5\n",
    "test_df['label'] = test_df['Intensity Score'] > 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F8ktamj9Lzvw"
   },
   "source": [
    "Now, let's download the **complete** word2voc word embedding, (which is not filtered only to lowercased words), and load it using the `gensim` Python package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RJLEhGpaLyH2"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "wget https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz \\\n",
    "     -O GoogleNews-vectors-negative300.bin.gz -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UjmXQTC9Lw3U"
   },
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "\n",
    "# Limit vocabulary to top-500K most frequent words\n",
    "VOCAB_SIZE = 500000\n",
    "\n",
    "# Load the word2vec\n",
    "w2v_model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin.gz',\n",
    "                                              binary=True,\n",
    "                                              limit=VOCAB_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mv2O-OS6KCsq"
   },
   "outputs": [],
   "source": [
    "# Get the vector embedding for a word\n",
    "w2v_model['home']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T-eIToHETM6o"
   },
   "outputs": [],
   "source": [
    "# Check whether there is an embedding for a word\n",
    "'bazinga' in w2v_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S8LfyJnzNIy1"
   },
   "source": [
    "### Preprocessing & feature extraction\n",
    "\n",
    "Before we transform a tweet into a vector of 300 dimensions, it should be broken into tokens (\"words\") and be cleaned. You can do that with various Python packages for NLP, such as [NLTK](https://www.nltk.org/) and \n",
    "[spaCy](https://spacy.io/). Feel free to use them if you would like to! We will use the basic preprocessing functionality that comes with [`gensim`](https://radimrehurek.com/gensim/parsing/preprocessing.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SbQRZlFIshUb"
   },
   "outputs": [],
   "source": [
    "from gensim.parsing.preprocessing import (preprocess_string,\n",
    "                                          strip_tags,\n",
    "                                          strip_punctuation,\n",
    "                                          strip_multiple_whitespaces,\n",
    "                                          strip_numeric,\n",
    "                                          remove_stopwords)\n",
    "\n",
    "\n",
    "# We pick a subset of the default filters,\n",
    "# in particular, we do not take\n",
    "# strip_short() and stem_text().\n",
    "FILTERS = [strip_punctuation,\n",
    "           strip_tags,\n",
    "           strip_multiple_whitespaces,\n",
    "           strip_numeric,\n",
    "           remove_stopwords]\n",
    "\n",
    "# See how the sentece is transformed into tokes (words)\n",
    "preprocess_string('This is a \"short\" text!', FILTERS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o_pi5WFOOcsb"
   },
   "source": [
    "After preprocessing all the tweets, we get tokens. We transform each token into a 300d vector using the word embedding and then compute the *average* vector. It will have 300 dimensions as well. This vector serves as the values of the features for each tweet. \n",
    "\n",
    "Note for these two possible pitfalls:\n",
    "\n",
    "1. Make sure that the token exists int he word embedding.\n",
    "2. Sometimes, there are tweets without any token found in the word embedding. Discard these tweets from the data. Keep in mind that you should discard the labels as well.\n",
    "\n",
    "Write the function `generate_text_features(text, w2v)` that gets a string `text` and a word embedding `w2v` and produces the features of this text according to the method xdescribed above. The function should return an Numpy array with lengh of 300."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mEn8ecmrsutk"
   },
   "outputs": [],
   "source": [
    "def generate_text_features(text, w2v):\n",
    "    pass  # Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JDSaVGfzP0Em"
   },
   "source": [
    "Now, use this function to produce the features for all three datasets (training, validation, test)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sl0aYxLNQTs3"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MEM-riKyvdKo"
   },
   "source": [
    "### Training a classifier\n",
    "\n",
    "The next step is straightforward, train logistic regression on the dataset. Report the accuracy of the training and the test dataset.\n",
    "\n",
    "We recommend using [`sklearn.linear_model.LogisticRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9eMRIjj3REvG"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bdDLoHIBRV_B"
   },
   "source": [
    "### Evaluate gender bias in the downstream appliation\n",
    "\n",
    "The **Equity Evaluation Corpus (EEC)** consists of 8,640 English sentences carefully chosen to tease out biases towards certain races and genders.\n",
    "\n",
    "We foucs on the sentences releated to gender. Every sentence is a build out of three elements:\n",
    "\n",
    "1. Person (e.g., `he`, `this woman`, `my uncly`, `my mother`)\n",
    "2. Emotion word (e.g., `anger`, `happy`, `gloomy`, `amazing`)\n",
    "3. Template (e.g., `<person subject> feels <emotion word>`).\n",
    "\n",
    "that are mixed together to form a sentence, for examples:\n",
    "* he feels anger\n",
    "* she feels anger\n",
    "* this woman feels happy\n",
    "* this man feels happy\n",
    "\n",
    "Thanks to this systemic constraction from templates, the sentence are paired by gender, i.e. the EEC data is built of pairs of sentences that are all the same except for a gender noun (e.g., `she`-`he`, `my mother`-`my father`). If we think about sentiment analysis, there is no reason that the a system would assign different prediction to the paird sentences! So if we find differce in that, it could point for a potential gender bias in the downstream application.\n",
    "\n",
    "#### Keep in mind, this is only an operalization of the gender bias in a sentiment analysis system. All the issues with a concreate and single measurement arise also here! We should always take into accout the human contex in which the system is deployed!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mI7tcLmhqzzI"
   },
   "source": [
    "First, let's download the "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N3_K-EWQqvZr"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "wget https://learn.responsibly.ai/word-embedding/data/Equity-Evaluation-Corpus.zip \\\n",
    "     -O Equity-Evaluation-Corpus.zip -q\n",
    "\n",
    "wget https://learn.responsibly.ai/word-embedding/data/SemEval2018-Task1-all-data.zip \\\n",
    "     -O SemEval2018-Task1-all-data.zip -q\n",
    "\n",
    "unzip -qq -o Equity-Evaluation-Corpus.zip -d ./data\n",
    "\n",
    "unzip -qq -o SemEval2018-Task1-all-data.zip -d ./data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3Z60x-8iuWGM"
   },
   "outputs": [],
   "source": [
    "# 🛠 Prepare the EEC data, no need to dig into this cell\n",
    "\n",
    "eec_df = pd.read_csv('./data/Equity-Evaluation-Corpus/Equity-Evaluation-Corpus.csv')\n",
    "\n",
    "# Remove the sentences for evaluating racial bias\n",
    "gender_eec_df = eec_df[eec_df['Race'].isna()][:]\n",
    "\n",
    "# Create identifier to mach sentence pairs\n",
    "# The EEC data comes withot this matching\n",
    "MALE_PERSONS = ('he', 'this man', 'this boy', 'my brother', 'my son', 'my husband',\n",
    "                'my boyfriend', 'my father', 'my uncle', 'my dad', 'him')\n",
    "\n",
    "FEMALE_PERSONS = ('she', 'this woman', 'this girl', 'my sister', 'my daughter', 'my wife',\n",
    "                  'my girlfriend', 'my mother', 'my aunt', 'my mom', 'her')\n",
    "\n",
    "MALE_IDENTIFIER = dict(zip(MALE_PERSONS, FEMALE_PERSONS))\n",
    "FEMALE_IDENTIFIER = dict(zip(FEMALE_PERSONS, FEMALE_PERSONS))\n",
    "\n",
    "PERSON_MATCH_WORDS = {**MALE_IDENTIFIER,\n",
    "                      **FEMALE_IDENTIFIER}\n",
    "\n",
    "gender_eec_df['PersonIdentifier'] = gender_eec_df['Person'].map(PERSON_MATCH_WORDS)\n",
    "\n",
    "gender_eec_df = gender_eec_df.sort_values(['Gender', 'Template', 'Emotion word', 'PersonIdentifier'])\n",
    "\n",
    "gender_split_index = len(gender_eec_df) // 2\n",
    "\n",
    "# Create two DataFrames, one for \n",
    "female_eec_df = gender_eec_df[:gender_split_index].reset_index(False)\n",
    "male_eec_df = gender_eec_df[gender_split_index:].reset_index(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5yKi4SYCPcc4"
   },
   "outputs": [],
   "source": [
    "female_eec_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wwxLn-WbxbwV"
   },
   "outputs": [],
   "source": [
    "male_eec_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7KOoYeQCQ9JU"
   },
   "source": [
    "Note that the two DataFrames are paired by index. If we take that *i*-th row in each one of them, then they are different only in the matched person word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8ZCc8eadRH-k"
   },
   "outputs": [],
   "source": [
    "k = 543  # change my value and run the cell again!\n",
    "female_eec_df.iloc[k]['Sentence'], male_eec_df.iloc[k]['Sentence']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-TH3wHOe5xN4"
   },
   "source": [
    "Compute the probability estimations of the classifier for the female and male parts in the EEC data. If you use `sklearn`, then the classifier's method `predict_proba` is your friend for that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ASdKJWk_C9rq"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4hGJspuayGmq"
   },
   "source": [
    "### Do the same for the neutralize-\"debiased\" word2vec\n",
    "\n",
    "Perform the all the previous steps for the neutralize-\"debiased\" word2vec to produce the probability estimations of the EEC data for the classifier using that word-embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1HAqdHCO7KhQ"
   },
   "source": [
    "#### Neutralize-\"debias\" the word embedding\n",
    "\n",
    "Hints:\n",
    "1. Use [`responsibly.we.GenderBiasWE`](https://docs.responsibly.ai/word-embedding-bias.html). \n",
    "2. Look for the method `debias`.\n",
    "3. Set the `method` argument to `'neutralize'`. \n",
    "4. Make sure that you set `inplace=True` to save memory. Note that you won't be able to work with the original word embedding after that.\n",
    "5. Validate the neutralize-\"debias\" was applied by computing the direct bias measure with the method `calc_direct_bias`.\n",
    "6. After the bias mitigating, the word embedding itself (as a `KeyedVectors` of `gensim`), is accessible through the attribute `model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T7CQ4Xz-7hBx"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NuCEd3hVDNEF"
   },
   "source": [
    "#### Generate features with the \"debiased\" word embedding and train a new classifier\n",
    "\n",
    "Check the classifier's accuracy on the training and the test data - did the \"debiasing\" of the word embeddings hurt the classifier performance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EOe-fwnnDT7C"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0PyZXIUiDU3l"
   },
   "source": [
    "#### Compute the probability estimations for the male and female sentences in the EEC data with the new classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mTLXhU0RDjNy"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cNldyVC_4O5k"
   },
   "source": [
    "### Gender bias analysis\n",
    "\n",
    "Now we are ready to blend all together. You have two classifiers, each one of them was trained on the same dataset, but with a different word embedding. The first used the original word2vec, and the other was undergone the neutralize-\"debias\" process. We computed the probability estimates for the EEC data twice for each one of the classifiers.\n",
    "\n",
    "\n",
    "**Think about how to evaluate the impact of replacing the word embedding concerning gender bias. Keep in mind that the female and male EEC data is paired!**\n",
    "\n",
    "#### Your analysis can take two points of view (there are more, but you start with that):\n",
    "1. Analyze the difference between the female and male probability estimations for each system *separately* and compare the results.\n",
    "2. Analyze the difference of differences; start with the difference of probability estimations between the paired female and male sentences for each system, and then compare the two differences.\n",
    "\n",
    "\n",
    "#### Few possible ideas of what to do:\n",
    "1. Plot distributions  ([`seaborn.displot`](https://seaborn.pydata.org/generated/seaborn.displot.html#seaborn.displot))\n",
    "2. Compute the [effect size](https://en.wikipedia.org/wiki/Effect_size#Cohen's_d)\n",
    "3. Perform statistical hypothesis testings to check whether means are eqaul using the paired t-test ([`scipy.stats.ttest_rel`])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Rz8SdPr9C-3p"
   },
   "outputs": [],
   "source": [
    "# Your Code Here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xpwTNQr7Oe-a"
   },
   "source": [
    "#### What is your conclusion? What would be your next steps?\n",
    "\n",
    "Consider:\n",
    "\n",
    "1. Group by the analysis according to the EEC columns (e.g., by emotion)\n",
    "2. Try another classifier (e.g., `sklearn.ensemble.RandomForestClassifier`)\n",
    "3. Change the mitigation bias to *hard* instead of *neutralize*.\n",
    "4. Analyze the training data from gender prespective\n",
    "\n",
    "\n",
    "\n",
    "Refer to this paper for some ideas:\n",
    "[Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](http://saifmohammad.com/WebDocs/EEC/ethics-StarSem-final_with_appendix.pdf). Svetlana Kiritchenko and Saif M. Mohammad. In Proceedings of *Sem, New Orleans, LA, USA, June 2018."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cyeapv6bFQu2"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rOYeEZl_C3vd"
   },
   "source": [
    "# Part Twelve: Examples of Representation Bias in the Context of Gender\n",
    "\n",
    "<img src=\"./images/examples-gender-bias-nlp.png\" />\n",
    "\n",
    "<small>Source: Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., ... & Wang, W. Y. (2019). [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://arxiv.org/pdf/1906.08976.pdf). arXiv preprint arXiv:1906.08976.</small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OSKHrpPbFRo_"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VsnVe19mFSHl"
   },
   "source": [
    "# Part Thirteen: Takeaways\n",
    "<big>💎💎</big>\n",
    "\n",
    "1. **Downstream application** - putting a system into a human context, be aware of the [abstraction error](http://friedler.net/papers/sts_fat2019.pdf)\n",
    "\n",
    "2. **Measurements** (a.k.a \"what is a *good* system?\")\n",
    "\n",
    "3. **Data** (generation process, corpus building, collection selection bias, features (measurement and operationalization), train vs. validation vs. test datasets)\n",
    "\n",
    "4. **Impact** of a system on individuals, groups, society, and humanity; **long-term**, **scale-up** and **automated**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "24dj9Z5fFPlB"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dqSvbe2sC3vd"
   },
   "source": [
    "# Part Fourteen: Resources\n",
    "<big>💎💎</big>\n",
    "\n",
    "## [Doing Data Science Responsibly - Resources](https://handbook.responsibly.ai/appendices/resources.html)\n",
    "\n",
    "In particular:\n",
    "\n",
    "- Timnit Gebru and Emily Denton - CVPR 2020 - [FATE Tutorial](https://youtu.be/-xGvcDzvi7Q) [Video]\n",
    "\n",
    "- Rachel Thomas - fast.ai - [Algorithmic Bias (NLP video 16)](https://youtu.be/pThqge9QDn8) [Video]\n",
    "\n",
    "- Solon Barocas, Moritz Hardt, Arvind Narayanan - [Fairness and machine learning - Limitations and Opportunities](https://fairmlbook.org/) [Textbook]\n",
    "\n",
    "## [Course // Responsible AI, Law, Ethics & Society](https://learn.responsibly.ai/)\n",
    "\n",
    "## Non-Technical Overview with More Downstream Application Examples\n",
    "- [Google - Text Embedding Models Contain Bias. Here's Why That Matters.](https://developers.googleblog.com/2018/04/text-embedding-models-contain-bias.html)\n",
    "- [Kai-Wei Chang (UCLA) - What It Takes to Control Societal Bias in Natural Language Processing](https://www.youtube.com/watch?v=RgcXD_1Cu18)\n",
    "- Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., ... & Wang, W. Y. (2019). [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://arxiv.org/pdf/1906.08976.pdf). arXiv preprint arXiv:1906.08976."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4LtaZOPmC3vd"
   },
   "source": [
    "## Critical Prespective on Bias in NLP\n",
    "\n",
    "Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020). [Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP](https://arxiv.org/pdf/2005.14050.pdf). arXiv preprint arXiv:2005.14050.\n",
    "\n",
    "## Additional Related Work\n",
    "\n",
    "- **Software Framework for Word Embedding Bias**\n",
    "  - [WEFE: The Word Embeddings Fairness Evaluation Framework](https://wefe.readthedocs.io/en/latest/)\n",
    "\n",
    "- **Understanding Bias**\n",
    "    - Ethayarajh, K., Duvenaud, D., & Hirst, G. (2019, July). [Understanding Undesirable Word Embedding Associations](https://arxiv.org/pdf/1908.06361.pdf). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1696-1705). - **Including critical analysis of the current metrics and debiasing methods (quite technical)**\n",
    "\n",
    "  - Brunet, M. E., Alkalay-Houlihan, C., Anderson, A., & Zemel, R. (2019, May). [Understanding the Origins of Bias in Word Embeddings](https://arxiv.org/pdf/1810.03611.pdf). In International Conference on Machine Learning (pp. 803-811).\n",
    "\n",
    "- **Discovering Bias**\n",
    "  - Swinger, N., De-Arteaga, M., Heffernan IV, N. T., Leiserson, M. D., & Kalai, A. T. (2019, January). [What are the biases in my word embedding?](https://arxiv.org/pdf/1812.08769.pdf). In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 305-311). ACM.\n",
    "    Measuring Gender Bias in Word Embeddings across Domains and Discovering New Gender Bias Word Categories\n",
    "  \n",
    "  - Chaloner, K., & Maldonado, A. (2019, August). [Measuring Gender Bias in Word Embeddings across Domains and Discovering New Gender Bias Word Categories](https://www.aclweb.org/anthology/W19-3804). In Proceedings of the First Workshop on Gender Bias in Natural Language Processing (pp. 25-32).\n",
    "\n",
    "- **Mitigating Bias**\n",
    "  - Maudslay, R. H., Gonen, H., Cotterell, R., & Teufel, S. (2019). [It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution](https://arxiv.org/pdf/1909.00871.pdf). arXiv preprint arXiv:1909.00871.\n",
    "  \n",
    "  - Shin, S., Song, K., Jang, J., Kim, H., Joo, W., & Moon, I. C. (2020). [Neutralizing Gender Bias in Word Embedding with Latent Disentanglement and Counterfactual Generation](https://arxiv.org/pdf/2004.03133.pdf). arXiv preprint arXiv:2004.03133.\n",
    "  \n",
    "  - Zhang, B. H., Lemoine, B., & Mitchell, M. (2018, December). [Mitigating unwanted biases with adversarial learning](https://dl.acm.org/doi/pdf/10.1145/3278721.3278779?casa_token=yd1KGvVDBGwAAAAA:YzUT7d8Fq4bOV2b5M-CB43NLqIReW7wx2EaZj0omJ0ncbZF_pkPFoyV6WHWIBnG_HKIRqiG7FWFjsA). In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 335-340). [Demo](https://colab.research.google.com/notebooks/ml_fairness/adversarial_debiasing.ipynb)\n",
    "  \n",
    "- **Fairness in Classification**\n",
    "  - Prost, F., Thain, N., & Bolukbasi, T. (2019, August). [Debiasing Embeddings for Reduced Gender Bias in Text Classification](https://arxiv.org/pdf/1908.02810.pdf). In Proceedings of the First Workshop on Gender Bias in Natural Language Processing (pp. 69-75).\n",
    "  \n",
    "  - Romanov, A., De-Arteaga, M., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., ... & Kalai, A. (2019, June). [What's in a Name? Reducing Bias in Bios without Access to Protected Attributes](https://arxiv.org/pdf/1904.05233.pdf). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4187-4195).\n",
    "\n",
    "- **Grammatical Gender**\n",
    "  - Zhou, P., Shi, W., Zhao, J., Huang, K. H., Chen, M., & Chang, K. W. [Analyzing and Mitigating Gender Bias in Languages with Grammatical Gender and Bilingual Word Embeddings](https://aiforsocialgood.github.io/icml2019/accepted/track1/pdfs/47_aisg_icml2019.pdf). ICML 2019 - AI for Social Good. [Poster](https://aiforsocialgood.github.io/icml2019/accepted/track1/posters/47_aisg_icml2019.pdf)\n",
    "\n",
    "  - Zhao, J., Mukherjee, S., Hosseini, S., Chang, K. W., & Awadallah, A. [Gender Bias in Multilingual Embeddings](https://www.researchgate.net/profile/Subhabrata_Mukherjee/publication/340660062_Gender_Bias_in_Multilingual_Embeddings/links/5e97428692851c2f52a6200a/Gender-Bias-in-Multilingual-Embeddings.pdf).\n",
    "\n",
    "  - Gonen, H., Kementchedjhieva, Y., & Goldberg, Y. (2019). [How does Grammatical Gender Affect Noun Representations in Gender-Marking Languages?](https://arxiv.org/pdf/1910.14161.pdf). arXiv preprint arXiv:1910.14161.\n",
    "\n",
    "- **Other**  \n",
    "  - Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., & Chang, K. W. (2019, June). [Gender Bias in Contextualized Word Embeddings](https://arxiv.org/pdf/1904.03310.pdf). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 629-634). [slides](https://jyzhao.net/files/naacl19.pdf)\n",
    "\n",
    "\n",
    "##### Complete example of using `responsibly` with Word2Vec, GloVe and fastText: http://docs.responsibly.ai/notebooks/demo-gender-bias-words-embedding.html\n",
    "\n",
    "\n",
    "## Bias in NLP\n",
    "\n",
    "Around dozen of papers on this field until 2019, but nowdays plenty of work is done. Two venues from back then:\n",
    "- [1st ACL Workshop on Gender Bias for Natural Language Processing](https://genderbiasnlp.talp.cat/)\n",
    "- [NAACL 2019](https://naacl2019.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LKwah1OlFWgQ"
   },
   "source": [
    "<img src=\"./images/banner.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8HMi4HiOFaJO"
   },
   "source": [
    "<center><h1>THE END!</h1></center>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "tutorial-bias-word-embedding.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}