bAbI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question Answering using GRU with bAbI Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We are going to implement a question answering algorithm to solve the first 20 tasks in the bAbI dataset. These tasks are a good start towards a more sophisticated automatic text understanding and reasoning.\n",
    "\n",
    "For a more comprehensive info about the dataset visit https://research.fb.com/downloads/babi/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### The principal components are the embedding matrix created using Glove Pretrained Word Embedding, a GRU layer which takes a story and the relative question concatenated in a sequencial way and a softmax over all the words in the vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import some useful library used through the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import regex as re\n",
    "from tensorflow.python.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.python.keras.preprocessing.sequence import pad_sequences\n",
    "from tqdm import tqdm\n",
    "from datetime import datetime\n",
    "import zipfile\n",
    "from urllib.request import urlretrieve\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unzip_single_file(zip_file_name, output_file_name):\n",
    "    '''\n",
    "    Unzip a file\n",
    "    \n",
    "    Input:\n",
    "    zip_file_name: path file to unzip\n",
    "    output_file_name: path to unzip\n",
    "    '''\n",
    "    \n",
    "    with open(output_file_name, 'wb') as out_file:\n",
    "        zipped = zipfile.ZipFile(zip_file_name)\n",
    "        for info in zipped.infolist():\n",
    "            if output_file_name in info.filename:\n",
    "                out_file.write(zipped.open(info).read())\n",
    "                return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrive and unzip glove word embedding\n",
    "\n",
    "glove_zip_file = \"glove.6B.zip\"\n",
    "glove_vectors_file = \"glove.6B.50d.txt\"\n",
    "\n",
    "urlretrieve (\"http://nlp.stanford.edu/data/glove.6B.zip\", glove_zip_file)\n",
    "unzip_single_file(glove_zip_file, glove_vectors_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function is used to create an embedding matrix for the words find in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embedding_matrix(emb_file, word_index, embedding_dim):\n",
    "    '''\n",
    "    Create an embedding matrix\n",
    "    \n",
    "    Input:\n",
    "    emb_file: string containing the path to the embedding file\n",
    "    word_index: dictionary with words as key and index for the word as value (e.g. {'word1': index1, 'word2':index2 ..})\n",
    "    embedding_dim: embedding dimension\n",
    "    \n",
    "    Output: embedding matrix. The n° row contains the embedding of the n° index word\n",
    "    '''\n",
    "    \n",
    "    # Load the embedding file\n",
    "    print('Loading embedding file', emb_file, '..')\n",
    "    embeddings_index = {}\n",
    "    f = open(emb_file, encoding=\"utf8\")\n",
    "    \n",
    "    for line in f:\n",
    "        try:\n",
    "            values = line.split()\n",
    "            word = values[0]\n",
    "            coefs = np.asarray(values[1:], dtype='float32')\n",
    "            embeddings_index[word] = coefs\n",
    "        except ValueError:\n",
    "            continue\n",
    "\n",
    "    f.close()\n",
    "    \n",
    "    print('Found %s word vectors.' % len(embeddings_index))\n",
    "    \n",
    "    # Create the embedding matrix needed for our dataset\n",
    "    global nb_words \n",
    "    nb_words = min(vocabulary_size, len(word_index)+1)\n",
    "    print('nb_words',nb_words)\n",
    "    \n",
    "    # Initialized the embedding matrix with normal random value\n",
    "    all_embs = np.hstack(word_index.values())\n",
    "    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n",
    "    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embedding_dim))\n",
    "    \n",
    "    count = 0\n",
    "    # For every word on the dataset, find the respective pre-trained word vectors.\n",
    "    for word, i in word_index.items():\n",
    "        if i >= vocabulary_size: continue\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None:\n",
    "            embedding_matrix[i] = embedding_vector\n",
    "            count += 1\n",
    "\n",
    "    print('Found', count, 'words corrispondances')\n",
    "\n",
    "    return embedding_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following methods preprocess the dataset and convert a string to a sequence of integers "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dataset(name):\n",
    "    '''\n",
    "    Return the Babl dataset\n",
    "    \n",
    "    Input\n",
    "    name: path of the babl task\n",
    "    \n",
    "    Output: Babl dataset\n",
    "    '''\n",
    "    \n",
    "    dataset = []\n",
    "    \n",
    "    with open(name) as file:\n",
    "        story = []\n",
    "        \n",
    "        for line in file.readlines():\n",
    "            fp, sp = line.split(' ', 1)\n",
    "            if fp == '1':\n",
    "                story = []\n",
    "                story.append(sp)\n",
    "            elif '\\t' not in sp:\n",
    "                story.append(sp)\n",
    "            else:\n",
    "                qst, answer, supp_facts = line.split('\\t')\n",
    "                dataset.append([story[:], qst.split(' ', 1)[1], qst.split(' ', 1)[0], answer, supp_facts])\n",
    "    \n",
    "    return dataset\n",
    "       \n",
    "def preprocess_dataset(dataset):\n",
    "    '''\n",
    "    Preprocess the babI dataset and return the story, the question and the answer\n",
    "    \n",
    "    Input\n",
    "    dataset: Babl dataset\n",
    "    \n",
    "    Output: the Babl dataset preprocessed\n",
    "    '''\n",
    "    \n",
    "    preproc_dataset = []\n",
    "    \n",
    "    for stry, qst, _, ans, _ in dataset:\n",
    "        stry = re.sub('\\n', ' ', ''.join(stry))\n",
    "        preproc_dataset.append((stry.lower(), qst.lower(), re.sub(',','',ans.lower())))\n",
    "        \n",
    "    return preproc_dataset\n",
    "\n",
    "def tokenize(tr_dataset, vl_dataset):\n",
    "    '''\n",
    "    Given a train and validation dataset, return the respective stories, questions and answers as sequence of indices. Every index represent the position of the give word in the embedding matrix \n",
    "    \n",
    "    Input:\n",
    "    tr_dataset: Train dataset. Should be an array containing as column the story, question and answer. \n",
    "    vl_dataset: Validation dataset. Should be an array containing as column the story, question and answer.\n",
    "    \n",
    "    Output:\n",
    "    1) Tokenizer\n",
    "    2) Sequences of indices corresponding to the train's stories\n",
    "    3) Sequences of indices corresponding to the train's questions\n",
    "    4) Sequences of answers corresponding to the train's answers\n",
    "    5) Sequences of indices corresponding to the validation's stories\n",
    "    6) Sequences of indices corresponding to the validation's questions\n",
    "    7) Sequences of answers corresponding to the validation's answers\n",
    "    '''\n",
    "    \n",
    "    # Tokenize both train & validation set\n",
    "    tokenizer = Tokenizer(num_words=vocabulary_size, lower=True, filters='.?')\n",
    "    tokenizer.fit_on_texts([x+y+z for x, y, z in tr_dataset]+[x+y+z for x, y, z in vl_dataset])\n",
    "\n",
    "    # Convert train texts into sequences\n",
    "    stories_tokenized_train = tokenizer.texts_to_sequences([x for x, _, _ in tr_dataset])\n",
    "    stories_seq = pad_sequences(stories_tokenized_train, maxlen=story_max_len)\n",
    "    \n",
    "    questions_tokenized_train = tokenizer.texts_to_sequences([x for _, x, _ in tr_dataset])\n",
    "    questions_seq = pad_sequences(questions_tokenized_train, maxlen=question_max_len)\n",
    "    \n",
    "    answers_tokenized_train = tokenizer.texts_to_sequences([x for _, _, x in tr_dataset])\n",
    "    \n",
    "    # Convert validation texts into sequences\n",
    "    valid_stories_tokenized_train = tokenizer.texts_to_sequences([x for x, _, _ in vl_dataset])\n",
    "    valid_stories_seq = pad_sequences(valid_stories_tokenized_train, maxlen=story_max_len)\n",
    "    \n",
    "    valid_questions_tokenized_train = tokenizer.texts_to_sequences([x for _, x, _ in vl_dataset])\n",
    "    valid_questions_seq = pad_sequences(valid_questions_tokenized_train, maxlen=question_max_len)\n",
    "    \n",
    "    valid_answers_tokenized_train = tokenizer.texts_to_sequences([x for _, _, x in vl_dataset])\n",
    "    \n",
    "    return tokenizer, stories_seq, questions_seq, np.squeeze(np.array(answers_tokenized_train)),\\\n",
    "            valid_stories_seq, valid_questions_seq, np.squeeze(np.array(valid_answers_tokenized_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define some useful constants "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocabulary_size = 2000\n",
    "story_max_len = 500\n",
    "question_max_len = 8\n",
    "EMBEDDING_DIM = 50\n",
    "batch_size = 64\n",
    "\n",
    "# Model hyperparameters\n",
    "learning_rate = 0.002\n",
    "iterations = 10000\n",
    "test_on_valid = 800\n",
    "gru_num_units = 100\n",
    "gru_rev_num_units = 50\n",
    "drop_input_prob = 0.9\n",
    "drop_output_prob = 0.9\n",
    "print_summary_step = 8\n",
    "summary_on_valid = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Acquire, preprocess and tokenize 1-20 bAbI tasks.\n",
    "train_dataset = []\n",
    "valid_dataset = []\n",
    "\n",
    "root = 'tasks_1-20_v1-2/en/'\n",
    "qa_tasks = ['qa1_single-supporting-fact_{}.txt', 'qa2_two-supporting-facts_{}.txt', 'qa3_three-supporting-facts_{}.txt', 'qa4_two-arg-relations_{}.txt', 'qa5_three-arg-relations_{}.txt', \n",
    "            'qa6_yes-no-questions_{}.txt', 'qa7_counting_{}.txt', 'qa8_lists-sets_{}.txt', 'qa9_simple-negation_{}.txt', 'qa10_indefinite-knowledge_{}.txt', \n",
    "            'qa11_basic-coreference_{}.txt','qa12_conjunction_{}.txt', 'qa13_compound-coreference_{}.txt', 'qa14_time-reasoning_{}.txt', 'qa15_basic-deduction_{}.txt',\n",
    "            'qa16_basic-induction_{}.txt', 'qa17_positional-reasoning_{}.txt', 'qa18_size-reasoning_{}.txt', 'qa19_path-finding_{}.txt', 'qa20_agents-motivations_{}.txt']\n",
    "\n",
    "for qa_task in qa_tasks:\n",
    "    if train_dataset == []:\n",
    "        train_dataset = get_dataset(root + qa_task.format('train'))\n",
    "        valid_dataset = get_dataset(root + qa_task.format('test'))\n",
    "    else:\n",
    "        train_dataset = np.concatenate([train_dataset, get_dataset(root + qa_task.format('train'))], axis=0)\n",
    "        valid_dataset = np.concatenate([valid_dataset, get_dataset(root + qa_task.format('test'))], axis=0)\n",
    "\n",
    "print('len train_dataset:', len(train_dataset))\n",
    "print('len valid_dataset:', len(train_dataset))\n",
    "\n",
    "train_dataset = preprocess_dataset(train_dataset)\n",
    "valid_dataset = preprocess_dataset(valid_dataset)\n",
    "\n",
    "# Tokenize and convert the dataset from a list of string to a list of index sequences\n",
    "tokenizer, stories_seq, questions_seq, answers_seq, valid_stories_seq, valid_question_seq, valid_answer_seq = tokenize(train_dataset, valid_dataset)\n",
    "\n",
    "# Create an embedding matrix using glove50 pretrained word embeddings\n",
    "glove_embedding_matrix = get_embedding_matrix(glove_vectors_file, tokenizer.word_index, EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <center> NETWORK MODEL</center>\n",
    "<center>![title](images/gru_model.png)</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tensorboard variables\n",
    "now = datetime.utcnow().strftime(\"%Y%m%d-%H-%M-%S\")\n",
    "root_dir = \"tf_logs\"\n",
    "logdir = \"{}/run-{}/\".format(root_dir, now)\n",
    "\n",
    "# Create the graph\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# Initialize the cells\n",
    "gru_cell = tf.nn.rnn_cell.GRUCell(gru_num_units)\n",
    "gru_drop = tf.contrib.rnn.DropoutWrapper(gru_cell, drop_input_prob, drop_output_prob)\n",
    "\n",
    "gru_cell_rev = tf.nn.rnn_cell.GRUCell(gru_rev_num_units)\n",
    "gru_drop_rev = tf.contrib.rnn.DropoutWrapper(gru_cell_rev, drop_input_prob, drop_output_prob)\n",
    "\n",
    "# Create two placeholders for the input\n",
    "X_story = tf.placeholder(tf.int32, [None, story_max_len], 'X_story')\n",
    "X_question = tf.placeholder(tf.int32, [None, question_max_len], 'X_question')\n",
    "\n",
    "\n",
    "with tf.variable_scope('embeddings'):\n",
    "    # initialize the embedding variable like the matrix previously created\n",
    "    glove_weights_initializer = tf.constant_initializer(np.array(glove_embedding_matrix))\n",
    "    embedding_weights = tf.get_variable('embedding_weights', shape=(nb_words, EMBEDDING_DIM), initializer=glove_weights_initializer, trainable=False)\n",
    "\n",
    "    # embeddings lookup for the story\n",
    "    ## tf.nn.embedding_lookup: lookup for every index in X_story, the respective word embedding in the embedding_weights variable\n",
    "    embedding_story = tf.nn.embedding_lookup(embedding_weights, X_story)\n",
    "\n",
    "    # embeddings lookup for the question\n",
    "    embedding_question = tf.nn.embedding_lookup(embedding_weights, X_question)\n",
    "\n",
    "    emb_qa = tf.concat([embedding_story,embedding_question], axis=1)\n",
    "\n",
    "with tf.variable_scope('GRU'):\n",
    "    # GRU layer\n",
    "    outputs_conc, state_conc = tf.nn.dynamic_rnn(gru_drop, emb_qa, dtype=tf.float32)\n",
    "\n",
    "    # Last layer\n",
    "    attend_init = tf.random_normal_initializer(stddev=0.1)\n",
    "    w = tf.get_variable(\"w\", [100, nb_words], tf.float32, initializer=attend_init)\n",
    "    b = tf.get_variable(\"b\", [nb_words], tf.float32, initializer=attend_init)\n",
    "    o = tf.matmul(state_conc, w) + b\n",
    "    summ_out = tf.summary.histogram('output', o)\n",
    "\n",
    "y = tf.placeholder(tf.int32, [None], 'y')\n",
    "y_label = tf.one_hot(y, nb_words)\n",
    "\n",
    "with tf.variable_scope('loss'):\n",
    "    # Evaluate the Softmax Cross Entropy loss\n",
    "    loss = tf.nn.softmax_cross_entropy_with_logits_v2(labels=y_label, logits=o)\n",
    "    summ_losses = tf.summary.histogram('losses', loss)\n",
    "    loss = tf.reduce_mean(loss)\n",
    "    summ_train_loss = tf.summary.scalar('train_loss', loss)\n",
    "    summ_valid_loss = tf.summary.scalar('valid_loss', loss)\n",
    "\n",
    "with tf.variable_scope('train'):\n",
    "    # Optimizer and training phase\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "    training_op = optimizer.minimize(loss)\n",
    "\n",
    "with tf.variable_scope('correct'):\n",
    "    # Compute the accuracy\n",
    "    y_p_labels = tf.squeeze(tf.argmax(y_label, axis=-1))\n",
    "    o_p_labels = tf.argmax(o, axis=-1)\n",
    "\n",
    "    correct = tf.cast(tf.equal(y_p_labels, o_p_labels), tf.int32)\n",
    "\n",
    "    summ_correct = tf.summary.histogram('correct', correct)\n",
    "    summ_valid_correct = tf.summary.scalar('valid_acc', tf.reduce_mean(tf.cast(correct, tf.float32)))\n",
    "    \n",
    "# tensorbaord initialization\n",
    "mrg_train_summary = tf.summary.merge([summ_losses, summ_train_loss, summ_out, summ_correct])\n",
    "mrg_valid_summary = tf.summary.merge([summ_valid_loss, summ_valid_correct])\n",
    "file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    embedding_weights.initializer.run()\n",
    "      \n",
    "    # Shuffle arrays\n",
    "    shuffled = np.arange(len(stories_seq))\n",
    "    np.random.shuffle(shuffled)\n",
    "    stories_seq = stories_seq[shuffled]\n",
    "    questions_seq = questions_seq[shuffled]\n",
    "    answers_seq = answers_seq[shuffled]\n",
    "    \n",
    "    # Train and evaluate the model training_it times\n",
    "    for it in tqdm(range(0, iterations)):\n",
    "        # Create the batch\n",
    "        batch = np.random.randint(len(stories_seq), size=batch_size)\n",
    "        stories_batch = stories_seq[batch]\n",
    "        questions_batch = questions_seq[batch]\n",
    "        answers_batch = answers_seq[batch]\n",
    "        \n",
    "        # Train the network and retrive the loss\n",
    "        batch_loss, _ = sess.run([loss, training_op], feed_dict={X_story:stories_batch, X_question:questions_batch, y:answers_batch})\n",
    "        \n",
    "        # Update the tensorboard summary\n",
    "        if it % print_summary_step == 0:\n",
    "            train_summary = mrg_train_summary.eval(feed_dict={X_story:stories_batch, X_question:questions_batch, y:answers_batch})\n",
    "            file_writer.add_summary(train_summary, it)\n",
    "\n",
    "        # Test on the validation set\n",
    "        if it % test_on_valid == 0:\n",
    "            valid_losses = []\n",
    "            acc_values = []\n",
    "            \n",
    "            # Compute the accuracy and loss for every tasks (1-20th)\n",
    "            for it_v in range(0,len(valid_stories_seq),1000):\n",
    "                valid_loss, acc_value = sess.run([loss, correct], feed_dict={X_story:valid_stories_seq[it_v:it_v+1000], X_question:valid_question_seq[it_v:it_v+1000], y:valid_answer_seq[it_v:it_v+1000]})\n",
    "                print('Task',int((it_v/1000)+1), 'Accuracy:', np.round(np.mean(acc_value)*100,2), '%')\n",
    "                valid_losses.append(valid_loss)\n",
    "                acc_values.append(np.mean(acc_value))\n",
    "\n",
    "            # Print the mean accuracy and loss\n",
    "            print('Train loss:', batch_loss, '\\tValidation loss:', np.round(np.mean(valid_losses), 4), 'Accuracy:', np.round(np.mean(acc_values)*100,2), '%')\n",
    "            \n",
    "        # Validation tensorboard summary\n",
    "        if it % summary_on_valid == 0:\n",
    "            # Compute the accuracy and loss for every tasks (1-20th)\n",
    "            for it_v in range(0,len(valid_stories_seq),1000):\n",
    "                valid_summary = mrg_valid_summary.eval(feed_dict={X_story:valid_stories_seq[it_v:it_v+1000], X_question:valid_question_seq[it_v:it_v+1000], y:valid_answer_seq[it_v:it_v+1000]})\n",
    "                file_writer.add_summary(valid_summary, it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Task | bAbl LSTM Baseline | Keras QA | Our Accuracy   |\n",
    "|------|------|------|\n",
    "|   1  | 50 % | 100 % | 50 %     |\n",
    "|   2  | 20 % | 50 % | 40 % |\n",
    "|   3  | 20 % | 20 % | 38 % |\n",
    "|   4  | 61 % | 63 % | 68 % |\n",
    "|   5  | 70 %  | 62 % | 66 % |\n",
    "|   6  | 48 %  | 51 % | 72 % |\n",
    "|   7  | 49 % | 79 % | 75 % |\n",
    "|   8  | 45 % | 77 % | 66 % |\n",
    "|   9  | 64 % | 64 % | 73 % |\n",
    "|   10  | 44 % | 48 % | 63 % |\n",
    "|   11  | 72 % | 75 % | 71 % |\n",
    "|   12  | 74 % | 76 % | 66 % |\n",
    "|   13  | 94 % | 94 % | 93 % |\n",
    "|   14  | 27 % | 35 % | 40 % |\n",
    "|   15  | 21 % | 32 % | 49 % |\n",
    "|   16  | 23 % | 51 % | 47 % |\n",
    "|   17  | 51 % | 49 % | 61 % |\n",
    "|   18  | 52 % | 91 % | 91 % |\n",
    "|   19  | 8 % | 9 % | 9 % |\n",
    "|   20  | 91 % | 91 % | 94 % |\n",
    "|  Mean | 49 % | 61 % | 62 % |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VALIDATION ACCURACY\n",
    "![title](images/valid_acc1.png)\n",
    "## VALIDATION LOSS\n",
    "![title](images/valid_loss1.png)\n",
    "## TRAIN LOSS\n",
    "![title](images/train_loss1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}