7.2_unsupervised_training_tsdae_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised sentence embedding learning - TSDAE\n",
    "\n",
    "** This notebook exposes some of the inner workings of the `DenoisingAutoEncoderLoss`(from the sentence_transformers library). The core components are borrowed from the sentence_transformers library. We just use the essential parts here to get a better understanding of the underlying architecture.\n",
    "\n",
    "In this notebook, we will look at the work of `Kexin Wang, Nils Reimers, Iryna Gurevych` on their paper [TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](https://arxiv.org/pdf/2104.06979.pdf).\n",
    "\n",
    "Here is the summary of the paper by the authors.\n",
    "\n",
    "> Learning sentence embeddings often requires\n",
    "a large amount of labeled data. However,\n",
    "for most tasks and domains, labeled data is\n",
    "seldom available and creating it is expensive.\n",
    "In this work, we present a new state-of-the-art unsupervised method based on pre-trained\n",
    "Transformers and Sequential Denoising AutoEncoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of indomain supervised approaches. Further, we\n",
    "show that TSDAE is a strong domain adaptation and pre-training method for sentence\n",
    "embeddings, significantly outperforming other approaches like Masked Language Model.\n",
    ">\n",
    "> A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.\n",
    "\n",
    "The techniques we have discussed so far - Bi-Encoders, Cross-Encoders, etc. required labeled data. While they show great performace on in-domain data, their performance declines rapidly on out-of-domain data.\n",
    "\n",
    "| ![](assets/models_generalize.png) | \n",
    "|:--:| \n",
    "| Fig. 1. Illustration of generalizability of Neural IR models on 18 IR datasets from the [BIER benchmark](https://arxiv.org/abs/2104.08663). (Image source: https://www.youtube.com/watch?v=xbdLowiQTlk&t=658s) |\n",
    "\n",
    "We can see that BM25(retriever/candidate generator) and Cross-Encoder(Re-Ranker) works the best while many dense retrievers fail to outperform just the BM25 on most on the datasets.\n",
    "\n",
    "So why not use BM25 + Cross-Encoders?\n",
    "\n",
    "As we mentioned earlier, Cross-Encoders are expensive as we can't index the corpus beforehand. For each query, we would need to score the query against all the retrieved candidates(which could be in 100s).\n",
    "\n",
    "So we again come back to Bi-Encoders as they are fast at both indexing the corpus and inference(using ANNs).\n",
    "\n",
    "In TSDAE, the authors train encoder based models with pre-training objective similar to `Masked Language Modeling` but with slight variation.\n",
    "In MLM, we mask some tokens and train the encoders to predict the masked tokens, but in TSDAE, we delete some tokens from the input sentences, create a pooled representation of the sentence(MEAN-pooling or CLS-embedding) and pass that to a Denoising Auto-Encoder to recreate the original input text.\n",
    "\n",
    "| ![](assets/tsdae.png) | \n",
    "|:--:| \n",
    "| Fig. 2. Illustration of TSDAE. (Image source: https://arxiv.org/pdf/2104.06979.pdf) |\n",
    "\n",
    "The authors tried a bunch of configurations for adding noise. The best results came from deletion with a deletion ratio of 0.6.\n",
    "Note that the models were trained on a combination of SNLI and MultiNLI datasets without labels and evaluated on the STS benchmark with the metric Spearman rank correlation.\n",
    "CLS and Mean-pooling worked the best with similar performance. They recommend choosing CLS-pooling so we will also use that.\n",
    "\n",
    "| ![](assets/tsdae_config.png) |\n",
    "|:--:|\n",
    "| Fig. 3. Results with differnt noise types, noise ratios and pooling methods. (Image source: https://arxiv.org/pdf/2104.06979.pdf) |\n",
    "\n",
    "Here we will use the `sentence_transformers` library to train this architecture. Lets start ...\n",
    "\n",
    "## Data preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from typing import Callable\n",
    "\n",
    "import datasets as hf_datasets\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import nltk\n",
    "from nltk.tokenize.treebank import TreebankWordDetokenizer\n",
    "import pandas as pd\n",
    "from sentence_transformers import InputExample, SentenceTransformer, LoggingHandler\n",
    "from sentence_transformers import models, util, datasets, evaluation, losses\n",
    "from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n",
    "from scipy import stats\n",
    "import torch\n",
    "from torch.utils.data import Dataset, DataLoader, random_split\n",
    "from transformers.optimization import get_linear_schedule_with_warmup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoConfig, AutoModelForCausalLM, BertModel, BertTokenizerFast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/utsav/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "nltk.download(\"punkt\")\n",
    "pd.set_option(\"display.max_colwidth\", None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 10\n",
    "\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "\n",
    "torch.manual_seed(seed)\n",
    "torch.cuda.manual_seed_all(seed)\n",
    "\n",
    "torch.backends.cudnn.deterministic = True\n",
    "torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset snli (/home/utsav/.cache/huggingface/datasets/snli/plain_text/1.0.0/1f60b67533b65ae0275561ff7828aad5ee4282d0e6f844fd148d05d3c6ea251b)\n",
      "Parameter 'function'=<function <lambda> at 0x7fb873803a60> of the transform datasets.arrow_dataset.Dataset.filter@2.0.1 couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "Loading cached processed dataset at /home/utsav/.cache/huggingface/datasets/snli/plain_text/1.0.0/1f60b67533b65ae0275561ff7828aad5ee4282d0e6f844fd148d05d3c6ea251b/cache-08577eb1924770d3.arrow\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(109812,\n",
       " {'premise': 'Children smiling and waving at camera',\n",
       "  'hypothesis': 'They are smiling at their parents',\n",
       "  'label': 1})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = hf_datasets.load_dataset(\"snli\", split=\"train\")\n",
    "dataset = dataset.filter(lambda _: True if random.random() > 0.9 else False)\n",
    "\n",
    "len(dataset), dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "77550"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_sentences = [d[\"premise\"] for d in dataset]\n",
    "train_sentences = list(set(train_sentences))\n",
    "len(train_sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sentence_transformers/datasets/DenoisingAutoEncoderDataset.py\n",
    "\n",
    "def get_noisy_text(text: str, del_ratio: float=0.6):\n",
    "    words = nltk.word_tokenize(text)\n",
    "    n = len(words)\n",
    "    if n == 0:\n",
    "        return text\n",
    "\n",
    "    keep_or_not = np.random.rand(n) > del_ratio\n",
    "    if sum(keep_or_not) == 0:\n",
    "        keep_or_not[np.random.choice(n)] = True\n",
    "    words_processed = TreebankWordDetokenizer().detokenize(np.array(words)[keep_or_not])\n",
    "\n",
    "    return words_processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "noisy_train_sentences = [get_noisy_text(txt) for txt in train_sentences]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at some examples of the noise added by this method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>noisy</th>\n",
       "      <th>original</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Two try to on quad, falls off.</td>\n",
       "      <td>Two students try to walk a tightrope on a campus quad, but one falls off.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>at work boy a on the beach</td>\n",
       "      <td>Hard at work, this teenage boy is handling a fish on the beach.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A cast something video another is at</td>\n",
       "      <td>A man with an arm cast films something on video while another man is looking at the camera.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>toddler yellow bathing is toe</td>\n",
       "      <td>A toddler in a yellow bathing suit is dipping her toe in a wading pool.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>huge</td>\n",
       "      <td>Children climbing huge tree.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>baby an adults fingers the</td>\n",
       "      <td>A baby is hold an adults fingers on the beach.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>black are at.</td>\n",
       "      <td>A woman in a pink bikini and a man in black swim trunks are jumping in the water at the beach.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A many</td>\n",
       "      <td>A dancer performs an aerial move with many others watch.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>at pool</td>\n",
       "      <td>A family at a swimming pool.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>A lays lot.</td>\n",
       "      <td>A man in blue shorts lays down outside in a parking lot.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>at local school game</td>\n",
       "      <td>A play happening at a local high school football game.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>polo while a</td>\n",
       "      <td>A polo player cradles a ball while riding a horse.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>people, businesses in the background.</td>\n",
       "      <td>A photograph of a street with people walking and sitting, and businesses in the background.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>man sleeping floor.</td>\n",
       "      <td>A man and two children are sleeping on the floor.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>licking</td>\n",
       "      <td>Brown dog with black collar licking nose.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>a bowl with chopsticks selling of meat</td>\n",
       "      <td>Woman eating a bowl of soup with chopsticks while selling raw slabs of meat.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>One female and white all of in of concrete stair the</td>\n",
       "      <td>One black female and one white female plus three white males, all wearing forms of black, in front of a concrete stair structure with other people sitting on the stairs.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>A a is down hill with a on his back</td>\n",
       "      <td>A skier in a blue outfit is skiing down the hill with a red pack on his back.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>wearing the and blue jean jacket smiling</td>\n",
       "      <td>The gal wearing the black sunglasses and blue jean jacket is smiling.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>reading names have passed.</td>\n",
       "      <td>The professor reading the names of students who have passed the exam.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   noisy  \\\n",
       "0                         Two try to on quad, falls off.   \n",
       "1                             at work boy a on the beach   \n",
       "2                   A cast something video another is at   \n",
       "3                          toddler yellow bathing is toe   \n",
       "4                                                   huge   \n",
       "5                             baby an adults fingers the   \n",
       "6                                          black are at.   \n",
       "7                                                 A many   \n",
       "8                                                at pool   \n",
       "9                                            A lays lot.   \n",
       "10                                  at local school game   \n",
       "11                                          polo while a   \n",
       "12                 people, businesses in the background.   \n",
       "13                                   man sleeping floor.   \n",
       "14                                               licking   \n",
       "15                a bowl with chopsticks selling of meat   \n",
       "16  One female and white all of in of concrete stair the   \n",
       "17                   A a is down hill with a on his back   \n",
       "18              wearing the and blue jean jacket smiling   \n",
       "19                            reading names have passed.   \n",
       "\n",
       "                                                                                                                                                                     original  \n",
       "0                                                                                                   Two students try to walk a tightrope on a campus quad, but one falls off.  \n",
       "1                                                                                                             Hard at work, this teenage boy is handling a fish on the beach.  \n",
       "2                                                                                 A man with an arm cast films something on video while another man is looking at the camera.  \n",
       "3                                                                                                     A toddler in a yellow bathing suit is dipping her toe in a wading pool.  \n",
       "4                                                                                                                                                Children climbing huge tree.  \n",
       "5                                                                                                                              A baby is hold an adults fingers on the beach.  \n",
       "6                                                                              A woman in a pink bikini and a man in black swim trunks are jumping in the water at the beach.  \n",
       "7                                                                                                                    A dancer performs an aerial move with many others watch.  \n",
       "8                                                                                                                                                A family at a swimming pool.  \n",
       "9                                                                                                                    A man in blue shorts lays down outside in a parking lot.  \n",
       "10                                                                                                                     A play happening at a local high school football game.  \n",
       "11                                                                                                                         A polo player cradles a ball while riding a horse.  \n",
       "12                                                                                A photograph of a street with people walking and sitting, and businesses in the background.  \n",
       "13                                                                                                                          A man and two children are sleeping on the floor.  \n",
       "14                                                                                                                                  Brown dog with black collar licking nose.  \n",
       "15                                                                                               Woman eating a bowl of soup with chopsticks while selling raw slabs of meat.  \n",
       "16  One black female and one white female plus three white males, all wearing forms of black, in front of a concrete stair structure with other people sitting on the stairs.  \n",
       "17                                                                                              A skier in a blue outfit is skiing down the hill with a red pack on his back.  \n",
       "18                                                                                                      The gal wearing the black sunglasses and blue jean jacket is smiling.  \n",
       "19                                                                                                      The professor reading the names of students who have passed the exam.  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame([(noisy_train_sentences[i], train_sentences[i]) for i in range(20)],\n",
    "                  columns=[\"noisy\", \"original\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"bert-base-uncased\"\n",
    "MAX_LENGTH = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizerFast.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_original_texts = tokenizer(train_sentences, max_length=MAX_LENGTH,\n",
    "                                     padding=\"max_length\", truncation=True,\n",
    "                                     return_tensors=\"pt\")\n",
    "\n",
    "tokenized_noisy_texts = tokenizer(noisy_train_sentences, max_length=MAX_LENGTH,\n",
    "                                  padding=\"max_length\", truncation=True,\n",
    "                                  return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NoisyDataset(Dataset):\n",
    "    def __init__(self, tokenized_original_texts: dict, tokenized_noisy_texts: dict):\n",
    "        self.tokenized_original_texts = tokenized_original_texts\n",
    "        self.tokenized_noisy_texts = tokenized_noisy_texts\n",
    "    \n",
    "    def __getitem__(self, ix: int) -> dict[str, torch.tensor]:\n",
    "        return {\"original_input_ids\": self.tokenized_original_texts[\"input_ids\"][ix],\n",
    "                \"original_attention_mask\": self.tokenized_original_texts[\"attention_mask\"][ix],\n",
    "                \"noisy_input_ids\": self.tokenized_noisy_texts[\"input_ids\"][ix],\n",
    "                \"noisy_attention_mask\": self.tokenized_noisy_texts[\"attention_mask\"][ix]}\n",
    "    \n",
    "    def __len__(self) -> int:\n",
    "        return len(self.tokenized_noisy_texts[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = NoisyDataset(tokenized_original_texts, tokenized_noisy_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ratio = 0.9\n",
    "n_total = len(dataset)\n",
    "n_train = int(n_total * train_ratio)\n",
    "n_val = n_total - n_train\n",
    "\n",
    "train_dataset, val_dataset = random_split(dataset, [n_train, n_val])\n",
    "\n",
    "batch_size = 16\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "val_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'original_input_ids': tensor([[ 101, 1037, 3899,  ...,    0,    0,    0],\n",
       "         [ 101, 1037, 2177,  ...,    0,    0,    0],\n",
       "         [ 101, 1037, 4845,  ...,    0,    0,    0],\n",
       "         ...,\n",
       "         [ 101, 2195, 2402,  ...,    0,    0,    0],\n",
       "         [ 101, 2879, 4147,  ...,    0,    0,    0],\n",
       "         [ 101, 1037, 2810,  ...,    0,    0,    0]]),\n",
       " 'original_attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         ...,\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0]]),\n",
       " 'noisy_input_ids': tensor([[  101,  1037,  3899,  ...,     0,     0,     0],\n",
       "         [  101,  1037,  2177,  ...,     0,     0,     0],\n",
       "         [  101,  4845,  3564,  ...,     0,     0,     0],\n",
       "         ...,\n",
       "         [  101,  3328, 20593,  ...,     0,     0,     0],\n",
       "         [  101,  4147,  3756,  ...,     0,     0,     0],\n",
       "         [  101,  2810,  7309,  ...,     0,     0,     0]]),\n",
       " 'noisy_attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         ...,\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0],\n",
       "         [1, 1, 1,  ..., 0, 0, 0]])}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(train_dataloader))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model config\n",
    "\n",
    "Here we will setup out custom `DenoisingAutoEncoder` model as detailed in the diagram from the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sentence_transformers/losses/DenoisingAutoEncoderLoss.py\n",
    "\n",
    "class DenoisingAutoEncoder(torch.nn.Module):\n",
    "    def __init__(self, model_name: str, device: str=\"cuda\"):\n",
    "        super().__init__()\n",
    "        self.model_name = model_name\n",
    "        self.device = device\n",
    "        self.encoder = BertModel.from_pretrained(self.model_name)\n",
    "        \n",
    "        decoder_config = AutoConfig.from_pretrained(self.model_name)\n",
    "        decoder_config.is_decoder = True\n",
    "        decoder_config.add_cross_attention = True\n",
    "        kwargs_decoder = {'config': decoder_config}\n",
    "        self.decoder = AutoModelForCausalLM.from_pretrained(self.model_name, **kwargs_decoder)\n",
    "    \n",
    "    def forward(self, x: torch.tensor) -> tuple[torch.tensor, torch.tensor]:\n",
    "        x[\"noisy_input_ids\"] = x[\"noisy_input_ids\"].to(self.device)\n",
    "        x[\"noisy_attention_mask\"] = x[\"noisy_attention_mask\"].to(self.device)\n",
    "        x[\"original_input_ids\"] = x[\"original_input_ids\"].to(self.device)\n",
    "\n",
    "        pooled_embeddings = self.encoder(x[\"noisy_input_ids\"], x[\"noisy_attention_mask\"]).pooler_output\n",
    "        \n",
    "        original_length = x[\"original_input_ids\"].shape[1]\n",
    "        decoder_input_ids = x[\"original_input_ids\"].clone()[:, :original_length - 1]\n",
    "        label_ids = x[\"original_input_ids\"][:, 1:]\n",
    "\n",
    "        decoder_outputs = self.decoder(\n",
    "            input_ids=decoder_input_ids,\n",
    "            encoder_hidden_states=pooled_embeddings.unsqueeze(1),  # (bsz, hdim) -> (bsz, 1, hdim)\n",
    "            encoder_attention_mask=x[\"noisy_attention_mask\"][:, 0:1]\n",
    "        )\n",
    "\n",
    "        lm_logits = decoder_outputs[0]\n",
    "        return lm_logits, label_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "model = DenoisingAutoEncoder(model_name, device)\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  optimizer, lr, num_warmup steps have been picked from the paper\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=3e-5)\n",
    "total_steps = len(train_dataset) // batch_size\n",
    "warmup_steps = int(0.1 * total_steps)\n",
    "scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps,\n",
    "                                            num_training_steps=total_steps - warmup_steps)\n",
    "\n",
    "loss_fn = torch.nn.CrossEntropyLoss(ignore_index=tokenizer.pad_token_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training loop "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_train_step_fn(\n",
    "    model: torch.nn.Module, optimizer: torch.optim.Optimizer,\n",
    "    scheduler: torch.optim.lr_scheduler.LambdaLR, loss_fn: torch.nn.CrossEntropyLoss\n",
    ") -> Callable[[torch.tensor], float]:\n",
    "\n",
    "    def train_step_fn(x: torch.tensor) -> float:\n",
    "        model.train()\n",
    "        logits, label_ids = model(x)\n",
    "        loss = loss_fn(logits.view(-1, logits.shape[-1]), label_ids.reshape(-1))\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        scheduler.step()\n",
    "        optimizer.zero_grad()\n",
    "        return loss.item()\n",
    "\n",
    "    return train_step_fn\n",
    "\n",
    "\n",
    "def get_val_step_fn(\n",
    "    model: torch.nn.Module, loss_fn: torch.nn.CrossEntropyLoss\n",
    ") -> Callable[[torch.tensor], float]:\n",
    "\n",
    "    def val_step_fn(x: torch.tensor) -> float:\n",
    "        model.eval()\n",
    "        logits = model(x)\n",
    "        logits, label_ids = model(x)\n",
    "        loss = loss_fn(logits.view(-1, logits.shape[-1]), label_ids.reshape(-1))\n",
    "        return loss.item()\n",
    "\n",
    "    return val_step_fn\n",
    "\n",
    "\n",
    "def mini_batch(\n",
    "    dataloader: DataLoader, step_fn: Callable[[torch.tensor], float], is_training: bool = True\n",
    ") -> tuple[np.array, list[float]]:\n",
    "\n",
    "    mini_batch_losses = []\n",
    "\n",
    "    if is_training:\n",
    "        print(\"\\nTraining ...\")\n",
    "    else:\n",
    "        print(\"\\nValidating ...\")\n",
    "    n_steps = len(dataloader)\n",
    "    for i, data in enumerate(dataloader):\n",
    "        loss = step_fn(data)\n",
    "        mini_batch_losses.append(loss)\n",
    "        if i % (batch_size * 50) == 0:\n",
    "            print(f\"step {i:>5}/{n_steps}, loss = {loss: .3f}\")\n",
    "\n",
    "    return np.mean(mini_batch_losses), mini_batch_losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 1\n",
      "\n",
      "Training ...\n",
      "step     0/4363, loss =  9.710\n",
      "step   800/4363, loss =  3.500\n",
      "step  1600/4363, loss =  2.766\n",
      "step  2400/4363, loss =  2.966\n",
      "step  3200/4363, loss =  2.297\n",
      "step  4000/4363, loss =  2.226\n",
      "\n",
      "Validating ...\n",
      "step     0/485, loss =  2.409\n",
      "\n",
      "Epoch 2\n",
      "\n",
      "Training ...\n",
      "step     0/4363, loss =  2.402\n",
      "step   800/4363, loss =  2.225\n",
      "step  1600/4363, loss =  2.484\n",
      "step  2400/4363, loss =  2.481\n",
      "step  3200/4363, loss =  2.845\n",
      "step  4000/4363, loss =  2.321\n",
      "\n",
      "Validating ...\n",
      "step     0/485, loss =  2.409\n",
      "\n",
      "Epoch 3\n",
      "\n",
      "Training ...\n",
      "step     0/4363, loss =  2.360\n",
      "step   800/4363, loss =  2.684\n",
      "step  1600/4363, loss =  2.410\n",
      "step  2400/4363, loss =  2.713\n",
      "step  3200/4363, loss =  2.580\n",
      "step  4000/4363, loss =  2.776\n",
      "\n",
      "Validating ...\n",
      "step     0/485, loss =  2.409\n",
      "CPU times: user 24min 16s, sys: 6min 4s, total: 30min 20s\n",
      "Wall time: 30min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "n_epochs = 3\n",
    "\n",
    "train_step_fn = get_train_step_fn(model, optimizer, scheduler, loss_fn)\n",
    "val_step_fn = get_val_step_fn(model, loss_fn)\n",
    "\n",
    "train_losses, train_mini_batch_losses = [], []\n",
    "val_losses, val_mini_batch_losses = [], []\n",
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    print(f\"\\nEpoch {epoch}\")\n",
    "    train_loss, _train_mini_batch_losses = mini_batch(train_dataloader, train_step_fn)\n",
    "    train_mini_batch_losses += _train_mini_batch_losses\n",
    "    train_losses.append(train_loss)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        val_loss, _val_mini_batch_losses = mini_batch(val_dataloader, val_step_fn, is_training=False)\n",
    "        val_mini_batch_losses += _val_mini_batch_losses\n",
    "        val_losses.append(val_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally we look at losses over multiple epochs, but here we have only 3 epochs. One way to look at the mini batch losses is to use a running mean(smoothing) to reduce noise from per batch loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "window_size = 32\n",
    "\n",
    "train_mb_running_loss = []\n",
    "for i in range(len(train_mini_batch_losses)-window_size):\n",
    "    train_mb_running_loss.append(np.mean(train_mini_batch_losses[i:i+window_size]))\n",
    "\n",
    "val_mb_running_loss = []\n",
    "for i in range(len(val_mini_batch_losses)-window_size):\n",
    "    val_mb_running_loss.append(np.mean(val_mini_batch_losses[i:i+window_size]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAygAAAI/CAYAAACcUP1mAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABtQklEQVR4nO3dd3hUZd7G8fvMTHpCB+mGotI7CNKkKM1eseyKrt217qLB3mV37a7ltfeuqCuIiKKAIL0XqZEOoaX3Oe8fM3Myk8wkAROmfT/XxcXMmTOTZ3ImybnP7ymGaZoCAAAAgFBgC3YDAAAAAMCDgAIAAAAgZBBQAAAAAIQMAgoAAACAkEFAAQAAABAyCCgAAAAAQoajNl60UaNGZmpqam28NAAAAIAIsGTJkv2maTYuv71WAkpqaqoWL15cGy8NAAAAIAIYhvGHv+108QIAAAAQMggoAAAAAEIGAQUAAABAyKiVMSgAAADAkSouLtaOHTtUUFAQ7KagBsXHx6tly5aKiYmp1v4EFAAAAISEHTt2KCUlRampqTIMI9jNQQ0wTVMHDhzQjh071KZNm2o9hy5eAAAACAkFBQVq2LAh4SSCGIahhg0bHlFVjIACAACAkEE4iTxHekwJKAAAAICkw4cP66WXXjqq544dO1aHDx+udJ/7779fM2fOPKrXLy81NVX79++vkdcKNQQUAAAAQJUHlNLS0kqfO23aNNWrV6/SfR5++GGNHDnyaJsXNQgoAAAAgKS0tDRt3rxZPXr00MSJE/Xzzz9r2LBhuvTSS9W1a1dJ0jnnnKPevXurc+fOevXVV63neioa6enp6tixo6655hp17txZp59+uvLz8yVJEyZM0Oeff27t/8ADD6hXr17q2rWr1q9fL0nKyMjQaaedpl69eum6667T8ccfX2Wl5Omnn1aXLl3UpUsXPfvss5Kk3NxcjRs3Tt27d1eXLl30ySefWO+xU6dO6tatm/75z39aX/P8889X37591bdvX/3666+SpF9++UU9evRQjx491LNnT2VnZ9fQd7pyzOIFAAAASJo8ebJWr16t5cuXS5J+/vlnLVy4UKtXr7ZmoHrzzTfVoEED5efnq2/fvjr//PPVsGFDn9fZuHGjPvroI7322mu66KKL9MUXX+jyyy+v8PUaNWqkpUuX6qWXXtKTTz6p119/XQ899JCGDx+uSZMmafr06T4hyJ8lS5borbfe0oIFC2Sapk4++WQNHTpUW7ZsUfPmzTV16lRJUmZmpg4ePKgpU6Zo/fr1MgzD6pJ266236vbbb9egQYO0bds2jRo1SuvWrdOTTz6pF198UQMHDlROTo7i4+P/5He4eggoAAAACDkP/W+N1u7KqtHX7NS8jh44s/MRPadfv34+0+M+//zzmjJliiRp+/bt2rhxY4WA0qZNG/Xo0UOS1Lt3b6Wnp/t97fPOO8/a58svv5QkzZ0713r90aNHq379+pW2b+7cuTr33HOVlJRkveacOXM0evRo/fOf/9Rdd92lM844Q4MHD1ZJSYni4+N19dVXa9y4cTrjjDMkSTNnztTatWut18zKylJ2drYGDhyoO+64Q5dddpnOO+88tWzZsjrfsj+NLl4AAABAAJ4Tf8lVUZk5c6bmz5+vFStWqGfPnn6nz42Li7Nu2+12lZSU+H1tz37e+5imeUTtC7T/iSeeqCVLlqhr166aNGmSHn74YTkcDi1cuFDnn3++vvrqK40ePVqS5HQ6NX/+fC1fvlzLly/Xzp07lZKSorS0NL3++uvKz89X//79rW5otY0KCgAAAELOkVY6akJKSkql4ywyMzNVv359JSYmav369frtt99qvA2DBg3Sp59+qrvuukszZszQoUOHKt1/yJAhmjBhgtLS0mSapqZMmaL33ntPu3btUoMGDXT55ZcrOTlZb7/9tnJycpSXl6exY8eqf//+at++vSTp9NNP13//+19NnDhRkrR8+XL16NFDmzdvVteuXdW1a1fNnz9f69evV4cOHWr8PZdXrYBiGMatkq6RZEh6zTTNZ2uzUQAAAMCx1rBhQw0cOFBdunTRmDFjNG7cOJ/HR48erVdeeUXdunXTSSedpP79+9d4Gx544AFdcskl+uSTTzR06FA1a9ZMKSkpAffv1auXJkyYoH79+kmSrr76avXs2VPff/+9Jk6cKJvNppiYGL388svKzs7W2WefrYKCApmmqWeeeUaSq9vaTTfdpG7duqmkpERDhgzRK6+8omeffVazZs2S3W5Xp06dNGbMmBp/v/4YVZWRDMPoIuljSf0kFUmaLukG0zQ3BnpOnz59zMWLF9dkOwEAABDh1q1bp44dOwa7GUFVWFgou90uh8Oh+fPn64YbbrAG7Yczf8fWMIwlpmn2Kb9vdSooHSX9ZppmnvuFfpF0rqR/10BbAQAAALht27ZNF110kZxOp2JjY/Xaa68Fu0nHXHUCympJjxmG0VBSvqSxkiiPAAAAADXshBNO0LJly4LdjKCqMqCYprnOMIx/SfpBUo6kFZIqTEVgGMa1kq6VpNatW9dwMwEAAABEg2pNM2ya5humafYyTXOIpIOSKow/MU3zVdM0+5im2adx48Y13U4AAAAAUaC6s3g1MU1zn2EYrSWdJ2lA7TYLAAAAQDSq7jooX7jHoBRLusk0zconZAYAAACAo1DdLl6DTdPsZJpmd9M0f6ztRtWUh/63Rle+tTDYzQAAAECESk5OliTt2rVLF1xwgd99Tj31VFW1BMezzz6rvLw86/7YsWN1+PDhP92+Bx98UE8++eSffp1jqVoBJVztyy7UHwfzqt4RAAAA+BOaN2+uzz///KifXz6gTJs2TfXq1auBloWfiA4oDpshp7PyhSgBAAAASbrrrrv00ksvWfcffPBBPfXUU8rJydGIESPUq1cvde3aVV9//XWF56anp6tLly6SpPz8fI0fP17dunXTxRdfrPz8fGu/G264QX369FHnzp31wAMPSHKt5L5r1y4NGzZMw4YNkySlpqZq//79kqSnn35aXbp0UZcuXfTss89aX69jx4665ppr1LlzZ51++uk+X8ef5cuXq3///urWrZvOPfdcHTp0yPr6nTp1Urdu3TR+/HhJ0i+//KIePXqoR48e6tmzp7KzsyVJ//nPf9S3b19169bNan9ubq7GjRun7t27q0uXLvrkk0+O7BtfTnXHoIQlu2GohIACAACAahg/frxuu+023XjjjZKkTz/9VNOnT1d8fLymTJmiOnXqaP/+/erfv7/OOussGYbh93VefvllJSYmauXKlVq5cqV69eplPfbYY4+pQYMGKi0t1YgRI7Ry5UrdcsstevrppzVr1iw1atTI57WWLFmit956SwsWLJBpmjr55JM1dOhQ1a9fXxs3btRHH32k1157TRdddJG++OILXX755QHf31//+le98MILGjp0qO6//3499NBDevbZZzV58mRt3bpVcXFxVreyJ598Ui+++KIGDhyonJwcxcfHa8aMGdq4caMWLlwo0zR11llnafbs2crIyFDz5s01depUSVJmZuafOQwRHlBshkoJKAAAAOFn+nRpz56afc2mTaXRowM+3LNnT+3bt0+7du1SRkaG6tevr9atW6u4uFh33323Zs+eLZvNpp07d2rv3r1q2rSp39eZPXu2brnlFklSt27d1K1bN+uxTz/9VK+++qpKSkq0e/durV271ufx8ubOnatzzz1XSUlJkqTzzjtPc+bM0VlnnaU2bdqoR48ekqTevXsrPT094OtkZmbq8OHDGjp0qCTpiiuu0IUXXmi18bLLLtM555yjc845R5I0cOBA3XHHHbrssst03nnnqWXLlpoxY4ZmzJihnj17SpJycnK0ceNGDR48WP/85z9111136YwzztDgwYMDtqM6IjqgOOwEFAAAAFTfBRdcoM8//1x79uyxujt98MEHysjI0JIlSxQTE6PU1FQVFBRU+jr+qitbt27Vk08+qUWLFql+/fqaMGFCla9jmoHPZePi4qzbdru9yi5egUydOlWzZ8/WN998o0ceeURr1qxRWlqaxo0bp2nTpql///6aOXOmTNPUpEmTdN1111V4jSVLlmjatGmaNGmSTj/9dN1///1H1RYpwgOKzSCgAAAAhKVKKh21afz48brmmmu0f/9+/fLLL5Jc1YcmTZooJiZGs2bN0h9//FHpawwZMkQffPCBhg0bptWrV2vlypWSpKysLCUlJalu3brau3evvvvuO5166qmSpJSUFGVnZ1fo4jVkyBBNmDBBaWlpMk1TU6ZM0XvvvXfE76tu3bqqX7++5syZo8GDB+u9997T0KFD5XQ6tX37dg0bNkyDBg3Shx9+qJycHB04cEBdu3ZV165dNX/+fK1fv16jRo3Sfffdp8suu0zJycnauXOnYmJiVFJSogYNGujyyy9XcnKy3n777SNun7eIDigOG2NQAAAAUH2dO3dWdna2WrRooWbNmkmSLrvsMp155pnq06ePevTooQ4dOlT6GjfccIOuvPJKdevWTT169FC/fv0kSd27d1fPnj3VuXNntW3bVgMHDrSec+2112rMmDFq1qyZZs2aZW3v1auXJkyYYL3G1VdfrZ49e1banSuQd955R9dff73y8vLUtm1bvfXWWyotLdXll1+uzMxMmaap22+/XfXq1dN9992nWbNmyW63q1OnThozZozi4uK0bt06DRjgWrM9OTlZ77//vjZt2qSJEyfKZrMpJiZGL7/88hG3zZtRWdnoaPXp08esaq7nY+Hh/63VZ4u3a9VDo4LdFAAAAFRh3bp16tixY7CbgVrg79gahrHENM0+5feN6GmG7TZRQQEAAADCSIQHFJtKa6FCBAAAAKB2RHRAcTDNMAAAABBWIjqg2NwBpTbG2QAAAKDmcd4WeY70mEZ0QHHYXPNPU0QBAAAIffHx8Tpw4AAhJYKYpqkDBw4oPj6+2s+J6GmG7e6AUuJ0ym6zB7k1AAAAqEzLli21Y8cOZWRkBLspqEHx8fFq2bJltfePioDCOBQAAIDQFxMTozZt2gS7GQiyqOjiRUABAAAAwkNEBxQqKAAAAEB4iYqAwmKNAAAAQHiIioDiJKAAAAAAYSGiA4qDCgoAAAAQViI6oNgMxqAAAAAA4SSiA4rDTkABAAAAwklEBxS7zfX26OIFAAAAhIfIDih08QIAAADCSmQHFNZBAQAAAMIKAQUAAABAyIjogFI2zbAzyC0BAAAAUB0RHVCshRpNKigAAABAOIiKgFJSSkABAAAAwkFUBBTGoAAAAADhIaIDStkYFAIKAAAAEA4iOqDYPBUUxqAAAAAAYSGiA4qnguKkggIAAACEhYgOKDaDLl4AAABAOInogOKwU0EBAAAAwklkBxQGyQMAAABhJaIDiqeLF9MMAwAAAOEhogOKw+Z6ewQUAAAAIDxEdEBx5xMCCgAAABAmIjqgWBUU1kEBAAAAwkJEBxRPBYVB8gAAAEB4iOiAYlVQSp1BbgkAAACA6ojogGJ3TzNcSgEFAAAACAvREVCcVFAAAACAcBDRAcVhBZQgNwQAAABAtUR0QClbqJGEAgAAAISDiA4oVFAAAACA8BLRAcXGGBQAAAAgrER0QJFcVRTWQQEAAADCQ8QHFLvNYCV5AAAAIExER0BhIRQAAAAgLERHQKGCAgAAAISF6AgojEEBAAAAwkLEBxQHAQUAAAAIGxEfUGwGAQUAAAAIFxEfUJhmGAAAAAgfER9Q7HZDTgIKAAAAEBYiP6AYVFAAAACAcBH5AYVphgEAAICwER0BhYUaAQAAgLAQBQHFRgUFAAAACBNREFCkklJnsJsBAAAAoBqiIKDYRA8vAAAAIDxEfEBx2JhmGAAAAAgXER9QXNMM08ULAAAACAeRH1BshsgnAAAAQHioVkAxDON2wzDWGIax2jCMjwzDiK/thtUUu40KCgAAABAuqgwohmG0kHSLpD6maXaRZJc0vrYbVlPsNkOljEEBAAAAwkJ1u3g5JCUYhuGQlChpV+01qWY5WEkeAAAACBtVBhTTNHdKelLSNkm7JWWapjmjthtWU2w2QyXMMwwAAACEhep08aov6WxJbSQ1l5RkGMblfva71jCMxYZhLM7IyKj5lh4lh82QkwoKAAAAEBaq08VrpKStpmlmmKZZLOlLSaeU38k0zVdN0+xjmmafxo0b13Q7j5rNZqiEMSgAAABAWKhOQNkmqb9hGImGYRiSRkhaV7vNqjkOBskDAAAAYaM6Y1AWSPpc0lJJq9zPebWW21VjmMULAAAACB+O6uxkmuYDkh6o5bbUCrtBQAEAAADCRcSvJO+wE1AAAACAcBHxAcVGBQUAAAAIGxEfUBzM4gUAAACEjYgPKHabTU4CCgAAABAWoiCgiAoKAAAAECaiIKDYVMpK8gAAAEBYiIKAIgbJAwAAAGEiCgKKTaVOUyZVFAAAACDkRXxAcdgMSRJFFAAAACD0RXxAsbsDSonTGeSWAAAAAKhK1AQU8gkAAAAQ+iI/oBhUUAAAAIBwEfkBxV1BYSYvAAAAIPRFfEBx2AkoAAAAQLiI+IBiMwgoAAAAQLiI+IDimWaY1eQBAACA0BfxAcXmmWa4lIACAAAAhLqIDygOBskDAAAAYSPiA4qdLl4AAABA2IiegEIFBQAAAAh5ER9Q6OIFAAAAhI+IDyhMMwwAAACEj4gPKJ6FGksIKAAAAEDIi/iAYre53iIVFAAAACD0RX5AoYsXAAAAEDYiP6AwSB4AAAAIGwQUAAAAACEjagJKidMZ5JYAAAAAqErEBxTPOihOVpIHAAAAQl7EBxSrglJKQAEAAABCXdQEFCooAAAAQOiLmoDCQo0AAABA6IuagMIsXgAAAEDoi/iA4iCgAAAAAGEj4gOKzaCLFwAAABAuIj6gOOzuQfIEFAAAACDkRXxA8YxBKSagAAAAACEv4gOKw+Z6i6WlrCQPAAAAhLrIDyh2xqAAAAAA4SLiA0qMu4JSzEryAAAAQMiL/IDirqAU08ULAAAACHkRH1CsleQJKAAAAEDIi/iAYhiGYuwGs3gBAAAAYSDiA4rkmsmLCgoAAAAQ+qIjoNgNBskDAAAAYSAqAkqM3aYSJxUUAAAAINRFSUAxVFxCBQUAAAAIdVERUBw2m4qpoAAAAAAhLyoCSozdUAljUAAAAICQFxUBxcEYFAAAACAsREVAibHbVMQYFAAAACDkRUVAiXXYVMQ6KAAAAEDIi4qAEme3qaikNNjNAAAAAFCFqAgosQ6bikqooAAAAAChLnoCCl28AAAAgJAXHQHFTgUFAAAACAfREVDo4gUAAACEBQIKAAAAgJARPQGFMSgAAABAyIuOgGK3qZAKCgAAABDyoiKgxNHFCwAAAAgLURFQPF28TNMMdlMAAAAAVCI6AordJtOUSpwEFAAAACCURUdAcbjeJt28AAAAgNBGQAEAAAAQMqIroDDVMAAAABDSqgwohmGcZBjGcq9/WYZh3HYM2lZjYu1UUAAAAIBw4KhqB9M0f5fUQ5IMw7BL2ilpSu02q2Z5KiishQIAAACEtiPt4jVC0mbTNP+ojcbUljjGoAAAAABh4UgDynhJH9VGQ2oTY1AAAACA8FDtgGIYRqyksyR9FuDxaw3DWGwYxuKMjIyaal+NiLXbJVFBAQAAAELdkVRQxkhaaprmXn8Pmqb5qmmafUzT7NO4ceOaaV0NYZphAAAAIDwcSUC5RGHYvUvy7uJVGuSWAAAAAKhMtQKKYRiJkk6T9GXtNqd2MM0wAAAAEB6qnGZYkkzTzJPUsJbbUmuYZhgAAAAID1GxkjzTDAMAAADhISoCCtMMAwAAAOEhOgIKY1AAAACAsBAdAYUuXgAAAEBYIKAAAAAACBlREVAcNkOGwRgUAAAAINRFRUAxDEOxdhsVFAAAACDERUVAkVzdvFgHBQAAAAhtURNQ4hw2ungBAAAAIS5qAkpynEPZBSXBbgYAAACASkRNQKmXGKvDeUXBbgYAAACASkRNQKmbEKPM/OJgNwMAAABAJaImoNRLjFEWAQUAAAAIaVETUOrEU0EBAAAAQl3UBJTEOLvyikqD3QwAAAAAlYiagJIQY1dhiVNOpxnspgAAAAAIIKoCiiTlF1NFAQAAAEJV1ASUxFgCCgAAABDqoiagxHsqKIxDAQAAAEJW1ASUBHcFpYAKCgAAABCyoieguCsozOQFAAAAhK6oCyiMQQEAAABCV/QEFAbJAwAAACEv6gLK4byiILcEAAAAQCBRE1AcNtdb/XTRjiC3BAAAAEAgURNQ2jZKkiR1a1U3yC0BAAAAEEjUBBSbzVCj5Dhl5RcHuykAAAAAAoiagCJJjZJjtT+HMSgAAABAqIqqgFIvMUaZeVRQAAAAgFAVXQElIVaH86mgAAAAAKEqugJKYowOUUEBAAAAQlZUBZS67i5epmkGuykAAAAA/IiqgFI/MVZFpU5WkwcAAABCVFQFlHoJMZKkw3TzAgAAAEJSVAWUBkmxkqS9WQVBbgkAAAAAf6IqoLRumChJWrc7O8gtAQAAAOBPVAWUpFiHJOm5HzcEuSUAAAAA/ImqgNKyfoIkaVzX5kFuCQAAAAB/oiqgGIah+okxKixhFi8AAAAgFEVVQJGkugkxyiooCXYzAAAAAPgRfQElMVaZ+UwzDAAAAIQiR7AbcKyt2H442E0AAAAAEEDUVVBi7VH3lgEAAICwEXVn61cPbiOHzZBpmsFuCgAAAIByoi6gJMc7VOI0VVDsDHZTAAAAAJQTdQElJT5GkpRdyEB5AAAAINREX0CJc80LkM1UwwAAAEDIib6AEk9AAQAAAEJV1AWUhFi7JCl9f26QWwIAAACgvKgLKK3qJ0qSCktKg9wSAAAAAOVFXUCpl+gaJM9q8gAAAEDoibqAkuweJP/xou1BbgkAAACA8qIuoBiGIUnaksEYFAAAACDURF1AAQAAABC6ojKgnNbpuGA3AQAAAIAfURlQTjouRTZDMk0z2E0BAAAA4CUqA0pyvENOU8ovZqphAAAAIJREZUBJcs/klcNq8gAAAEBIicqAkuIJKIUEFAAAACCURGVAqZPgCiiH8lisEQAAAAglURlQ6ibESpKyCwgoAAAAQCiJyoASH+N62wUMkgcAAABCSlQGlIQYuySpoNgZ5JYAAAAA8BadASXWE1CooAAAAAChJCoDSrzDFVBW7MgMcksAAAAAeIvKgOKpoHy0cFuQWwIAAADAW7UCimEY9QzD+NwwjPWGYawzDGNAbTesNsW7x6CM7NgkyC0BAAAA4M1Rzf2ekzTdNM0LDMOIlZRYi206Jjo2qxPsJgAAAAAop8qAYhhGHUlDJE2QJNM0iyQV1W6zal9ynJ2V5AEAAIAQU50uXm0lZUh6yzCMZYZhvG4YRlItt6vWJcU5lFvILF4AAABAKKlOQHFI6iXpZdM0e0rKlZRWfifDMK41DGOxYRiLMzIyariZNc8VUKigAAAAAKGkOgFlh6QdpmkucN//XK7A4sM0zVdN0+xjmmafxo0b12Qba0VKnIMuXgAAAECIqTKgmKa5R9J2wzBOcm8aIWltrbbqGEgioAAAAAAhp7qzeN0s6QP3DF5bJF1Ze006NhJi7CoscQa7GQAAAAC8VCugmKa5XFKf2m3KsRXnsKnUaaqk1CmHPSrXqwQAAABCTtSemecWuWbwyiqgmxcAAAAQKqI2oKTEu4pHuw7nB7klAAAAADyiNqB0bVFXkpRfzFooAAAAQKiI2oCSGGuXJOUVEVAAAACAUBHFAcXVxSu/iDEoAAAAQKiI2oCS4K6grNyRGeSWAAAAAPCI2oByXJ04SdLmjJwgtwQAAACAR9QGlMRYh1LiHWpWNyHYTQEAAADgFrUBRZJS4hzKZh0UAAAAIGREd0CJj1FOYXGwmwEAAADALaoDSlGpU9+v2RvsZgAAAABwi+qAsnV/brCbAAAAAMBLVAeU/m0bSJI27s0OcksAAAAASFEeUC7p11qSdCC3KMgtAQAAACBFeUBp0yhJkpTDTF4AAABASIjqgJIc55Ak5RYRUAAAAIBQEN0BJd4VUFgLBQAAAAgN0R1Q3BWUnEICCgAAABAKojqgJMTYJUmrdmQGuSUAAAAApCgPKIZhSJKmrtod5JYAAAAAkKI8oAAAAAAILVEfUC7o3VLN68YHuxkAAAAAREBRSrxDWcziBQAAAIQEAkp8jHIKS1TqNIPdFAAAACDqEVBYrBEAAAAIGVEfUJI8AYW1UAAAAICgI6DEudZCIaAAAAAAwRf1ASXZqqCUBrklAAAAAKI+oNDFCwAAAAgdUR9QPBWUbAIKAAAAEHQEFCooAAAAQMggoMS7AkoOAQUAAAAIOgJKHAEFAAAACBVRH1DiHDY5bIZyCggoAAAAQLBFfUAxDEOJsXblFTHNMAAAABBsUR9QJNdUwwySBwAAAIKPgCJ3QCkioAAAAADBRkCRlBRrZyV5AAAAIAQQUCQlxjqURwUFAAAACDoCijxjUKigAAAAAMFGQJGUEu9QdmFxsJsBAAAARD0CiqQ68Q5lsw4KAAAAEHQEFEl1EmKUlV8s0zSD3RQAAAAgqhFQJNWJj5HTlHJZrBEAAAAIKgKKpDoJDklSZj7jUAAAAIBgIqDIVUGRpCwCCgAAABBUBBS5xqBIBBQAAAAg2Ago8qqgMJMXAAAAEFQEFEl1qaAAAAAAIYGAorJB8lkFBBQAAAAgmAgokpLjmMULAAAACAUEFEkOu03JcQ5l5TMGBQAAAAgmAopbSrxD2XTxAgAAAIKKgOJWJz6GMSgAAABAkBFQ3Ook0MULAAAACDYCiltKfIyyC6mgAAAAAMFEQHGrE08FBQAAAAg2AopbnQTGoAAAAADBRkBxc83iVSLTNIPdFAAAACBqEVDc6sTHqNRpKq+oNNhNAQAAAKIWAcUtJT5GkujmBQAAAAQRAcWtToJDkpRdwEB5AAAAIFgIKG51PBWUfCooAAAAQLAQUNxS4l0VFLp4AQAAAMFDQHFLinMFFAbJAwAAAMFDQHFLiLFLkvIJKAAAAEDQOKqzk2EY6ZKyJZVKKjFNs09tNioY4mJcWa2gxBnklgAAAADRq1oBxW2YaZr7a60lQeapoKzZmRnklgAAAADRiy5ebomxrqy2YW92kFsCAAAARK/qBhRT0gzDMJYYhnFtbTYoWOw2Qw2TYtWxWZ1gNwUAAACIWtXt4jXQNM1dhmE0kfSDYRjrTdOc7b2DO7hcK0mtW7eu4WYeG0lxDmbxAgAAAIKoWhUU0zR3uf/fJ2mKpH5+9nnVNM0+pmn2ady4cc228hhJjLUrr4iV5AEAAIBgqTKgGIaRZBhGiue2pNMlra7thgWDK6BQQQEAAACCpTpdvI6TNMUwDM/+H5qmOb1WWxUkSXEO5RZSQQEAAACCpcqAYprmFkndj0Fbgi4x1q6M7MJgNwMAAACIWkwz7CUp1qFcxqAAAAAAQUNA8ZIQa1c+Y1AAAACAoCGgeHGNQSGgAAAAAMFCQPGSGGtXfnGpSp1msJsCAAAARCUCipekWNecAfnFVFEAAACAYCCgeEmItUuS8phqGAAAAAgKAoqXpDhXQMlloDwAAAAQFAQUL54uXizWCAAAAAQHAcVLUhwBBQAAAAgmAooXT0DJo4sXAAAAEBQEFC/J7jEo2VRQAAAAgKAgoHhJiY+RJGUXFAe5JQAAAEB0IqB4qeMOKJn5BBQAAAAgGAgoXuJjbLLbDAbJAwAAAEFCQPFiGIbiHTYVFDuD3RQAAAAgKhFQyomPsaugmFm8AAAAgGAgoJQTRwUFAAAACBoCSjnxMXYVlFBBAQAAAIKBgFJOXIxdhXTxAgAAAILCEewGhJp1u7O0bndWsJsBAAAARCUqKAAAAABCBgElgEO5RcFuAgAAABB1CCjl3DW6gyQpI6cwyC0BAAAAog8BpZzOzetIkrLyi4PcEgAAACD6EFDKqZsQI0nKJKAAAAAAxxwBpRxPQDmcR0ABAAAAjjUCSjlUUAAAAIDgIaCUU4eAAgAAAAQNAaUcu81QSryDgAIAAAAEAQHFj+yCEr332x/BbgYAAAAQdQgoAZQ6TZU6zWA3AwAAAIgqBJRK5BWVBLsJAAAAQFQhoPhxWqfjJEnTVu0OcksAAACA6EJA8WPOxgxJ0l1frApySwAAAIDoQkDx44oBqZKkri3qBrchAAAAQJQhoPjxz1EnSZK6EFAAAACAY4qA4keM3aa2jZKUU8ggeQAAAOBYIqAEkBTnUE4BizUCAAAAxxIBJYDkOAcVFAAAAOAYcwS7AaFq/pYDkqT8olIlxNqD3BoAAAAgOlBBqcLzP20MdhMAAACAqEFAqcKHC7YFuwkAAABA1CCgBPCP006UJF3Up2WQWwIAAABEDwJKAH8f3l6SNGXZriC3BAAAAIgeBJQADMOQJO3PKQxySwAAAIDoQUABAAAAEDIIKJVo1zhJklRU4gxySwAAAIDoQECpRN/UBpKkA7l08wIAAACOBQJKJfq1cQUUKigAAADAsUFAqURuYYkk6f3f/ghySwAAAIDoQECpxL5sV9eu1+ZsldNpBrk1AAAAQOQjoFTi1hEnWLfnbtofxJYAAAAA0YGAUgmHvezbQ0ABAAAAah8BpQoPn91ZkvTVsp1BbgkAAAAQ+QgoVRjbtZkkaeiJjYPcEgAAACDyEVCq0DApVoYhNa0bH+ymAAAAABGPgFIFwzBkmtILP21ScSnroQAAAAC1iYByBBalHwx2EwAAAICIRkA5AoXFVFAAAACA2kRAOQJZBcXBbgIAAAAQ0Qgo1fDWlX0lSYdyi4LcEgAAACCyEVCqYcgJjWUY0sE8KigAAABAbSKgVIPdZqhuQgwVFAAAAKCWEVCqKSXeoZzCkmA3AwAAAIhoBJRqSop1aMqynfp9T3awmwIAAABELAJKNa13B5NRz84OcksAAACAyFXtgGIYht0wjGWGYXxbmw0CAAAAEL2OpIJyq6R1tdWQUHftkLaSpMEnNApySwAAAIDIVa2AYhhGS0njJL1eu80JXXeP7ahTT2qszHymGgYAAABqS3UrKM9KulOSs/aaEvrshqGVOzJVVBLV3wYAAACg1lQZUAzDOEPSPtM0l1Sx37WGYSw2DGNxRkZGjTUwlPy4fp8k6YFv1gS5JQAAAEBkqk4FZaCkswzDSJf0saThhmG8X34n0zRfNU2zj2mafRo3blzDzQwtHy3cFuwmAAAAABGpyoBimuYk0zRbmqaZKmm8pJ9M07y81lsWgv5zQbdgNwEAAACIaKyDcgTO79Uy2E0AAAAAItoRBRTTNH82TfOM2mpMqLPZDJ3W6ThJktNpBrk1AAAAQOShgnKEfli7V5LU9u5phBQAAACghhFQjpB3N6+Xf9kcxJYAAAAAkYeAcoTGdm1q3f5+zZ4gtgQAAACIPASUI9Tn+Abq1rKuJOnUk5oEuTUAAABAZCGgHKG6iTH65u+DJEnP/7gxyK0BAAAAIgsBBQAAAEDIIKD8SS/8uFEPfL062M0AAAAAIgIB5SjdcGo7SdJTP2zQO/P/YMphAAAAoAYQUI5SvMPucz+7sCRILQEAAAAiBwHlKJWavhWTQ7lFQWoJAAAAEDkIKEepQWKMz/3ZGzOC1BIAAAAgchBQjtLFfVv73L//6zVavTNTqWlT9eA3a4LUKgAAACC8EVCOUkKsXW9c0cdn27+mr5ckvT0vPQgtAgAAAMIfAeVPGNHxOKVPHmfdn7Nxv3V7S0ZOMJoEAAAAhDUCSg14bnyPCtsO5TFoHgAAADhSBJQakBznqLAtu4BphwEAAIAjRUCpAcNOalJh25u/ph/7hgAAAABhjoBSA2w2Qx9f21+S9NaEvpKkto2SgtkkAAAAICxV7JuEo9K/bUNrwHxSrF12mxHkFgEAAADhhwpKLUiOdyiHMSgAAADAESOg1ILkOIdyCgkoAAAAwJEioNSCOgkxyswvDnYzAAAAgLBDQKkFDZNidSCXdVAAAACAI0VAqQV1EmK0bneWTNMMdlMAAACAsEJAqQVfLt0pSWozaZp6PfJDkFsDAAAAhA8CSi1o3SDRun2Qrl4AAABAtRFQasG7V/ULdhMAAACAsERAqQWp5VaRLy51aufh/CC1BgAAAAgfBJRj4O8fLtXAyT/R3QsAAACoAgHlGPh+zV5J0uE8AgoAAABQGQJKLZlz57AK29bsygpCSwAAAIDwQUCpJa0aJOrM7s19tk38fEWQWgMAAACEBwJKLbrs5NY+909u0zBILQEAAADCAwGlFvVv21DL7jtNXVrUkST9siEjyC0CAAAAQhsBpZbVT4rVtzcPtu4XlTiD2BoAAAAgtBFQjpFrh7SVJGUXFAe5JQAAAEDoIqAcIx2bpUiSDgRYC2VPZgHVFQAAAEQ9AsoxUlxqSpJe+GlThccKikvV/4kfdc+UVce6WQAAAEBIIaAcI2O6NJUkNUyK1aQvV2n66t3WY9sO5kmSPluyIyhtAwAAAEKFI9gNiBYp8TFqnBKnt+elS5I+WrhN6ZPHSZJ2ZxYEsWUAAABA6KCCcgw1So7zuT/r933KKijW4byycSlOp3msmwUAAACEDCoox9C63Vk+9698a5EkqVFyrLXt5w37NLzDcce0XQAAAECooIISBO2bJPvc359TVkEpKaWCAgAAgOhFQDmGrhhwvCRpXNdmAff5mdXmAQAAEMUIKMfQQ2d3UfrkcbpyYGrAfT5csO3YNQgAAAAIMQSUIKiXGKtPru2v4+qUDZpf9/DoILYIAAAACA0ElCA5uW1DLbh7pHU/IdZu3S4pZUV5AAAARCcCSggY1L6Rz/30A7lBagkAAAAQXASUINv42Bi9c1U/SdJTF3aXJI18enYwmwQAAAAEDQElyGLsNtlthiSpsMS3a9eezAIdyCkMRrMAAACAoGChxhAyoF1Dn/v9n/hRkrT1ibEyDCMYTQIAAACOKSooIaRNoyRd2LulJCmnsMTann4gL1hNAgAAAI4pAkqIqZMQI0mat2m/tW3Ykz8HqTUAAADAsUVACTHjurlWmb/2vSU+27MLipWaNlWpaVNV6jSD0TQAAACg1hFQQkxddwWlvK4PzrBuvzRrk89jS7cdUmraVM1av69W2wYAAADUNgJKiGnXONnnfnJcxXkMnvphg8/9816aJ0m68u1FtdcwAAAA4BhgFq8QF+uwSX5mGr7uvcVavTNLfx1wvM920zSZ8QsAAABhiwpKCErxqprMSxvud5/v1+zVzsP5euK79T7btx/Mr7BvbmGJ/v7hUu3OrPgYAAAAEEoIKCFo2q2DJUk3D2+v+Bi7z2OtGiT4fU7bRkmSpEN5Rdq0L0epaVO1zT098f/N3qJvV+7W49PW+30uAAAAECoIKCGoVYNEpU8ep3+cfpIk6YsbBkiSxnRpqt2HC/w+518XdJMkZRUU68wX5kqSznnpV0nS8z9ulCTlea2tAgAAAIQiAkoY6NW6viaN6aDHz+2qWf881dp+0nEpkqThHZpYg+lzCkqUX1xq3fbmNJmeGAAAAKGNQfJhwDAMXTe0nSSpflKs3riijz5auE2v/bWPNSB++0FXd65srypJUanT53Va1PffPQwAAAAIFQSUMDSi43Ea0fE4n21J7grKnZ+v9NmeVVBs3f7DPSYFAAAACFV08YoQgRZ47Oa1wOOcjfuPVXMAAACAo0JAiRB2m+/aJxf1ael3v79/uFSpaVP1wNerlVfEoHkAAACEFgJKhPr3Bd39bv925W5J0jvz/9DLP28+lk0CAAAAqlRlQDEMI94wjIWGYawwDGONYRgPHYuG4ci9cUWfgI99dE3/Ctte+GmTMvOKfbat2pGpjvdN194s/9MZAwAAALWpOhWUQknDTdPsLqmHpNGGYVQ820XQnXpSE0lSj1b1KjzWN7W+3+f0fGSGz/3bPlmm/OJSzVq/r8bbBwAAAFSlylm8TNM0JeW478a4/7GgRgiy2wxNv22wWtRzTSfcqVkdrd2dJUly2P1nUacpmaapV2dv0RPfla00v3Ffjt/9AQAAgNpUrTEohmHYDcNYLmmfpB9M01xQq63CUevQtI5S4l0zek29ZZCS4xy6ZcQJkqSxXZtKkj651rcAtnDrQZ9wIklvzN16DFoLAAAA+DLMI1hd3DCMepKmSLrZNM3V5R67VtK1ktS6devef/zxRw02EzWhuNSpzRk56tC0jiQpNW1qpfunTx5XYVtBcaniY+ySpLyiEh3MLVLL+ok131gAAABENMMwlpimWWEQ9RHN4mWa5mFJP0sa7eexV03T7GOaZp/GjRsfbTtRi2LsNiucSNLEUSdVun92ge8A+iemrVOH+6Zr5Y7DmrMxQ53u/16D/jVLB3IKa6W9AAAAiD7VmcWrsbtyIsMwEiSNlLS+0ichLNx4ajvFOnw/Av/3l966bkhbSdIXS3b4PjZ7iyTpri9W+Sz6uIcZvwAAAFBDqlNBaSZplmEYKyUtkmsMyre12ywcC4ZhqKjEad3f+sRYjercVENPdFXA8oudPvu3bZQkSRpyQiO9Oz/d2v7ot+tqv7EAAACICtWZxWulpJ7HoC0IIsNwBRZJ6u2ekri41BVQDuUWqX5SrGzu1eo9lRSPRekHj2FLAQAAEMlYST7KdW7uGpPy9EVlK8/HOVyD4F+ctUkn3vudej7yg5ZtO6TM/GK/r1HiNLV6Z2btNxYAAAARj4AS5b6+aaBevqyXzunRosJjhSVOqwvY+j3ZFQLKre7piyWx8jwAAABqBAElyjnsNo3p2szq3lWZohKnxnRpat1v3SDRWlPlb+8s1oGcQm3Ym61PFm2z9q+OXYfz9fcPlyq3sOQo3gEAAAAiSZVjUABJmvTlKklSy/oJ1rYGybFq0zjJut/70ZnW7byiUj30v7Ua17WZXrysl7V9X1aBmtSJlySd9d+5WrkjU+f3aqlvV+7WqSc10QW9W9b2WwEAAEAIo4ICv56/xP+8CGt2ZVm3GyXFqXFynN/9HvrfWknS1FW7rcH2j09bp36P/6hfN+3XvqwCrdzhGrcyc91eSdLB3LL1VEpKnbr/69VK35/7598MAAAAwgYBBX6d2a2ZJKlRuQDy5oS+1u3WDRJlGIbSJ49TSnzgYlzaF67qy9SVuyVJV769SPtziqzHPWNbHp+2Xu/MS5ckrdqZqXfn/6HbPln+p98LAAAAwgcBBX4ZhqGvbxqoabcOUnxM2cckPsaue8Z21K0jTlDdxBhr+9k9mgd8rS+W7lBq2lQNbN9QkmtsyoS3Fvrd94Fv1kiSth3MkyRlBZg5DAAAAJGJgIKAureqpyYp8frm74MkyaqSXDOkrW4/7USffe8d16nK1/t0cdnK9PEx9oD7lZQ6rS5iSXG+lZnCklJt3Jstp9Os3psAAABAWGGQPKp04nEpeubi7hp+0nEB94mPsWvOncP07vx09WhVX07T1NiuzdTu7ml+9/dUSI6rE6e9WYVq1zhJmzNc400+XrRdB3NdXcB+35Pt87x7pqzW50tcQWfjY2MUYydjAwAARBLO7lAt5/Zs6dOly59WDRJ1z7hOGtetmc7s3lx2m6FVD56ua4e09dmvRb2ymcDevrKfhp7YWP/3lz7Wtnu/Wm3dLip1qqC41LrvCSeSdNvHy4/27QAAACBEEVBQq1LiY3TFKanW/X+f3007D+db99s3SdY7V/VT+ybJmnvXML+vMfrZ2dbtNo3KpjWeump3zTcYAAAAQUVAQa1rUS9Bmx4bo98mjdBFfVv5PObdRat53QSfx07v5OpSln4gz9q2tdy0wxv3+nYBk6TF6Qf11bKdf7rdAAAAOPYIKDgmHHabmtZ1LdA4f9Jwv/vYbL6r2XdvVc+6XVTi1Po9WSrvtGdmV9h2wSvzddsny61xLNU1+N8/aexzc1RYUlr1zgAAAKgVBBQcc83qJuiz6wdo6i2DKjy2+qFR1u3rh7azbn+6eLu+cI8/sRlS1xZ1rcdGPztbqWlT9eXSHTqcVxZK/rdiV6Xt2JddoOXbDyuvqEQHc4u0/WC+1u7O0kn3Tj/q9wYAAIA/h1m8EBR9Uxv43Z4c59BDZ3VWZn6x7DZDb07oo6veXqx7v1qts7q71lp5bnxPvThrk/Wc9e6Zvu74dIXPay3545DP+JfyTn9mtg7nFatOvENZBSXVbvucjRm66u1FWnzPaVVOHAAAAIAjQwUFIeeKU1J1y4gTJEl14ssCwDfuisiZ3ZtboaQy36zYpemrywbSf7tyl/ZlF1j3D+e5FoEsH066tayryjw7c6OKS02/Xc4AAADw5xBQENJaNUj0u/2nfwyt1vOvf3+pUtOmKiO7UH//cJn6PfZjpfu3rJ8ge7mxMI9NXavPFm+37pe4F4k0DN/9ytt5OF8DJ//kM00yAAAAKkdAQUg7rk68Xr6sl3Xf4Q4PbRsna0DbhtV+nVs/XmbdNk1Ty7Yd8rtfz9b1Kwyuf23OVk38fKV1f8X2w5KkPVkFCuSjhds0cPJP2nk4Xw98vaba7QQAAIh2BBSEvBOOS7Zur3zwdOv2y5eXBRfvQfMX9WmpzY+P9XmNeZsPWLe/WbFL5740z+/Xapwcp/3Zhdb9klKndds0TZ/7Ow+VredS6jSVmjZVo9yzik36clVZ26roMlZdpU5Tny/Z4dOGo7H9YJ4yvN4jAABAKCGgIOS1aeQKKGf3aK7E2LJ5HeolxmrKjadIkoZ3aKInL+wuSfrH6SfJbjO0/P7T/L6eaVbcNv22wXr7yr5qnBKn3KJSHchxncDvzymrpmw7mOdz/0BOofKKSjRt1W6t2HFYkvT73myVOn2/wL1frdaGvdlKTZuqTPe4lyV/HNLF/zf/iKY0/nzJdv3zsxV6e166JKmguNRn0cvqGvzvWTr9mV8q3SevqER7K6kQHYnXZm/Rr5v218hrAQCAyMcsXgh5dpuh9Mnj/D7Ws3V9fX79APVoVU8Ou03n92phjQ2plxirpfedpl6P/ODznP05heraoq4O5BRq3qQR1vYOTetowdaDkqTej85U+uRx+uNA2cKQW/fnql5irHX/08XbdSC3SFOW7dRDZ3W2ts/3qtZ4nO6urHR/eIbSJ4/T+S+7KjjbD+apfZOUan0fVu3MlCSt3e0anP/Sz5v1/I8bdeXAVN1/Rqcqx8RIUmraVEnSIXdQCuSadxfr100HAn7fq8s0TT02bZ0kadl9p6l+UmwVzwAAANGOCgrCXp/UBnK4V6Qvf5Je3880wI9OXafdmQUa2L5RhceO9xqUf8I903Txq79Z9ye8tUhXv7PYup9VUKIp7hXrH/imbJzJ5W8s0PEN/Q/ul8pCgiS9NntrwP0+XrjNZ12XWLtdkvTl0p1KTZuq53/cKEl669d07ThUdSWl+Ai6hv26yRWy/uwA/w17c6zb/Z+ofIICAAAAiYCCCOcdWK4a2Ma6vT+nUG0bJ1fYf3y/1tbt4tKKfcH2u7t+1atk/ZPkOIdyC0t1Sb9Wun3kiZW27xOv2cG89Xn0B6V9uUrnvVw2Via7IHDVIyOn6jElP63f53P/3fnpVT7HXzcv0zR92pJXVKKNe/1P+zzq2dnW7cKSigFp3qb9mvzd+iPq6gYAACIbAQURb+HdIzT1lkG6e2wHnzVOAlU53riiT5Wv6d2lq7ycwhJl5hepTkKMLuvfOuB+HqlpU5WaNtWnwuEZ67Ilw9XFbPrq3fpsyY6Ar7E30zdIzFy7Vyt3HJbpNeDmuveW+Oxzf4DZxbyf46mkePtq+U51fXCGfv7dFXiufXeJTntmtu79apU1w1l1Xfr6Ar3yy2a94x5XAwAAQEBBxGtSJ16dm9eVw27TJ9cOsLa3DrDGyoiOx+m2kSdY9+8cfZLP42O7NtXpnZpWeN5doztYt4tLTSXFOtQwKVbJcdUb6nUor0hfL9/p0wXM4/r3l1b63Bs+cK33sjkjRwXFpbr63cU667+/WotbHs4rUmKsq4uYvwBWUFyq3o/8oOmr9yjfq1tXk5S4Cvve8ekKSdJTMzbINE3NdQ+Af/+3bTr7xV914weuIJRfVLEq4h3CvKsmv205WOn7AyLNjkN5wW4CAIQsAgqiSoL7JF2SUhslBdzv5uFlAeVvg9ro25sHaVD7Rtry+Fi9dFlvJcTalVIueIzo2ERjupQFl2mrdsswDBV5dW26elAbBfL41HW69ePlFbb7Cyweb13Z17cNT/2iDvdNt+7PWr9PszdkqMfDPyjPHRiGd2hS4XU63DddB3KLdP37S7Rud1l3rdyiEp/9lm07ZM2Ctmpnps/X8pi2ao9+XLdXB3Jd3c4mn9fVemxx+iF9s2KXDuYW6fU5ZeNvflq/jwUtq7BmV6bemLtVL/28yafKhfBz3XuLNehfszSrXLfL2jBnY4b2ZdfMjHxHIzO/2O/Fimix7UCernhzoTWDI4DqYRYvRJ2VD56uzLziSisbdpuhqbcM0tyN+xXnsKtLi7p6/+qTffb5eeKpyi4oUaOUODlNU3XiY1Qnvmxsimfa4yKvqsG9Z3TSP04/STsP52vk075T/X61fFeVbf9t0ggt+eOQbvpwqW4ZcYKGnVQxbJR/zfKv6z0uJzVtqtY/Mtrn8fO9xr14Tiw2Z+SofmJshTEp/saVSNLf3lmsfm0aSJIaJsfpyoGpeuvXdE3+bp1W7MjU4BMaac5G36mHO9w3/U/PGhbJxj0/17p9SrtG6tGqnnW/oLhUsXabbLaqZ3ILFZ7gXdkxzy0sUU5hiY6rE/+nvtaK7YfVIClWrQJUTWtC+v5cnfnfufr25kE6vmHgix+S9P2avZKk/63YpWF+LhjUFNM09Zc3FkpyTaXeoWmdWvtagXR/aIakyo9zJHtl9mb9siFD36zYqb8MSA12cyLO9NV7dP37S/TtzYPUpUXNrDl2LBzIKVRWQYnaVHKhNNpRQUHUqRMfU60Tlc7N6+q6oe0CPt4wOU6pjZKUHOewgknbxq5fNmljOlT4ZfndrYMluao47Zsk6/lLelb69Yec2LjCtqZ14zW2a1N9ePXJut3dDe0VrwUrqzK+bytJ0jWDyyo5L/+8OeD+aV+ukmmaGvHUL+r1yA9677c//O5346ntNHGUb1e4he4pm+slxujKU1xfb8UO11TJ3uFk8AkVZ1OLJJv25VT6+JpdmUfc3cd7+mvTNNXhvum67+vVR9W+YFseYNxSUYlTnR/4Xic//uOfvgJ/9ou/avC/Zx3Vc6ev3uPz9X/bcsAaf+Xt1Cd/VnZBiYb+52drm9O9gOuT3/9ubfOufn3pngXQ448DuUpNmxrwe+JtcfpBpaZNVU5hScB9vLtrjn52TpWv+Wet3HFYqWlTdfsnyys89n+/BP49U1sy84o18bMVf7riaJqmnM6je414h6tqvyj90J9qQ3mmaQacnOTPWL0zU6lpU49qjS2PvKIS/bR+bw22KrDr33d1KQ63cYy9H52pYU/+rLyiwD+/krQ7M9/vDJwPfrNGpz1d+Xpm4Y6AAtSgqwa10aPndNHf/HTlSor1rdic1b255t41TFuf8F31/n9/H6T3/tZP71zZV3UTyioyni5lhmHolPaNrErI6C7NdMWA431e456xHf22b/L53SRJ3VrWs7Y9556uOJAT7vnOuu0ZNO999b5van1NHHWSbhrWXt/ePKjC8zs0TVGTOhXHsni897eyytSNHyzRq7M3h0UXpvV7sjRnY4Z1/3BeUYVFOn9av1cjn/5F7wcIdpKrMjLoX1WfPLeol2DdvvXj5dp+0BVqPF33Pliw7Yjaf6yYplnh++I9/uifn63w+zzvk9z91ZilTpKem7lRn5abGa+yk7iVOw7r9z2BH1+/J0vXv79E90xZJclVJRn/6m+a8NYivfDjRi35o/KxU/O3uH5e/jtrk7UtP0BXxjW7Mq1wM+nLVZW+riRd8Mp8SVLPh2cE3Key8FIbfv7d9fPgmX7d+6T+ie/W1+jX2rQvRz+srfwkuPvDM/TZkh3Wz19GdqF2BTjxNk1XmLzjk+V6c67v9O9tJk3TSfd95/d55eUWlujS137T5gzXhYkC92f9mxW7fLr7/lk/rd+n056ZrfeqMRtjdRzIKVRRiVNnvOCq1A6c/NNRv1an+7/XVW8vtn5H1Rbvz1dlk8gE8vPv+7Rhb7be++0Pa7KaTftcvw/e/nWr+j42U6Zp6oo3F6rbg9/XWLu9fVFJu3MKSzTgiZ/04De+E9qUlDr19rx0bfRz8auoxKk7Pl2uKcv8v26Ph2eo/+PhMeU/AQWoQTF2my7vf7xi7GU/Wj/cPkRXDDherf3MGtayfmKFtVu6tqyrwSc0lmEY6tm6nrU9u5KTjYfO7qIf/zHUun/NkLYadlJZBSbWbvMJDw39LJj44JmdrOd0aFq2eGSJnyuHCTFlY3luG3mi9R66tKhbYWB9SnyM4r32r8y0VXv0+LT1Af84ZuYXH/Uf+Ts/X6F356ersKRUz/+4UdsP5imnsET/+X59pVM4+/PblgMa/ewcq/tMbmGJejz8gyZ/t87axzRNXfW2a92ce79araxyX+POz1f4XPVzOk2/weyMF+ZYVzTP6dHc2u6pCBzMLfLZv9Rpat4m3+5zHtNX79GoZ2br4f+tPSYhsNRpqs2kaWp39zSf7Ye9+uMHqjBNXbXbul1UxRo+xaVOXf3OIj0zc4Pu/HylJFc1oqjEqdOeme33OYUlpTrrv79q1LOzA45/Kih2fd21u7P0+pwtOvXJn63Hnvphg85/eb51v1WDsgC5/WCeXp+zRdO83sOGvdn6Ye1eTV1Ztk2SVQXxrrL0Ta0f8L0u23bIZ1yav+nQhz/1s276YKnyCo/N2I/f92TrwlfmqcTrOO3LKqjwO6vkCNZi2nEoT79tqTiLoMfIp3/RNe8uDvi49+e7YbLrd9LAf/2kU9y/WwqKS62T0sy8Yr3oDpFfLtuph79da1219oRQf99nyXVin5o2VbPcVbXOD3yveZsP6IKX5+nXTfv1odeFg71ZBSoqcSo1barGPjenWj+DTqdpXWX33v9v7jW57vOajdF73axASkqdOuOFOVaFY9O+HM36fZ96PzpTF/7ffJ99F7i//yWlzqMaIzilXIWwpmXmV/57O7ewRPM27dcniypevCl1mprw1iKd/sxs3fdVWfV55NOzNW/Tfj34v7XKyHZ1w/plQ4ayCkr0+pwtNdJu79/7B3IDH7MD7gszHyzYZgVeqawngqQKn6ET7/1OXy7dqds/WaEN5S7OrN6ZqcN5xdqTVXDUFcFjiYAC1LITjkvRQ2d3qXSfJfeO1DMXd9fUW3wrEM9e3KPaX6dd42S9c1U/a0zJvWd0sh6bc9cwny5n/lZ0nzCwjdV3fn0lV5Ul6S9eFZsTmviuJ/PZ9QPK7y7Jdx0aj5l3DPWzp7Qr0/+g3u4PzdCJ936n3KO4Mvzp4h26/+s1Oune6Xr6hw0a/O9Zend+ul6ctVmfLd4h0zR180fLqjVwebzXAp6udWFc7flyadkf5G3lrh52e3CGVUkoKnHq08U7fBb4fHbmBrWZNE2PT/MNOat3Zln35/oJHg/9r+w1th/M0yWv/qZLX19QoWrzf79s1vXvL9Hve7P15q9bNW9z4JO/qlR2YrVw60FrUPbuzLKr1d4nOJWdWJSUOiv88azqeG/cm6OZ68qO245DeRr6n5914r2+V7292z3Z64q+v/V+JGl/dqHV9kenrvO7z87D+XI6TW0/WPZe//HpCj06dZ1PVev0Z2brmncXWzPreev76EzN+r2sGvfufP8Vt9U7M3XuS/P8PuZRUFyqLRm5mrpqd4UKypEEBG/vzEvXlozAXRVHPTtbi9IP6fmfyipF/R7/UVnljnN2QYkmfblKh9wnZZ8u2q5bPlpmPf7ot2uVmjZVpmlq0L9mafyrv6mk1Kll2w5plddJmaf7qOf9+uNdGb7xg6UqdZo+FzdOe6ase0z3h2foyRkbfJ7v+cx5h1B/bvrQNcPilW8t8umKcyivWJe9vsBn38H/nqWh/3FdWFi7O8tvwNq0L1t//3Cpfli7V18u3aG2d09Tp/u/V2raVLWZNM1vd9C9WQV68Js16vHwD0pNm6rpq/cEbO+B3CKt3pmlq95erP05hRr59C+68q1FklRhmnjPQsU3fbjU74Qo5ZX/vfD0DxsqVE//LO+LVN4n931T62v1zkzN9KqqdX7ge136+gLd9cWqCm0L9DMfYzf04s9ln2PvKlCg3wFHyvf3fuAeDJ97VVcu9gqPTq/3sierQKZpKu2LlZpbblzn6eUuzqz0+hmq6qJPKCCgACGgYXKczu3ZUp2b+45bqZcYq1UPnl7t1xl6YmOrWtHOayHK8hWTE5ok6/xeLfXWBN9ZwLyv8r/3t35+v8ZXNw3U2K7NNKBtQw1q30hNyg1gPr5hkm4a5hq782vacGv7/Wd2UvrkcTq3Zws1So7TxsfGqL073Cy6Z2SFr5NdUKyVOw7rK/dVOO8/dJe+9luF/SsTqJvPv6e7rlq/OnuL2kyapv+t2KUr315U6WuV/0O35I9D1hXOA7lF1h/5ie4r+d7udV+p8z5x9/Cc3L06u+wqXfl+6/UTfY/jA1+v9pmo4NyX5lnh4Ps1vicp5bvYlD95qq5Zv+9Tm0nTrO/p3qwCpaZN1Z2fu7pqXfR/89XvMVcXgj1eQTPDfbJvmmaFP5weWQXFan/Pd2pbruKSW0UloPxsc88F+KP/47p9cjpNvTp7s976Nd3aHqgL2dXuE8j0A4G7qgyc/FOF9q7fkxVgb+mk41zVSe8qpb9uX+Oen6M1uzL15Pe/a5T7++XpflOed1cn7xNJT3D2OJhbpMnfrbdCcK9HXCe02QXF1om+ZwyC53htO5CnB75Zo+FP/aInvltnXdXdcShPqWlT9eki/4vNSmW/T1Ld1eMXZ23SRwu3qecjP0iS7vxipb5ZsUtOp6kNe7P1urtrVZtJZd/P9vd8p3Nfmqcz/ztXWQXFyi0s8RkD5Gnn3VNWKTVtqh75dq0Wpx+scOLnXcX765sLfQKlP2t2uY6h92yN5au3TqfpM0X6TR9UPh28JO32+pnwDtUeI5+erW9X7tY17y62pnT3dsP7SyuMSVi3O0tve12Vv/79JRr97Gylpk1VXlGJnE7X4rq5hSX6YmnZSW+fR2dW2V6pbFIHz8m6p/IkuSpMJaVOPTdzozrcN73C77Z2d0/zO2bLW6l7rJbnd0ggTqep7g/N0I3u77N39XhR+iFd+Mp8Xf3uYpklJTpYbua6rHzfnwXvn/nb5n6gUze7fu83So7zWf/rjblbFevVI8Lz3r9evlMyTZUUFmn+5gPWeLMxz80J3LVy6lQ573+g0vfozfsCx6GsfHmm0PSu+A944icdzivWx4u26/I3XL/T6+Vn6ba5Hyj1YNkFs12H83W3u6vqnDuHVbtXQzARUIAQ5xm7cuJxyVXsGZjDbqtw/6mLumtYhyb69LoB+sndPcxz5fr8Xi01+ISyLmKe7mHXDWlrjT/56Nr+FWY285g4qoPSJ4/zGTfh8czFPbT43pE+3eAa+1lvpeuDM3TWf3/VbZ8s1z8+XeFzNTZQhWXNrkwNe/LnCldVb6jixGFPuatpmXnFSk2bqs73T9dLP2/SvM1lV6bKd6m64JX5WrrtsHXfM6bCE6g8YU2SPlq4TSWlTp+B1OV5/zG8qFyXi6m3DPYJrO/M/8NnwoH9OYXW8alqcL4krd0V+ETaI7ewRKlpU61uY9bVVvfVuJPd/Zk/XbzDp+vRzsP5+ofX+BJPcPI35im/qFSmaarbg77jKdq6Z7i55LXfKr0Se7jcFK7l+6N3bu6avSqvuFRt756mx6f5hrXFRziA+aZh7TSy43EBH+/QLPBsWZ6TcH+fealsfNeaXVka9/xc/XfWJv3uJ2A/fm5X3Xiq67P1P3dVpvzn3tMtxDM5Rr/Hf9Qrv2zWq7O3aPaGDOuzPP7V39ThvulKTZtqhaBL3BcBvH82/u+Xsm5unq50d35RMYh3b1lXXVvUtSplnort615jO7z7wW/ZnxNwbIi3U574SZ0f+N6nkjhz3V45nabVleqNuVut8TmBzN6QUenjUtnn1LubmveJ4Z7MggrB9HCAyuCISmZqe9FrfFJ1rNqZqbfd4dozaV9BsVNDy02q4qmCd7r/e132+gJ1fXCGej7yg3VRpjJNvS46eYeywf+e5dNt8Z+frdD5L89X+3u+0zMzN6iwxKnb/EyV75lS/rXZWypcOJHKwuOniysfR+I57jPX7dXB3CIddE9l3yj3kPrsWCNnvusztPaGifrozGt8nnsgt1ClTlOPfrtW2w/m6az//ipJsjldPzMjs11VSytAmqbszlJ1albHqjbYnKXqv22lGuccci0J8OWX+u2qOzThpV+sz8K63Vnq8sD31liWw3lFGv7Uz5r42Qrt+3GO0g/kWkHDY+nW/dY2z9+dnDXr1GrjKsk0FVtSrJvnfazSaa6KcPkKtCfwe7TIdAXCXjll1dr0/bkyTKdsztJanc2wJhFQgBBnsxn67PoB+uia/kf83G/+PlAvVDFbWL82DdTWXW3x/NosP7NWlxZ1temxMUob00G1xXOy5c8XS3f4DH7OyC7UXZ+v9JnNSnINON+6P1cd7puuLRk5enqG64/xkXZt6e4eeJxbVKp/T/9dl762QLPW71N2QbHVpSPWUfbr03ug98Z9OdqSkaMlfxxSi3oJmjiqg2beMcR6vP09vt2Olt53mp4b38O6P/Skxtrnrkp4u3dcR8U6bEqJj/E7GYFHlvuq+e7MAmsAZqA1GMY+7zuz08TPVujV2Zv11IzflZo2VRe9Mt8KSZe+vkCPfrvW2jcx1l4hrHnbkpGjP7wqD5szcvX+b3/4XNke3dm1btCGvdk+V809JgxMtW6v35OlDXuztXV/boX9Drn73ntPKuHtCfdaPDsP+T8JfuK79X67rcXH+P6JbFY3XlseH6uJozrodT8LnnqOo+eqvhR4woq+qQ38bvd+rrfyn4dLT26tq9yTcZQ4nZqzMUOvlJsp69uVrhOUUV7rM3l4d5Na4yeoJsXaZZpmhYHOnqpMoC6CN57aTit2ZGrVzkxNeNMVZj3H2Zt38Hls6rry52wV2G2GdWXau6vK63O2VggKHpseG6P//T3wz4qnvR6ntGuo5fefJsk11qe8XzcfsK6g93+i4kBjz/f03av66YEzy7rYXjukbYV96ye6Pqv/cY892pNZoIuqCFYej7krYOf3ainJVTH5pZLQ5ZmsIdD4vdkTh/lM7HIgt1DXDXW1uVe5k98bvS72fO5ngPcCr8+VRzv37JaPTVun695b4vNY+TZVNsbwr28utG73euQHHczM06mbF+murbM0KH25zlg3R4bp1A/r9qq4xDesn/X4d3r8s0V6fe5WTb7yYY3+/Vd12bNJg9OXq2FSrMZ1beaz/22/fqib532sHdszlFyYp3HFu3XLvI/Vf9sqXbbc9XkzV67U4j8OKqGk7GfW5ixVw9zDGvn0bJnr1+ulcdfLvnatFsxcpGmrdqvUaSqhuGz/4w/t0srr/il99ZXMwkIdl71fw1bM0k/3PKNhWxar076tusR0/Ry/8MT7yiks0Tx3hadBXqYcpSVylJboysVf6+Rtq9R5zya902iPUhsmyX74kFXRuvS133Trrx/ptT+qN9lDKCCgAGGgb2oDa6DnkejWsp7O7N686h3d0sZ00Fndm2uU+4Tilct76eGzO0tyVV3KD+ivSXeO7qCtT4zVBwGqMuW7KH2yeLtVifhs8fYK3XSGP/WLnv9pk1LTplpddKbeMkgL7h6hD91fI9Ze/V+BV769SL0e+cHq0vHEuV0D7jv8KVf/ds9UnYGuWCXG2tUgKVZn92hhbfth7V71KzfLyhc3DNDVg8tOcrzHE7VplKQzupX9cf3JawzNa+5BnQfdJ/CJsXalTx6nf5x2orVPcalTq3Zk6uH/rdVnS3bo8Wnr9YK7u9nC9IM+J6/eV8CXbz+sN+YGHjTqmUDA487PV1pd3CTp9pEnqo97MPjZL/7q9zV6tS4bLD7u+bk6/ZnZGuY1UN3DE5RuHt7e7+t4xlb9a3rgmaTaenVFWbH9sB78Zo01SN7jrSv7+qw1s/S+03we7328q72eEJU+eZyu8XNyKrlOjD++tr8+vKbs8/7pdQM09KTGfvf39ti5rjFtjZLjFB9jU0KMQ395Y6EV/jwnm57P6oC2DSu8xs8byj4nSbEVu3us2JGpNpOm+VTBPG77eFmFbU9d2F3/Pr+b7hxddhHDc+V5YPvKpxLv0KyOFTLP7uH/91WgCtrxfiYf8XDYberasm7A34Et6iX4fMbmbT6geu5ulMWlptWVyvM1vMfLeCu/xktirF0pXmti9UltoNtGli3+e3GfVtbvWMl1AaH/Ez9qYXrZyf1Z5dr83PgePtVYSXq4irGN1dW6YaLuO6OTNfX9c+N7aoc7zB/tTHDeYw7fmf+Hzxiiw3lF+t+KXTJNU4vSfQPNyKd/kWmauvCVeer72EzN3bg/4Ex/Dd57Uz12b7B+H7bK3KNbf/2obAfT1P/9pbck6epFU5T8wnOSpBMObFOHjHSN3LRAb3Qo0V8GpKp+Uqz+7yLX37o6Ba7Ko8NmU+Kvs3X1oikal+7b9ffKxV/rkPvCj3fgGLp1qf6ybKqSCvO0/aU3lVhcoHG/z9V5a37S4fxiOeyGTjiwTY+d20ULJg7RqA3zXZ+zFStU+PCjumTF92p/YLu27He14fGYdA3aXlalfPL73/XZkh2KLSnWX5d+qxGbFqpR7mHVLcjRgG0rddqmBdbvhRZZGRrw+I/KzCtWs2xX9WnQCY2kvcdmCug/i4ACwNKsboKev6SnEtwnLKO7NNNfj+HiYoZh6JR2FU+mKrNgywFN/HxltfpTd25eV8fVidcp7Rvp4bM7a8btQ45ooSzvmXzO7tFc5/Vs4fP45f1bSyrrmuThLwid1b251j5ctkjmnDuHBfy6rRtUbOMVA45XnXiHdmfmq0lKvN+Z2dbvyVZq2lQrtHiqaX/3OpG/8/OVOvO/c/Xmr1srPL8yr87eohdnua7YexYllaSxXX2vlnuqF+XdOvIEvwt9dmlRRy9d1ktPXthdXVrU9ak+eXifrG7JyLEGvHtfBe7Wsq4mjemglvUTVCfe/6Ksn3tN6GCa0oS3Fmn66t06+8VfrT79z43vocv7t9a8tOEVFjpskBSr9Mnj9J8LuummYe3UwM8xkKRl951mnSh5OOw29W/b0GfcWb82DfTXclOGlzf9tsG67OSyfeonxmrnYd8qR5/jfWcBi4+x64sbTvHZ5j35Qm411pmZfF5X60Tf36KyqY2SdJG7K1n5NZzal5tI42T3Iq6eLqAxdps13uIOr/B8/xmdlBLg2Hmuygeq5HRpUXasvKvIvbxmRhzQrqHivKpknq5YF/VpqeQ4h9WVpjoLhTq8gmt8jF3n92qhawa30eJ7R8puM3TbyLL39bfBbXzeZ/kupl/fNFDPX9JTWx53TUHftlGSzu7RosIYtIRywfL2kSeqn1dlzl8XW0m6PsD6Xmd1b66tT4zV2K7N9JTXz7Tk+twdifu9KkiSdOZ/y8ZP9Xj4B9380TJN/HylNZbN83t/28E8ZeWXaFH6IWVkF+ryNxbo8yU7/FaBd2/eoVi7zWc8l7c+u9arz/H1df4q1wUfQ6ZP96rOzevKYSs7/kO/eEOd92zSVYu/VrvGyUqOsytx9XJJ8plRs3FKnOoW5Ohd9xTPF690Vdwb5GWq+27XZAsN8zL15dKKFaavl+/S8M2LNOKP5TruhaeUWFxgTciwzKursMcJDRPUJKXs85d78LBkmrrxt091QpNkdczYKodZ9vPr+YwM7+Dqgjpqw3z9/crJVhvjHHbp5Zf9fr9CDQEFQEgxDMMaVCvJqnZ4lA8U/gaS+nPnaN+FJP86IFWpjZL0/tUn68kLu+vrmwZq9UOjrMe7t3SdOJ54XLL+c0G3Cq/nsNt0V7kub54/dlvcV9A/uba/9Z7WPjzKZ1/PVXCPVg0Sfbp6eax+aJTf8Qr1k2KVVVCigmKnmtWN1+NeQcBz8ubxiLtrlud1vCthNTEVaCevMRePlLuqmxTn/wRTkq72WjBUcp1QvXlFX43t2kwX9HZ1X2nfpOLJx4a92fp44TbtPJzv0yfeMAyd1sn1h3nljkxdN7Sd5t41vELl7/qh7fTtzYPUJ7WBLnR/Heux933HK53RrbkePaermgc42ZOkC/u00sRRHZTotdaR9wly/aRYjerc1KdboEey+/vTzf1569C0jtY9PFo//mOo38VaU8utUl83IcZnoLakCpNtSFKcn68tubpOlXedn6rP+H6t9eSFFX8OPOolllUM/u9y3zAWH2PXhkfH6KXLemnZfadZ3YDuGefq/va817ikVvUTtege16yGVw1qo1UPjvIbcjdnVOzql+h1wl6+S+ybE/oo1mHzqZg8cV5XnzWd/npKqiTX1Oo5hSVW8L20X+uA79sT2sb3a2Vt69KirgzD0D3jOqmRV+V72i2D9cUNA3TicSlqUideX9zgCsjlB5F3d7fJZjM0f9JwTblpoCT/MyF6LopIrtD/6fUD1LK+67N6gp9xi3eOPklpYzro9b+6uii+dJnvQr+en5Xyg6g7NK1T7W7Gb13pmnxlZMfAY28kVxexTPfgde+LJs//VHGc2g/rXFf9Hz2ni/U3ITO/WEWlThmGYf3u7epVXb65ZKsa2J1qlVk27qXTvrILMZ2a+15wiHPYdNom10Dz1IZJSkmIsS6GJMc5rEq1Z6ILbw+c2UkvHCybYe+8NT9pT3Lgi22NVvl2dTucV6QFWyuG7aQ4h7q1rGt97e0zZqvzPlfluq876McVF1lVyobJroAS67Cped0EdczYqt47/cw+FgZrjRFQAIScnyeWVRNOKdc95H739MmeoFJ+xWNPl7TyTm7j/49Fi3oJuqB3S3VvVc86WZSk/u0aqnWDRD1wZmdd2KeVz3OOcy886X1SNuufp/pMe1k3IUYne3WtSYx1WCckkny6gHh4d/WSXFWV5AAn+N49XgpLSq2TEkn6/PpT/DyjrKtTdXhOYCRpjJ8xDB6PntPF5w99w+Q4/f5oWWXooFfXu/9c0E2vXN5bX7lPuOIcZSdBVw1so1/ThleYFU6SNS7Aas9zc5T25Spd+PI8bSl3ojooQHeiRsllV5+vG9LW6hbyn3JXisvzdwJfmVtGuLry3Dz8hAqPPenna9lthqbceIreu6osiCfE2tWucbLevaqfptx4irY8PlYL7x6h9/92coUTR39TgrdumKiP3eG4tbt74UlNUxRrt1WopPjrOjVpbEdrunJJWnj3CEm+x0tyHc9VD56uNyf08Zk10PvKvqdvf6zDprFdm6l+UqwVlvx9Z202Q41TXLMaegw7qeKJbuOUOJ/KjN1maNWDZRcByv98De9wnDY8OkbneFU9Y+w2n/2GuMfeXXmKKwh4xlg0So7ze1IqlVVnZqyputtMp+Z11Pv4sgqHJ/R6d1/97lbfSkWzugnW2CqbzbBmRvSciA9o62rz815VoivdQcZz4eDaIW2VFGvXc+N76MZTXUFgZKfjlD55nMaWG3vhrfzg/gHtGuqrmwZqzp3DrAqQ5+dteIcmSp88TpsfH2sdr9ev6Otz0cefZ2a6Kg6ntGtkdZF8Y27Faq6nm5fdZqi/+/fqO73OsB4/8z8TdUa3Zj4XCYYM7SrjwAGfn+Fn4tIlSTvrNLZ+j6ufa8ZKwzCsttdJcPhMvpEY61D7Jim6bfwp6lhuEoyL+rTSlb2aamD7Rrpt5Im61f07oGmOK3Dceu1oa1uBw/U1PRPXeIKV9yxsJ7o/a6ubtrfa5blQU+CIU8d9W3Vm9+ZWZeXM9bNVPzFGvS8/y7pAo8su06nluoue6n3Bo6jqNXOCjYACICTdd0YnPXKO79X4mXcM1bAOTbT58bEBg8iQE/z34U+tpK+6N8+MR3XiYzT7zmF++897ZnDyPmFr0yjJZ0C0v7U+eh/fQN/8fWClg9x/mXiq+rdtoC9vPKXS2VY83WQk18BS7yvr/ta5cb2nsj/esydW7FLm+X5/dv0AjfT8oZP0stcV8cnndfXpGnXZya4ruF/eeIp1Ndb7+3J+75ba8OgYvfe3frqwTyuN7tLU56q1R/+2/geNS7LGBZS3K7NAX7orQMvc40EuO7m16ifGWCHI4zSv9xPo+1MT7jjtRKVPHuczxsDjrO7NNfiERnr3Kt8pvHu2rq+6iRUDq+cxm81Qkzrxrv7jlZg/abhWPOCa5a1/24aalzZc09wnvDF2mzY8Nka9j6+vabeUnQR7JgLo37aBurWsa1Xf4mNc45XSJ4/zGxolV+UoJT7G6k7iz80jKo4Lmj9phG4a1s7nmEiuMW/+1E+KqbDPJ9f2191jyyqYax8eZZ2IBuryI7mCWnme9+mpHnRt6VuBapgcW6ECK0kL3MFNKqt6lD8prEyzur5VuWuHtK1w8ltei3oJevHSXnrTPUX8uG7N9PM/T/UZs/K3QW2UPnmcrhrURn1T62vCKala8/DoChdAqvKa+yKF96xePVrVU6sGibppWHt9cPXJevWvvXVm9+Z63D0mr3ygD3SBxZ9/nR+4QucxtkszaxzYocSy43R8i4Zq3yRFBcWlmnbSQO2s01iGaUqvv66bhrXXVYPa6JrBba3j/2tqDzlS3J8Fe9nvq24t62p839Y6vmk9zW5Z1h57O3dVcfhwJcU51LaRKxz/cEJ/Na0bL/3rX9a+RrOy0Lc7pZGMiy6yPlvfdhjk87nv16aBFrQq+zs3/cRTrMcHJ3qFiBtvlOSaoatl5l5rdk+Pdo2TNfjai8p+9zZposYpcWpZv+xvSI/T+ktnny1NnCjFHfmY1mOt+p8cADiGvMcTpE8eJ9M0rV/ydpvhc6W8fmKMzu3ZUveM6+jzB/KVy3vr+vddpfRAJ7nl3X7aifrjQJ4u7utbNbl95InW1b7WXsHhlct7KynO9Uehad14n/386dayXqVf//iGSfr4Wv+LXXrr4tWN5/pT2ykx1qGf/jHU+oP07MU99NXynfrZvQBg3YQYn65OrcsFtqEnNtZf+h+v8X1bWVNAe19F79isjtbtztKYLs1UNzFG/ds20O97sq3X9O46I0k///NU7c0qsK5QDw4QHD2qqlR8fdNAOU0z4EKFntDhsNu07P6Kawdd2u94fbTQ/5odX900UB8u+EOntGuk2z5ZromjTtJZ3ZsH7Bb1Z7z3N/+TQBytFQ+cru4PufqXlz/hDdQtzVN9vG5IWy3YelDLtx/WPWM7VTgx92fTY2N06esLKnTjCsT7BMmjQVKsJo5yhYvvbxuiUc+61nkpP8bCI85h16PndNGAdg3VrnGytjw+VjabobaNk7Xi/tNVJ8FhfQ5/TRuuBpX8rMfYbereqp4uLlcVrUyj5Dh1aJqipy/qrm4t62rk07N18/D2PmNTnr6ouxZsOegT7I/U7V7jVCozrptv1SM1wDi6Rslx+ixANbU6bDZD89KG+1SKPew2w7p4U9VMkZf0a62PFm7TMxd31+2fBO6SW36skj+eIL/i/tOVkVOo9td2lGJjpZauiluzVk20wZ6qLns3W4PBbYahOn+5VPrsM0nSFQNSNWzCcCm+RHrvPalNG6lpU2nKFBmGoaajh0mjRqn3jxv17Yfx+vvFp0gDT5R+/13q2FFKTdXQ/GLdctJZandwh2zlJ49pWVb9a9q4jhTv+pz0799RnzVqow557vVNkpOV9fe/aP4X67W2SVu1yMrQ+iZtFHP+II28tIGOT7RJb70unXqq1KSJzuuXqi8XpktyLSipv/5Vp+/6rxx2w/X5t9mk3r2lJUuk5GQZzZrpgt7utp1/vtQ18MQuoYiAAiAslB9HYBiGurSoo9U7s/TLncNUJ973Kuv9X6/RsA5lJ8XV7apzXJ14fXRtxb7Wt4xobwUP7wXwRpfr/nRZ/9b6duUu6wpnbfG+4u7p1tDWq5vNOT1b6JyeLbRpX45enb3Z74w/c+4cppFP/6IpNw60uml5r0/j3Z1o2i2DVOo0ra4JVYWo1EZJAU+cvK19eJS+X7NHp/rpxuOtu5+qi0dVA8slKSHW1W5/lbQerepZVZ1zeh7ZVeZgq5sQo7O6N9fpnat/YpwQa9fqh0YpMcauolKnZq7bW61wIrkC4KfXVR2gp9x4ilbvyqryCrp3RaOyytbl/cuOsfdMauUrT4EGhnv7ulx1zZ8fbh+i056ZrRi7YY3dOs89re8vE09Vq3LBKyU+5qjCyQuX9NTN7tnByg96DwWVjb2qrrTRHdQ4OVZndGuut+f9oRXbD+u0TsdZC4y+eGlZ5Wz6bYM1+lnX9Oc//WOoNSNieXUTY1zH3jvU3Hmn2tsd0gM/6FBCuUpUp7JB+/WTYlW/U2vJMKR//ENKSZFKS6UDB6SBA63qws3D2+uiPq1cFRJJ6uVu51/+ojr5+bpjwW6d17aX9M6rZV+nf38pJkbj+7bWx4u2qd9Yd7U8LU397Xb9GhMjrTlOSkyU2rTRqaVOdZq3S2t3S5kJKa5xPu0ayvptfccdrvZJap3aVHIHlLgYu1S/foVxNBo7Vho+3FUVGj9eWr3a9Z5qcQbO2mL4m/v9z+rTp4+5ePHiGn9dAPBmmqZKnKbPSXV5Xy3bqbW7s3R3gPUojsTHC7cp7ctVmnnHEL+Dt4+19XuyVOo0/Q6KjlR/e3uRflzvO6i4/DSvgXy3ardOPalJSJ4IRjPPGi/VPY6RpKjEqRPvda1NES3vf192gRJjHRr25M96+qLuFaqrnnFRnotKns9HqwYJmnPn8CpfPzVtqq5OOKh7Tfe6QMOGSUOHSg8+6Lp/+eVSe/9Tkh+x0lLpkUfK7j/wgPTNN9KyZSooLlX8Y48Efq7b1v251vTplX4GpkzRs0+5qkC3/e9FV8h54gnXY3ff7aokhSHDMJaYpllhYSkqKADClmEYrlJ3JTyVhJowvl9rja9kRp9jrfy0t9HgDXdl6sMF23T3lFUV1oaozJhKBgQjeKbfNjjgIpuRLtZh09MXdfc75Xak8gzuXnTPSL+Pl692b31irA7nFVd77Njvj46WY/Nm6SN3QBlSbqrydtX/nVElr/Er+stfXJWKGNdnOf6U6s165qnqll8guYKYGF3Yu5XrAkv9+r4zcYVpOKkMAQUAEHYuPbm1eh1fLypDWqSJ9mPo6ToG/wzDOKKJLeIcdinFq+uXp3tT167Sjh21193JMzi+Wzdp4UJrdrCqGIZRvepZfr5a1E+Q6tb1PFFKSJC61MyCnaGGLl4AAACIHCUl0qefurp3Navlyulvv0mFha5uZLVp/35p+nTpwgvDYhau6grUxYuAAgAAAOCYCxRQWAcFAAAAQMggoAAAAAAIGQQUAAAAACGDgAIAAAAgZBBQAAAAAIQMAgoAAACAkEFAAQAAABAyCCgAAAAAQgYBBQAAAEDIIKAAAAAACBkEFAAAAAAhg4ACAAAAIGQQUAAAAACEDAIKAAAAgJBBQAEAAAAQMggoAAAAAEIGAQUAAABAyCCgAAAAAAgZBBQAAAAAIYOAAgAAACBkEFAAAAAAhAwCCgAAAICQQUABAAAAEDIM0zRr/kUNI0PSHzX+wkenkaT9wW4EjgjHLPxwzMILxyv8cMzCD8cs/HDMjr3jTdNsXH5jrQSUUGIYxmLTNPsEux2oPo5Z+OGYhReOV/jhmIUfjln44ZiFDrp4AQAAAAgZBBQAAAAAISMaAsqrwW4AjhjHLPxwzMILxyv8cMzCD8cs/HDMQkTEj0EBAAAAED6ioYICAAAAIExEdEAxDGO0YRi/G4axyTCMtGC3J1oZhtHKMIxZhmGsMwxjjWEYt7q3NzAM4wfDMDa6/6/v9ZxJ7uP2u2EYo7y29zYMY5X7secNwzCC8Z6igWEYdsMwlhmG8a37PscrhBmGUc8wjM8Nw1jv/lkbwDELbYZh3O7+nbjaMIyPDMOI55iFFsMw3jQMY59hGKu9ttXYMTIMI84wjE/c2xcYhpF6TN9gBApwzP7j/t240jCMKYZh1PN6jGMWgiI2oBiGYZf0oqQxkjpJusQwjE7BbVXUKpH0D9M0O0rqL+km97FIk/SjaZonSPrRfV/ux8ZL6ixptKSX3MdTkl6WdK2kE9z/Rh/LNxJlbpW0zus+xyu0PSdpummaHSR1l+vYccxClGEYLSTdIqmPaZpdJNnlOiYcs9Dytip+P2vyGP1N0iHTNNtLekbSv2rtnUSPt1XxmP0gqYtpmt0kbZA0SeKYhbKIDSiS+knaZJrmFtM0iyR9LOnsILcpKpmmuds0zaXu29lynTi1kOt4vOPe7R1J57hvny3pY9M0C03T3Cppk6R+hmE0k1THNM35pmvw1Ltez0ENMgyjpaRxkl732szxClGGYdSRNETSG5JkmmaRaZqHxTELdQ5JCYZhOCQlStoljllIMU1ztqSD5TbX5DHyfq3PJY2gAvbn+DtmpmnOME2zxH33N0kt3bc5ZiEqkgNKC0nbve7vcG9DELlLoT0lLZB0nGmauyVXiJHUxL1boGPXwn27/HbUvGcl3SnJ6bWN4xW62krKkPSW4eqW97phGEnimIUs0zR3SnpS0jZJuyVlmqY5QxyzcFCTx8h6jvsEOlNSw1prOSTpKknfuW9zzEJUJAcUf2mWKcuCyDCMZElfSLrNNM2synb1s82sZDtqkGEYZ0jaZ5rmkuo+xc82jtex5ZDUS9LLpmn2lJQrd7eTADhmQeYet3C2pDaSmktKMgzj8sqe4mcbxyy0HM0x4vgdQ4Zh3CNXt/MPPJv87MYxCwGRHFB2SGrldb+lXOVzBIFhGDFyhZMPTNP80r15r7uMKvf/+9zbAx27HSory3pvR80aKOkswzDS5eoaOdwwjPfF8QplOyTtME1zgfv+53IFFo5Z6BopaatpmhmmaRZL+lLSKeKYhYOaPEbWc9xd/eqqYpcy1ADDMK6QdIaky8yyNTY4ZiEqkgPKIkknGIbRxjCMWLkGQX0T5DZFJXffzDckrTNN82mvh76RdIX79hWSvvbaPt49U0YbuQanLXSX0rMNw+jvfs2/ej0HNcQ0zUmmabY0TTNVrp+bn0zTvFwcr5BlmuYeSdsNwzjJvWmEpLXimIWybZL6G4aR6P5ej5BrfB7HLPTV5DHyfq0L5Pp9y9X4GmYYxmhJd0k6yzTNPK+HOGahyjTNiP0naaxcszVslnRPsNsTrf8kDZKr/LlS0nL3v7Fy9dn8UdJG9/8NvJ5zj/u4/S5pjNf2PpJWux/7r9yLjfKv1o7dqZK+dd/meIXwP0k9JC12/5x9Jak+xyy0/0l6SNJ69/f7PUlxHLPQ+ifpI7nGCBXLdeX8bzV5jCTFS/pMrsHZCyW1DfZ7Dvd/AY7ZJrnGjXjOQV7hmIX2P1aSBwAAABAyIrmLFwAAAIAwQ0ABAAAAEDIIKAAAAABCBgEFAAAAQMggoAAAAAAIGQQUAAAAACGDgAIAAAAgZBBQAAAAAISM/wdUmc2oEEXKxgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1008x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fix, ax = plt.subplots(figsize=(14, 10))\n",
    "n_train_losses = len(train_mb_running_loss)\n",
    "ax.plot(range(n_train_losses), train_mb_running_loss, label=\"training losses\");\n",
    "ax.plot(range(n_train_losses-len(val_mb_running_loss), n_train_losses),\n",
    "        val_mb_running_loss, c=\"r\", alpha=0.5, label=\"validation losses\")\n",
    "ax.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "We will evaluate the trained model on a STS dataset. We will compare the finetuned model and the bert-base model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset glue (/home/utsav/.cache/huggingface/datasets/glue/stsb/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1500,\n",
       " {'sentence1': 'A man with a hard hat is dancing.',\n",
       "  'sentence2': 'A man wearing a hard hat is dancing.',\n",
       "  'label': 5.0,\n",
       "  'idx': 0})"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sts = hf_datasets.load_dataset(\"glue\", \"stsb\", split=\"validation\")\n",
    "len(sts), sts[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence1s = [data[\"sentence1\"] for data in sts]\n",
    "sentence2s = [data[\"sentence2\"] for data in sts]\n",
    "normalized_labels = [data[\"label\"] / 5 for data in sts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode(\n",
    "    input_texts: list[str], tokenizer: BertTokenizerFast, model: BertModel, device: str = \"cpu\"\n",
    ") -> torch.tensor:\n",
    "\n",
    "    model.eval()\n",
    "    tokenized_texts = tokenizer(input_texts, max_length=MAX_LENGTH,\n",
    "                                padding='max_length', truncation=True, return_tensors=\"pt\")\n",
    "    pooled_embeds = model(tokenized_texts[\"input_ids\"].to(device),\n",
    "                          tokenized_texts[\"attention_mask\"].to(device)).pooler_output\n",
    "    return pooled_embeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cos_sim_in_batches(\n",
    "    batch_size: int, s1_texts: list[str], s2_texts: list[str],\n",
    "    tokenizer: BertTokenizerFast, model: BertModel,\n",
    "    cos_sim_f: Callable[[torch.tensor, torch.tensor], torch.tensor],\n",
    "    device: str = \"cuda\"\n",
    ") -> torch.tensor:\n",
    "\n",
    "    cos_sims = None\n",
    "    n_batches = len(s1_texts) // batch_size + int(len(s1_texts) % batch_size != 0)\n",
    "    for i in range(n_batches):\n",
    "        s1_batch = s1_texts[i*batch_size: (i+1)*batch_size]\n",
    "        s2_batch = s2_texts[i*batch_size: (i+1)*batch_size]\n",
    "        if i == 0:\n",
    "            cos_sims = cos_sim_f(encode(s1_batch, tokenizer, model, device).detach(),\n",
    "                                 encode(s2_batch, tokenizer, model, device).detach())\n",
    "        else:\n",
    "            _cos_sims = cos_sim_f(encode(s1_batch, tokenizer, model, device).detach(),\n",
    "                                  encode(s2_batch, tokenizer, model, device).detach())\n",
    "            cos_sims = torch.cat([cos_sims, _cos_sims])\n",
    "    \n",
    "    return cos_sims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "cos_sim_f = torch.nn.CosineSimilarity()\n",
    "sbert_cos_sims = cos_sim_in_batches(16, sentence1s, sentence2s, tokenizer, model.encoder, cos_sim_f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SpearmanrResult(correlation=0.6562854102049397, pvalue=1.742454940612208e-185)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.spearmanr(sbert_cos_sims.cpu().detach().numpy(), np.array(normalized_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "bert_model = BertModel.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SpearmanrResult(correlation=0.15565973492337204, pvalue=1.3559203499038571e-09)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bert_cos_sims = cos_sim_in_batches(16, sentence1s, sentence2s, tokenizer, bert_model, cos_sim_f, \"cpu\")\n",
    "stats.spearmanr(bert_cos_sims.cpu().detach().numpy(), np.array(normalized_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our unsupervised finetuned model is doing much better on this semantic similarity dataset as compared to the bert-base model. It's not performing as good as the supervised training we did for Bi and Cross encoders(where we saw average spearman rank correlation of ~0.80) but that's to be expected.\n",
    "\n",
    "** Note that we achieve .74 spearman rank correlation(vs 0.66 here) in the previous notebook, using all the utils from sbert for training. Here we are using a different scheduler, but there might be other factors affecting our score which I'm not sure about.\n",
    "\n",
    "Let's look how this model can be used for retrieval tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ranked_docs(\n",
    "    model: BertModel, query: str, corpus_emebds: np.array, device: str=\"cuda\"\n",
    ") -> None:\n",
    "    tokenized_query = tokenizer(query, max_length=MAX_LENGTH,\n",
    "                                padding=\"max_length\", truncation=True,\n",
    "                                return_tensors=\"pt\")\n",
    "    query_embed = model(**tokenized_query.to(device)).pooler_output\n",
    "    scores = util.cos_sim(query_embed, corpus_embeds)\n",
    "    print(f\"Query - {query}\\n---\")\n",
    "    scores = scores.cpu().detach().numpy()[0]\n",
    "    scores_ix = np.argsort(scores)[::-1]\n",
    "    for ix in scores_ix:\n",
    "        print(f\"{scores[ix]: >.2f}\\t{corpus[ix]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"A man is eating food.\",\n",
    "    \"A man is eating a piece of bread.\",\n",
    "    \"The girl is carrying a baby.\",\n",
    "    \"A man is riding a horse.\",\n",
    "    \"A woman is playing violin.\",\n",
    "    \"Two men pushed carts through the woods.\",\n",
    "    \"A man is riding a white horse on an enclosed ground.\",\n",
    "    \"A monkey is playing drums.\",\n",
    "    \"A cheetah is running behind its prey.\"\n",
    "]\n",
    "\n",
    "query = \"A man is eating pasta.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_corpus = tokenizer(corpus, max_length=MAX_LENGTH,\n",
    "                             padding=\"max_length\", truncation=True,\n",
    "                             return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - A man is eating pasta.\n",
      "---\n",
      "0.81\tA man is eating food.\n",
      "0.76\tA man is eating a piece of bread.\n",
      "0.49\tA man is riding a white horse on an enclosed ground.\n",
      "0.48\tA man is riding a horse.\n",
      "0.43\tA woman is playing violin.\n",
      "0.33\tA monkey is playing drums.\n",
      "0.13\tA cheetah is running behind its prey.\n",
      "0.02\tTwo men pushed carts through the woods.\n",
      "0.02\tThe girl is carrying a baby.\n"
     ]
    }
   ],
   "source": [
    "corpus_embeds = model.encoder(**tokenized_corpus.to(\"cuda\")).pooler_output\n",
    "get_ranked_docs(model.encoder, query, corpus_embeds, \"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - A man is eating pasta.\n",
      "---\n",
      "1.00\tA man is eating a piece of bread.\n",
      "1.00\tA woman is playing violin.\n",
      "1.00\tA man is riding a horse.\n",
      "1.00\tA monkey is playing drums.\n",
      "0.99\tA man is riding a white horse on an enclosed ground.\n",
      "0.99\tThe girl is carrying a baby.\n",
      "0.99\tA man is eating food.\n",
      "0.98\tTwo men pushed carts through the woods.\n",
      "0.97\tA cheetah is running behind its prey.\n"
     ]
    }
   ],
   "source": [
    "corpus_embeds = bert_model(**tokenized_corpus.to(\"cpu\")).pooler_output\n",
    "get_ranked_docs(bert_model, query, corpus_embeds, \"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"A man is eating food.\",\n",
    "    \"A man is eating a piece of bread.\",\n",
    "    \"A woman is playing violin.\",\n",
    "    \"Two men pushed carts through the woods.\",\n",
    "    \"A woman is practicing jumps with her horse.\",\n",
    "    \"A horse is running around the track.\"\n",
    "]\n",
    "\n",
    "query = \"Horse jumped over the obstacle.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_corpus = tokenizer(corpus, max_length=MAX_LENGTH,\n",
    "                             padding=\"max_length\", truncation=True,\n",
    "                             return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - Horse jumped over the obstacle.\n",
      "---\n",
      "0.60\tA horse is running around the track.\n",
      "0.45\tA woman is practicing jumps with her horse.\n",
      "0.27\tTwo men pushed carts through the woods.\n",
      "0.12\tA man is eating food.\n",
      "0.10\tA man is eating a piece of bread.\n",
      "-0.02\tA woman is playing violin.\n"
     ]
    }
   ],
   "source": [
    "corpus_embeds = model.encoder(**tokenized_corpus.to(device)).pooler_output\n",
    "get_ranked_docs(model.encoder, query, corpus_embeds, \"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - Horse jumped over the obstacle.\n",
      "---\n",
      "0.98\tTwo men pushed carts through the woods.\n",
      "0.98\tA horse is running around the track.\n",
      "0.98\tA man is eating food.\n",
      "0.97\tA woman is playing violin.\n",
      "0.96\tA man is eating a piece of bread.\n",
      "0.95\tA woman is practicing jumps with her horse.\n"
     ]
    }
   ],
   "source": [
    "corpus_embeds = bert_model(**tokenized_corpus.to(\"cpu\")).pooler_output\n",
    "get_ranked_docs(bert_model, query, corpus_embeds, \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## References\n",
    "\n",
    "[1] Kexin Wang, Nils Reimers and Iryna Gurevych. \"[TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](https://arxiv.org/pdf/2104.06979.pdf)\"\n",
    "\n",
    "[2] https://github.com/UKPLab/sentence-transformers\n",
    "\n",
    "[3] [BIER benchmark](https://arxiv.org/abs/2104.08663)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}