9_unsupervised_training_gpl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised sentence embedding learning - GPL\n",
    "\n",
    "In this notebook we will implement [GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval](https://arxiv.org/abs/2112.07577) by `Kexin Wang, Nandan Thakur, Nils Reimers and Iryna Gurevych`.\n",
    "\n",
    "Here is the summary of the paper by the authors.\n",
    "\n",
    "> Dense retrieval approaches can overcome the lexical gap and lead to significantly improved search results. However, they require large amounts of training data which is not available for most domains. As shown in previous work (Thakur et al., 2021b), the performance of dense retrievers severely degrades under a domain shift. This limits the usage of dense retrieval approaches to only a few domains with large training datasets.\n",
    "In this paper, we propose the novel unsupervised domain adaptation method Generative Pseudo Labeling (GPL), which combines a query generator with pseudo labeling from a cross-encoder. On six representative domain-specialized datasets, we find the proposed GPL can outperform an out-of-the-box state-of-the-art dense retrieval approach by up to 9.3 points nDCG@10. GPL requires less (unlabeled) data from the target domain and is more robust in its training than previous methods.\n",
    ">\n",
    "> We further investigate the role of six recent pre-training methods in the scenario of domain adaptation for retrieval tasks, where only three could yield improved results. The best approach, TSDAE (Wang et al., 2021) can be combined with GPL, yielding another average improvement of 1.4 points nDCG@10 across the six tasks. The code and the models are available at [this https URL](https://github.com/UKPLab/gpl).\n",
    "\n",
    "GPL consists of the followings steps:\n",
    "- Collecting passages from the target domain\n",
    "- Query generation from the passages using a finetuned t5 model(`doc2query/msmarco-t5-base-v1`)\n",
    "- Using a finetuned model(`msmarco-distilbert-base-tas-b`) for retrieval to retrieve top passages for each query\n",
    "- Uing a finetuned Cross-Encoder(`cross-encoder/ms-marco-MiniLM-L-6-v2`) to mine hard negatives from the retrieved passages\n",
    "- Training a Bi-Encoder using the (query, positive, hard negative, Cross-Encoder score) on MarginMSE loss\n",
    "\n",
    "![](assets/gpl.png)\n",
    "\n",
    "In short, we are generating queries, mining hard negatives and finetuning a Bi-Encoder to mimic the Cross-Encoder.\n",
    "MarginMSE loss is perfect for such teacher-student model training.\n",
    "\n",
    "$$ L(Q, P^{+}, P^{-}) = MSE(M_{s}(Q, P^{+}) - M_{s}(Q, P^{-}),  M_{t}(Q, P^{+}) - M_{t}(Q, P^{-}) )) $$\n",
    "\n",
    "𝑃+ : Positive passage<br>\n",
    "𝑃- : Negative passage<br>\n",
    "𝑀𝑡 : Teacher model(Cross-Encoder)<br>\n",
    "𝑀𝑠 : Student model(Bi-Encoder)\n",
    "\n",
    "\n",
    "Hard negatives play a key role here. Hard negatives are passages that are related to the query but don't exactly answer the question but mention some other fact maybe. This depends a lot on the domain as there can't be universal sentence encoder.<br>\n",
    "For example do we encode these two sentences closer in the vector space or farther? `Nuclear energy in safe!`, `Nuclear energy is dangerous`. The answer will depend on the domain, if we are focusing on nuclear energy then yes, if we are focusing on safety issues then maybe not.\n",
    "\n",
    "| ![](assets/hard_negatives.png) | \n",
    "|:--:| \n",
    "| Effects of using hard negatives in retrieval systems. (Image source: https://arxiv.org/pdf/2010.08191.pdf)  |\n",
    "\n",
    "GPL can be used both to finetune a pretrained model or as a domain adaptation technique for a finetuned Bi-Encoder model. Here we will do the domain adaptation.\n",
    "\n",
    "Ideally we would need lots in-domain data, here we will explore the core idea with a small dataset(due to the resource constraints)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "HZbWmxIjW-sc"
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "import random\n",
    "from tqdm import tqdm\n",
    "\n",
    "import datasets\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sentence_transformers import CrossEncoder, InputExample, losses, SentenceTransformer, util\n",
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cuda'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105,
     "referenced_widgets": [
      "fee64bf203e54ba98126ac3b1b806faa",
      "66310509ead04ae39bf8b7ffb523c411",
      "59e0ac4998844191b2fe55d82353781e",
      "b2033ee9411f4449a593bea0ca798288",
      "fdafb465ce8b4bd5b548acba3eb50d43",
      "dd6bc5c2e5d44e90b0816eb2a4c5d874",
      "746c5c6161d447c1b605b40996d1158d",
      "78f65d5dd16c4972b80ad57842688d2f",
      "1e6d3d085091481ab4dc4b98e8a07cca",
      "c02376dfc7c046c989a102d0d6bb9efb",
      "e06f39294b4c410bb2abb489f1f8da2c",
      "b1027d3c96034f2d81868fd6267bb87f",
      "b443e89c08a64d87b2c7703bf6b929d3",
      "da6dcd75615e4258a65e47141fb9092f",
      "1599afe703c54f2d8be7b9666a511857",
      "8b2d0f5df788426c967de270dd5fbbeb",
      "1e434b72393e4b3b8d464c409b66ae95",
      "6ac6af5ca9d14075915a7a37200e63be",
      "c99312ef350e41d7938955ee4a09d81b",
      "ae08a9ef72ea45e6bba9caa72a995a51",
      "761e339bda144b218685608c1df710c9",
      "fb8b139665f348d99caa7406639c55ab",
      "459d2c124c4f42f988f6e8d7de362e59",
      "6624effa65bd4985a8e0aab8fcbe5403",
      "3fb165f1258f425a80e6457a5a6bec8f",
      "c0b75b080c1147ac84d04c5e358377cc",
      "8ba201b0d21e40d09919e38b8055cf9f",
      "eed0d294051e40199a299006c7766805",
      "1287975a9a1948e6ac098f9f0f70ec9c",
      "609ec846c5fe48a09f5ea1e8fb899fe4",
      "c6d42da408e04ae0bb5fe9c2371d5477",
      "8c202783ca984fa28186b119f659bab6",
      "15705887479a4b7896de71d59d9b1574",
      "3f1812129cd840b6a0c171bd019386e5",
      "18dced07268e4bc287c19bfa111ad737",
      "2fbaa6537fbd489f8430cc91b3a3cd17",
      "54d3010a0c964605b9aae3d94c4d43bf",
      "c53ef72839714eaf977e3c4d4299ad6b",
      "5d2d9b7782334cb4875a9c021cb735d0",
      "ecbfc203abdb41d38499d28e5139191f",
      "4563aad9bff94843b472a8a382d4d990",
      "862369200c944df9ba48b01a23fbc160",
      "ae60fb6564a3453e95308bdce7a0688b",
      "4aa84adb5bf94c94b4c50776efcdf004"
     ]
    },
    "id": "ZC3qflE7HW3f",
    "outputId": "8563f0fc-71d5-48b1-c0fa-f4d015d519ad"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration corpus-data_dir=corpus\n",
      "Reusing dataset nfcorpus (/home/utsav/.cache/huggingface/datasets/BeIR___nfcorpus/corpus-data_dir=corpus/0.0.0/093f1fe2ffa7a9c72fa48239c8f279b51d6b171abd77737c7fd1406125307599)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f2fa0bb866cc4604876efe28d9d865d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(3633,\n",
       " {'_id': 'MED-10',\n",
       "  'title': 'Statin Use and Breast Cancer Survival: A Nationwide Cohort Study from Finland',\n",
       "  'text': 'Recent studies have suggested that statins, an established drug group in the prevention of cardiovascular mortality, could delay or prevent breast cancer recurrence but the effect on disease-specific mortality remains unclear. We evaluated risk of breast cancer death among statin users in a population-based cohort of breast cancer patients. The study cohort included all newly diagnosed breast cancer patients in Finland during 1995–2003 (31,236 cases), identified from the Finnish Cancer Registry. Information on statin use before and after the diagnosis was obtained from a national prescription database. We used the Cox proportional hazards regression method to estimate mortality among statin users with statin use as time-dependent variable. A total of 4,151 participants had used statins. During the median follow-up of 3.25 years after the diagnosis (range 0.08–9.0 years) 6,011 participants died, of which 3,619 (60.2%) was due to breast cancer. After adjustment for age, tumor characteristics, and treatment selection, both post-diagnostic and pre-diagnostic statin use were associated with lowered risk of breast cancer death (HR 0.46, 95% CI 0.38–0.55 and HR 0.54, 95% CI 0.44–0.67, respectively). The risk decrease by post-diagnostic statin use was likely affected by healthy adherer bias; that is, the greater likelihood of dying cancer patients to discontinue statin use as the association was not clearly dose-dependent and observed already at low-dose/short-term use. The dose- and time-dependence of the survival benefit among pre-diagnostic statin users suggests a possible causal effect that should be evaluated further in a clinical trial testing statins’ effect on survival in breast cancer patients.'})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = datasets.load_dataset(\"BeIR/nfcorpus\", \"corpus\", \"corpus\")\n",
    "\n",
    "len(dataset[\"corpus\"]), dataset[\"corpus\"][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9EpFpFxCV8OG",
    "outputId": "20590cd2-fd07-4019-df26-0d905d5172a4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3633"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "passages = []\n",
    "\n",
    "for text in dataset[\"corpus\"]:\n",
    "    passages.append(text[\"text\"])\n",
    "\n",
    "len(passages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 265
    },
    "id": "eGeCHORFXrRh",
    "outputId": "08ea4aa4-4c94-4457-cf3a-ba4f43cdf937"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzwAAAFlCAYAAAAu6O34AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAVeklEQVR4nO3db8yd9X3f8c93OGU0KRsZBlHbmVnnbQWkkmF5bEhTNqbhNtNMpUUyUgOakFwhsiVTpMnkSfrEEpPaZEMqSLQwyJaFWkkq0AJtGYsUVWIQQ1HBUBQrMLiDB+6yNWwP6Ey+e3Au4MTctm/7/md+fr2ko3Od37muc35H/IR5c65zubo7AAAAI/oL6z0BAACA1SJ4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhbVjvCZzMhRde2Fu3bl3vaQAAAGeop5566k+7e+Niz53xwbN169YcOHBgvacBAACcoarqvx/vOae0AQAAwxI8AADAsAQPAAAwLMEDAAAMS/AAAADDEjwAAMCwBA8AADAswQMAAAxL8AAAAMMSPAAAwLAEDwAAMCzBAwAADEvwAAAAw9qw3hNgDFv3fmu9p3DGevn2T673FAAAzlq+4QEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYJw2eqtpSVd+uqheq6mBVfXYa/7Wq+kFVPTPdfmnumNuq6lBVvVhV182NX1VVz07P3VFVtTofCwAAINmwhH2OJvl8dz9dVT+T5KmqenR67svd/evzO1fVZUl2J7k8yc8m+S9V9Te6++0kdyXZk+S/JXk4yc4kj6zMRwEAAPhJJ/2Gp7sPd/fT0/abSV5IsukEh+xK8kB3v9XdLyU5lGRHVV2S5Pzufry7O8lXkly/3A8AAABwPKf0G56q2prk40memIY+U1V/XFX3VtUF09imJK/OHbYwjW2ato8dX+x99lTVgao6cOTIkVOZIgAAwLuWHDxV9ZEk30jyue7+UWanp/1ckiuTHE7yG+/susjhfYLx9w92393d27t7+8aNG5c6RQAAgJ+wpOCpqg9lFjtf7e5vJkl3v97db3f3j5P8VpId0+4LSbbMHb45yWvT+OZFxgEAAFbFUq7SVknuSfJCd39pbvySud1+Oclz0/ZDSXZX1blVdWmSbUme7O7DSd6sqqun17wxyYMr9DkAAADeZylXabsmyaeTPFtVz0xjX0hyQ1VdmdlpaS8n+dUk6e6DVbU/yfOZXeHt1ukKbUlyS5L7kpyX2dXZXKENAABYNScNnu7+wyz++5uHT3DMviT7Fhk/kOSKU5kgAADA6Tqlq7QBAAB8kAgeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWCcNnqraUlXfrqoXqupgVX12Gv9oVT1aVd+b7i+YO+a2qjpUVS9W1XVz41dV1bPTc3dUVa3OxwIAAFjaNzxHk3y+u38+ydVJbq2qy5LsTfJYd29L8tj0ONNzu5NcnmRnkjur6pzpte5KsifJtum2cwU/CwAAwE84afB09+HufnrafjPJC0k2JdmV5P5pt/uTXD9t70ryQHe/1d0vJTmUZEdVXZLk/O5+vLs7yVfmjgEAAFhxp/QbnqramuTjSZ5IcnF3H05mUZTkomm3TUlenTtsYRrbNG0fO77Y++ypqgNVdeDIkSOnMkUAAIB3LTl4quojSb6R5HPd/aMT7brIWJ9g/P2D3Xd39/bu3r5x48alThEAAOAnLCl4qupDmcXOV7v7m9Pw69Npapnu35jGF5JsmTt8c5LXpvHNi4wDAACsiqVcpa2S3JPkhe7+0txTDyW5adq+KcmDc+O7q+rcqro0s4sTPDmd9vZmVV09veaNc8cAAACsuA1L2OeaJJ9O8mxVPTONfSHJ7Un2V9XNSV5J8qkk6e6DVbU/yfOZXeHt1u5+ezruliT3JTkvySPTDQAAYFWcNHi6+w+z+O9vkuTa4xyzL8m+RcYPJLniVCYIAABwuk7pKm0AAAAfJIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAY1kmDp6rurao3quq5ubFfq6ofVNUz0+2X5p67raoOVdWLVXXd3PhVVfXs9NwdVVUr/3EAAADes5RveO5LsnOR8S9395XT7eEkqarLkuxOcvl0zJ1Vdc60/11J9iTZNt0We00AAIAVc9Lg6e7vJPnhEl9vV5IHuvut7n4pyaEkO6rqkiTnd/fj3d1JvpLk+tOcMwAAwJIs5zc8n6mqP55OebtgGtuU5NW5fRamsU3T9rHji6qqPVV1oKoOHDlyZBlTBAAAzmanGzx3Jfm5JFcmOZzkN6bxxX6X0ycYX1R3393d27t7+8aNG09zigAAwNnutIKnu1/v7re7+8dJfivJjumphSRb5nbdnOS1aXzzIuMAAACr5rSCZ/pNzjt+Ock7V3B7KMnuqjq3qi7N7OIET3b34SRvVtXV09XZbkzy4DLmDQAAcFIbTrZDVX0tySeSXFhVC0m+mOQTVXVlZqelvZzkV5Okuw9W1f4kzyc5muTW7n57eqlbMrvi23lJHpluAAAAq+akwdPdNywyfM8J9t+XZN8i4weSXHFKswMAAFiG5VylDQAA4IwmeAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGGdNHiq6t6qeqOqnpsb+2hVPVpV35vuL5h77raqOlRVL1bVdXPjV1XVs9Nzd1RVrfzHAQAAeM9SvuG5L8nOY8b2Jnmsu7cleWx6nKq6LMnuJJdPx9xZVedMx9yVZE+SbdPt2NcEAABYUScNnu7+TpIfHjO8K8n90/b9Sa6fG3+gu9/q7peSHEqyo6ouSXJ+dz/e3Z3kK3PHAAAArIrT/Q3Pxd19OEmm+4um8U1JXp3bb2Ea2zRtHzsOAACwalb6ogWL/S6nTzC++ItU7amqA1V14MiRIys2OQAA4OxyusHz+nSaWqb7N6bxhSRb5vbbnOS1aXzzIuOL6u67u3t7d2/fuHHjaU4RAAA4251u8DyU5KZp+6YkD86N766qc6vq0swuTvDkdNrbm1V19XR1thvnjgEAAFgVG062Q1V9LcknklxYVQtJvpjk9iT7q+rmJK8k+VSSdPfBqtqf5PkkR5Pc2t1vTy91S2ZXfDsvySPTDQAAYNWcNHi6+4bjPHXtcfbfl2TfIuMHklxxSrMDAABYhpW+aAEAAMAZQ/AAAADDEjwAAMCwBA8AADAswQMAAAxL8AAAAMMSPAAAwLAEDwAAMCzBAwAADEvwAAAAwxI8AADAsAQPAAAwLMEDAAAMS/AAAADDEjwAAMCwBA8AADAswQMAAAxL8AAAAMMSPAAAwLAEDwAAMCzBAwAADEvwAAAAwxI8AADAsAQPAAAwLMEDAAAMS/AAAADDEjwAAMCwBA8AADAswQMAAAxL8AAAAMMSPAAAwLA2rPcEPki27v3Wek8BAAA4Bb7hAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhLSt4qurlqnq2qp6pqgPT2Eer6tGq+t50f8Hc/rdV1aGqerGqrlvu5AEAAE5kJb7h+QfdfWV3b58e703yWHdvS/LY9DhVdVmS3UkuT7IzyZ1Vdc4KvD8AAMCiVuOUtl1J7p+2709y/dz4A939Vne/lORQkh2r8P4AAABJlh88neQPquqpqtozjV3c3YeTZLq/aBrflOTVuWMXpjEAAIBVsWGZx1/T3a9V1UVJHq2qPznBvrXIWC+64yye9iTJxz72sWVOEQAAOFst6xue7n5tun8jye9mdora61V1SZJM929Muy8k2TJ3+OYkrx3nde/u7u3dvX3jxo3LmSIAAHAWO+3gqaoPV9XPvLOd5B8neS7JQ0lumna7KcmD0/ZDSXZX1blVdWmSbUmePN33BwAAOJnlnNJ2cZLfrap3Xuc/dffvVdV3k+yvqpuTvJLkU0nS3Qeran+S55McTXJrd7+9rNkDAACcwGkHT3d/P8kvLDL+P5Nce5xj9iXZd7rvCQAAcCpW47LUAAAAZwTBAwAADEvwAAAAwxI8AADAsAQPAAAwLMEDAAAMS/AAAADDEjwAAMCwBA8AADAswQMAAAxL8AAAAMMSPAAAwLAEDwAAMCzBAwAADEvwAAAAwxI8AADAsAQPAAAwLMEDAAAMS/AAAADDEjwAAMCwNqz3BGB0W/d+a72ncMZ6+fZPrvcUAIDB+YYHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABiW4AEAAIYleAAAgGEJHgAAYFiCBwAAGJbgAQAAhiV4AACAYQkeAABgWIIHAAAYluABAACGJXgAAIBhCR4AAGBYggcAABjWmgdPVe2sqher6lBV7V3r9wcAAM4eaxo8VXVOkt9M8otJLktyQ1VdtpZzAAAAzh4b1vj9diQ51N3fT5KqeiDJriTPr/E8gDPA1r3fWu8pnLFevv2T6z0FABjCWgfPpiSvzj1eSPJ31ngOAGc8MXh8YvD4rJvjs27g7LXWwVOLjPX7dqrak2TP9PD/VNWLqzqr97swyZ+u8XsyDuuH5bB+TqL+zXrP4Ixm/RyHdbNk1hDLsZ7r568e74m1Dp6FJFvmHm9O8tqxO3X33UnuXqtJHauqDnT39vV6fz7YrB+Ww/phOawflssaYjnO1PWz1ldp+26SbVV1aVX9VJLdSR5a4zkAAABniTX9hqe7j1bVZ5L8fpJzktzb3QfXcg4AAMDZY61PaUt3P5zk4bV+31O0bqfTMQTrh+WwflgO64flsoZYjjNy/VT3+64ZAAAAMIS1/g0PAADAmhE8c6pqZ1W9WFWHqmrves+HM0NVbamqb1fVC1V1sKo+O41/tKoerarvTfcXzB1z27SOXqyq6+bGr6qqZ6fn7qiqxS7VzoCq6pyq+qOq+s/TY+uHJamqv1xVX6+qP5n+PfR3rR+Wqqr+1fRn13NV9bWq+ovWDydSVfdW1RtV9dzc2Iqtmao6t6p+Zxp/oqq2rvZnEjyTqjonyW8m+cUklyW5oaouW99ZcYY4muTz3f3zSa5Ocuu0NvYmeay7tyV5bHqc6bndSS5PsjPJndP6SpK7Mvs7prZNt51r+UFYV59N8sLcY+uHpfp3SX6vu/9Wkl/IbB1ZP5xUVW1K8i+TbO/uKzK7YNTuWD+c2H15/z/flVwzNyf5X93915N8Ocmq/y1Zguc9O5Ic6u7vd/efJ3kgya51nhNngO4+3N1PT9tvZvYfG5syWx/3T7vdn+T6aXtXkge6+63ufinJoSQ7quqSJOd39+M9+/HcV+aOYWBVtTnJJ5P89tyw9cNJVdX5Sf5+knuSpLv/vLv/d6wflm5DkvOqakOSn87s7z+0fjiu7v5Okh8eM7ySa2b+tb6e5NrV/sZQ8LxnU5JX5x4vTGPwrulr148neSLJxd19OJlFUZKLpt2Ot5Y2TdvHjjO+f5vkXyf58dyY9cNS/LUkR5L8++mUyN+uqg/H+mEJuvsHSX49yStJDif5s+7+g1g/nLqVXDPvHtPdR5P8WZK/smozj+CZt1hZuoQd76qqjyT5RpLPdfePTrTrImN9gnEGVlX/JMkb3f3UUg9ZZMz6OXttSPK3k9zV3R9P8n8znUpyHNYP75p+Z7EryaVJfjbJh6vqV050yCJj1g8ncjprZs3Xk+B5z0KSLXOPN2f2tS+kqj6UWex8tbu/OQ2/Pn1lm+n+jWn8eGtpYdo+dpyxXZPkn1bVy5mdKvsPq+o/xvphaRaSLHT3E9Pjr2cWQNYPS/GPkrzU3Ue6+/8l+WaSvxfrh1O3kmvm3WOmUy3/Ut5/Ct2KEjzv+W6SbVV1aVX9VGY/wHponefEGWA6r/SeJC9095fmnnooyU3T9k1JHpwb3z1dheTSzH6o9+T0FfCbVXX19Jo3zh3DoLr7tu7e3N1bM/v3yn/t7l+J9cMSdPf/SPJqVf3NaejaJM/H+mFpXklydVX99PTP/drMfodq/XCqVnLNzL/WP8vsz8VV/YZnw2q++AdJdx+tqs8k+f3MrmJyb3cfXOdpcWa4JsmnkzxbVc9MY19IcnuS/VV1c2Z/qHwqSbr7YFXtz+w/So4mubW7356OuyWzq5+cl+SR6cbZyfphqf5Fkq9O/zPu+0n+eWb/w9L64YS6+4mq+nqSpzNbD3+U5O4kH4n1w3FU1deSfCLJhVW1kOSLWdk/s+5J8h+q6lBm3+zsXvXPtMpBBQAAsG6c0gYAAAxL8AAAAMMSPAAAwLAEDwAAMCzBAwAADEvwAAAAwxI8AADAsAQPAAAwrP8PW6OQAB70qwgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(14, 6))\n",
    "ax.hist([len(passage) for passage in passages]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "XqdVfEN4WKMI"
   },
   "outputs": [],
   "source": [
    "def generate_questions(\n",
    "    tokenizer: AutoTokenizer, model: AutoModelForSeq2SeqLM, passages: list[str],\n",
    "    batch_size: int=32, device: str=\"cuda\", n_ques_per_passage: int = 3\n",
    ") -> list[str]:\n",
    "\n",
    "    outputs = []\n",
    "    n_batches = len(passages) // batch_size + int(len(passages) % batch_size != 0)\n",
    "\n",
    "    for n in tqdm(range(n_batches)):\n",
    "        passages_batch = passages[n*batch_size: (n+1)*batch_size]\n",
    "        inputs = tokenizer(passages_batch, padding=True, truncation=True,\n",
    "                           max_length=512, return_tensors=\"pt\")\n",
    "        output = model.generate(\n",
    "            input_ids=inputs['input_ids'].to(device),\n",
    "            attention_mask=inputs['attention_mask'].to(device),\n",
    "            max_length=64,\n",
    "            do_sample=True,\n",
    "            top_p=0.9,\n",
    "            num_return_sequences=n_ques_per_passage\n",
    "        )\n",
    "        outputs += output\n",
    "\n",
    "    questions = [tokenizer.decode(txt, skip_special_tokens=True) for txt in outputs]\n",
    "    return questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"doc2query/msmarco-t5-base-v1\"\n",
    "\n",
    "doc2q_tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "doc2q_model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rLzUV_uQXeuu",
    "outputId": "a4b6be41-58f2-4e60-957c-9b06bd1493c5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████| 114/114 [01:41<00:00,  1.12it/s]\n"
     ]
    }
   ],
   "source": [
    "n_ques_per_passage = 3\n",
    "questions = generate_questions(\n",
    "    doc2q_tokenizer, doc2q_model, passages, n_ques_per_passage=n_ques_per_passage\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "Wne71Gd_l-wp"
   },
   "outputs": [],
   "source": [
    "q2pix = {}  # query to passage index mapping\n",
    "\n",
    "pix = 0\n",
    "for i in range(len(questions)):\n",
    "    if i > 0 and i % n_ques_per_passage == 0:\n",
    "        pix += 1\n",
    "    q2pix[questions[i]] = pix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EBpvr_9em7q3",
    "outputId": "4818c7ae-51dd-4f85-ed79-32671f7c376f"
   },
   "source": [
    "Let's look at some passages and the generated questions ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recent studies have suggested that statins, an established drug group in the prevention of cardiovascular mortality, could delay or prevent breast cancer recurrence but the effect on disease-specific mortality remains unclear. We evaluated risk of breast cancer death among statin users in a population-based cohort of breast cancer patients. The study cohort included all newly diagnosed breast cancer patients in Finland during 1995–2003 (31,236 cases), identified from the Finnish Cancer Registry. Information on statin use before and after the diagnosis was obtained from a national prescription database. We used the Cox proportional hazards regression method to estimate mortality among statin users with statin use as time-dependent variable. A total of 4,151 participants had used statins. During the median follow-up of 3.25 years after the diagnosis (range 0.08–9.0 years) 6,011 participants died, of which 3,619 (60.2%) was due to breast cancer. After adjustment for age, tumor characteristics, and treatment selection, both post-diagnostic and pre-diagnostic statin use were associated with lowered risk of breast cancer death (HR 0.46, 95% CI 0.38–0.55 and HR 0.54, 95% CI 0.44–0.67, respectively). The risk decrease by post-diagnostic statin use was likely affected by healthy adherer bias; that is, the greater likelihood of dying cancer patients to discontinue statin use as the association was not clearly dose-dependent and observed already at low-dose/short-term use. The dose- and time-dependence of the survival benefit among pre-diagnostic statin users suggests a possible causal effect that should be evaluated further in a clinical trial testing statins’ effect on survival in breast cancer patients.\n",
      "[Q] do statins lower mortality ?\n",
      "---\n",
      "\n",
      "Recent studies have suggested that statins, an established drug group in the prevention of cardiovascular mortality, could delay or prevent breast cancer recurrence but the effect on disease-specific mortality remains unclear. We evaluated risk of breast cancer death among statin users in a population-based cohort of breast cancer patients. The study cohort included all newly diagnosed breast cancer patients in Finland during 1995–2003 (31,236 cases), identified from the Finnish Cancer Registry. Information on statin use before and after the diagnosis was obtained from a national prescription database. We used the Cox proportional hazards regression method to estimate mortality among statin users with statin use as time-dependent variable. A total of 4,151 participants had used statins. During the median follow-up of 3.25 years after the diagnosis (range 0.08–9.0 years) 6,011 participants died, of which 3,619 (60.2%) was due to breast cancer. After adjustment for age, tumor characteristics, and treatment selection, both post-diagnostic and pre-diagnostic statin use were associated with lowered risk of breast cancer death (HR 0.46, 95% CI 0.38–0.55 and HR 0.54, 95% CI 0.44–0.67, respectively). The risk decrease by post-diagnostic statin use was likely affected by healthy adherer bias; that is, the greater likelihood of dying cancer patients to discontinue statin use as the association was not clearly dose-dependent and observed already at low-dose/short-term use. The dose- and time-dependence of the survival benefit among pre-diagnostic statin users suggests a possible causal effect that should be evaluated further in a clinical trial testing statins’ effect on survival in breast cancer patients.\n",
      "[Q] the effect of statins on breast cancer ?\n",
      "---\n",
      "\n",
      "Recent studies have suggested that statins, an established drug group in the prevention of cardiovascular mortality, could delay or prevent breast cancer recurrence but the effect on disease-specific mortality remains unclear. We evaluated risk of breast cancer death among statin users in a population-based cohort of breast cancer patients. The study cohort included all newly diagnosed breast cancer patients in Finland during 1995–2003 (31,236 cases), identified from the Finnish Cancer Registry. Information on statin use before and after the diagnosis was obtained from a national prescription database. We used the Cox proportional hazards regression method to estimate mortality among statin users with statin use as time-dependent variable. A total of 4,151 participants had used statins. During the median follow-up of 3.25 years after the diagnosis (range 0.08–9.0 years) 6,011 participants died, of which 3,619 (60.2%) was due to breast cancer. After adjustment for age, tumor characteristics, and treatment selection, both post-diagnostic and pre-diagnostic statin use were associated with lowered risk of breast cancer death (HR 0.46, 95% CI 0.38–0.55 and HR 0.54, 95% CI 0.44–0.67, respectively). The risk decrease by post-diagnostic statin use was likely affected by healthy adherer bias; that is, the greater likelihood of dying cancer patients to discontinue statin use as the association was not clearly dose-dependent and observed already at low-dose/short-term use. The dose- and time-dependence of the survival benefit among pre-diagnostic statin users suggests a possible causal effect that should be evaluated further in a clinical trial testing statins’ effect on survival in breast cancer patients.\n",
      "[Q] how does statin affect mortality ?\n",
      "---\n",
      "\n",
      "BACKGROUND: Preclinical studies have shown that statins, particularly simvastatin, can prevent growth in breast cancer cell lines and animal models. We investigated whether statins used after breast cancer diagnosis reduced the risk of breast cancer-specific, or all-cause, mortality in a large cohort of breast cancer patients. METHODS: A cohort of 17,880 breast cancer patients, newly diagnosed between 1998 and 2009, was identified from English cancer registries (from the National Cancer Data Repository). This cohort was linked to the UK Clinical Practice Research Datalink, providing prescription records, and to the Office of National Statistics mortality data (up to 2013), identifying 3694 deaths, including 1469 deaths attributable to breast cancer. Unadjusted and adjusted hazard ratios (HRs) for breast cancer-specific, and all-cause, mortality in statin users after breast cancer diagnosis were calculated using time-dependent Cox regression models. Sensitivity analyses were conducted using multiple imputation methods, propensity score methods and a case-control approach. RESULTS: There was some evidence that statin use after a diagnosis of breast cancer had reduced mortality due to breast cancer and all causes (fully adjusted HR = 0.84 [95% confidence interval = 0.68-1.04] and 0.84 [0.72-0.97], respectively). These associations were more marked for simvastatin 0.79 (0.63-1.00) and 0.81 (0.70-0.95), respectively. CONCLUSIONS: In this large population-based breast cancer cohort, there was some evidence of reduced mortality in statin users after breast cancer diagnosis. However, these associations were weak in magnitude and were attenuated in some sensitivity analyses.\n",
      "[Q] when to use statins after breast cancer diagnosis ?\n",
      "---\n",
      "\n",
      "BACKGROUND: Preclinical studies have shown that statins, particularly simvastatin, can prevent growth in breast cancer cell lines and animal models. We investigated whether statins used after breast cancer diagnosis reduced the risk of breast cancer-specific, or all-cause, mortality in a large cohort of breast cancer patients. METHODS: A cohort of 17,880 breast cancer patients, newly diagnosed between 1998 and 2009, was identified from English cancer registries (from the National Cancer Data Repository). This cohort was linked to the UK Clinical Practice Research Datalink, providing prescription records, and to the Office of National Statistics mortality data (up to 2013), identifying 3694 deaths, including 1469 deaths attributable to breast cancer. Unadjusted and adjusted hazard ratios (HRs) for breast cancer-specific, and all-cause, mortality in statin users after breast cancer diagnosis were calculated using time-dependent Cox regression models. Sensitivity analyses were conducted using multiple imputation methods, propensity score methods and a case-control approach. RESULTS: There was some evidence that statin use after a diagnosis of breast cancer had reduced mortality due to breast cancer and all causes (fully adjusted HR = 0.84 [95% confidence interval = 0.68-1.04] and 0.84 [0.72-0.97], respectively). These associations were more marked for simvastatin 0.79 (0.63-1.00) and 0.81 (0.70-0.95), respectively. CONCLUSIONS: In this large population-based breast cancer cohort, there was some evidence of reduced mortality in statin users after breast cancer diagnosis. However, these associations were weak in magnitude and were attenuated in some sensitivity analyses.\n",
      "[Q] what are statins used for ?\n",
      "---\n",
      "\n",
      "BACKGROUND: Preclinical studies have shown that statins, particularly simvastatin, can prevent growth in breast cancer cell lines and animal models. We investigated whether statins used after breast cancer diagnosis reduced the risk of breast cancer-specific, or all-cause, mortality in a large cohort of breast cancer patients. METHODS: A cohort of 17,880 breast cancer patients, newly diagnosed between 1998 and 2009, was identified from English cancer registries (from the National Cancer Data Repository). This cohort was linked to the UK Clinical Practice Research Datalink, providing prescription records, and to the Office of National Statistics mortality data (up to 2013), identifying 3694 deaths, including 1469 deaths attributable to breast cancer. Unadjusted and adjusted hazard ratios (HRs) for breast cancer-specific, and all-cause, mortality in statin users after breast cancer diagnosis were calculated using time-dependent Cox regression models. Sensitivity analyses were conducted using multiple imputation methods, propensity score methods and a case-control approach. RESULTS: There was some evidence that statin use after a diagnosis of breast cancer had reduced mortality due to breast cancer and all causes (fully adjusted HR = 0.84 [95% confidence interval = 0.68-1.04] and 0.84 [0.72-0.97], respectively). These associations were more marked for simvastatin 0.79 (0.63-1.00) and 0.81 (0.70-0.95), respectively. CONCLUSIONS: In this large population-based breast cancer cohort, there was some evidence of reduced mortality in statin users after breast cancer diagnosis. However, these associations were weak in magnitude and were attenuated in some sensitivity analyses.\n",
      "[Q] how do statins work in humans ?\n",
      "---\n",
      "\n",
      "The aims of this study were to determine the concentrations of 4-nonylphenol (NP) and 4-octylphenol (OP) in 59 human milk samples and to examine related factors including mothers' demographics and dietary habits. Women who consumed over the median amount of cooking oil had significantly higher OP concentrations (0.98 ng/g) than those who consumed less (0.39 ng/g) (P < 0.05). OP concentration was significantly associated with the consumption of cooking oil (beta = 0.62, P < 0.01) and fish oil capsules (beta = 0.39, P < 0.01) after adjustment for age and body mass index (BMI). NP concentration was also significantly associated with the consumption of fish oil capsules (beta = 0.38, P < 0.01) and processed fish products (beta = 0.59, P < 0.01). The food pattern of cooking oil and processed meat products from factor analysis was strongly associated with OP concentration in human milk (P < 0.05). These determinations should aid in suggesting foods for consumption by nursing mothers in order to protect their infants from NP/OP exposure. 2010 Elsevier Ltd. All rights reserved.\n",
      "[Q] what is op in human milk ?\n",
      "---\n",
      "\n",
      "The aims of this study were to determine the concentrations of 4-nonylphenol (NP) and 4-octylphenol (OP) in 59 human milk samples and to examine related factors including mothers' demographics and dietary habits. Women who consumed over the median amount of cooking oil had significantly higher OP concentrations (0.98 ng/g) than those who consumed less (0.39 ng/g) (P < 0.05). OP concentration was significantly associated with the consumption of cooking oil (beta = 0.62, P < 0.01) and fish oil capsules (beta = 0.39, P < 0.01) after adjustment for age and body mass index (BMI). NP concentration was also significantly associated with the consumption of fish oil capsules (beta = 0.38, P < 0.01) and processed fish products (beta = 0.59, P < 0.01). The food pattern of cooking oil and processed meat products from factor analysis was strongly associated with OP concentration in human milk (P < 0.05). These determinations should aid in suggesting foods for consumption by nursing mothers in order to protect their infants from NP/OP exposure. 2010 Elsevier Ltd. All rights reserved.\n",
      "[Q] what is the concentration of op in human milk ?\n",
      "---\n",
      "\n",
      "The aims of this study were to determine the concentrations of 4-nonylphenol (NP) and 4-octylphenol (OP) in 59 human milk samples and to examine related factors including mothers' demographics and dietary habits. Women who consumed over the median amount of cooking oil had significantly higher OP concentrations (0.98 ng/g) than those who consumed less (0.39 ng/g) (P < 0.05). OP concentration was significantly associated with the consumption of cooking oil (beta = 0.62, P < 0.01) and fish oil capsules (beta = 0.39, P < 0.01) after adjustment for age and body mass index (BMI). NP concentration was also significantly associated with the consumption of fish oil capsules (beta = 0.38, P < 0.01) and processed fish products (beta = 0.59, P < 0.01). The food pattern of cooking oil and processed meat products from factor analysis was strongly associated with OP concentration in human milk (P < 0.05). These determinations should aid in suggesting foods for consumption by nursing mothers in order to protect their infants from NP/OP exposure. 2010 Elsevier Ltd. All rights reserved.\n",
      "[Q] op content in milk ?\n",
      "---\n",
      "\n"
     ]
    }
   ],
   "source": [
    "n = 0\n",
    "\n",
    "for question, passage_ix in q2pix.items():\n",
    "    print(passages[passage_ix])\n",
    "    print(f\"[Q] {question} ?\\n---\\n\")\n",
    "    n += 1\n",
    "    if n == (n_ques_per_passage * 3):\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CmKwZH_wfg44",
    "outputId": "9ad868a4-3d61-4afb-9c73-8b500916d257"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10692"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_questions = list(set(questions))\n",
    "len(unique_questions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Negative mining "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "d7FZxHDBZOgU"
   },
   "outputs": [],
   "source": [
    "retriever = SentenceTransformer(\"msmarco-distilbert-base-tas-b\", device=\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "Q3pNxq5_Z8UD"
   },
   "outputs": [],
   "source": [
    "passage_embeddings = retriever.encode(passages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "pkaLmCWQacEj"
   },
   "outputs": [],
   "source": [
    "def get_topk_passages(\n",
    "    model: SentenceTransformer, query: str, passage_embeds: torch.tensor, n: int=10\n",
    ") -> tuple[np.array, list[float]]:\n",
    "    query_embed = model.encode(query)\n",
    "    sim_scores = util.cos_sim(query_embed, passage_embeds).numpy()\n",
    "    top_passages_ix = np.argsort(sim_scores)[0][::-1][:n]\n",
    "    return top_passages_ix, [sim_scores[0][ix] for ix in top_passages_ix]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "8IGv4OgNe-cO"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████| 10692/10692 [01:10<00:00, 150.91it/s]\n"
     ]
    }
   ],
   "source": [
    "query_passage_pairs = []\n",
    "\n",
    "for query in tqdm(unique_questions):\n",
    "    top_passages_ixs, _ = get_topk_passages(retriever, query, passage_embeddings, 3)\n",
    "    query_passage_pairs += [(query, passages[ix]) for ix in top_passages_ixs if ix != q2pix[query]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tR-ZgiyenM3N",
    "outputId": "c7368c90-222d-4bbb-e50a-be7714a343c2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25082"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(query_passage_pairs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YtOajkA5pY7z",
    "outputId": "d440c2c6-25b4-4d3f-9a4c-30e4245b599f"
   },
   "source": [
    "Let's look at some query, positive and negative passages ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Q] is rapamycin a fungi\n",
      "\n",
      "[NEGATIVE] Proteins regulating the mammalian target of rapamycin (mTOR), as well as some of the targets of the mTOR kinase, are overexpressed or mutated in cancer. Rapamycin, the naturally occurring inhibitor of mTOR, along with a number of recently developed rapamycin analogs (rapalogs) consisting of synthetically derived compounds containing minor chemical modifications to the parent structure, inhibit the growth of cell lines derived from multiple tumor types in vitro, and tumor models in vivo. Results from clinical trials indicate that the rapalogs may be useful for the treatment of subsets of certain types of cancer. The sporadic responses from the initial clinical trials, based on the hypothesis of general translation inhibition of cancer cells are now beginning to be understood owing to a more complete understanding of the dynamics of mTOR regulation and the function of mTOR in the tumor microenvironment. This review will summarize the preclinical and clinical data and recent discoveries of the function of mTOR in cancer and growth regulation.\n",
      "\n",
      "[POSITIVE] A streptomycete was isolated from an Easter Island soil sample and found to inhibit Candida albicans, Microsporum gypseum and Trichophyton granulosum. The antibiotic-producing microorganism was characterized and identified as Streptomyces hygroscopicus. The antifungal principle was extracted with organic solvent from the mycelium, isolated in crystalline form and named rapamycin. Rapamycin is mainly active against Candida albicans; minimum inhibitory concentration against ten strains ranged from 0.02 to 0.2 mug/ml. Its apparent activity against Microsporum gypseum and Trichophyton granulosum is lower because of its instability in culture media on prolonged incubation required by these fungi. No activity was observed against gram-positive and gram-negative bacteria. Acute toxicity in mice is low.\n",
      "---\n",
      "\n",
      "[Q] what are the results of having vegetarians\n",
      "\n",
      "[NEGATIVE] A number of studies have evaluated the health of vegetarians. Others have studied the health effects of foods that are preferred or avoided by vegetarians. The purpose of this review is to look critically at the evidence on the health effects of vegetarian diets and to seek possible explanations where results appear to conflict. There is convincing evidence that vegetarians have lower rates of coronary heart disease, largely explained by low LDL cholesterol, probable lower rates of hypertension and diabetes mellitus, and lower prevalence of obesity. Overall, their cancer rates appear to be moderately lower than others living in the same communities, and life expectancy appears to be greater. However, results for specific cancers are much less convincing and require more study. There is evidence that risk of colorectal cancer is lower in vegetarians and in those who eat less meat; however, results from British vegetarians presently disagree, and this needs explanation. It is probable that using the label “vegetarian” as a dietary category is too broad and that our understanding will be served well by dividing vegetarians into more descriptive subtypes. Although vegetarian diets are healthful and are associated with lower risk of several chronic diseases, different types of vegetarians may not experience the same effects on health.\n",
      "\n",
      "[POSITIVE] We studied 10 vegetarian and 10 nonvegetarian premenopausal women on four occasions approximately four months apart. During each study period, the participants kept three-day dietary records, and estrogens were measured in plasma, urinary, and fecal samples. Vegetarians consumed less total fat than omnivores did (30 per cent of total calories, as compared with 40 per cent) and more dietary fiber (28 g per day, as compared with 12 g). There was a positive correlation between fecal weight and fecal excretion of estrogens in both groups (P less than 0.001), with vegetarians having higher fecal weight and increased fecal excretion of estrogens. Urinary excretion of estriol was lower in vegetarians (P less than 0.05), and their plasma levels of estrone and estradiol were negatively correlated with fecal excretion of estrogen (P = 0.005). Among the vegetarians the beta-glucuronidase activity of fecal bacteria was significantly reduced (P = 0.05). We conclude that vegetarian women have an increased fecal output, which leads to increased fecal excretion of estrogen and a decreased plasma concentration of estrogen.\n",
      "---\n",
      "\n",
      "[Q] what is the purpose of a nutrition intervention\n",
      "\n",
      "[NEGATIVE] Nutrient profiling of foods, described as the science of ranking foods based on their nutrient content, is fast becoming the basis for regulating nutrition labels, health claims, and marketing and advertising to children. A number of nutrient profile models have now been developed by research scientists, regulatory agencies, and by the food industry. Whereas some of these models have focused on nutrients to limit, others have emphasized nutrients known to be beneficial to health, or some combination of both. Although nutrient profile models are often tailored to specific goals, the development process ought to follow the same science-driven rules. These include the selection of index nutrients and reference amounts, the development of an appropriate algorithm for calculating nutrient density, and the validation of the chosen nutrient profile model against healthy diets. It is extremely important that nutrient profiles be validated rather than merely compared to prevailing public opinion. Regulatory agencies should act only when they are satisfied that the scientific process has been followed, that the algorithms are transparent, and that the profile model has been validated with respect to objective measures of a healthy diet.\n",
      "\n",
      "[POSITIVE] PURPOSE: To determine whether a multicomponent nutrition intervention program at a corporate site reduces body weight and improves other cardiovascular risk factors in overweight individuals. DESIGN: Prospective clinical intervention study. SUBJECTS/SETTING: Employees of the Government Employees Insurance Company (GEICO) (N = 113), aged 21 to 65 years, with a body mass index > or =25 kg/m(2) and/or previous diagnosis of type 2 diabetes. INTERVENTION: A 22-week intervention including a low-fat, vegan diet. MEASURES: Changes in body weight, anthropometric measures, blood pressure, lipid profile, and dietary intake. ANALYSIS: Multivariate analyses of variance were calculated for clinical and nutrient measures, followed by univariate analyses of variance, to determine the significance of differences between groups in changes over time. RESULTS: Intervention-group participants experienced greater weight changes compared with control-group participants (mean, -5.1 [SE, .6] kg vs. + .1 [SE, .6] kg, p < .0001), as well as greater changes in waist circumference (mean, -4.7 [SE, .6] cm vs. + .8 [SE, .6] cm, p < .0001) and waistratiohip ratio (mean, -.006 [SE, .003] vs. + .014 [SE, .005], p = .0007). Weight loss of 5% of body weight was more frequently observed in the intervention group (48.5%) compared with the control group (11.1%) (chi(2)[1, N = 113] = 16.99, p < .0001). CONCLUSIONS: Among individuals volunteering for a 22-week worksite research study, an intervention using a low-fat, vegan diet effectively reduced body weight and waist circumference.\n",
      "---\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for query, negative_passage in random.sample(query_passage_pairs, 3):\n",
    "    print(f\"[Q] {query}\\n\\n[NEGATIVE] {negative_passage}\\n\\n[POSITIVE] {passages[q2pix[query]]}\\n---\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "FWCDNc3fsC9t"
   },
   "outputs": [],
   "source": [
    "ce = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2', device=\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zl_OGGV4sQJu",
    "outputId": "8cc161a9-e98f-4dbb-de93-7a7821080100"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████| 25082/25082 [03:26<00:00, 121.70it/s]\n"
     ]
    }
   ],
   "source": [
    "query_passage_margins = []\n",
    "\n",
    "for query, passage in tqdm(query_passage_pairs):\n",
    "    p_score = ce.predict((query, passages[q2pix[query]]))\n",
    "    n_score = ce.predict((query, passage))\n",
    "    margin = p_score - n_score\n",
    "    query_passage_margins.append((query, passages[q2pix[query]], passage, margin))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, I save the data, restart the kernel to free up the GPU memory as I kept running into OOM.\n",
    "\n",
    "> If you face the same issue, you'll need to \n",
    "> - load the libraries\n",
    "> - load the the dataset\n",
    "> - create the passages\n",
    "> - load the retriever\n",
    "> - redefine `get_topk_passages()`\n",
    "> - create the passage embeddings we created using the retriever\n",
    ">\n",
    "> We will use the passage embeddings to compare the retrieval with the finetuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"query_passage_margins.pkl\", \"wb\") as f:\n",
    "    pickle.dump(query_passage_margins, f)\n",
    "\n",
    "#  restart kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"query_passage_margins.pkl\", \"rb\") as f:\n",
    "    query_passage_margins = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "irFNZW5lxZY6"
   },
   "outputs": [],
   "source": [
    "training_data = []\n",
    "for query, passage_pos, passage_neg, margin in query_passage_margins:\n",
    "    training_data.append(InputExample(texts=[query, passage_pos, passage_neg], label=float(margin)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "rdUTuswOx6oi"
   },
   "outputs": [],
   "source": [
    "loader = torch.utils.data.DataLoader(training_data, batch_size=8, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finetuning the Bi-Encoder "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "vBiCO_KGyL6m"
   },
   "outputs": [],
   "source": [
    "bi_encoder = SentenceTransformer(\"msmarco-distilbert-base-tas-b\", device=\"cuda\")\n",
    "bi_encoder.max_seq_length = 512"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "oaw44LgyybVQ"
   },
   "outputs": [],
   "source": [
    "loss = losses.MarginMSELoss(bi_encoder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81,
     "referenced_widgets": [
      "2846925295294d819a62655c784df8c2",
      "ce0536884a594ddca425c0c3862caea0",
      "2de210f3638a44448fd5ad3c4c24c7dd",
      "546f2c471c6247fd94d1dccb8db919f1",
      "87e0a0e7bc0e4dd59163b5cb596b3add",
      "df2a51cde40a46c9bad49d46315f9246",
      "8be3d3f1da664bada9646012b1690117",
      "3904769c197942c793a5d60326637504",
      "55fef994c7de4cf7aa038581130ae9d2",
      "3959def343f2412ca41c9d19ba843bb4",
      "4c2f7f623d424af2b1b4f5d5e8d5653a",
      "962de3077c54405baf3238e223c79342",
      "c4f0a26b52924e4886fa44ea6e57b9a8",
      "4f7c1f51f14e46c2bdebc89130bc0f32",
      "56a4c8c9ec1a4029b056950a48adfddd",
      "81e92ba9faff4ab1bcfe64ba3df96328",
      "1f573fe5b8054ab6b4467d7154ff7a69",
      "e18574af80d94d85a1920fd2bc138566",
      "507f63350c004bbe88794c61f17d435a",
      "5ffa8f28ebe0436884c451ca8fbb3b19",
      "e3810172078e47149dabea209a690e29",
      "0b6da3ef076841f8bde62dfc8d343cb6"
     ]
    },
    "id": "NrqEXGToyjZ_",
    "outputId": "9114c1dd-6989-4959-836e-dbed9ba4b90b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b1860b262ddd46fea97617dfb6be0e20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "621b309e2758474596238ceaf98dbbff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/3136 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc05c9e30cbb4d189cc9314b7a66fad2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/3136 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bi_encoder.fit(\n",
    "    train_objectives=[(loader, loss)],\n",
    "    epochs=2,\n",
    "    warmup_steps=int(len(loader)*0.1),\n",
    "    show_progress_bar=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_passage_embeddings = bi_encoder.encode(passages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pLKgYBwzzeAg"
   },
   "source": [
    "For a proper evaluation, we will need some labeled data from our domain. Here I've cherry picked some examples to show how the in-domain fintetuned retriever is doing better than the general purpose finetuned Bi-Encoder we started with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q6x_wKjm1Z4U"
   },
   "source": [
    "**Passage**<br>\n",
    "BACKGROUND: Oral L-citrulline is efficiently converted to L-arginine, the precursor for endothelial nitric oxide (NO) synthesis. Oral L-arginine supplementation reduces brachial blood pressure (BP). We evaluated the effects of watermelon supplementation on aortic BP and arterial function in individuals with prehypertension. METHODS: Heart rate (HR), brachial systolic BP (bSBP), brachial pulse pressure (bPP), aortic SBP (aSBP), aortic PP (aPP), augmentation index (AIx), AIx adjusted for HR of 75 beats/min (AIx@75), amplitude of the first (P1) and second (P2) systolic peaks, reflection time (Tr), and carotid-femoral pulse wave velocity (PWV) were evaluated in the supine position in nine subjects (four men/five women, age 54 ± 3 years) with prehypertension (134/77 ± 5/3 mm Hg). Subjects were randomly assigned to 6 weeks of watermelon supplementation (L-citrulline/L arginine, 2.7 g/1.3 g/day) or placebo followed by a 4-week washout period and then crossover. RESULTS: There was a significant treatment effect (change in the value of watermelon minus placebo from baseline to 6 weeks) on bPP (-8 ± 3 mm Hg, P < 0.05), aSBP (-7 ± 2 mm Hg, P < 0.05), aPP (-6 ± 2 mm Hg, P < 0.01), AIx (-6 ± 3%, P < 0.05), AIx@75 (-4 ± 2%, P < 0.05), and P2 (-2 ± 1 mm Hg, P < 0.05). There was no significant treatment effect (P > 0.05) on bSBP, brachial diastolic BP (DBP), aortic DBP, Tr, P1, HR, and carotid-femoral PWV. CONCLUSIONS: This pilot study shows that watermelon supplementation improves aortic hemodynamics through a decrease in the amplitude of the reflected wave in individuals with prehypertension. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bi6Fzsbk1ct4",
    "outputId": "3723d993-3910-42ae-e37c-666b9709aaa5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.84084153 OBJECTIVE: Watermelon is a rich source of citrulline, an amino acid that can be metabolized to arginine, a conditionally essential amino acid for humans. Arginine is the nitrogenous substrate used in the synthesis of nitric oxide and plays an essential role in cardiovascular and immune functions. No detailed studies have been conducted to evaluate plasma arginine response in humans after long-term feeding of citrulline from natural plant sources. This study investigated if watermelon juice consumption increases fasting concentrations of plasma arginine, ornithine, and citrulline in healthy adult humans. METHODS: Subjects (n = 12-23/treatment) consumed a controlled diet and 0 (control), 780, or 1560 g of watermelon juice per day for 3 wk in a crossover design. The treatments provided 1 and 2 g of citrulline per day. Treatment periods were preceded by washout periods of 2 to 4 wk. RESULTS: Compared with the baseline, fasting plasma arginine concentrations increased 12% after 3 wk of the lower-dose watermelon treatment; arginine and ornithine concentrations increased 22% and 18%, respectively, after 3 wk of the higher-dose watermelon treatment. Fasting citrulline concentrations did not increase relative to the control but remained stable throughout the study. CONCLUSION: The increased fasting plasma concentrations of arginine and ornithine and stable concentrations of plasma citrulline in response to watermelon juice consumption indicated that the citrulline from this plant origin was effectively converted into arginine. These results demonstrate that plasma concentration of arginine can be increased through intake of citrulline from watermelon. \n",
      "\n",
      "0.8386627 BACKGROUND: Oral L-citrulline is efficiently converted to L-arginine, the precursor for endothelial nitric oxide (NO) synthesis. Oral L-arginine supplementation reduces brachial blood pressure (BP). We evaluated the effects of watermelon supplementation on aortic BP and arterial function in individuals with prehypertension. METHODS: Heart rate (HR), brachial systolic BP (bSBP), brachial pulse pressure (bPP), aortic SBP (aSBP), aortic PP (aPP), augmentation index (AIx), AIx adjusted for HR of 75 beats/min (AIx@75), amplitude of the first (P1) and second (P2) systolic peaks, reflection time (Tr), and carotid-femoral pulse wave velocity (PWV) were evaluated in the supine position in nine subjects (four men/five women, age 54 ± 3 years) with prehypertension (134/77 ± 5/3 mm Hg). Subjects were randomly assigned to 6 weeks of watermelon supplementation (L-citrulline/L arginine, 2.7 g/1.3 g/day) or placebo followed by a 4-week washout period and then crossover. RESULTS: There was a significant treatment effect (change in the value of watermelon minus placebo from baseline to 6 weeks) on bPP (-8 ± 3 mm Hg, P < 0.05), aSBP (-7 ± 2 mm Hg, P < 0.05), aPP (-6 ± 2 mm Hg, P < 0.01), AIx (-6 ± 3%, P < 0.05), AIx@75 (-4 ± 2%, P < 0.05), and P2 (-2 ± 1 mm Hg, P < 0.05). There was no significant treatment effect (P > 0.05) on bSBP, brachial diastolic BP (DBP), aortic DBP, Tr, P1, HR, and carotid-femoral PWV. CONCLUSIONS: This pilot study shows that watermelon supplementation improves aortic hemodynamics through a decrease in the amplitude of the reflected wave in individuals with prehypertension. \n",
      "\n",
      "0.8262954 l-Citrulline is an excellent candidate to reduce muscle soreness, and watermelon is a fruit rich in this amino acid. This study investigated the potential of watermelon juice as a functional drink for athletes. An in vitro study of intestinal absorption of l-citrulline in Caco-2 cells was performed using unpasteurized (NW), pasteurized (80 °C for 40 s) watermelon juice (PW) and, as control, a standard of l-citrulline. l-citrulline bioavailability was greater when it was contained in a matrix of watermelon and when no heat treatment was applied. In the in vivo experiment (maximum effort test in a cycloergometer), seven athletes were supplied with 500 mL of natural watermelon juice (1.17 g of l-citrulline), enriched watermelon juice (4.83 g of l-citrulline plus 1.17 g from watermelon), and placebo. Both watermelon juices helped to reduce the recovery heart rate and muscle soreness after 24 h. \n",
      "\n",
      "0.82332575 Pharmacological antioxidant vitamins have previously been investigated for a prophylactic effect against exercise-induced oxidative stress. However, large doses are often required and may lead to a state of pro-oxidation and oxidative damage. Watercress contains an array of nutritional compounds such as β-carotene and α-tocopherol which may increase protection against exercise-induced oxidative stress. The present randomised controlled investigation was designed to test the hypothesis that acute (consumption 2 h before exercise) and chronic (8 weeks consumption) watercress supplementation can attenuate exercise-induced oxidative stress. A total of ten apparently healthy male subjects (age 23 (SD 4) years, stature 179 (SD 10) cm and body mass 74 (SD 15) kg) were recruited to complete the 8-week chronic watercress intervention period (and then 8 weeks of control, with no ingestion) of the experiment before crossing over in order to compete the single-dose acute phase (with control, no ingestion). Blood samples were taken at baseline (pre-supplementation), at rest (pre-exercise) and following exercise. Each subject completed an incremental exercise test to volitional exhaustion following chronic and acute watercress supplementation or control. The main findings show an exercise-induced increase in DNA damage and lipid peroxidation over both acute and chronic control supplementation phases (P< 0.05 v. supplementation), while acute and chronic watercress attenuated DNA damage and lipid peroxidation and decreased H₂O₂ accumulation following exhaustive exercise (P< 0.05 v. control). A marked increase in the main lipid-soluble antioxidants (α-tocopherol, γ-tocopherol and xanthophyll) was observed following watercress supplementation (P< 0.05 v. control) in both experimental phases. These findings suggest that short- and long-term watercress ingestion has potential antioxidant effects against exercise-induced DNA damage and lipid peroxidation. \n",
      "\n",
      "0.8218895 The pharmacology, clinical efficacy, adverse effects, drug interactions, and place in therapy of bitter melon are described. Bitter melon (Momordica charantia) is an alternative therapy that has primarily been used for lowering blood glucose levels in patients with diabetes mellitus. Components of bitter melon extract appear to have structural similarities to animal insulin. Antiviral and antineoplastic activities have also been reported in vitro. Four clinical trials found bitter melon juice, fruit, and dried powder to have a moderate hypoglycemic effect. These studies were small and were not randomized or double-blind, however. Reported adverse effects of bitter melon include hypoglycemic coma and convulsions in children, reduced fertility in mice, a favism-like syndrome, increases in gamma-glutamyltransferase and alkaline phosphatase levels in animals, and headaches. Bitter melon may have additive effects when taken with other glucose-lowering agents. Adequately powered, randomized, placebo-controlled trials are needed to properly assess safety and efficacy before bitter melon can be routinely recommended. Bitter melon may have hypoglycemic effects, but data are not sufficient to recommend its use in the absence of careful supervision and monitoring. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "query = \"effects of watermelon supplementation\"\n",
    "\n",
    "top_passages_ixs, scores = get_topk_passages(bi_encoder, query, new_passage_embeddings, 5)\n",
    "for ix, score in zip(top_passages_ixs, scores):\n",
    "    print(score, passages[ix], \"\\n\")\n",
    "\n",
    "#  we got the correct passage at the third stop, also 2 out of the 5 retrieved documents\n",
    "#  are focused on some fruit/vegetable supplementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3wyXUp209jzZ",
    "outputId": "35e65f2a-f21b-48c0-e0d0-6cb9409aa727"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.79298615 OBJECTIVE: Watermelon is a rich source of citrulline, an amino acid that can be metabolized to arginine, a conditionally essential amino acid for humans. Arginine is the nitrogenous substrate used in the synthesis of nitric oxide and plays an essential role in cardiovascular and immune functions. No detailed studies have been conducted to evaluate plasma arginine response in humans after long-term feeding of citrulline from natural plant sources. This study investigated if watermelon juice consumption increases fasting concentrations of plasma arginine, ornithine, and citrulline in healthy adult humans. METHODS: Subjects (n = 12-23/treatment) consumed a controlled diet and 0 (control), 780, or 1560 g of watermelon juice per day for 3 wk in a crossover design. The treatments provided 1 and 2 g of citrulline per day. Treatment periods were preceded by washout periods of 2 to 4 wk. RESULTS: Compared with the baseline, fasting plasma arginine concentrations increased 12% after 3 wk of the lower-dose watermelon treatment; arginine and ornithine concentrations increased 22% and 18%, respectively, after 3 wk of the higher-dose watermelon treatment. Fasting citrulline concentrations did not increase relative to the control but remained stable throughout the study. CONCLUSION: The increased fasting plasma concentrations of arginine and ornithine and stable concentrations of plasma citrulline in response to watermelon juice consumption indicated that the citrulline from this plant origin was effectively converted into arginine. These results demonstrate that plasma concentration of arginine can be increased through intake of citrulline from watermelon. \n",
      "\n",
      "0.7893438 The pharmacology, clinical efficacy, adverse effects, drug interactions, and place in therapy of bitter melon are described. Bitter melon (Momordica charantia) is an alternative therapy that has primarily been used for lowering blood glucose levels in patients with diabetes mellitus. Components of bitter melon extract appear to have structural similarities to animal insulin. Antiviral and antineoplastic activities have also been reported in vitro. Four clinical trials found bitter melon juice, fruit, and dried powder to have a moderate hypoglycemic effect. These studies were small and were not randomized or double-blind, however. Reported adverse effects of bitter melon include hypoglycemic coma and convulsions in children, reduced fertility in mice, a favism-like syndrome, increases in gamma-glutamyltransferase and alkaline phosphatase levels in animals, and headaches. Bitter melon may have additive effects when taken with other glucose-lowering agents. Adequately powered, randomized, placebo-controlled trials are needed to properly assess safety and efficacy before bitter melon can be routinely recommended. Bitter melon may have hypoglycemic effects, but data are not sufficient to recommend its use in the absence of careful supervision and monitoring. \n",
      "\n",
      "0.78203934 PURPOSE OF REVIEW: To summarize recent findings and current concepts in the beneficial effects of berry consumption on brain function during aging. RECENT FINDINGS: Berryfruit supplementation has continued to demonstrate efficacy in reversing age-related cognitive decline in animal studies. In terms of the mechanisms behind the effects of berries on the central nervous system, recent studies have demonstrated the bioavailability of berry polyphenols in several animal models. These studies have revealed that flavonoids and polyphenols from berries do accumulate in the brain following long-term consumption. Finally, several compelling studies have revealed that berries can influence cell-signaling cascades both in vivo and in cell culture systems. These studies underscore the developing theory that berries and antioxidant-rich foods may be acting as more than just oxygen radical neutralizers in the aging central nervous system. SUMMARY: Antioxidant-rich berries consumed in the diet can positively impact learning and memory in the aged animal. This effect on cognition is thought to be due to the direct interaction of berry polyphenols with aging neurons, reducing the impact of stress-related cellular signals and increasing the capacity of neurons to maintain proper functioning during aging. \n",
      "\n",
      "0.78111243 Grapefruit is a healthy addition to a well-balanced diet. However, the fruit has been shown to affect the metabolism of many medications, increasing the risk of toxicity and adverse effects. Characteristics of oral medications that may interact with grapefruit include extensive metabolism through the intestinal cytochrome P450 3A4 system, low bioavailability, and a narrow therapeutic index. Prominent medications known to interact with grapefruit include statins, antiarrhythmic agents, immunosuppressive agents, and calcium channel blockers. There are equally effective alternatives to these drug classes that do not have the potential to interact with grapefruit. These alternative drugs may be substituted if a patient experiences or is at risk of a grapefruit-drug interaction. Patients also may choose to exclude grapefruit from their diets and consume other fruits, including other types of citrus, to avoid an interaction. \n",
      "\n",
      "0.770946 Beneficial effects of consumption of fruit and vegetables on the cardiovascular system have been reported. Fruit and vegetable components affect the cardiovascular system in both antioxidant and nonantioxidant ways. The mechanisms of their actions are, however, still not well understood. The compounds present in fruits and vegetables may function individually or in concert to protect lipoproteins and vascular cells from oxidation or by other mechanisms such as reducing plasma lipid levels, high blood pressure, and platelet hyperactivity. Emerging data indicate that kiwifruit is beneficial in the prevention of cardiovascular disease, as consumption of two or three fruit per day for 28 days or more lowers platelet hyperactivity, plasma lipids, and blood pressure in human volunteers. These studies suggest that kiwifruit may provide a new dietary means as part of a preventive or therapeutic strategy to favorably modify cardiovascular risk factors. The relevance of lowering the cardiovascular risk factors by kiwifruit in human health is discussed. Copyright © 2013 Elsevier Inc. All rights reserved. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "top_passages_ixs, scores = get_topk_passages(retriever, query, passage_embeddings, 5)\n",
    "for ix, score in zip(top_passages_ixs, scores):\n",
    "    print(score, passages[ix], \"\\n\")\n",
    "\n",
    "#  here we failed to get the correct passage and only 1 out of 5 retrieved passages\n",
    "#  is talking about supplementaion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Passage**<br>\n",
    "The use of over-the-counter supplements is commonplace in today's health conscious society. We present an unusual case of intrahepatic cholestasis caused by vitamin A intoxication. The patient consumed one Herbalife shake with two multivitamin tablets of the same brand for 12 years. When calculated this equated to more than the recommended daily allowance for vitamin A consumption. Deranged liver function tests were consistent with a cholestatic process. Liver biopsy was obtained and revealed features pathognomonic of vitamin A toxicity, without the usual fibrosis. When the supplements were ceased, his jaundice and alkaline phosphatase completely normalized. This case highlights the importance of health care providers documenting non-prescribed dietary supplements and considering them in the etiology of cholestatic liver disease. Copyright 2009 Elsevier Inc. All rights reserved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.84828144 The use of over-the-counter supplements is commonplace in today's health conscious society. We present an unusual case of intrahepatic cholestasis caused by vitamin A intoxication. The patient consumed one Herbalife shake with two multivitamin tablets of the same brand for 12 years. When calculated this equated to more than the recommended daily allowance for vitamin A consumption. Deranged liver function tests were consistent with a cholestatic process. Liver biopsy was obtained and revealed features pathognomonic of vitamin A toxicity, without the usual fibrosis. When the supplements were ceased, his jaundice and alkaline phosphatase completely normalized. This case highlights the importance of health care providers documenting non-prescribed dietary supplements and considering them in the etiology of cholestatic liver disease. Copyright 2009 Elsevier Inc. All rights reserved. \n",
      "\n",
      "0.8270186 Our understanding of coronary artery disease risk and the atherosclerotic process has changed greatly in recent years. For example, it is now known that angiographically apparent coronary artery plaque is not the major cause of myocardial infarction (MI). Rather, it is unstable, soft plaque that cannot be seen angiographically that is prone to rupture and result in infarction. Also important are changes in vascular reactivity resulting from diet. Cholesterol levels by themselves reveal little about a patient's coronary artery disease risk. Most infarctions occur in patients who have normal total cholesterol levels. At-risk patients can be identified using the ratio of total-to-high-density lipoprotein (HDL) cholesterol levels. The ratio of triglyceride to HDL cholesterol levels is also important. Simple steps to assess patients' risk in practice are outlined. Primary prevention trials demonstrate that coronary artery disease risk can be lowered dramatically with diet and drug therapy. \n",
      "\n",
      "0.81834847 Cardiovascular diseases are one of the leading causes of morbidity and mortality in industrialized countries, and although many processes play a role in the development of vascular disease, thrombosis is the primary event that precipitates stroke and acute coronary syndromes. The blood platelets are of significant importance in medicine. These cells are involved in many physiological processes, particularly haemostasis through their ability to aggregate and form clots in response to activation. In addition, these dynamic cells display activities that extend beyond thrombosis, including an important role in initiating and sustaining vascular inflammation. The expansion of knowledge from basic and clinical research has highlighted the critical position of platelets in several inflammatory diseases such as arthritis and atherosclerosis. Platelets are emerging as important mediators of inflammation and provide important signals to mediate phenotype of other blood and vascular cells. The important role of platelets in arterial thrombosis and the onset of acute myocardial infarction after atherosclerotic plaque rupture make inhibition of platelet aggregation a critical step in preventing thrombotic events associated with stroke, heart attack, and peripheral arterial thrombosis. However, the use of platelet inhibitors for thrombosis prevention must seek a delicate balance between inhibiting platelet activation and an associated increased bleeding risk. The aim of this review is to up-date the knowledge on platelets physiology and dysfunction in pathologies, such as diabetes mellitus, hypercholesterolemia, and hypertension, emphasizing the link between platelets and the inflammation-related atherosclerosis. The review evaluates the opportunities offered by the novel platelet inhibitors to efficiently alleviate the thrombotic events. Copyright © 2011 Elsevier Ltd. All rights reserved. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "query = \"what causes intrahepatic cholestasis\"\n",
    "\n",
    "top_passages_ixs, scores = get_topk_passages(bi_encoder, query, new_passage_embeddings, 3)\n",
    "for ix, score in zip(top_passages_ixs, scores):\n",
    "    print(score, passages[ix], \"\\n\")\n",
    "\n",
    "#  we got the correct passage at top with a high score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.79298615 OBJECTIVE: Watermelon is a rich source of citrulline, an amino acid that can be metabolized to arginine, a conditionally essential amino acid for humans. Arginine is the nitrogenous substrate used in the synthesis of nitric oxide and plays an essential role in cardiovascular and immune functions. No detailed studies have been conducted to evaluate plasma arginine response in humans after long-term feeding of citrulline from natural plant sources. This study investigated if watermelon juice consumption increases fasting concentrations of plasma arginine, ornithine, and citrulline in healthy adult humans. METHODS: Subjects (n = 12-23/treatment) consumed a controlled diet and 0 (control), 780, or 1560 g of watermelon juice per day for 3 wk in a crossover design. The treatments provided 1 and 2 g of citrulline per day. Treatment periods were preceded by washout periods of 2 to 4 wk. RESULTS: Compared with the baseline, fasting plasma arginine concentrations increased 12% after 3 wk of the lower-dose watermelon treatment; arginine and ornithine concentrations increased 22% and 18%, respectively, after 3 wk of the higher-dose watermelon treatment. Fasting citrulline concentrations did not increase relative to the control but remained stable throughout the study. CONCLUSION: The increased fasting plasma concentrations of arginine and ornithine and stable concentrations of plasma citrulline in response to watermelon juice consumption indicated that the citrulline from this plant origin was effectively converted into arginine. These results demonstrate that plasma concentration of arginine can be increased through intake of citrulline from watermelon. \n",
      "\n",
      "0.7893438 The pharmacology, clinical efficacy, adverse effects, drug interactions, and place in therapy of bitter melon are described. Bitter melon (Momordica charantia) is an alternative therapy that has primarily been used for lowering blood glucose levels in patients with diabetes mellitus. Components of bitter melon extract appear to have structural similarities to animal insulin. Antiviral and antineoplastic activities have also been reported in vitro. Four clinical trials found bitter melon juice, fruit, and dried powder to have a moderate hypoglycemic effect. These studies were small and were not randomized or double-blind, however. Reported adverse effects of bitter melon include hypoglycemic coma and convulsions in children, reduced fertility in mice, a favism-like syndrome, increases in gamma-glutamyltransferase and alkaline phosphatase levels in animals, and headaches. Bitter melon may have additive effects when taken with other glucose-lowering agents. Adequately powered, randomized, placebo-controlled trials are needed to properly assess safety and efficacy before bitter melon can be routinely recommended. Bitter melon may have hypoglycemic effects, but data are not sufficient to recommend its use in the absence of careful supervision and monitoring. \n",
      "\n",
      "0.78203934 PURPOSE OF REVIEW: To summarize recent findings and current concepts in the beneficial effects of berry consumption on brain function during aging. RECENT FINDINGS: Berryfruit supplementation has continued to demonstrate efficacy in reversing age-related cognitive decline in animal studies. In terms of the mechanisms behind the effects of berries on the central nervous system, recent studies have demonstrated the bioavailability of berry polyphenols in several animal models. These studies have revealed that flavonoids and polyphenols from berries do accumulate in the brain following long-term consumption. Finally, several compelling studies have revealed that berries can influence cell-signaling cascades both in vivo and in cell culture systems. These studies underscore the developing theory that berries and antioxidant-rich foods may be acting as more than just oxygen radical neutralizers in the aging central nervous system. SUMMARY: Antioxidant-rich berries consumed in the diet can positively impact learning and memory in the aged animal. This effect on cognition is thought to be due to the direct interaction of berry polyphenols with aging neurons, reducing the impact of stress-related cellular signals and increasing the capacity of neurons to maintain proper functioning during aging. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "top_passages_ixs, scores = get_topk_passages(retriever, query, passage_embeddings, 3)\n",
    "for ix, score in zip(top_passages_ixs, scores):\n",
    "    print(score, passages[ix], \"\\n\")\n",
    "\n",
    "#  we got the correct passage at the third place and the score distrubtion is pretty flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Passage**<br>\n",
    "Epidemiologic studies suggest that dietary lutein and zeaxanthin may be of benefit in maintaining cognitive health. Among the carotenoids, lutein and zeaxanthin are the only two that cross the blood-retina barrier to form macular pigment (MP) in the eye. They also preferentially accumulate in the human brain. Lutein and zeaxanthin in macula from nonhuman primates were found to be significantly correlated with their concentrations in matched brain tissue. Therefore, MP can be used as a biomarker of lutein and zeaxanthin in primate brain tissue. This is of interest given that a significant correlation was found between MP density and global cognitive function in healthy older adults. An examination of a relation between cognition and lutein and zeaxanthin concentrations in the brain tissue of decedents from a population-based study in centenarians found that zeaxanthin concentrations in brain tissue were significantly related to antemortem measures of global cognitive function, memory retention, verbal fluency, and dementia severity after adjustment for age, sex, education, hypertension, and diabetes. In univariate analyses, lutein was related to recall and verbal fluency, but the strength of the associations was attenuated with adjustment for covariates. However, lutein concentrations in the brain were significantly lower in individuals with mild cognitive impairment than in those with normal cognitive function. Last, in a 4-mo, double-blinded, placebo-controlled trial in older women that involved lutein supplementation (12 mg/d), alone or in combination with DHA (800 mg/d), verbal fluency scores improved significantly in the DHA, lutein, and combined-treatment groups. Memory scores and rate of learning improved significantly in the combined-treatment group, who also showed a trend toward more efficient learning. When all of these observations are taken into consideration, the idea that lutein and zeaxanthin can influence cognitive function in older adults warrants further study."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.83842903 Epidemiologic studies suggest that dietary lutein and zeaxanthin may be of benefit in maintaining cognitive health. Among the carotenoids, lutein and zeaxanthin are the only two that cross the blood-retina barrier to form macular pigment (MP) in the eye. They also preferentially accumulate in the human brain. Lutein and zeaxanthin in macula from nonhuman primates were found to be significantly correlated with their concentrations in matched brain tissue. Therefore, MP can be used as a biomarker of lutein and zeaxanthin in primate brain tissue. This is of interest given that a significant correlation was found between MP density and global cognitive function in healthy older adults. An examination of a relation between cognition and lutein and zeaxanthin concentrations in the brain tissue of decedents from a population-based study in centenarians found that zeaxanthin concentrations in brain tissue were significantly related to antemortem measures of global cognitive function, memory retention, verbal fluency, and dementia severity after adjustment for age, sex, education, hypertension, and diabetes. In univariate analyses, lutein was related to recall and verbal fluency, but the strength of the associations was attenuated with adjustment for covariates. However, lutein concentrations in the brain were significantly lower in individuals with mild cognitive impairment than in those with normal cognitive function. Last, in a 4-mo, double-blinded, placebo-controlled trial in older women that involved lutein supplementation (12 mg/d), alone or in combination with DHA (800 mg/d), verbal fluency scores improved significantly in the DHA, lutein, and combined-treatment groups. Memory scores and rate of learning improved significantly in the combined-treatment group, who also showed a trend toward more efficient learning. When all of these observations are taken into consideration, the idea that lutein and zeaxanthin can influence cognitive function in older adults warrants further study. \n",
      "\n",
      "0.8261617 Presence of omega-3, omega-6 rich oil, alpha-linoleic acid, dietary fibers, secoisolariciresinol diglucoside, protein and minerals in flaxseed constitute a very strong basis for the utilization of flaxseed in various food preparations as a curative agent. An extensive body of literature illustrates that flaxseed has gained a significant position in the domain of nutritional sciences owing to its pivotal role as an antioxidant agent. The review discusses at length, numerous health benefits of flaxseed typically focusing its preventive role against cardiovascular diseases, cancer, diabetes and enhancement of spatial memory. Massive increase in the size of population with a special emphasize to the developing countries, there is an urge for exploration of the alternative dietary resources that can meet the dietary and nutritional needs of forthcoming generations. With respect to its remarkable nutritional importance, the review in question enables researchers engaged in nutritional sciences to further investigate the therapeutic value of flaxseed functional components and their dietary application in various food products and availability in processed foods as well as in the human cell line. \n",
      "\n",
      "0.82421887 Lutein and zeaxanthin are thought to decrease the incidence of age-related macular degeneration (AMD); however, findings have been inconsistent. We conducted a systematic literature review and meta-analysis to evaluate the relationship between dietary intake of lutein and zeaxanthin and AMD risk. Relevant studies were identified by searching five databases up to April 2010. Reference lists of articles were retrieved, and experts were contacted. Literature search, data extraction and study quality assessment were performed independently by two reviewers and results were pooled quantitatively using meta-analysis methods. The potential sources of heterogeneity and publication bias were also estimated. The search yielded six longitudinal cohort studies. The pooled relative risk (RR) for early AMD, comparing the highest with the lowest category of lutein and zeaxanthin intake, was 0·96 (95 % CI 0·78, 1·17). Dietary intake of these carotenoids was significantly related with a reduction in risk of late AMD (RR 0·74; 95 % CI 0·57, 0·97); and a statistically significant inverse association was observed between lutein and zeaxanthin intake and neovascular AMD risk (RR 0·68; 95 % CI 0·51, 0·92). The results were essentially consistent among subgroups stratified by participant characteristics. The findings of the present meta-analysis indicate that dietary lutein and zeaxanthin is not significantly associated with a reduced risk of early AMD, whereas an increase in the intake of these carotenoids may be protective against late AMD. However, additional studies are needed to confirm these relationships. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "query = \"benefits of zeaxanthin\"\n",
    "\n",
    "top_passages_ixs, scores = get_topk_passages(bi_encoder, query, new_passage_embeddings, 3)\n",
    "for ix, score in zip(top_passages_ixs, scores):\n",
    "    print(score, passages[ix], \"\\n\")\n",
    "\n",
    "#  we got the correct passage at top and the third passage also looks like a correct answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8006925 Presence of omega-3, omega-6 rich oil, alpha-linoleic acid, dietary fibers, secoisolariciresinol diglucoside, protein and minerals in flaxseed constitute a very strong basis for the utilization of flaxseed in various food preparations as a curative agent. An extensive body of literature illustrates that flaxseed has gained a significant position in the domain of nutritional sciences owing to its pivotal role as an antioxidant agent. The review discusses at length, numerous health benefits of flaxseed typically focusing its preventive role against cardiovascular diseases, cancer, diabetes and enhancement of spatial memory. Massive increase in the size of population with a special emphasize to the developing countries, there is an urge for exploration of the alternative dietary resources that can meet the dietary and nutritional needs of forthcoming generations. With respect to its remarkable nutritional importance, the review in question enables researchers engaged in nutritional sciences to further investigate the therapeutic value of flaxseed functional components and their dietary application in various food products and availability in processed foods as well as in the human cell line. \n",
      "\n",
      "0.79755646 Flaxseed is one of the most important oilseed crops for industrial as well as food, feed, and fiber purposes. Almost every part of the flaxseed plant is utilized commercially, either directly or after processing. The stem yields good quality fiber having high strength and durability. The seed provides oil rich in omega-3, digestible proteins, and lignans. In addition to being one of the richest sources of α-linolenic acid oil and lignans, flaxseed is an essential source of high quality protein and soluble fiber and has considerable potential as a source of phenolic compounds. Flaxseed is emerging as an important functional food ingredient because of its rich contents of α-linolenic acid (ALA), lignans, and fiber. Lignans appear to be anti-carcinogenic compounds. The omega-3s and lignan phytoestrogens of flaxseed are in focus for their benefits for a wide range of health conditions and may possess chemo-protective properties in animals and humans. This paper presents a review of literature on the nutritional composition of flaxseed, its health benefits, and disease-prevention qualities, utilization of flaxseed for food, feed, and fiber, and processing of flaxseed. \n",
      "\n",
      "0.77763236 Epidemiologic studies suggest that dietary lutein and zeaxanthin may be of benefit in maintaining cognitive health. Among the carotenoids, lutein and zeaxanthin are the only two that cross the blood-retina barrier to form macular pigment (MP) in the eye. They also preferentially accumulate in the human brain. Lutein and zeaxanthin in macula from nonhuman primates were found to be significantly correlated with their concentrations in matched brain tissue. Therefore, MP can be used as a biomarker of lutein and zeaxanthin in primate brain tissue. This is of interest given that a significant correlation was found between MP density and global cognitive function in healthy older adults. An examination of a relation between cognition and lutein and zeaxanthin concentrations in the brain tissue of decedents from a population-based study in centenarians found that zeaxanthin concentrations in brain tissue were significantly related to antemortem measures of global cognitive function, memory retention, verbal fluency, and dementia severity after adjustment for age, sex, education, hypertension, and diabetes. In univariate analyses, lutein was related to recall and verbal fluency, but the strength of the associations was attenuated with adjustment for covariates. However, lutein concentrations in the brain were significantly lower in individuals with mild cognitive impairment than in those with normal cognitive function. Last, in a 4-mo, double-blinded, placebo-controlled trial in older women that involved lutein supplementation (12 mg/d), alone or in combination with DHA (800 mg/d), verbal fluency scores improved significantly in the DHA, lutein, and combined-treatment groups. Memory scores and rate of learning improved significantly in the combined-treatment group, who also showed a trend toward more efficient learning. When all of these observations are taken into consideration, the idea that lutein and zeaxanthin can influence cognitive function in older adults warrants further study. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "top_passages_ixs, scores = get_topk_passages(retriever, query, passage_embeddings, 3)\n",
    "for ix, score in zip(top_passages_ixs, scores):\n",
    "    print(score, passages[ix], \"\\n\")\n",
    "\n",
    "#  we get the correct passage at the third place and missed the other answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** When trying out other examples, remember to focus on questions where the answer can be found at the beginning of the passage as we had a tokenization limit of 512."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## References\n",
    "\n",
    "[1] Kexin Wang, Nandan Thakur, Nils Reimers and Iryna Gurevych. \"[GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval](https://arxiv.org/abs/2112.07577)\"\n",
    "\n",
    "[2] https://github.com/UKPLab/gpl\n",
    "\n",
    "[3] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu and Haifeng Wang. \"[RocketQA: An Optimized Training Approach to Dense Passage Retrieval\n",
    "for Open-Domain Question Answering](https://arxiv.org/pdf/2010.08191.pdf)\""
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "nlp_models.ipynb",
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0b6da3ef076841f8bde62dfc8d343cb6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1287975a9a1948e6ac098f9f0f70ec9c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "15705887479a4b7896de71d59d9b1574": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1599afe703c54f2d8be7b9666a511857": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_761e339bda144b218685608c1df710c9",
      "placeholder": "​",
      "style": "IPY_MODEL_fb8b139665f348d99caa7406639c55ab",
      "value": " 493/811 [00:00&lt;00:00, 2674.49 examples/s]"
     }
    },
    "18dced07268e4bc287c19bfa111ad737": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5d2d9b7782334cb4875a9c021cb735d0",
      "placeholder": "​",
      "style": "IPY_MODEL_ecbfc203abdb41d38499d28e5139191f",
      "value": "100%"
     }
    },
    "1e434b72393e4b3b8d464c409b66ae95": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1e6d3d085091481ab4dc4b98e8a07cca": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1f573fe5b8054ab6b4467d7154ff7a69": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2846925295294d819a62655c784df8c2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ce0536884a594ddca425c0c3862caea0",
       "IPY_MODEL_2de210f3638a44448fd5ad3c4c24c7dd",
       "IPY_MODEL_546f2c471c6247fd94d1dccb8db919f1"
      ],
      "layout": "IPY_MODEL_87e0a0e7bc0e4dd59163b5cb596b3add"
     }
    },
    "2de210f3638a44448fd5ad3c4c24c7dd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3904769c197942c793a5d60326637504",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_55fef994c7de4cf7aa038581130ae9d2",
      "value": 1
     }
    },
    "2fbaa6537fbd489f8430cc91b3a3cd17": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4563aad9bff94843b472a8a382d4d990",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_862369200c944df9ba48b01a23fbc160",
      "value": 3
     }
    },
    "3904769c197942c793a5d60326637504": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3959def343f2412ca41c9d19ba843bb4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3f1812129cd840b6a0c171bd019386e5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_18dced07268e4bc287c19bfa111ad737",
       "IPY_MODEL_2fbaa6537fbd489f8430cc91b3a3cd17",
       "IPY_MODEL_54d3010a0c964605b9aae3d94c4d43bf"
      ],
      "layout": "IPY_MODEL_c53ef72839714eaf977e3c4d4299ad6b"
     }
    },
    "3fb165f1258f425a80e6457a5a6bec8f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_609ec846c5fe48a09f5ea1e8fb899fe4",
      "max": 661,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c6d42da408e04ae0bb5fe9c2371d5477",
      "value": 661
     }
    },
    "4563aad9bff94843b472a8a382d4d990": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "459d2c124c4f42f988f6e8d7de362e59": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6624effa65bd4985a8e0aab8fcbe5403",
       "IPY_MODEL_3fb165f1258f425a80e6457a5a6bec8f",
       "IPY_MODEL_c0b75b080c1147ac84d04c5e358377cc"
      ],
      "layout": "IPY_MODEL_8ba201b0d21e40d09919e38b8055cf9f"
     }
    },
    "4aa84adb5bf94c94b4c50776efcdf004": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4c2f7f623d424af2b1b4f5d5e8d5653a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4f7c1f51f14e46c2bdebc89130bc0f32": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_507f63350c004bbe88794c61f17d435a",
      "max": 2939,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5ffa8f28ebe0436884c451ca8fbb3b19",
      "value": 2939
     }
    },
    "507f63350c004bbe88794c61f17d435a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "546f2c471c6247fd94d1dccb8db919f1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3959def343f2412ca41c9d19ba843bb4",
      "placeholder": "​",
      "style": "IPY_MODEL_4c2f7f623d424af2b1b4f5d5e8d5653a",
      "value": " 1/1 [34:48&lt;00:00, 2088.20s/it]"
     }
    },
    "54d3010a0c964605b9aae3d94c4d43bf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ae60fb6564a3453e95308bdce7a0688b",
      "placeholder": "​",
      "style": "IPY_MODEL_4aa84adb5bf94c94b4c50776efcdf004",
      "value": " 3/3 [00:00&lt;00:00, 79.50it/s]"
     }
    },
    "55fef994c7de4cf7aa038581130ae9d2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "56a4c8c9ec1a4029b056950a48adfddd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e3810172078e47149dabea209a690e29",
      "placeholder": "​",
      "style": "IPY_MODEL_0b6da3ef076841f8bde62dfc8d343cb6",
      "value": " 2939/2939 [34:48&lt;00:00,  1.42it/s]"
     }
    },
    "59e0ac4998844191b2fe55d82353781e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_78f65d5dd16c4972b80ad57842688d2f",
      "max": 6253,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1e6d3d085091481ab4dc4b98e8a07cca",
      "value": 6253
     }
    },
    "5d2d9b7782334cb4875a9c021cb735d0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5ffa8f28ebe0436884c451ca8fbb3b19": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "609ec846c5fe48a09f5ea1e8fb899fe4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6624effa65bd4985a8e0aab8fcbe5403": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_eed0d294051e40199a299006c7766805",
      "placeholder": "​",
      "style": "IPY_MODEL_1287975a9a1948e6ac098f9f0f70ec9c",
      "value": "Generating validation split:  72%"
     }
    },
    "66310509ead04ae39bf8b7ffb523c411": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dd6bc5c2e5d44e90b0816eb2a4c5d874",
      "placeholder": "​",
      "style": "IPY_MODEL_746c5c6161d447c1b605b40996d1158d",
      "value": "Generating train split:  97%"
     }
    },
    "6ac6af5ca9d14075915a7a37200e63be": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "746c5c6161d447c1b605b40996d1158d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "761e339bda144b218685608c1df710c9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "78f65d5dd16c4972b80ad57842688d2f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "81e92ba9faff4ab1bcfe64ba3df96328": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "862369200c944df9ba48b01a23fbc160": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "87e0a0e7bc0e4dd59163b5cb596b3add": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8b2d0f5df788426c967de270dd5fbbeb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8ba201b0d21e40d09919e38b8055cf9f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8be3d3f1da664bada9646012b1690117": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8c202783ca984fa28186b119f659bab6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "962de3077c54405baf3238e223c79342": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c4f0a26b52924e4886fa44ea6e57b9a8",
       "IPY_MODEL_4f7c1f51f14e46c2bdebc89130bc0f32",
       "IPY_MODEL_56a4c8c9ec1a4029b056950a48adfddd"
      ],
      "layout": "IPY_MODEL_81e92ba9faff4ab1bcfe64ba3df96328"
     }
    },
    "ae08a9ef72ea45e6bba9caa72a995a51": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "ae60fb6564a3453e95308bdce7a0688b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b1027d3c96034f2d81868fd6267bb87f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b443e89c08a64d87b2c7703bf6b929d3",
       "IPY_MODEL_da6dcd75615e4258a65e47141fb9092f",
       "IPY_MODEL_1599afe703c54f2d8be7b9666a511857"
      ],
      "layout": "IPY_MODEL_8b2d0f5df788426c967de270dd5fbbeb"
     }
    },
    "b2033ee9411f4449a593bea0ca798288": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c02376dfc7c046c989a102d0d6bb9efb",
      "placeholder": "​",
      "style": "IPY_MODEL_e06f39294b4c410bb2abb489f1f8da2c",
      "value": " 6066/6253 [00:01&lt;00:00, 3919.44 examples/s]"
     }
    },
    "b443e89c08a64d87b2c7703bf6b929d3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1e434b72393e4b3b8d464c409b66ae95",
      "placeholder": "​",
      "style": "IPY_MODEL_6ac6af5ca9d14075915a7a37200e63be",
      "value": "Generating test split:  61%"
     }
    },
    "c02376dfc7c046c989a102d0d6bb9efb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c0b75b080c1147ac84d04c5e358377cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8c202783ca984fa28186b119f659bab6",
      "placeholder": "​",
      "style": "IPY_MODEL_15705887479a4b7896de71d59d9b1574",
      "value": " 479/661 [00:00&lt;00:00, 2615.02 examples/s]"
     }
    },
    "c4f0a26b52924e4886fa44ea6e57b9a8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1f573fe5b8054ab6b4467d7154ff7a69",
      "placeholder": "​",
      "style": "IPY_MODEL_e18574af80d94d85a1920fd2bc138566",
      "value": "Iteration: 100%"
     }
    },
    "c53ef72839714eaf977e3c4d4299ad6b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c6d42da408e04ae0bb5fe9c2371d5477": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c99312ef350e41d7938955ee4a09d81b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ce0536884a594ddca425c0c3862caea0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_df2a51cde40a46c9bad49d46315f9246",
      "placeholder": "​",
      "style": "IPY_MODEL_8be3d3f1da664bada9646012b1690117",
      "value": "Epoch: 100%"
     }
    },
    "da6dcd75615e4258a65e47141fb9092f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c99312ef350e41d7938955ee4a09d81b",
      "max": 811,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ae08a9ef72ea45e6bba9caa72a995a51",
      "value": 811
     }
    },
    "dd6bc5c2e5d44e90b0816eb2a4c5d874": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "df2a51cde40a46c9bad49d46315f9246": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e06f39294b4c410bb2abb489f1f8da2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e18574af80d94d85a1920fd2bc138566": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e3810172078e47149dabea209a690e29": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ecbfc203abdb41d38499d28e5139191f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "eed0d294051e40199a299006c7766805": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fb8b139665f348d99caa7406639c55ab": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fdafb465ce8b4bd5b548acba3eb50d43": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fee64bf203e54ba98126ac3b1b806faa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_66310509ead04ae39bf8b7ffb523c411",
       "IPY_MODEL_59e0ac4998844191b2fe55d82353781e",
       "IPY_MODEL_b2033ee9411f4449a593bea0ca798288"
      ],
      "layout": "IPY_MODEL_fdafb465ce8b4bd5b548acba3eb50d43"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}