notebooks/Auto-generated MOCs via topic modelling.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "592f0f61",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir('..')  # go up to the project directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "97b7cc80",
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "import matplotlib.pyplot as plt\n",
    "from pathlib import Path\n",
    "\n",
    "import src\n",
    "from src.vault import get_vault\n",
    "from src.nlp import VaultNLP\n",
    "from src.moc import auto_generate_MOCs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "509c03ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8817ad",
   "metadata": {},
   "source": [
    "# Proof of concept"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f2e1e86",
   "metadata": {},
   "source": [
    "I have an Obsidian.md vault with these characteristics:\n",
    "- 550+ files (800+ notes when including files not yet created)\n",
    "- 130k+ words of content across the vault\n",
    "\n",
    "These notes were used for studying 11 modules as part of my degree.  I used it like a personal wiki.\n",
    "\n",
    "Hence the notes had natural ['categories'](https://notes.linkingyourthinking.com/Cards/Why+Categories+for+Your+Notes+are+a+Good+Idea), which I grouped into folders: I did not use Maps of Content (MOCs).\n",
    "\n",
    "The vault is an interesting test case for auto-generating MOCs from its content.  These are reasons why I want to develop MOCs (from [advantages](https://notes.linkingyourthinking.com/Cards/MOCs+Overview) in the LYT kit):\n",
    "- Better navigation of my vault\n",
    "- Convenient reference points for my modules\n",
    "- Help with spatial relationships"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e321bf9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 22.2 s, sys: 225 ms, total: 22.4 s\n",
      "Wall time: 23.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "VAULT = get_vault('MPHIL_VAULT_DIR')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4f107ae2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = VAULT.get_note_metadata()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "80050e85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(800, 7)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2df1910c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Notes with content: 561\n"
     ]
    }
   ],
   "source": [
    "print(f\"Notes with content: {df[df['note_exists']].shape[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e3e719f",
   "metadata": {},
   "source": [
    "This code is a **proof of concept** for _**auto-generating MOCs via topic modelling**_.\n",
    "\n",
    "I use the **[Anchored CorEx algorithm](https://github.com/gregversteeg/corex_topic)** to create one MOC per module, to mirror the modules I studied for my degree.  In essence, the aim for the MOCs is to generate an index of the most important notes for each module.  See the _References_ section at the end of this notebook for the paper.\n",
    "\n",
    "**The beauty of this approach:**\n",
    "- The user specifies keywords (anchor words) for each topic.  This is _semi-supervised learning_, so **the user can nudge the algorithm** to group notes into topics that have some context set by the user.  The user can do incremental changes to the implementation to aim for more intuitive results.\n",
    "- It relies solely on the text within notes: the implementation **does NOT use detail from wikilinks**, i.e. it has zero information on any relationships between notes.  I've noticed that some approaches for MOC generation in Obsidian.md plug-ins rely on wikilinks to build structure (e.g. [Robin Haupt's plug-in](https://github.com/Robin-Haupt-1/Obsidian-Map-of-Content)), whereas this approach does not!  If the user has neglected the creation of wikilinks over time, in a complex vault, that shouldn't matter for the MOC generation here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "327cdacb",
   "metadata": {},
   "source": [
    "The knowledge graph for my vault is shown below, with notes coloured by module.  The MOCs that were generated from this repo's code are coloured white.\n",
    "\n",
    "![Knowledge graph](../img/MPhil_knowledge-graph_2022-08.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c8b871f",
   "metadata": {},
   "source": [
    "# Topic modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c323563b",
   "metadata": {},
   "source": [
    "This section has the code that covers the topic modelling.  With a few tweaks to the code, it could be re-applied to another vault.\n",
    "\n",
    "These are some of the main inputs to tweak:\n",
    "- Number of topics\n",
    "- Anchor words for topics _(the most important for improving results)_\n",
    "- Anchor strength\n",
    "\n",
    "The tutorial notebook has more detail on how to choose the number of topics.  For this vault, it was straightforward to choose 11 as I have 11 modules on my degree, so I aim to have one topic to represent the content for one module."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f73ff233",
   "metadata": {},
   "source": [
    "## Anchor words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ec6b438",
   "metadata": {},
   "source": [
    "I create a list of **anchor words** (`ANCHOR_WORDS`), which will have one list of keywords per topic.  This is where the semi-supervised approach comes in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0c2f8166",
   "metadata": {},
   "outputs": [],
   "source": [
    "ANCHOR_WORDS = [\n",
    "    ['statistics', 'frequentist', 'bayesian', 'hypothesis'],  # stats\n",
    "    ['bayesian', 'statistics', 'jags'],  # bayesian stats\n",
    "    ['statistics', 'missing', 'regularisation', 'pca', 'bonferroni', 'sem'],  # advanced biostats\n",
    "    ['machine', 'caret', 'unsupervised', 'supervised'],  # ML 1\n",
    "    ['machine', 'ensemble', 'neural'],  # ML 2\n",
    "    ['data', 'tidyverse', 'ggplot2', 'dplyr'],  # data carpentry\n",
    "    ['methodology', 'research', 'synthesis'],  # research skills\n",
    "    ['epidemiology', 'study', 'risk', 'disease', 'bias'],  # epi\n",
    "    ['genetic', 'epidemiology', 'allele', 'phenotype', 'gwas'],  # gen epi\n",
    "    ['genomics', 'variant', 'pathogenicity', 'acmg'],  # genomics\n",
    "    ['public', 'prevention']  # public health\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "187fcd9b",
   "metadata": {},
   "source": [
    "It is an interesting experiment for this vault, as there can be a lot of overlap between some modules, whereas others are much more separable in the Obsidian.md graph.  For example:\n",
    "- These are pairings of modules that could be difficult to separate:\n",
    "    - Modules on genomics and genetic epidemiology both have genetics and genetic association as core concepts.\n",
    "    - I had two modules about machine learning, which have considerable overlap in concepts.\n",
    "    - One module covered frequentist & Bayesian stats, but another module went deeper on the Bayesian stats concepts.\n",
    "- One module is an introduction to many stats & machine learning concepts, so it is purposely broad in nature and has overlap with multiple quantitative modules.  It is difficult to characterise it by keywords that are relevant to it alone.\n",
    "- These are modules that have distinct concepts from the rest, so they:\n",
    "    - Most modules are quantitative, but the public health module is not so it has high separation from most modules.\n",
    "    - Two modules on: Research Skills; Data Analysis.  However, the amount of content is low relative to other modules."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e623e6c",
   "metadata": {},
   "source": [
    "## `VaultNLP` class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eecc6bc",
   "metadata": {},
   "source": [
    "The `VaultNLP` class has functions that rework some of the code from the CorEx tutorial notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a41c89bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = VaultNLP(VAULT)  # create object\n",
    "nlp.anchor_words = ANCHOR_WORDS  # pass in the list of anchor words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d6d3ba3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 93 ms, sys: 1.03 ms, total: 94 ms\n",
      "Wall time: 93.7 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nlp.generate_sparse_word_matrix()  # word matrix, via vectorisation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a90423",
   "metadata": {},
   "source": [
    "There are over 8k unique words in my vault (when excluding stopwords):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1b1aef0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique words in the vault: 8167\n"
     ]
    }
   ],
   "source": [
    "print(f\"Unique words in the vault: {nlp.word_matrix.shape[1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63127daf",
   "metadata": {},
   "source": [
    "## Model fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5ac8d93",
   "metadata": {},
   "source": [
    "See the tutorial for more detail about model fitting and tuning.\n",
    "\n",
    "One of the key parameter decisions I have made is to use a **high anchor strength** (>5).  I found this helped to create keyword lists that characterise the topics in the way I wanted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0a3c3f1d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.44 s, sys: 11.1 ms, total: 3.45 s\n",
      "Wall time: 3.46 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nlp.fit_anchored_topic_model(n_hidden=len(ANCHOR_WORDS))  # fit model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d37183c0",
   "metadata": {},
   "source": [
    "These are the top 10 keywords for each topic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3034897a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0: bayesian, hypothesis, statistics, frequentist, inference, values, testing, statistical, null, used\n",
      "1: bayesian, jags, statistics, posterior, distribution, likelihood, parameters, prior, model, parameter\n",
      "2: statistics, missing, regularisation, bonferroni, imputation, problems, mar, doing, mnar, problem\n",
      "3: machine, supervised, learning, caret, unsupervised, feature, algorithms, features, kernel, algorithm\n",
      "4: machine, ensemble, neural, method, does, hyperparameters, forest, methods, performance, hyperparameter\n",
      "5: data, dplyr, tidyverse, main, ggplot2, fit, datasets, ways, available, processing\n",
      "6: research, methodology, synthesis, review, need, analysis, examples, service, clinical, ethics\n",
      "7: risk, study, disease, epidemiology, bias, studies, people, population, exposure, control\n",
      "8: genetic, epidemiology, phenotype, allele, gwas, inheritance, significance, genotype, snps, dominant\n",
      "9: variant, pathogenicity, genomics, variants, day, ag, acmg, genome, gene, genes\n",
      "10: public, prevention, health, policy, intervention, interventions, strategy, food, air, consent\n"
     ]
    }
   ],
   "source": [
    "for ix, _ in enumerate(ANCHOR_WORDS):\n",
    "    topic_words, _, _ = zip(\n",
    "        *nlp.anchor_topic_model.get_topics(topic=ix, n_words=10))\n",
    "    print('{}: '.format(ix) + ', '.join(topic_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4538b82",
   "metadata": {},
   "source": [
    "Comments on my choice of anchor words and how well the top 10 words in each topic relate to the intended module:\n",
    "- Topic 10 (public health): I only used 2 anchor words, but the top 10 keywords are highly relevant for the module.\n",
    "- Topics 3 & 4 (machine learning modules): these modules are difficult to separate, as there is a lot of overlap in content.  However their keywords are highly relevant.\n",
    "- Topic 2 (advanced biostats): I needed to choose a high number of anchor words for this module, as there are so many loosely related concepts.  The module covers breadth rather than depth.  Using fewer anchor words would have made the topic list poor."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b9c30b",
   "metadata": {},
   "source": [
    "This plot, adapted from the CorEx tutorial, shows how 'informative' each topic is to what is called the 'total correlation': the more informative a topic is, the higher its TC value will be.\n",
    "\n",
    "Topic 10 (public health) has the highest informativeness.  This is not surprising as the module has content very different to the other modules, but it's impressive how it has the fewest anchor words and gets the highest informativeness.\n",
    "\n",
    "At the other extreme, Topic 2 (advanced biostats) has the lowest informativeness, even though it has the most anchor words.  As mentioned earlier, it is difficult to separate the module's content from the others.  This is a useful diagnostic to quantify how difficult the separation is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8c4b306e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmMAAAFCCAYAAAC90NpzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdm0lEQVR4nO3de7glVX3m8e8rFwEvE5BWFOw0EMQoipoOgo6IYtSkUYiJGR1BJJieMYmCl8egRiVqlBijopMx6RClE4g3dNQBRQSkjUkAuaMCw61B5NYIAqLc5Dd/VHU4Hvtc6py9T53e+/t5nvPsXbXqVP16b2zfXmvVqlQVkiRJ6sdD+i5AkiRpnBnGJEmSemQYkyRJ6pFhTJIkqUeGMUmSpB4ZxiRJknq0ad8FzNW2225by5Yt67sMSZKkGZ177rm3VNWSDbVttGFs2bJlnHPOOX2XIUmSNKMk10zV5jClJElSjwxjkiRJPTKMSZIk9cgwJkmS1CPDmCRJUo8MY5IkST0yjEmSJPXIMCZJktQjw5gkSVKPDGOSJEk9MoxJkiT1aKN9NqUkSRp9y444aejXWHvUiqFfYzr2jEmSJPXIMCZJktQjw5gkSVKPDGOSJEk9MoxJkiT1yLspJUlDMw53wknzZc+YJElSjwxjkiRJPTKMSZIk9cgwJkmS1CPDmCRJUo8MY5IkST0yjEmSJPVoQcNYkk8muTnJdyfs2ybJN5Jc3r5uvZA1SZIk9Wmhe8aOBV48ad8RwGlVtQtwWrstSZI0FhY0jFXVt4BbJ+3eH1jdvl8NHLCQNUmSJPVpMcwZe0xV3QDQvj6653okSZIWzGIIY7OWZGWSc5Kcs27dur7LkSRJmrfFEMZuSvJYgPb15qkOrKpVVbW8qpYvWbJkwQqUJEkalsUQxr4CHNy+Pxj4co+1SJIkLaiFXtri08B/ALsmuS7JocBRwG8luRz4rXZbkiRpLGy6kBerqldO0bTvQtYhSZK0WCyGYUpJkqSxZRiTJEnqkWFMkiSpR4YxSZKkHhnGJEmSemQYkyRJ6pFhTJIkqUeGMUmSpB4ZxiRJknpkGJMkSeqRYUySJKlHhjFJkqQezfpB4Un2BF4M7Ak8DtgSuAW4DFgDfKmqbhtGkZIkSaNqxp6xJAcnuRj4d+BwYCvgcuAs4DbgmcAxwA+THJtkx+GVK0mSNFqm7RlLciHwaOCfgFcDF1RVbeC4/wLsB7wK+F6SQ6rqs0OoV5IkaaTMNEz5KeDvquru6Q6qqtuB44Hjk+wObDeg+iRJkkbatGGsqj7a9YRVdSFw4VwLkiRJGifzupsyyTZJfiPJQwdVkCRJ0jiZdRhL8udJPjBhe29gLXA2cHmSXQZfniRJ0mjr0jN2IHDVhO0P0gxHHgDcBLx3cGVJkiSNh1mvMwZsT7OkBUmWAL8J7FtVZyTZHPjYEOqTJEnzsOyIk4Z6/rVHrRjq+cdBl56xnwObt+/3Bu4G/q3dXgdsM8C6JEmSxkKXMPY94MAkDwf+EFhTVfe1bY8Hbh50cZIkSaOuyzDle4Av0yzseh/wogltvwOcN8C6JEmSxsKsw1hVfT3JrwPPoFmJ/8oJzd8CLhhwbZIkSSOvy9IWrwbuqKovTApiAJ8Hdh1oZZIkSWOgy5yxTwE7T9G2Y9suSZKkDrqEsUzT9jDg/nnWIkmSNHamnTOW5Gk0c8TWe0mS3SYdtiXwCto1yCRJkjR7M03g3x94d/u+gHdMcdyPgEMHVZQkSdK4mCmMfRQ4lmaI8irgZcD5k465B7ipqmrQxUmSJI26acNYVd0O3A6QZEfghqq6dyEKkyRJGgdd1hm7ZpiFSJIkjaMud1OSZGWS85P8NMnPJ/8Mq0hJkqRR1XXR148D3wG2oFlX7DjgDuBKmsclSZIkqYMuPWOHAx8AXtdu/++qOhjYCfgZzR2VkiRJ6qBLGNuF5hmUD7Q/mwNU1W3AXwKHDbw6SZKkEdcljP0MeEi7hMWNND1i6/0EeNwgC5MkSRoHs76bErgY+DXgVOBfgbcnuZrmMUhHApcOvDpJkqQR1yWMreLB3rB30oSyb7fbdwIHzKeQJG8EXkuz0v/FwCFVdfd8zilYdsRJQz3/2qNWDPX8kiSNui7rjH12wvsrkjwZ2AvYCvj3qrplrkUk2R54A/CkqvpZks/RPO/y2LmeU5IkaWPQpWfsF1TVXTS9Y4OsZcsk99EEvOsHeG5JkqRFqXMYS7IdsJRmrbFfUFXfmksRVfXDJB8CrqW5UeCUqjplLueSJEnamMw6jLVDiccBe2+omWau1yZzKSLJ1sD+wI7Aj4HPJzmwqo6bdNxKYCXA0qVL53IpSZKkRaVLz9gngN2At9JMsL9ngHW8ALi6qtYBJPki8Cya8PefqmoVzY0ELF++vAZ4fUmSpF50CWPPAd5QVf88hDquBfZMshXNMOW+wDlDuI4kSdKi0nXR15uHUURVnQWcAJxH0+v2ENoeMEmSpFHWJYz9A3DQsAqpqndX1ROrareqOqiqBjkMKkmStCh1Gab8IXBQktOBrwK3Tj6gqj45qMIkSZLGQZcw9nft6zJgnw20F2AYkyRJ6qBLGNtxaFVIkiSNqS6PQ7pmmIVIkiSNoy4T+CVJkjRg04axJBck+d0kmc3JkuyQ5GNJ3jqY8iRJkkbbTD1j/0yzpMUPk3wkycuS7JzkkUkemmS7JM9KcniS04C1wBOALw23bEmSpNEw7ZyxqvqbJMcArwUOBQ6juWtyotA8GunLwL5VtWYYhUqSJI2iGSfwV9XtwN8Af5Pk8cBewOOALYAfAZcCZ7tIqyRJUnddlragqn4A/GBItUiSJI0d76aUJEnqUaeeMUndLDvipKGef+1RK4Z6fknS8NkzJkmS1CPDmCRJUo8MY5IkST0yjEmSJPWo0wT+JDsBfwAspVlnbKKqqkMHVZgkSdI4mHUYS7I/8Hma3rSbaVbdn2jyyvySpHnyjlxp9HXpGXsfcAbwqqpaN5xyJEmSxkuXMLYT8GaDmCRJ0uB0mcB/KfCoYRUiSZI0jrqEsbcCb28n8UuSJGkAugxTHknTM3ZJksuBWye1V1U9d1CFSZIkjYMuYeznwGXDKkSSJGkczTqMVdU+Q6xDkiRpLLkCvyRJUo86hbEkj03yoSTfSXJlkrOTfDDJdsMqUJIkaZTNOowleQJwAfAG4CfA2cBdwGHABUl2GUaBkiRJo6zLBP6/Au4AnllVa9fvTPKrwClt+8sGWp0kSdKI6zJM+TzgnRODGEBVXUOz7MXzBleWJEnSeOgSxjYH7pyi7c62XZIkSR10CWMXAK9P8gu/kyTAH7ftkiRJ6qDLnLH3ACfSrMD/WeAGYDvg5cAuwIrBlydJkjTauiz6enKS/YD3Ae8AAhRwLrBfVZ0ynBIlSZJGV5eeMarqZODkJFsBWwO3VdVPh1KZJEnSGOgUxtZrA5ghTJIkaZ6mDWNJ3gUcU1XXt++nU1X13sGVJkmSNPpm6hk7EjgZuL59P50CDGOSJEkdTBvGquohG3ovSRuLZUecNNTzrz3KG8klzU+XZ1MuTbLZFG2bJlk6uLIkSZLGQ5ferquBp0/RtnvbPmdJfiXJCUkuTXJJkr3mcz5JkqSNQZe7KTNN22bAA/Os5Wjg5Kr6/SSbA1vN83ySJEmL3kx3U/4KsM2EXdsn2WnSYVsCBwM3zrWIJI8E9gZeA1BV9wL3zvV8kiRJG4uZesYOA95Nc6dkASdMcVza4+ZqJ2Ad8Kkku9Os6n9YVd31CxdJVgIrAZYudYqaJEna+M0Uxr4ErKUJW5+keRTSlZOOuQf4flVdNM86ngG8vqrOSnI0cATwzokHVdUqYBXA8uXLax7XkyRJWhRmWtriQuBCgCQFnFhVPxpCHdcB11XVWe32CTRhTJIkaaTN+m7Kqlo9pCBGVd0I/CDJru2ufYHvD+NakiRJi0mnZ1Mm2Q04FNgV2GJSc1XVvvOo5fXA8e2dlFcBh8zjXJIkDYQLB2vYZh3GkjwTWEMzh2wX4CJga2ApzTDjFfMppKouAJbP5xySJEkbmy6Lvr4f+CLwZJoJ/YdW1TLgBcAmNJP7JUmS1EGXMPZU4DiaJS6gCWBU1ek0QewDgy1NkiRp9HUJY5sBd1XVA8CtwGMntF0G7DbIwiRJksZBlzB2JbB9+/4i4A+TPCTJQ2gm2895BX5JkqRx1eVuyv8L7AP8C838sZOAO4CfAw8H3jDo4iRJkkbdrMNYVR054f2pSfYEfo/mgd4nV9Upgy9PkiRptHVaZ2yiqjofOH+AtUiSJI2dLnPGJEmSNGDT9owluZoHl7KYSVXVzvMvSZIkaXzMNEy5htmHMUmSJHU0bRirqtcsUB2SJEljyTljkiRJPeoUxpI8PckXk9yS5P4kz2j3vz/Ji4dToiRJ0uiadRhL8l+B/wCeSLPw68TffQD4n4MtTZIkafR1WWfsKODrwAE0Dwn/0wlt5wGvHlxZi8OyI04a6vnXHrViqOeXJEmLX5cw9gzgZVVVSSbfYXkLsGRwZUmSJI2HLnPG7qZ59NGGPBa4ff7lSJIkjZcuPWPfBg5P8uUJ+9b3kB0KnD6wqiQtGg7XS9JwdQlj7wT+DbgQOIEmiB2c5MPAbwC/OfjyJEmSRtushymr6kLgOcBNwDuA8OAk/udW1WWDL0+SJGm0zapnLMlmwO8AF1XVvkm2ALYBflxVPx1mgZIkSaNsVj1jVXUf8DlgWbt9d1VdbxCTJEmany53U14FPHpYhUiSJI2jLmHsg8A7kriemCRJ0oB0uZvy+TTzxK5OciZwAw8ubQFQVXXwIIuTJEkadV3C2HOA+4B1wM7tz0STV+WXJEnSDGYdxqpq2RDrkCRJGkuzmjOWZPMk5yV54bALkiRJGiezXdriXmBH4P7hliNJkjReutxN+Q3AnjFJkqQB6jKB/+PAcUk2Bb7EL99NSVVdNbjSJEmSRl+XMLamfX0T8MYpjtlkfuVIkiSNly5h7JChVSFJkjSmuixtsXqYhUiSJI2jLj1jACQJ8CSa1fh/BFxSVS74KkmSNAdd7qYkyWtpJu5fBJwBXAxcn+TQwZcmSZI0+mbdM5bkVcAq4DTgOOBGYDvgVcCqJD+tqk8PpUpJkqQR1WWY8q3A8VV10KT9q5P8M/BngGFMkiSpgy7DlLvS9IhtyHFtuyRJkjroEsbuBHaYom2Htn1ekmyS5PwkJ873XJIkSRuDLmHsa8D7kzxn4s4kewHva9vn6zDgkgGcR5IkaaPQJYy9FbgdOCPJtUnOSnIN8G3gjrZ9zpLsAKwAjpnPeSRJkjYmXRZ9vTHJ04A/BJ5Ds87YWprHJB1bVT+dZy0fpQl0j5jneSRJkjYanRZ9bQPX/2p/BibJfsDNVXVukn2mOW4lsBJg6dKlgyxBkiSpF7MepkyyZ5I/mKLt5UmeOY86ng28NMla4DPA85P80p2bVbWqqpZX1fIlS5bM43KSJEmLQ5eesQ8A35qi7deB1wHPn0sRVfU24G0Abc/YW6rqwLmcS6Nl2REnDfX8a49aMdTzS5I0ky4T+HcHzpyi7WzgqfMvR5Ikabx06RnbgqnD2ybAw+ZfDlTVGTTPvZQkSRp5XXrGLgFeOkXbS4HL5l+OJEnSeOnSM/Z3wN8nuQP4B+A6YHuauxsPBf548OVJkiSNti7rjP1Dkl2BNwJvmtgEfKSqVg26OEmSpFHXdZ2xtyT5BPAC4FHALcCpVXXVMIqTJEkadZ3CGEBVXQlcOYRaJEmSxk6XCfySJEkaMMOYJElSjwxjkiRJPTKMSZIk9cgwJkmS1CPDmCRJUo+mXdoiyQM0i7rORlVV56UyJEmSxtlM4ek9zD6MSZIkqaNpw1hVHblAdUiSJI0l54xJkiT1qNMcrySbA78N7ApsMam5quq9gypMkiRpHMw6jCV5HPBtYBnNPLK0TRPnlBnGJEmSOugyTPnXwDpgKU0QeyawE/CXwBXte0mSJHXQZZjyOcBbgOvb7Qeqai3wriSbAB8D9h9seZIkSaOtS8/Yo4Drq+oB4C5g6wltpwP7DLAuSZKksdAljF0HbNu+vxJ44YS2PYC7B1WUJEnSuOgyTPlN4LnAl4C/B/42ydOA+4AXtfskSZLUQZcw9ufANgBV9YkkmwL/DdgK+CDNav2SJEnqYNZhrKpuAW6ZsP1x4OPDKEqSJGlczHrOWJLTkzxxirYnJDl9cGVJkiSNhy4T+PcBHjlF2yNo5pNJkiSpg67Ppqwp9u8M/GSetUiSJI2daeeMJTkEOKTdLGBVkjsnHbYlsBtw2uDLkyRJGm0z9Yw9APy8/cmk7fU/PwI+ARw6vDIlSZJG07Q9Y1W1GlgNkOSbwOuq6tKFKEySJGkcdFna4nnDLESSJGkcdZrAn+QpSU5Isi7J/UluTvK5JE8ZVoGSJEmjbNY9Y0l+E1gD/Az4CnAjsB3wEmBFkr2r6tyhVClJkjSiujwO6QPAd4F9q+o/76hM8gjg1Lb9hVP8riRJkjagyzDlnsAHJgYxgHb7r4C9BlmYJEnSOOgSxqZa8HW27ZIkSZqkSxg7C3h7Oyz5n5I8DPgz4MxBFiZJkjQOZlqB/yrgd6vqQuDtwBnANUlOBG6gmcC/gmYV/n2GWqkkSdIImmkC/zLgoQBVdXaSPYF3AS8CtgFuBU4H3ltVFw+xTkmSpJHU5W5Kquoi4PcHXUSSxwP/RNPT9gCwqqqOHvR1JEmSFpvZhLGFmJh/P/DmqjqvnZN2bpJvVNX3F+DakiRJvZlNGPuLJLfM4riqqoPnUkRV3UAzB42qujPJJcD2gGFMkiSNtNmEsacB98ziuIH0oCVZBjyd5u5NSZKkkTabMHZAVZ099EqAJA8HvgAcXlV3bKB9JbASYOnSpQtRkiRJ0lB1elD4MCXZjCaIHV9VX9zQMVW1qqqWV9XyJUuWLGyBkiRJQ7AowliSAP8IXFJVH+67HkmSpIWyKMIY8GzgIOD5SS5of36n76IkSZKGbdo5Y1W1IGGtqr4NZCGuJUmStJgslp4xSZKksWQYkyRJ6pFhTJIkqUeGMUmSpB4ZxiRJknpkGJMkSeqRYUySJKlHhjFJkqQeGcYkSZJ6ZBiTJEnqkWFMkiSpR4YxSZKkHhnGJEmSemQYkyRJ6pFhTJIkqUeGMUmSpB4ZxiRJknpkGJMkSeqRYUySJKlHhjFJkqQeGcYkSZJ6ZBiTJEnqkWFMkiSpR4YxSZKkHhnGJEmSemQYkyRJ6pFhTJIkqUeGMUmSpB4ZxiRJknpkGJMkSeqRYUySJKlHhjFJkqQeGcYkSZJ6ZBiTJEnqkWFMkiSpR4YxSZKkHhnGJEmSemQYkyRJ6pFhTJIkqUeLJowleXGSy5JckeSIvuuRJElaCIsijCXZBPhb4LeBJwGvTPKkfquSJEkavkURxoA9gCuq6qqquhf4DLB/zzVJkiQN3WIJY9sDP5iwfV27T5IkaaSlqvqugSQvB15UVa9ttw8C9qiq1086biWwst3cFbhsQQud2bbALX0XMeb8Dvrnd9AvP//++R30bzF+B79aVUs21LDpQlcyheuAx0/Y3gG4fvJBVbUKWLVQRXWV5JyqWt53HePM76B/fgf98vPvn99B/za272CxDFN+B9glyY5JNgdeAXyl55okSZKGblH0jFXV/Un+FPg6sAnwyar6Xs9lSZIkDd2iCGMAVfVV4Kt91zFPi3YIdYz4HfTP76Bffv798zvo30b1HSyKCfySJEnjarHMGZMkSRpLhrEB8XFO/Ury+CTfTHJJku8lOazvmsZRkk2SnJ/kxL5rGUdJfiXJCUkubf+3sFffNY2TJG9s//75bpJPJ9mi75pGXZJPJrk5yXcn7NsmyTeSXN6+bt1njbNhGBsAH+e0KNwPvLmqfh3YE/gTv4NeHAZc0ncRY+xo4OSqeiKwO34XCybJ9sAbgOVVtRvNzWiv6LeqsXAs8OJJ+44ATquqXYDT2u1FzTA2GD7OqWdVdUNVnde+v5Pm/4R8isMCSrIDsAI4pu9axlGSRwJ7A/8IUFX3VtWPey1q/GwKbJlkU2ArNrBepgarqr4F3Dpp9/7A6vb9auCAhaxpLgxjg+HjnBaRJMuApwNn9VzKuPko8FbggZ7rGFc7AeuAT7VDxcckeVjfRY2Lqvoh8CHgWuAG4PaqOqXfqsbWY6rqBmj+oQ48uud6ZmQYG4xsYJ+3qfYgycOBLwCHV9UdfdczLpLsB9xcVef2XcsY2xR4BvCJqno6cBcbwfDMqGjnJe0P7Ag8DnhYkgP7rUobC8PYYMzqcU4ariSb0QSx46vqi33XM2aeDbw0yVqaYfrnJzmu35LGznXAdVW1vkf4BJpwpoXxAuDqqlpXVfcBXwSe1XNN4+qmJI8FaF9v7rmeGRnGBsPHOfUsSWjmylxSVR/uu55xU1Vvq6odqmoZzX//p1eVvQILqKpuBH6QZNd2177A93ssadxcC+yZZKv276N98QaKvnwFOLh9fzDw5R5rmZVFswL/xszHOS0KzwYOAi5OckG77+3tkx2kcfF64Pj2H4VXAYf0XM/YqKqzkpwAnEdzd/f5bGSrwG+Mknwa2AfYNsl1wLuBo4DPJTmUJiS/vL8KZ8cV+CVJknrkMKUkSVKPDGOSJEk9MoxJkiT1yDAmSZLUI8OYJElSjwxjkuYkSc3iZ+0Ar/ea9pzLBnXOxSLJWhfJlcaX64xJmqu9Jm3/H+BC4MgJ++4Z4PVOaq95wwDPKUm9M4xJmpOqOnPidpJ7gFsm7x/g9dbRPAhbc5DkoVU1yHAsaUAcppQ0NEn2SHJqkp8kuSvJaUn2mHTMsUmuS/KsJN9Jcnc7bPf6ScdtcJgyyR8lOS/Jz5LclmRNkmmfCdie531J3pDk6iR3tr/35EnHrU1y7BS/f+SE7SPbfU9M8vX2z3ptkkPa9oOSXNp+Dt9MsvMUdf1Rkivaz+C8JM/bwDHPbT/HO9vrfD3JbpOOOSPJt5O8JMn5bVD+4+k+E0n9MYxJGookTwXWAFsDrwFeDTwSWJNk90mHPxL4LLAaOAA4A/hYktfMcI0P0Txy5jzgD4ADgW8BS2dR4oHACuAwmscGLQW+nGQ+IwafpxlOPQA4F/hkkvcDrwOOaK+zK/AvG/jd5wJvAt5B83zPe4CvTXjWJElWAKcBP2nr/+/AI4B/TfL4Sed7AvAx4OPAi9rfk7QIOUwpaVjeRRMo9q2qHwMk+Qawlub5cS+bcOwjgJVV9Zl2++Qk2wN/kWR1beC5bUl+DXgj8JGqetOEppNmWd99wH5VdV97PmjC1B7Av8/yHJP9dVX9U3u+c4CXAP8D2LGq7mj3PxY4OsmvVtU1E373McCzq+ra9rjTgGuAP6d57irA0cCaqtp//S8l+SbNcyjfDBw+4XzbAi+sqgvm+GeRtEDsGZM0LHsDJ64PYgBtIPkKTS/QRD8HvjBp32doequ2n+L8L6D5O2yuD2P+xvog1rq4fZ1Nr9pUvrb+TVXdBtwMnLk+iLUubV8n92SduT6Itb9/Jw/etECSXYCdaR4Evun6H+CnwH/QfN4TrTWISRsHw5ikYdmGDd/5eCPN0OVEt00KRgA3ta9ThbFHta/Xza08bp20vX5y+xZzPB/AbZO2751i34aucxO/7CYe/PM/un39R5pevYk/+/Hg57Ged51KGwmHKSUNy63AdhvYvx2/HIS2TrLZpED2mPb1h1Oc/5b2dXvgsjlXOb27gc0n7kiyzZCu9Zgp9q3/8/+ofX0bcOoGjr130vYvDe1KWpzsGZM0LGuAFUkesX5H+/4lbdtEmwC/N2nfK4BrmTqMnQo8AKwcSLUbdg2w26R9+w3pWntOnITfflYraIYgoQmca4EnV9U5G/i5aEh1SRoye8YkDct7aYLLaUn+iqan5s+ArYD3TDr2TuCDSbYFLgdeSTMn7DUbmrwPUFVXJvkI8KY2uHyFZu7ZHsClVfXZAfwZPkNzR+RHgBOB3WnuDB2Gm4BT2iUz7qH5rB5G8zlSVZXkT2ju+Nwc+BxN7+BjgGcB11bVh4dUm6QhMoxJGoqquijJPsBf0ixZEeBM4LlVdeGkw++g6Qk7GngKTTA5rKpWz3CNtyS5gmYNrYOBu4CLgFMG9MdYTTPR/lCauyL/Ffhd4IoBnX+iNTRLerwf2AH4PvDbVfX/1h9QVV9NsjfN8hfHAFvSzME7k2ZpEEkboUzxj05JWhDtoqovqKod+q5FkvrgnDFJkqQeGcYkSZJ65DClJElSj+wZkyRJ6pFhTJIkqUeGMUmSpB4ZxiRJknpkGJMkSeqRYUySJKlH/x94kz5CIcvQnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "ax.bar(range(nlp.anchor_topic_model.tcs.shape[0]),\n",
    "       nlp.anchor_topic_model.tcs, width=0.5)\n",
    "plt.xlabel('Topic number', fontsize=16)\n",
    "plt.ylabel('Total correlation (nats)', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2e76a11",
   "metadata": {},
   "source": [
    "# Auto-generate MOCs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "583b44b4",
   "metadata": {},
   "source": [
    "The `auto_generate_MOCs` function loops over each topic to find the notes that are most relevant to a topic.  In this implementation, I create one MD file per topic (e.g. `topic_0.md`, `topic_1.md`, ...), and write a list of wikilinks to the top 30 notes most relevant to the topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ab8615fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "out_dir = Path(os.getcwd()) / 'data/out/mphil-mocs'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0e409c8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove dir if exists (for ease of iterative changes)\n",
    "if out_dir.exists():\n",
    "    try:\n",
    "        shutil.rmtree(out_dir)\n",
    "    except OSError:\n",
    "        os.remove(out_dir)\n",
    "# make dir:\n",
    "out_dir.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c1e49f5",
   "metadata": {},
   "source": [
    "I sort by `tc`.  The other option with `logp` does not work as well, as dozens of notes are tied for the highest score, which leads to alphabetical sorting on filepaths (not ideal!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "45ec233e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "WARNING: sorting by logz not well tested\n",
      "CPU times: user 4.48 ms, sys: 8 µs, total: 4.49 ms\n",
      "Wall time: 4.23 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "auto_generate_MOCs(nlp, out_dir=out_dir, top_n_docs=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "753b1450",
   "metadata": {},
   "source": [
    "Let's look at the output for Topic 10's MOC (which was intended to cover the public health module)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4fd16eb7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Topic 10\n",
      "Anchor words: ['public', 'prevention']\n",
      "- [[Public health policy]]\n",
      "- [[Ethics]]\n",
      "- [[Evaluation]]\n",
      "- [[Risk communication]]\n",
      "- [[Research ethics]]\n",
      "- [[Determinants of health]]\n",
      "- [[Complex system]]\n",
      "- [[Emergency planning]]\n",
      "- [[Market]]\n",
      "- [[Public health]]\n"
     ]
    }
   ],
   "source": [
    "with open(out_dir / 'topic_10.md', 'r') as f:\n",
    "    lines = f.readlines()\n",
    "    for line in lines[:(2+10)]:  # show only top 10 for this output\n",
    "        print(line.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c760e6d",
   "metadata": {},
   "source": [
    "The notes that are assigned to the public health module folder are in blue:\n",
    "![#007bff](https://via.placeholder.com/15/007bff/007bff.png)\n",
    "\n",
    "In the MOC for this module, most of the notes do in fact relate to that module.  Nonetheless there are a few outside the module that entered the auto-generated MOC.  The earlier image in this notebook shows that there can be some overlap in concepts between modules.  For example, the note I made on 'Public health' was made when studying epidemiology (so I kept _first attribution_ of the note to that module!)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3510540e",
   "metadata": {},
   "source": [
    "![Topic 10 MOC](../img/MPhil_topic-10-MOC.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3822adac",
   "metadata": {},
   "source": [
    "The notes correspond very well to the main concepts covered in the module.  This makes the MOC very effective for my needs! 👍\n",
    "\n",
    "In a graph of 800+ notes, it would take me many days to create MOCs from scratch and review the results manually.  I would also be relying on: i) the connections between notes to make MOC generation more efficient; ii) the existing categories that I already use to organise my notes.\n",
    "\n",
    "In contrast, this approach via topic modelling only took a few hours to implement and tune.  The results it gives are better than what I could have done manually, as it uses all the text in the vault.  The MOC files generated can be 'pruned' by the user as well, if wanting to remove or add specific wikilinks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c59b502",
   "metadata": {},
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "887624cc",
   "metadata": {},
   "source": [
    "- Gallagher, R. J., Reing, K., Kale, D., and Ver Steeg, G. \"[Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge](https://www.transacl.org/ojs/index.php/tacl/article/view/1244).\" *Transactions of the Association for Computational Linguistics (TACL)*, 2017."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}