KeywordExtractionNotebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style = \"text-align:center\">\n",
    "<h1> Feature Extraction Tutorial </h1>\n",
    "<h3> Keyword extraction has six main seteps </h3>\n",
    "<img src=\"./steps.jpeg\" width = \"600px\" height= \"500px\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> We first need to import the needed packages </h2>\n",
    "<p>This includes: </p>\n",
    "<ol>\n",
    "    <li> <b>nltk.corpus.stopwords</b> : This package allows us to get curated stopwords for many different languages </li>\n",
    "    <li> <b> nltk.stem.wordnet.WordNetLemmatizer </b>: This package is used for the lemmatization process. The WordNetLemmatizer uses the Princeton University WordNet database to get the root of words. WordNet has many other uses as well. More information \n",
    "        <a href=\"https://pythonprogramming.net/wordnet-nltk-tutorial/\"> here </a> </li>\n",
    "    <li> <b> re </b>: a regular expression package for python and my prefered choice for text extraction, and cleaning. NLTK also has really utilities for text cleaning as well. More information on what regular expressions are can be found <a href = \"https://www.regular-expressions.info/\" > here </a></li>\n",
    "    <li> <b> sklearn.feature_extraction.text.CountVectorizer </b>: This package is used to generate the frequency matrix. More information can be found \n",
    " <a href = \"https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html\">here </a> </li>\n",
    "    <li> <b>pandas</b>: Since we'll be using a csv of comments today, one of the best way to manipulate this data is by using a pandas dataframe constructed from this csv. An awesome tutorial on pandas can be found <a href = \"https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python\"></a></li>\n",
    "    \n",
    "</ol>\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
     ]
    }
   ],
   "source": [
    "#If this is the first time importing the stopwords package /\n",
    "#you must first download the stopwords using nltk.download('stopwords')\n",
    "import re\n",
    "import pandas\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "nltk.download('stopwords')\n",
    "nltk.download('wordnet')\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Importing our data </h2>\n",
    "<p> I have provided a very popular dataset called the NIPS papers you can find more information about this dataset <a href= \"https://www.kaggle.com/benhamner/nips-papers/home\"> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows : 2999\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>id</th>\n",
       "      <th>year</th>\n",
       "      <th>title</th>\n",
       "      <th>event_type</th>\n",
       "      <th>pdf_name</th>\n",
       "      <th>abstract</th>\n",
       "      <th>paper_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>1987</td>\n",
       "      <td>A Mean Field Theory of Layer IV of Visual Cort...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10-a-mean-field-theory-of-layer-iv-of-visual-c...</td>\n",
       "      <td>Abstract Missing</td>\n",
       "      <td>683\\n\\nA MEAN FIELD THEORY OF LAYER IV OF VISU...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>100</td>\n",
       "      <td>1988</td>\n",
       "      <td>Storing Covariance by the Associative Long-Ter...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100-storing-covariance-by-the-associative-long...</td>\n",
       "      <td>Abstract Missing</td>\n",
       "      <td>394\\n\\nSTORING COVARIANCE BY THE ASSOCIATIVE\\n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1000</td>\n",
       "      <td>1994</td>\n",
       "      <td>Bayesian Query Construction for Neural Network...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000-bayesian-query-construction-for-neural-ne...</td>\n",
       "      <td>Abstract Missing</td>\n",
       "      <td>Bayesian Query Construction for Neural\\nNetwor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1001</td>\n",
       "      <td>1994</td>\n",
       "      <td>Neural Network Ensembles, Cross Validation, an...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1001-neural-network-ensembles-cross-validation...</td>\n",
       "      <td>Abstract Missing</td>\n",
       "      <td>Neural Network Ensembles, Cross\\nValidation, a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1002</td>\n",
       "      <td>1994</td>\n",
       "      <td>Using a neural net to instantiate a deformable...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1002-using-a-neural-net-to-instantiate-a-defor...</td>\n",
       "      <td>Abstract Missing</td>\n",
       "      <td>U sing a neural net to instantiate a\\ndeformab...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0    id  year                                              title  \\\n",
       "0           1    10  1987  A Mean Field Theory of Layer IV of Visual Cort...   \n",
       "1           2   100  1988  Storing Covariance by the Associative Long-Ter...   \n",
       "2           3  1000  1994  Bayesian Query Construction for Neural Network...   \n",
       "3           4  1001  1994  Neural Network Ensembles, Cross Validation, an...   \n",
       "4           5  1002  1994  Using a neural net to instantiate a deformable...   \n",
       "\n",
       "   event_type                                           pdf_name  \\\n",
       "0         NaN  10-a-mean-field-theory-of-layer-iv-of-visual-c...   \n",
       "1         NaN  100-storing-covariance-by-the-associative-long...   \n",
       "2         NaN  1000-bayesian-query-construction-for-neural-ne...   \n",
       "3         NaN  1001-neural-network-ensembles-cross-validation...   \n",
       "4         NaN  1002-using-a-neural-net-to-instantiate-a-defor...   \n",
       "\n",
       "           abstract                                         paper_text  \n",
       "0  Abstract Missing  683\\n\\nA MEAN FIELD THEORY OF LAYER IV OF VISU...  \n",
       "1  Abstract Missing  394\\n\\nSTORING COVARIANCE BY THE ASSOCIATIVE\\n...  \n",
       "2  Abstract Missing  Bayesian Query Construction for Neural\\nNetwor...  \n",
       "3  Abstract Missing  Neural Network Ensembles, Cross\\nValidation, a...  \n",
       "4  Abstract Missing  U sing a neural net to instantiate a\\ndeformab...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#here we construct a dataframe from the papers.csv in the nips-paper folder\n",
    "nips_papers = pandas.read_csv(\"nips-papers/papers.csv\")\n",
    "#lets look at some of the data\n",
    "print(\"Number of rows : \" + str(nips_papers.shape[0]))\n",
    "nips_papers.head()\n",
    "#Its clear we need to do some cleaning, we want only rows which have abstracts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows : 413\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>id</th>\n",
       "      <th>year</th>\n",
       "      <th>title</th>\n",
       "      <th>event_type</th>\n",
       "      <th>pdf_name</th>\n",
       "      <th>abstract</th>\n",
       "      <th>paper_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>940</th>\n",
       "      <td>941</td>\n",
       "      <td>1861</td>\n",
       "      <td>2000</td>\n",
       "      <td>Algorithms for Non-negative Matrix Factorization</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1861-algorithms-for-non-negative-matrix-factor...</td>\n",
       "      <td>Non-negative matrix factorization (NMF) has pr...</td>\n",
       "      <td>Algorithms for Non-negative Matrix\\nFactorizat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1066</th>\n",
       "      <td>1067</td>\n",
       "      <td>1975</td>\n",
       "      <td>2001</td>\n",
       "      <td>Characterizing Neural Gain Control using Spike...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1975-characterizing-neural-gain-control-using-...</td>\n",
       "      <td>Spike-triggered averaging techniques are effec...</td>\n",
       "      <td>Characterizing neural gain control using\\nspik...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2383</th>\n",
       "      <td>2384</td>\n",
       "      <td>3163</td>\n",
       "      <td>2007</td>\n",
       "      <td>Competition Adds Complexity</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3163-competition-adds-complexity.pdf</td>\n",
       "      <td>It is known that determinining whether a DEC-P...</td>\n",
       "      <td>Competition adds complexity\\n\\nJudy Goldsmith\\...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2384</th>\n",
       "      <td>2385</td>\n",
       "      <td>3164</td>\n",
       "      <td>2007</td>\n",
       "      <td>Efficient Principled Learning of Thin Junction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3164-efficient-principled-learning-of-thin-jun...</td>\n",
       "      <td>We present the first truly polynomial algorith...</td>\n",
       "      <td>Efficient Principled Learning of Thin Junction...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2387</th>\n",
       "      <td>2388</td>\n",
       "      <td>3167</td>\n",
       "      <td>2007</td>\n",
       "      <td>Regularized Boost for Semi-Supervised Learning</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3167-regularized-boost-for-semi-supervised-lea...</td>\n",
       "      <td>Semi-supervised inductive learning concerns ho...</td>\n",
       "      <td>Regularized Boost for Semi-Supervised Learning...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0    id  year  \\\n",
       "940          941  1861  2000   \n",
       "1066        1067  1975  2001   \n",
       "2383        2384  3163  2007   \n",
       "2384        2385  3164  2007   \n",
       "2387        2388  3167  2007   \n",
       "\n",
       "                                                  title  event_type  \\\n",
       "940    Algorithms for Non-negative Matrix Factorization         NaN   \n",
       "1066  Characterizing Neural Gain Control using Spike...         NaN   \n",
       "2383                        Competition Adds Complexity         NaN   \n",
       "2384  Efficient Principled Learning of Thin Junction...         NaN   \n",
       "2387     Regularized Boost for Semi-Supervised Learning         NaN   \n",
       "\n",
       "                                               pdf_name  \\\n",
       "940   1861-algorithms-for-non-negative-matrix-factor...   \n",
       "1066  1975-characterizing-neural-gain-control-using-...   \n",
       "2383               3163-competition-adds-complexity.pdf   \n",
       "2384  3164-efficient-principled-learning-of-thin-jun...   \n",
       "2387  3167-regularized-boost-for-semi-supervised-lea...   \n",
       "\n",
       "                                               abstract  \\\n",
       "940   Non-negative matrix factorization (NMF) has pr...   \n",
       "1066  Spike-triggered averaging techniques are effec...   \n",
       "2383  It is known that determinining whether a DEC-P...   \n",
       "2384  We present the first truly polynomial algorith...   \n",
       "2387  Semi-supervised inductive learning concerns ho...   \n",
       "\n",
       "                                             paper_text  \n",
       "940   Algorithms for Non-negative Matrix\\nFactorizat...  \n",
       "1066  Characterizing neural gain control using\\nspik...  \n",
       "2383  Competition adds complexity\\n\\nJudy Goldsmith\\...  \n",
       "2384  Efficient Principled Learning of Thin Junction...  \n",
       "2387  Regularized Boost for Semi-Supervised Learning...  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we select the rows only when the column \"abstract\" is not equal to the default value \n",
    "nips_papers = nips_papers[nips_papers['abstract'] != \"Abstract Missing\"]\n",
    "print(\"Number of rows : \" + str(nips_papers.shape[0]))\n",
    "nips_papers.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sum \t57452\n",
      "count    413.000000\n",
      "mean     139.108959\n",
      "std       44.134286\n",
      "min       19.000000\n",
      "25%      107.000000\n",
      "50%      133.000000\n",
      "75%      168.000000\n",
      "max      290.000000\n",
      "Name: word_count, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>abstract</th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>940</td>\n",
       "      <td>Non-negative matrix factorization (NMF) has pr...</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1066</td>\n",
       "      <td>Spike-triggered averaging techniques are effec...</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2383</td>\n",
       "      <td>It is known that determinining whether a DEC-P...</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2384</td>\n",
       "      <td>We present the first truly polynomial algorith...</td>\n",
       "      <td>143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2387</td>\n",
       "      <td>Semi-supervised inductive learning concerns ho...</td>\n",
       "      <td>119</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                                           abstract  word_count\n",
       "0    940  Non-negative matrix factorization (NMF) has pr...         107\n",
       "1   1066  Spike-triggered averaging techniques are effec...          81\n",
       "2   2383  It is known that determinining whether a DEC-P...          67\n",
       "3   2384  We present the first truly polynomial algorith...         143\n",
       "4   2387  Semi-supervised inductive learning concerns ho...         119"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#now lets remove all but the data we want\n",
    "abstract_dataset = pandas.DataFrame(columns=['abstract', 'word_count'])\n",
    "abstract_dataset['abstract'] = nips_papers['abstract']\n",
    "abstract_dataset.reset_index(inplace = True)\n",
    "abstract_dataset['word_count'] = abstract_dataset['abstract'].apply(lambda x: len(x.split(\" \")))\n",
    "total_wordcount = abstract_dataset['word_count'].sum()\n",
    "print(\"sum \\t\" + str(total_wordcount))\n",
    "print(abstract_dataset.word_count.describe())\n",
    "abstract_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src = \"./toomuch.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Looking at our data</h2>\n",
    "<p> lets look at the most common words without cleaning our data </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "abstract_raw_word_counts = pandas.Series(''.join(abstract_dataset['abstract']).split()).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "the                     2905\n",
       "of                      2196\n",
       "a                       1677\n",
       "and                     1293\n",
       "to                      1272\n",
       "in                       994\n",
       "is                       804\n",
       "that                     795\n",
       "for                      747\n",
       "on                       515\n",
       "We                       487\n",
       "we                       460\n",
       "with                     416\n",
       "as                       367\n",
       "are                      363\n",
       "this                     357\n",
       "an                       350\n",
       "learning                 309\n",
       "be                       305\n",
       "by                       305\n",
       "can                      302\n",
       "which                    299\n",
       "from                     265\n",
       "model                    253\n",
       "The                      241\n",
       "data                     239\n",
       "In                       226\n",
       "show                     211\n",
       "algorithm                206\n",
       "our                      190\n",
       "                        ... \n",
       "MB.                        1\n",
       "($p$                       1\n",
       "smoother                   1\n",
       "Performance                1\n",
       "attributes                 1\n",
       "within-subject             1\n",
       "constraints;               1\n",
       "extracted.                 1\n",
       "extant                     1\n",
       "perspective.               1\n",
       "research,                  1\n",
       "L1-norm.                   1\n",
       "written                    1\n",
       "low-load,                  1\n",
       "equally                    1\n",
       "0.86--                     1\n",
       "though,                    1\n",
       "mapping.                   1\n",
       "tree.Semi-supervised       1\n",
       "multi-component            1\n",
       "surround,                  1\n",
       "respecting                 1\n",
       "Sellie                     1\n",
       "polynomial,                1\n",
       "energy.In                  1\n",
       "(STDP).                    1\n",
       "datasets.Many              1\n",
       "hierarchies,               1\n",
       "(Kalai                     1\n",
       "creates                    1\n",
       "Length: 8881, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we can see we can't really get useful informative words the way that the data currently is\n",
    "abstract_raw_word_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Cleaning our data </h2>\n",
    "<p> To clean our data we need to do the following </p>\n",
    "<ul>\n",
    "    <li> remove punctuation and special symbols </li>\n",
    "    <li> remove tags </li>\n",
    "    <li> remove digits </li>\n",
    "    <li> make all characters lower case </li>\n",
    "<ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' Non negative matrix factorization NMF has previously been shown to be a useful decomposition for multivariate data Two different multi plicative algorithms for NMF are analyzed They differ only slightly in the multiplicative factor used in the update rules One algorithm can be shown to minimize the conventional least squares error while the other minimizes the generalized Kullback Leibler divergence The monotonic convergence of both algorithms can be proven using an auxiliary func tion analogous to that used for proving convergence of the Expectation Maximization algorithm The algorithms can also be interpreted as diag onally rescaled gradient descent where the rescaling factor is optimally chosen to ensure convergence  Spike triggered averaging techniques are effective for linear characterization of neural responses But neurons exhibit important nonlinear behaviors such as gain control that are not captured by such analyses We describe a spike triggered covariance method for retrievi'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we loop through all our rows\n",
    "corpus = \"\"\n",
    "for abstract in abstract_dataset['abstract']:\n",
    "    #remove all punctuation\n",
    "    abstract = re.sub('[^a-zA-Z]', ' ', abstract)\n",
    "    #empty tags \n",
    "    abstract = re.sub(\"&lt;/?.*?&gt;\", \" &lt;&gt; \", abstract)\n",
    "    #remove special characters and digits\n",
    "    abstract = re.sub(\"(\\\\d|\\\\W)+\",\" \", abstract)\n",
    "    #append to corpus\n",
    "    corpus = \" \".join((corpus, abstract))\n",
    "corpus[:1000]  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Normalization and Lemmatization </h2>\n",
    "<p> Here we remove all stopwords and get the lemmas for our words</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Non negative matrix factorization NMF previously shown useful decomposition multivariate data Two different multi plicative algorithm NMF analyzed They differ slightly multiplicative factor used update rule One algorithm shown minimize conventional least square error minimizes generalized Kullback Leibler divergence The monotonic convergence algorithm proven using auxiliary func tion analogous used proving convergence Expectation Maximization algorithm The algorithm also interpreted diag onally rescaled gradient descent rescaling factor optimally chosen ensure convergence',\n",
       " 'Spike triggered averaging technique effective linear characterization neural response But neuron exhibit important nonlinear behavior gain control captured analysis We describe spike triggered covariance method retrieving suppressive component gain control signal neuron We demonstrate method simulation retinal ganglion cell data Analysis physiological data reveals significant suppressive ax explains neural nonlinearities This method applicable sensory area modality']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = []\n",
    "#initialize the lemmatizer\n",
    "lem = WordNetLemmatizer()\n",
    "#create a set of stopwords\n",
    "stop_words = set(stopwords.words(\"english\"))\n",
    "for abstract in abstract_dataset['abstract']:\n",
    "    #remove all punctuation\n",
    "    abstract = re.sub('[^a-zA-Z]', ' ', abstract)\n",
    "    #empty tags \n",
    "    abstract = re.sub(\"&lt;/?.*?&gt;\", \" &lt;&gt; \", abstract)\n",
    "    #remove special characters and digits\n",
    "    abstract = re.sub(\"(\\\\d|\\\\W)+\",\" \", abstract)\n",
    "    #make all chars lowercase\n",
    "    abstract.lower()\n",
    "    #create a list of words \n",
    "    words = abstract.split()\n",
    "    text = \" \".join([lem.lemmatize(word) for word in words if \\\n",
    "                    word not in stop_words])\n",
    "    corpus.append(text)\n",
    "# we have now generated a normalized corpus with only word lemmas and \n",
    "corpus[:2]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Create the count vector </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer(stop_words= stop_words, max_features = 10000, ngram_range= (1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<413x4534 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 26681 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#load the corpus into the CountVectorizer\n",
    "cv = cv.fit(corpus)\n",
    "frequency_matrix = cv.transform(corpus)\n",
    "frequency_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum_words = frequency_matrix.sum(axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_frequencies = [(word, sum_words[0,idx]) for word, idx in cv.vocabulary_.items()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('model', 485),\n",
       " ('algorithm', 413),\n",
       " ('learning', 398),\n",
       " ('data', 356),\n",
       " ('method', 338),\n",
       " ('problem', 304),\n",
       " ('show', 234),\n",
       " ('based', 234),\n",
       " ('approach', 206),\n",
       " ('function', 185),\n",
       " ('using', 167),\n",
       " ('result', 164),\n",
       " ('paper', 149),\n",
       " ('task', 142),\n",
       " ('image', 141),\n",
       " ('set', 134),\n",
       " ('feature', 133),\n",
       " ('kernel', 130),\n",
       " ('present', 129),\n",
       " ('new', 126)]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_frequencies = sorted(word_frequencies, key = lambda x: x[1], reverse = True)\n",
    "word_frequencies[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}