2-Exploratory-Data-Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the data cleaning step where we put our data into a few standard formats, the next step is to take a look at the data and see if what we're looking at makes sense. Before applying any fancy algorithms, it's always important to explore the data first.\n",
    "\n",
    "When working with numerical data, some of the exploratory data analysis (EDA) techniques we can use include finding the average of the data set, the distribution of the data, the most common values, etc. The idea is the same when working with text data. We are going to find some more obvious patterns with EDA before identifying the hidden patterns with machines learning (ML) techniques. We are going to look at the following for each comedian:\n",
    "\n",
    "1. **Most common words** - find these and create word clouds\n",
    "2. **Size of vocabulary** - look number of unique words and also how quickly someone speaks\n",
    "3. **Amount of profanity** - most common terms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Most Common Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the document-term matrix\n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_pickle('dtm.pkl')\n",
    "data = data.transpose()\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the top 30 words said by each comedian\n",
    "top_dict = {}\n",
    "for c in data.columns:\n",
    "    top = data[c].sort_values(ascending=False).head(30)\n",
    "    top_dict[c]= list(zip(top.index, top.values))\n",
    "\n",
    "top_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the top 15 words said by each comedian\n",
    "for comedian, top_words in top_dict.items():\n",
    "    print(comedian)\n",
    "    print(', '.join([word for word, count in top_words[0:14]]))\n",
    "    print('---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** At this point, we could go on and create word clouds. However, by looking at these top words, you can see that some of them have very little meaning and could be added to a stop words list, so let's do just that.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the most common top words --> add them to the stop word list\n",
    "from collections import Counter\n",
    "\n",
    "# Let's first pull out the top 30 words for each comedian\n",
    "words = []\n",
    "for comedian in data.columns:\n",
    "    top = [word for (word, count) in top_dict[comedian]]\n",
    "    for t in top:\n",
    "        words.append(t)\n",
    "        \n",
    "words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's aggregate this list and identify the most common words along with how many routines they occur in\n",
    "Counter(words).most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If more than half of the comedians have it as a top word, exclude it from the list\n",
    "add_stop_words = [word for word, count in Counter(words).most_common() if count > 6]\n",
    "add_stop_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's update our document-term matrix with the new list of stop words\n",
    "from sklearn.feature_extraction import text \n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Read in cleaned data\n",
    "data_clean = pd.read_pickle('data_clean.pkl')\n",
    "\n",
    "# Add new stop words\n",
    "stop_words = text.ENGLISH_STOP_WORDS.union(add_stop_words)\n",
    "\n",
    "# Recreate document-term matrix\n",
    "cv = CountVectorizer(stop_words=stop_words)\n",
    "data_cv = cv.fit_transform(data_clean.transcript)\n",
    "data_stop = pd.DataFrame(data_cv.toarray(), columns=cv.get_feature_names())\n",
    "data_stop.index = data_clean.index\n",
    "\n",
    "# Pickle it for later use\n",
    "import pickle\n",
    "pickle.dump(cv, open(\"cv_stop.pkl\", \"wb\"))\n",
    "data_stop.to_pickle(\"dtm_stop.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's make some word clouds!\n",
    "# Terminal / Anaconda Prompt: conda install -c conda-forge wordcloud\n",
    "from wordcloud import WordCloud\n",
    "\n",
    "wc = WordCloud(stopwords=stop_words, background_color=\"white\", colormap=\"Dark2\",\n",
    "               max_font_size=150, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Reset the output dimensions\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rcParams['figure.figsize'] = [16, 6]\n",
    "\n",
    "full_names = ['Ali Wong', 'Anthony Jeselnik', 'Bill Burr', 'Bo Burnham', 'Dave Chappelle', 'Hasan Minhaj',\n",
    "              'Jim Jefferies', 'Joe Rogan', 'John Mulaney', 'Louis C.K.', 'Mike Birbiglia', 'Ricky Gervais']\n",
    "\n",
    "# Create subplots for each comedian\n",
    "for index, comedian in enumerate(data.columns):\n",
    "    wc.generate(data_clean.transcript[comedian])\n",
    "    \n",
    "    plt.subplot(3, 4, index+1)\n",
    "    plt.imshow(wc, interpolation=\"bilinear\")\n",
    "    plt.axis(\"off\")\n",
    "    plt.title(full_names[index])\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Ali Wong says the s-word a lot and talks about her husband. I guess that's funny to me.\n",
    "* A lot of people use the F-word. Let's dig into that later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the number of unique words that each comedian uses\n",
    "\n",
    "# Identify the non-zero items in the document-term matrix, meaning that the word occurs at least once\n",
    "unique_list = []\n",
    "for comedian in data.columns:\n",
    "    uniques = data[comedian].to_numpy().nonzero()[0].size\n",
    "    unique_list.append(uniques)\n",
    "\n",
    "# Create a new dataframe that contains this unique word count\n",
    "data_words = pd.DataFrame(list(zip(full_names, unique_list)), columns=['comedian', 'unique_words'])\n",
    "data_unique_sort = data_words.sort_values(by='unique_words')\n",
    "data_unique_sort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the words per minute of each comedian\n",
    "\n",
    "# Find the total number of words that a comedian uses\n",
    "total_list = []\n",
    "for comedian in data.columns:\n",
    "    totals = sum(data[comedian])\n",
    "    total_list.append(totals)\n",
    "    \n",
    "# Comedy special run times from IMDB, in minutes\n",
    "run_times = [60, 59, 80, 60, 67, 73, 77, 63, 62, 58, 76, 79]\n",
    "\n",
    "# Let's add some columns to our dataframe\n",
    "data_words['total_words'] = total_list\n",
    "data_words['run_times'] = run_times\n",
    "data_words['words_per_minute'] = data_words['total_words'] / data_words['run_times']\n",
    "\n",
    "# Sort the dataframe by words per minute to see who talks the slowest and fastest\n",
    "data_wpm_sort = data_words.sort_values(by='words_per_minute')\n",
    "data_wpm_sort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's plot our findings\n",
    "import numpy as np\n",
    "\n",
    "y_pos = np.arange(len(data_words))\n",
    "\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.barh(y_pos, data_unique_sort.unique_words, align='center')\n",
    "plt.yticks(y_pos, data_unique_sort.comedian)\n",
    "plt.title('Number of Unique Words', fontsize=20)\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.barh(y_pos, data_wpm_sort.words_per_minute, align='center')\n",
    "plt.yticks(y_pos, data_wpm_sort.comedian)\n",
    "plt.title('Number of Words Per Minute', fontsize=20)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Vocabulary**\n",
    "   * Ricky Gervais (British comedy) and Bill Burr (podcast host) use a lot of words in their comedy\n",
    "   * Louis C.K. (self-depricating comedy) and Anthony Jeselnik (dark humor) have a smaller vocabulary\n",
    "\n",
    "\n",
    "* **Talking Speed**\n",
    "   * Joe Rogan (blue comedy) and Bill Burr (podcast host) talk fast\n",
    "   * Bo Burnham (musical comedy) and Anthony Jeselnik (dark humor) talk slow\n",
    "   \n",
    "Ali Wong is somewhere in the middle in both cases. Nothing too interesting here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Amount of Profanity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Earlier I said we'd revisit profanity. Let's take a look at the most common words again.\n",
    "Counter(words).most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's isolate just these bad words\n",
    "data_bad_words = data.transpose()[['fucking', 'fuck', 'shit']]\n",
    "data_profanity = pd.concat([data_bad_words.fucking + data_bad_words.fuck, data_bad_words.shit], axis=1)\n",
    "data_profanity.columns = ['f_word', 's_word']\n",
    "data_profanity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's create a scatter plot of our findings\n",
    "plt.rcParams['figure.figsize'] = [10, 8]\n",
    "\n",
    "for i, comedian in enumerate(data_profanity.index):\n",
    "    x = data_profanity.f_word.loc[comedian]\n",
    "    y = data_profanity.s_word.loc[comedian]\n",
    "    plt.scatter(x, y, color='blue')\n",
    "    plt.text(x+1.5, y+0.5, full_names[i], fontsize=10)\n",
    "    plt.xlim(-5, 155) \n",
    "    \n",
    "plt.title('Number of Bad Words Used in Routine', fontsize=20)\n",
    "plt.xlabel('Number of F Bombs', fontsize=15)\n",
    "plt.ylabel('Number of S Words', fontsize=15)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Averaging 2 F-Bombs Per Minute!** - I don't like too much swearing, especially the f-word, which is probably why I've never heard of Bill Bur, Joe Rogan and Jim Jefferies.\n",
    "* **Clean Humor** - It looks like profanity might be a good predictor of the type of comedy I like. Besides Ali Wong, my two other favorite comedians in this group are John Mulaney and Mike Birbiglia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Side Note"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What was our goal for the EDA portion of our journey? **To be able to take an initial look at our data and see if the results of some basic analysis made sense.**\n",
    "\n",
    "My conclusion - yes, it does, for a first pass. There are definitely some things that could be better cleaned up, such as adding more stop words or including bi-grams. But we can save that for another day. The results, especially the profanity findings, are interesting and make general sense, so we're going to move on.\n",
    "\n",
    "As a reminder, the data science process is an interative one. It's better to see some non-perfect but acceptable results to help you quickly decide whether your project is a dud or not, instead of having analysis paralysis and never delivering anything.\n",
    "\n",
    "**Alice's data science (and life) motto: Let go of perfectionism!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Additional Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. What other word counts do you think would be interesting to compare instead of the f-word and s-word? Create a scatter plot comparing them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}