project/reports/election_prediction/ElectionPrediction_Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting an Election from Tweets     \n",
    "\n",
    "[Michaël Juillard](michael.juillard@epfl.ch), \n",
    "[Mikhail Vorobiev](mikhail.vorobiev@epfl.ch),\n",
    "[Chiara Ercolani](chiara.ercolani@epfl.ch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Problem Definition\n",
    "\n",
    "Nowadays social medias like Twitter and Facebook are means by which people continuously express their opinion on any matter. Thus, data mining combined with data analysis could be a great way to undersand the general feeling of the users on a certian matter.\n",
    "With this in mind, we decided to try to predict the result of an election using data from Twitter.\n",
    "\n",
    "We focused on the US Senate Election of 2016, since this election would provide enough meaningful data for our analysis. In fact there are many candidates and a lot of people are tweeting about them, allowing us to have a big dataset to train our algorithm with. We started by mining data from Twitter, collecting every Tweet regarding most of the Republican and Democrate Senate candidates during the two months preceeding the elections. We collected tweets aimed at each candidate, meaning tweets containing the candidate's twitter name . \n",
    "\n",
    "The second step was to perform a sentiment analysis on these tweets to understand if the user was expressing a positive or negative feeling towards the candidate. Sentiment analysis combines natural language processing, text analysis and computational linguistics to assess the attitude of a writer towards a topic. There are various tools for it, we picked one called Pattern developed by the University of Antwerp, in Belgium. \n",
    "\n",
    "Afterwards we trained two variations of the same machine learning algorithm with the sentiment analysis data and some other features like number of followers of the candidate. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 Resources\n",
    "\n",
    "For the project we used a variety of tools, here are the links to their websites.\n",
    "\n",
    "Links used for Twitter data mining\n",
    "* [US Senate Elections 2016 Wikipedia page](https://en.wikipedia.org/wiki/United_States_Senate_elections,_2016)\n",
    "* [Twitter REST API](https://dev.twitter.com/rest/public)\n",
    "* [Tool to get older Tweets](https://github.com/Jefferson-Henrique/GetOldTweets-python)\n",
    "\n",
    "\n",
    "Tool used for the sentiment analysis\n",
    "* [Sentiment Analysis Tool](http://www.clips.ua.ac.be/pattern)\n",
    "\n",
    "Papers about election prediction with tweets\n",
    "* [Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment](https://www.google.ch/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjqp9O79qPRAhWCvhQKHb-SAq4QFggcMAA&url=http%3A%2F%2Fwww.aaai.org%2Focs%2Findex.php%2FICWSM%2FICWSM10%2Fpaper%2Fdownload%2F1441%2F1852&usg=AFQjCNFnJCLUH96xhdmhtDbDnJs4Dtx8jg)\n",
    "* [Limits of Electoral Predictions Using Twitter](http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/viewFile/2862/3254)\n",
    "* [On Using Twitter to Monitor Political Sentimentand Predict Election Results](https://www.aclweb.org/anthology/W/W11/W11-3702.pdf)\n",
    "* [Predicting US Primary Elections with Twitter](http://snap.stanford.edu/social2012/papers/shi.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 Web scraping\n",
    "\n",
    "We started by finding the list of candidates for the United States Senate Elections in 2016 and we looked for their twitter accounts. We decided to only use Democratic and Republican candidates, as the other parties are not very relevant in the US. \n",
    "\n",
    "Twitter APIs only allow data mining in the past week, thus we used a tool to bypass this limitation and collect data from the 8th of September 2016 to the 8th of November 2016, the day of the elections.\n",
    "\n",
    "The tool is a Python script found on GitHub (link in the *Resources* section). It enabled us to get all the tweets in a desired time frame according to some custom parameters. For example to mine tweets about Evan Bayh from the 8th of Septemebr 2016 to the 8th of November 2016 we simply ran the following command:\n",
    "\n",
    "*python Exporter.py --querysearch \"SenEvanBayh\" --since 2016-09-08 --until 2016-11-08*\n",
    "\n",
    "The outputs of this analysis are kept in the *data* folder of our project in .csv format. The files contain the date of the tweet, the username of the author of the tweet, the tweet itself, the number of likes and retweets for the tweet and some other information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 Data Analysis\n",
    "\n",
    "This section will present the analysis that was performed on the data to make it usable for the machine learning part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Imported libraries\n",
    "from pattern.en import sentiment\n",
    "import numpy as np \n",
    "import os\n",
    "import csv\n",
    "from sklearn import linear_model\n",
    "import time\n",
    "from statsmodels.tsa.ar_model import AR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Sentiment Analysis\n",
    "\n",
    "Sentiment analysis is used to measure the polarity of a text. By polarity it is intended whether the text leans to a negative or positive attitute towards the topic it contains. \n",
    "\n",
    "Pattern, used to perform sentiment analysis for this project, is a natural language processing toolkit. In particular, we used pattern.en, which was made for the English language. \n",
    "\n",
    "The **sentiment( )** function returns a **(polarity, subjectivity)**-tuple for the given sentence, based on the adjectives it contains, where polarity is a value between -1.0 and +1.0 and subjectivity between 0.0 and 1.0. \n",
    "\n",
    "Before performing the sentiment analysis on the data, we got rid of the hashtags and of the @ to facilitate the sentiment analysis in case a meaningful word followed these symbols.\n",
    "\n",
    "**Important note** : this sentiment analysis tool requires the usage of Python 2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output_gotCampbellforLa.csv\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "pattern/text/__init__.py:1943: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal\n",
      "  if w in imap(lambda e: e.lower(), e):\n",
      "pattern/text/__init__.py:979: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal\n",
      "  and tokens[j] in (\"'\", \"\\\"\", u\"”\", u\"’\", \"...\", \".\", \"!\", \"?\", \")\", EOS):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output_gotCatherineForNV.csv\n",
      "output_gotChrisvance123.csv\n",
      "output_gotChrisVanHollen.csv\n",
      "output_gotChuckGrassley.csv\n",
      "output_gotDanCarterCT.csv\n",
      "output_gotGovernorHassan.csv\n",
      "output_gotJasonKander.csv\n",
      "output_gotJerryMoran.csv\n",
      "output_gotjimbarksdale.csv\n",
      "output_gotJimGrayLexKY.csv\n",
      "output_gotJohnKennedyLA.csv\n",
      "output_gotKamalaHarris.csv\n",
      "output_gotKathyforMD.csv\n",
      "output_gotKellyAyotte.csv\n",
      "output_gotLorettaSanchez.csv\n",
      "output_gotmarcorubio.csv\n",
      "output_gotMikeCrapo.csv\n",
      "output_gotPatrickMurphyFL.csv\n",
      "output_gotpattyforiowa.csv\n",
      "output_gotPattyMurray.csv\n",
      "output_gotRandPaul.csv\n",
      "output_gotRepJoeHeck.csv\n",
      "output_gotRepKirkpatrick.csv\n",
      "output_gotRoyBlunt.csv\n",
      "output_gotSenatorIsakson.csv\n",
      "output_gotSenatorKirk.csv\n",
      "output_gotSenBlumenthal.csv\n",
      "output_gotSenEvanBayh.csv\n",
      "output_gotSenJohnMcCain.csv\n",
      "output_gotsenrobportman.csv\n",
      "output_gotSenSchumer.csv\n",
      "output_gotSturgill4Idaho.csv\n",
      "output_gotTammyforIL.csv\n",
      "output_gotTedStrickland.csv\n",
      "output_gotToddYoungIN.csv\n",
      "output_gotWendyLongNY.csv\n",
      "output_gotwiesner4senate.csv\n"
     ]
    }
   ],
   "source": [
    "# Use all the .csv files inside the data folder\n",
    "for file in os.listdir(\"./data\"):\n",
    "    if file.endswith(\".csv\"):\n",
    "        path = r'data/'+file\n",
    "        print(file)\n",
    "        var = os.path.basename(path)\n",
    "        var= str.split(var,'_')\n",
    "        var=str.split(var[1],'.')\n",
    "        sentiment_path = 'sentim/sentiment_'+var[0]+'.csv'\n",
    "\n",
    "        # read the text, the date and the number of retwetts and favourites from the data files\n",
    "        text=np.loadtxt(path,dtype=str, comments='++++',delimiter=';',skiprows=1,usecols=(4,))\n",
    "        date=np.loadtxt(path,dtype=str, comments='++++',delimiter=';',skiprows=1,usecols=(1,))\n",
    "        retweet=np.loadtxt(path,dtype=str, comments='++++',delimiter=';',skiprows=1,usecols=(2,))\n",
    "        favourites=np.loadtxt(path,dtype=str, comments='++++',delimiter=';',skiprows=1,usecols=(3,))\n",
    "\n",
    "        # remove hashtags and @ from the tweets \n",
    "        text=np.core.defchararray.replace(text,'#',' ')\n",
    "        text=np.core.defchararray.replace(text,'@',' ')\n",
    "\n",
    "        array_sentiment=np.zeros((len(text),2))\n",
    "\n",
    "\n",
    "        # perform sentiment analysis and filter out values that are too close to zero\n",
    "        for i in range(len(text)):\n",
    "            array_sentiment[i] = sentiment(text[i])\n",
    "            if (array_sentiment[i][0]<0.0000000000000001 and array_sentiment[i][0]> -0.0000000000000001 and array_sentiment[i][0]!=0.0):\n",
    "                array_sentiment[i]=0\n",
    "                \n",
    "\n",
    "        # save sentiment analysis to output file\n",
    "        np.savetxt(sentiment_path,np.transpose([date,array_sentiment[:,0],array_sentiment[:,1],retweet,favourites]),fmt=\"%s;%s;%s;%s;%s\",delimiter=';',header=\"date;sentiment;objectivity;retweets;favourites\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Unique User Identification\n",
    "\n",
    "We decided to identify the number of unique users who tweeted about a certain candidate and use this number as a feature for our machine learning algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output_gotCampbellforLa.csv 904 230\n",
      "output_gotCatherineForNV.csv 6595 2145\n",
      "output_gotChrisvance123.csv 723 252\n",
      "output_gotChrisVanHollen.csv 719 518\n",
      "output_gotChuckGrassley.csv 7642 2839\n",
      "output_gotDanCarterCT.csv 320 128\n",
      "output_gotGovernorHassan.csv 1296 672\n",
      "output_gotJasonKander.csv 6148 2573\n",
      "output_gotJerryMoran.csv 770 483\n",
      "output_gotjimbarksdale.csv 1058 354\n",
      "output_gotJimGrayLexKY.csv 505 306\n",
      "output_gotJohnKennedyLA.csv 722 259\n",
      "output_gotKamalaHarris.csv 5928 3200\n",
      "output_gotKathyforMD.csv 960 411\n",
      "output_gotKellyAyotte.csv 25556 11030\n",
      "output_gotLorettaSanchez.csv 1333 885\n",
      "output_gotmarcorubio.csv 73326 27106\n",
      "output_gotMikeCrapo.csv 3028 2105\n",
      "output_gotPatrickMurphyFL.csv 21705 5978\n",
      "output_gotpattyforiowa.csv 1596 540\n",
      "output_gotPattyMurray.csv 5059 1694\n",
      "output_gotRandPaul.csv 26216 13883\n",
      "output_gotRepJoeHeck.csv 2547 1471\n",
      "output_gotRepKirkpatrick.csv 968 534\n",
      "output_gotRoyBlunt.csv 2309 1202\n",
      "output_gotSenatorIsakson.csv 1626 834\n",
      "output_gotSenatorKirk.csv 6537 4546\n",
      "output_gotSenBlumenthal.csv 1568 911\n",
      "output_gotSenEvanBayh.csv 459 351\n",
      "output_gotSenJohnMcCain.csv 40786 21604\n",
      "output_gotsenrobportman.csv 3770 1567\n",
      "output_gotSenSchumer.csv 4953 2751\n",
      "output_gotSturgill4Idaho.csv 148 79\n",
      "output_gotTammyforIL.csv 4584 2853\n",
      "output_gotTedStrickland.csv 6301 2566\n",
      "output_gotToddYoungIN.csv 2959 1088\n",
      "output_gotWendyLongNY.csv 1872 1007\n",
      "output_gotwiesner4senate.csv 23 13\n"
     ]
    }
   ],
   "source": [
    "# Use all the .csv files inside the data folder\n",
    "for file in os.listdir(\"./data\"):\n",
    "    if file.endswith(\".csv\"):\n",
    "        cand = r'data/'+file\n",
    "        mydata = np.loadtxt(cand, dtype=str, comments='++++',delimiter=';',skiprows=1,usecols=(0,))\n",
    "        # Collect the number of tweets for one deputee and number of unique authors of tweets about this candidate\n",
    "        print(file + \" \" + str(len(mydata)) + \" \" + str(len(np.unique(mydata))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.3 Mean \n",
    "\n",
    "We decided to do a daily mean of the polarity value given by the sentiment analysis of the tweets. The mean is weighted on the number of likes and retweets that every tweet received. In fact we assumed that likes and retweets meant that people agreed with the content of the tweet and thus such tweets deserved to have a higher weight.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentiment_gotCampbellforLa.csv\n",
      "sentiment_gotCatherineForNV.csv\n",
      "sentiment_gotChrisvance123.csv\n",
      "sentiment_gotChrisVanHollen.csv\n",
      "sentiment_gotChuckGrassley.csv\n",
      "sentiment_gotDanCarterCT.csv\n",
      "sentiment_gotGovernorHassan.csv\n",
      "sentiment_gotJasonKander.csv\n",
      "sentiment_gotJerryMoran.csv\n",
      "sentiment_gotjimbarksdale.csv\n",
      "sentiment_gotJimGrayLexKY.csv\n",
      "sentiment_gotJohnKennedyLA.csv\n",
      "sentiment_gotKamalaHarris.csv\n",
      "sentiment_gotKathyforMD.csv\n",
      "sentiment_gotKellyAyotte.csv\n",
      "sentiment_gotLorettaSanchez.csv\n",
      "sentiment_gotmarcorubio.csv\n",
      "sentiment_gotMikeCrapo.csv\n",
      "sentiment_gotPatrickMurphyFL.csv\n",
      "sentiment_gotpattyforiowa.csv\n",
      "sentiment_gotPattyMurray.csv\n",
      "sentiment_gotRandPaul.csv\n",
      "sentiment_gotRepJoeHeck.csv\n",
      "sentiment_gotRepKirkpatrick.csv\n",
      "sentiment_gotRoyBlunt.csv\n",
      "sentiment_gotSenatorIsakson.csv\n",
      "sentiment_gotSenatorKirk.csv\n",
      "sentiment_gotSenBlumenthal.csv\n",
      "sentiment_gotSenEvanBayh.csv\n",
      "sentiment_gotSenJohnMcCain.csv\n",
      "sentiment_gotsenrobportman.csv\n",
      "sentiment_gotSenSchumer.csv\n",
      "sentiment_gotSturgill4Idaho.csv\n",
      "sentiment_gotTammyforIL.csv\n",
      "sentiment_gotTedStrickland.csv\n",
      "sentiment_gotToddYoungIN.csv\n",
      "sentiment_gotWendyLongNY.csv\n",
      "sentiment_gotwiesner4senate.csv\n"
     ]
    }
   ],
   "source": [
    "# Use all the .csv files inside the sentim folder\n",
    "for file in os.listdir(\"./sentim\"):\n",
    "    if file.endswith(\".csv\"):\n",
    "        path = r'sentim/'+file\n",
    "        print(file)\n",
    "        var = os.path.basename(path)\n",
    "        var= str.split(var,'_')\n",
    "        var=str.split(var[1],'.')\n",
    "        mean_path = 'means/mean_'+var[0]+'.csv'\n",
    "        \n",
    "        #read sentiment, date and number of retweets and favourites from the sentiment analysis files\n",
    "        sentim=np.loadtxt(path, comments='++++',delimiter=';',skiprows=1,usecols=(1,))\n",
    "        date=np.loadtxt(path,dtype=str, comments='++++',delimiter=';',skiprows=1,usecols=(0,))\n",
    "        retweet=np.loadtxt(path, comments='++++',delimiter=';',skiprows=1,usecols=(3,))\n",
    "        favourites=np.loadtxt(path, comments='++++',delimiter=';',skiprows=1,usecols=(4,))\n",
    "\n",
    "        array_length=62\n",
    "        mean=[0]*array_length\n",
    "\n",
    "\n",
    "        #create day and month arrays\n",
    "        array_day=np.append(np.arange(8,0,-1),(np.append(np.arange(31,0,-1),np.arange(30,7,-1))))\n",
    "        array_month=np.append(np.ones(8)*11,(np.append(np.ones(31)*10,np.ones(23)*9)))\n",
    "\n",
    "        # parse the date and the month and create arrays for them\n",
    "        day =np.zeros(len(date))\n",
    "        month=np.zeros(len(date))\n",
    "        for i in range(len(date)):\n",
    "            day[i]=np.datetime64(date[i]).astype(object).day\n",
    "            month[i]=np.datetime64(date[i]).astype(object).month\n",
    "\n",
    "\n",
    "        #create array of the weights (based on likes and favourites)\n",
    "        array_weight = np.zeros(len(sentim))\n",
    "        for i in range(len(retweet)):\n",
    "            if(retweet[i]!=0.0 or favourites[i]!=0.0):\n",
    "                array_weight[i]=retweet[i]+favourites[i]\n",
    "            else:\n",
    "                array_weight[i]= 1\n",
    "\n",
    "        #compute the mean\n",
    "        cnt_date=0\n",
    "        cnt_mean=[0]*array_length\n",
    "\n",
    "        for i in range(len(sentim)):\n",
    "            if (sentim[i]!=0.0):\n",
    "                if (array_day[cnt_date]==day[i]and array_month[cnt_date]==month[i]): \n",
    "                        mean[cnt_date] = mean[cnt_date]+array_weight[i]*sentim[i]\n",
    "                        cnt_mean[cnt_date]=cnt_mean[cnt_date]+array_weight[i]\n",
    "\n",
    "                else :\n",
    "                    \n",
    "                    while (array_day[cnt_date]!=day[i] or array_month[cnt_date]!= month[i]): \n",
    "                        if (cnt_date <len(array_day)-1):\n",
    "                            cnt_date=cnt_date + 1\n",
    "                        else:\n",
    "                            break\n",
    "                        \n",
    "                    mean[cnt_date]=mean[cnt_date]+array_weight[i]*sentim[i]\n",
    "                    cnt_mean[cnt_date]=cnt_mean[cnt_date]+array_weight[i]\n",
    "\n",
    "\n",
    "\n",
    "        weigthed_mean=[None]*array_length\n",
    "\n",
    "        for i in range(len(mean)):\n",
    "            if (mean[i]!=0.0):\n",
    "                weigthed_mean[i]=mean[i]/cnt_mean[i]\n",
    "            else:\n",
    "                weigthed_mean[i]=0\n",
    "                \n",
    "        # save output on file\n",
    "        np.savetxt(mean_path,np.transpose([array_day,array_month,mean,cnt_mean,weigthed_mean]),fmt=\"%d;%d;%s;%s;%s\",delimiter=';',header=\"day;month;sum;weight;mean\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating files containing the daily mean values, we built the data structures necessary to proceed with the machine learning algorithm. In addition to the previous features, we collect by hand the number of follower for every candidates. Concerning the structure, we used dictionaries with the twitter name of the candidate as a key identifier.\n",
    "\n",
    "D, name_dict and id_dict are dictionaries with the same keys as identifiers.\n",
    "For each key, d's values from 0 to 61 are the daily means, value 62 is the # of follower, value 63 is the amount of\n",
    "unique ids and value 64 in the total amount of tweets.\n",
    "For each key, id_dict's value 0 is the election result (1 for win, 0 for loss), value 1 is the percentage of votes that the candidate received and value 2 the pair identifier that pairs up candidates running in the same election.\n",
    "For each key, name_dict's value 0 is the name of the candidate and value 1 is the last name of the candidate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d={}\n",
    "id_dict={}\n",
    "name_dict={}\n",
    "\n",
    "# Use all the .csv files inside the means folder\n",
    "for file in os.listdir(\"./means\"):\n",
    "    if file.endswith(\".csv\"):\n",
    "        path = r'means/'+file\n",
    "        var = os.path.basename(path)\n",
    "        var= str.split(var,'_got')\n",
    "        var=str.split(var[1],'.')\n",
    "        key = var[0]\n",
    "        d.setdefault(key,[])\n",
    "        d[key]=np.loadtxt(path, comments='++++',delimiter=';',skiprows=1,usecols=(4,))\n",
    "        id_dict.setdefault(key,[])\n",
    "        name_dict.setdefault(key,[])\n",
    "\n",
    "with open('listDeputee.csv', 'rb') as csvfile:\n",
    "    reader = csv.reader(csvfile, delimiter=';', quotechar=';')\n",
    "    for row in reader:\n",
    "        if (row[0]!='Id'):\n",
    "            d[row[0]]= np.append(d[row[0]],[float(row[5]),float(row[6]),float(row[7])])\n",
    "            id_dict[row[0]]=np.append(id_dict[row[0]],[float(row[3]),float(row[4]),float(row[8])])\n",
    "            name_dict[row[0]]=np.append(name_dict[row[0]],[row[1],row[2],])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 Machine Learning Algorithm\n",
    "\n",
    "The first step of the machine learning algorithm that we used is to fit the 62 daily mean values with an autoregressive model. The goal of this step is to reduce the number of features that we will use for the prediction because we have too many of them compared to the amount of data available. We tried multiple orders for the AR fitting and chose the one that outputs the best prediction result.\n",
    "\n",
    "The second step consists in the prediction itself. First we concatenate the AR model coefficients with the extra features which are number of followers, number of unique author of tweets and total amount of tweets. The idea of using these extra features came from some papers we read about the Election Prediction topic. Then we fit a linear regression model between the features and the actual percentage of votes that the candidate received.\n",
    "\n",
    "The data set was devided in the training set and the test set. The splitting of the data is different between the two algorithms. It will be described individually.\n",
    "\n",
    "Concerning the training phase, we used the **leave-one-out** procedure to train the coefficients. This means that each time we train the coefficients on a reduced training set and test it on the value that was left out. If the prediction is correct, we keep the coefficients. At the end we average all the coefficients. \n",
    "After the model is trained, we apply it on the test set and we count the number of successful predictions and the total number of tries. The success of the prediction depends on the algorithm.\n",
    "\n",
    "The process described above is repeated multiple times, each time the division between the training and the test sets is done randomly. After all the repetitions, we estimate the accuracy by dividing the total number of successful predictions by the total number of tries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Algorithm 1\n",
    "\n",
    "In this algorithm, we consider every deputee individually and the goal is to predict whether they won or lost. In order to do this we follow the procedure described above with the output of the linear regression being the percentage of vote received by the candidate. The prediction is considered correct if a winning candidate is receiving a score higher than 50 from the algorithm. It is also considered correct in case a losing candidate gets a score lower than 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 0.586875\n",
      "2 0.585125\n",
      "3 0.572625\n",
      "4 0.565125\n",
      "5 0.56725\n",
      "6 0.554625\n",
      "7 0.565\n",
      "8 0.541\n",
      "9 0.533875\n",
      "10 0.58725\n",
      "11 0.588625\n",
      "12 0.584125\n",
      "13 0.58825\n",
      "14 0.579625\n",
      "15 0.579875\n",
      "16 0.61825\n",
      "17 0.6335\n",
      "18 0.626625\n",
      "19 0.645625\n",
      "20 0.6145\n",
      "21 0.594\n",
      "22 0.639125\n",
      "23 0.618\n",
      "24 0.615875\n"
     ]
    }
   ],
   "source": [
    "# Rearranging the result of the election\n",
    "Y_all_init =[]\n",
    "for keys in id_dict :\n",
    "    Y_all_init=np.append(Y_all_init,id_dict[keys][1])\n",
    "\n",
    "\n",
    "orders = np.array(range(1,25)) #variable to test different order of the AR model\n",
    "for order in orders:           # looping the whole algorithm with different orders for the AR\n",
    "    i=0\n",
    "    coeffs = np.zeros([38,order+4]) #initialization of autoregression coefficients\n",
    "    for keys in d:    \n",
    "        mean = d[keys]\n",
    "        ar_mod = AR(mean[0:61])  #initialoization of the AR model with the mean sentiment value of one candidate\n",
    "        ar_res = ar_mod.fit(maxlag = order,method = 'cmle',ic='aic',trend = 'c',tol = 1e-2) #fitting the AR model\n",
    "        for n in range (len(ar_res.params)):\n",
    "            coeffs[i][n] = ar_res.params[n]  #assignment of the AR model coefficients with zero padding in case of not max order\n",
    "        for m in range (3):\n",
    "            coeffs[i][m+order+1] = mean[62+m] # appending extra values\n",
    "        i += 1\n",
    "    shapeMean = np.shape(coeffs)\n",
    "    nbFeature = shapeMean[1]     #getting the number of features \n",
    "    MEANS_all_init_2 = coeffs\n",
    "    nbRight = 0\n",
    "    nbAll = 0\n",
    "\n",
    "    #Number of time we want to try our prediction with a different set of training and test data.\n",
    "    for iteration in range(1000):\n",
    "        MEANS_all = MEANS_all_init_2\n",
    "        Y_all = Y_all_init\n",
    "\n",
    "        #vector containing the data to test\n",
    "        Y_predict_final = np.zeros((8,1))\n",
    "        MEANS_predict_final = np.zeros((8,nbFeature))\n",
    "        for i in range(8):\n",
    "            #Choose randomly 8 person for the test data\n",
    "            selected = np.random.randint(0,38-i, 1)\n",
    "            Y_predict_final[i] = Y_all[selected]\n",
    "            MEANS_predict_final[i] = MEANS_all[selected]\n",
    "\n",
    "            #Supress the test data from the full vectors to create the training data\n",
    "            MEANS_all = np.delete(MEANS_all, selected, 0)\n",
    "            Y_all = np.delete(Y_all, selected, 0)\n",
    "\n",
    "\n",
    "        coef_all = np.zeros((nbFeature))\n",
    "        nbKeep = 0\n",
    "        #We loop on all the data of the training set\n",
    "        for i in range(30):\n",
    "            #Prediction phase - We create the prediction model\n",
    "            clf = linear_model.LinearRegression(fit_intercept=False)\n",
    "\n",
    "            #We remove one of the data from the training set\n",
    "            MEANS_fit = np.delete(MEANS_all, i, 0)\n",
    "            Y_fit = np.delete(Y_all, i, 0)\n",
    "\n",
    "            # We fit the data of 29 deputy to the model and we keep 1 for the testing.\n",
    "            clf.fit(MEANS_fit, Y_fit)\n",
    "\n",
    "            #The prediction with the data we reomoved\n",
    "            predIt = clf.predict(MEANS_all[i].reshape(1, -1))\n",
    "            #If the prediction works, we keep the coeficients of the linear regression. (We add them to an array)\n",
    "            if (Y_all[i] >50 and predIt > 50 ) or (Y_all[i] < 50 and predIt < 50) :\n",
    "                nbKeep += 1\n",
    "                coef_all += clf.coef_\n",
    "\n",
    "\n",
    "        #The average of all the coeficients we want to keep\n",
    "        coef_all = coef_all/nbKeep\n",
    "\n",
    "        #The preidiction using the average coefficient we computed before.\n",
    "        #We count the number of prediction tries and the number of succesful ones.\n",
    "        for i in range(len(Y_predict_final)):\n",
    "            nbAll +=1\n",
    "            pred1 = np.dot(MEANS_predict_final[i], coef_all)\n",
    "            if (Y_predict_final[i][0] > 50 and pred1 > 50 ) or (Y_predict_final[i][0] < 50 and pred1 < 50) :\n",
    "                nbRight += 1\n",
    "\n",
    "    print(str(order) + \" \" + str(float(nbRight)/float(nbAll)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Algorithm 2\n",
    "\n",
    "In this case we consider the deputees by pair (meaning that they were opponents in the same state). It means that the features involve the concatenation of two candidates and the prediction should give the percentage of each candidate with respect to the other. To determine if the prediction is successful, a winning candidate must be the one with the highest percentage in the pair of prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 0.5225\n",
      "2 0.62175\n",
      "3 0.5045\n",
      "4 0.50425\n",
      "5 0.58175\n",
      "6 0.583\n",
      "7 0.53475\n",
      "8 0.6165\n",
      "9 0.53025\n",
      "10 0.54725\n",
      "11 0.541\n",
      "12 0.59625\n",
      "13 0.5115\n",
      "14 0.565\n",
      "15 0.52725\n",
      "16 0.65175\n",
      "17 0.6495\n",
      "18 0.60425\n",
      "19 0.61775\n",
      "20 0.65775\n",
      "21 0.6585\n",
      "22 0.6155\n",
      "23 0.6005\n",
      "24 0.598\n"
     ]
    }
   ],
   "source": [
    "Y_all_init =[]\n",
    "\n",
    "MEANS_all_init=[]\n",
    "\n",
    "for i in range(1,20):      #creating the set of data by pair of deputees\n",
    "    first=0\n",
    "    first_array = []\n",
    "    first_predict=[]\n",
    "    for keys in id_dict:\n",
    "        if id_dict[keys][2]==i :\n",
    "            if first==0:\n",
    "                first=1\n",
    "                first_array = d[keys]  \n",
    "                first_predict = id_dict[keys][1]\n",
    "            else:\n",
    "                MEANS_all_init=np.append(MEANS_all_init,np.append(first_array,d[keys]))\n",
    "                Y_all_init = np.append(Y_all_init,np.append(first_predict,id_dict[keys][1]))\n",
    "\n",
    "Y_all_init = np.reshape(Y_all_init,[19,2])\n",
    "\n",
    "\n",
    "newmeans = np.reshape(MEANS_all_init,[38,65]) #list of all candidates individualy\n",
    "orders = np.array(range(1,25)) #variable to test different order of the AR model\n",
    "for order in orders:\n",
    "    coeffs = np.zeros([38,order+4]) #initialisation of autoregression coefficients\n",
    "    for i in range(38):    \n",
    "        mean = newmeans[i]\n",
    "        ar_mod = AR(mean[0:61]) # initialisation of the AR model with the mean sentiment values of 1 candidate\n",
    "        ar_res = ar_mod.fit(maxlag = order,method = 'cmle',ic='aic',trend = 'c',tol = 1e-2) #fitting of the AR model\n",
    "        for n in range (len(ar_res.params)):\n",
    "            coeffs[i][n] = ar_res.params[n] #assignement of the AR model coefficients, with 0 padding in case not max order\n",
    "        for m in range (3):\n",
    "            coeffs[i][m+order+1] = mean[62+m] #appending extra values (followers, total tweets, unique ID)\n",
    "    AR_coeff = np.reshape(coeffs,[19,2*order+8]) #reshaping coeff to reform pairs win/lose\n",
    "    \n",
    "    \n",
    "    #Prediction testing, By pairs, Using AR model\n",
    "\n",
    "    nbRight = 0 # correct predictions\n",
    "    nbAll = 0 # total predictions\n",
    "    for iteration in range(1000):\n",
    "        MEANS_all = AR_coeff #initialisation\n",
    "        Y_all = Y_all_init\n",
    "\n",
    "        Y_predict_final = np.zeros((4,2))\n",
    "        MEANS_predict_final = np.zeros((4,2*order+8))\n",
    "        \n",
    "        #removing data of some candidates to use as test set\n",
    "        for i in range(4):\n",
    "            selected = np.random.randint(0,19-i, 1)\n",
    "            Y_predict_final[i] = Y_all[selected]\n",
    "            MEANS_predict_final[i] = MEANS_all[selected]\n",
    "\n",
    "            MEANS_all = np.delete(MEANS_all, selected, 0)\n",
    "            Y_all = np.delete(Y_all, selected, 0)\n",
    "\n",
    "\n",
    "\n",
    "        coef_all = np.zeros((2,2*order+8))\n",
    "        nbKeep = 0\n",
    "        \n",
    "        #Training linear regression model using leave-one-out technique\n",
    "        for i in range(15):\n",
    "            #Prediction phase - We create the prediction model\n",
    "            clf = linear_model.LinearRegression(fit_intercept=False)\n",
    "\n",
    "            MEANS_fit = np.delete(MEANS_all, i, 0)\n",
    "            Y_fit = np.delete(Y_all, i, 0)\n",
    "\n",
    "            # We fit the data of 14 deputy to the model and we keep 1 for the testing.\n",
    "            clf.fit(MEANS_fit, Y_fit)\n",
    "\n",
    "            predIt = clf.predict(MEANS_all[i].reshape(1, -1))\n",
    "\n",
    "            if (Y_all[i][0] > Y_all[i][1] and predIt[0][0] > predIt[0][1] ) or (Y_all[i][0] < Y_all[i][1] and predIt[0][0] < predIt[0][1]) :\n",
    "                nbKeep += 1\n",
    "                coef_all += clf.coef_ #keeping only coefficients that result in a correct prediction\n",
    "\n",
    "        #averaging the coefficients that gave a correct prediction\n",
    "        coef_all = coef_all/nbKeep\n",
    "        \n",
    "        #testing the linear regression model on test set data\n",
    "        for i in range(len(Y_predict_final)):\n",
    "            nbAll +=1\n",
    "            pred1 = np.dot(MEANS_predict_final[i],coef_all[0])\n",
    "            pred2 = np.dot(MEANS_predict_final[i], coef_all[1])\n",
    "            if (Y_predict_final[i][0] > Y_predict_final[i][1] and pred1 > pred2 ) or (Y_predict_final[i][0] < Y_predict_final[i][1] and pred1 < pred2) :\n",
    "                nbRight += 1\n",
    "\n",
    "    print(str(order) + \" \" + str(float(nbRight)/float(nbAll)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 Conclusions\n",
    "\n",
    "Sentiment Analysis, like any other natural language processing tool, is hard to perform and does not give extremely accurate results. Performing it on Tweets is tricky because many slang words are used and often the analysis does not output anything. Moreover the sentiment analysis of tweets like \"I hate how @Trump denies the work of @Obama\" and \"I hate how @Obama denies the work of @Trump\" give the same result. However the real meaning is opposite. If a tweet is perceived as negative, it does not mean that the negativity is towards the topic of tweet itself. We were aware of this problem, but thought that it would have been interesting to play with this tool anyway.\n",
    "\n",
    "Another issue that we encountered was that the number of tweets differed a lot depending on the popularity of the candidate. However, since we looked at big elections in a big country, we were able to have a quite big dataset even though we did not consider every candidate, since some of them were not active at all on Twitter. \n",
    "\n",
    "Both algorithms reach with the right tunning a level of accuracy of 65%, which means that there is a weak correlation between the Twitter data and the result of the election. This poor result has multiple explainations, one of them is the imprecision of the sentiment analysis. Another reason is that the Twitter population is not a good sample of the voting people. Moreover tweets do not always represent the point of view of the author, sometimes they can just be provocative. Finally the author of tweets may not have the right to vote in the election they are talking about. All these reasons, combined with a relatively small dataset, contributed in the obtained result.\n",
    "\n",
    "For further improvements of this work, a bigger dataset would be helpful to train more the machine learning algorithm. Moreover, a different sentiment analysis tool could be exploited.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}