2.1 Supervised Learning - Classification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize the workings of machine learning algorithms, it is often helpful to study two-dimensional or one-dimensional data, that is data with only one or two features. While in practice, datasets usually have many more features, it is hard to plot high-dimensional data on two-dimensional screens.\n",
    "\n",
    "We will illustrate some very simple examples before we move on to more \"real world\" data sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification\n",
    "========\n",
    "First, we will look at a two class classification problem in two dimensions. We use the synthetic data generated by the ``make_blobs`` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_blobs\n",
    "X, y = make_blobs(centers=2, random_state=0)\n",
    "print(X.shape)\n",
    "print(y.shape)\n",
    "print(X[:5, :])\n",
    "print(y[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the data is two-dimensional, we can plot each sample as a point in two-dimensional space, with the first feature being the x-axis and the second feature being the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=40)\n",
    "plt.xlabel(\"first feature\")\n",
    "plt.ylabel(\"second feature\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As classification is a supervised task, and we are interested in how well the model generalizes, we split our data into a training set,\n",
    "to built the model from, and a test-set, to evaluate how well our model performs on new data. The ``train_test_split`` function form the ``cross_validation`` module does that for us, by randomly splitting of 25% of the data for testing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K Nearest Neighbors\n",
    "------------------------------------------------\n",
    "A popular and easy to understand classifier is K nearest neighbors (kNN).  It has one of the simplest learning strategies: given a new, unknown observation, look up in your reference database which ones have the closest features and assign the predominant class.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we set a parameter of the KNeighborsClassifier to tell it we only want to look at one nearest neighbor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We fit the model with out training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "knn.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from figures import plot_2d_separator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=40)\n",
    "plt.xlabel(\"first feature\")\n",
    "plt.ylabel(\"second feature\")\n",
    "plot_2d_separator.plot_2d_separator(knn, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knn.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a different classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll take a few minutes and try out another learning model.  Because of ``scikit-learn``'s uniform interface, the syntax is identical to that of ``LinearSVC`` above.\n",
    "\n",
    "There are many possibilities of classifiers; you could try any of the methods discussed at <http://scikit-learn.org/stable/supervised_learning.html>.  Alternatively, you can explore what's available in ``scikit-learn`` using just the tab-completion feature.  For example, import the ``linear_model`` submodule:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import linear_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And use the tab completion to find what's available.  Type ``linear_model.`` and then the tab key to see an interactive list of the functions within this submodule.  The ones which begin with capital letters are the models which are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now select a new classifier and try out a classification of the iris data.\n",
    "\n",
    "Some good choices are\n",
    "\n",
    "- ``sklearn.svm.LinearSVC`` :\n",
    "    Support Vector Machines without kernels based on liblinear\n",
    "\n",
    "- ``sklearn.svm.SVC`` :\n",
    "    Support Vector Machines with kernels based on libsvm\n",
    "\n",
    "- ``sklearn.linear_model.LogisticRegression`` :\n",
    "    Regularized Logistic Regression based on liblinear\n",
    "\n",
    "- ``sklearn.linear_model.SGDClassifier`` :\n",
    "    Regularized linear models (SVM or logistic regression) using a Stochastic Gradient Descent algorithm written in Cython\n",
    "\n",
    "- ``sklearn.neighbors.NeighborsClassifier`` :\n",
    "    k-Nearest Neighbors classifier based on the ball tree datastructure for low dimensional data and brute force search for high dimensional data\n",
    "\n",
    "- ``sklearn.naive_bayes.GaussianNB`` :\n",
    "    Gaussian Naive Bayes model. This is an unsophisticated model which can be trained very quickly. It is often used to obtain baseline results before moving to a more sophisticated classifier.\n",
    "\n",
    "- ``sklearn.tree.DecisionTreeClassifier`` :\n",
    "  A classifier based on a series of binary decisions.  This is another very fast classifier, which can be very powerful.\n",
    "\n",
    "Choose one of the above, import it, and use the ``?`` feature to learn about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now instantiate this model as we did with ``LinearSVC`` above.  Call it ``clf``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use our data ``X`` and ``y`` to train the model, using ``clf2.fit(X, y)``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now call the ``predict`` method, and find the classification of ``X_new``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use the code snippet in `Cell 16` to plot the corresponding graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the Iris Dataset\n",
    "=========\n",
    "**Exercise** Apply the KNeighborsClassifier to the ``iris`` dataset. Play with different values of the ``n_neighbors`` and observe how training and test score change.\n",
    "\n",
    "Note: If you finish early, you can try applying a different estimator: `sklearn.svm.SVC`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load scripts/knn_iris.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot_iris_knn()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another powerful and highly effective method can be used for both Classification and Regression.\n",
    "\n",
    "SVMs are a **discriminative** classifier: that is, they draw a boundary between clusters of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets.samples_generator import make_blobs\n",
    "X, y = make_blobs(n_samples=50, centers=2,\n",
    "                  random_state=0, cluster_std=0.60)\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There can be many seperators for the dataset above. How the find the best one?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "xfit = np.linspace(-1, 3.5)\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')\n",
    "\n",
    "for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:\n",
    "    yfit = m * xfit + b\n",
    "    plt.plot(xfit, yfit, '-k')\n",
    "    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none', color='#AAAAAA', alpha=0.4)\n",
    "\n",
    "plt.xlim(-1, 3.5);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SVM's to the rescue :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC  # \"Support Vector Classifier\"\n",
    "clf = SVC(kernel='linear')\n",
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_svc_decision_function(clf, ax=None):\n",
    "    \"\"\"Plot the decision function for a 2D SVC\"\"\"\n",
    "    if ax is None:\n",
    "        ax = plt.gca()\n",
    "    x = np.linspace(plt.xlim()[0], plt.xlim()[1], 30)\n",
    "    y = np.linspace(plt.ylim()[0], plt.ylim()[1], 30)\n",
    "    Y, X = np.meshgrid(y, x)\n",
    "    P = np.zeros_like(X)\n",
    "    for i, xi in enumerate(x):\n",
    "        for j, yj in enumerate(y):\n",
    "            P[i, j] = clf.decision_function([[xi, yj]])\n",
    "    # plot the margins\n",
    "    ax.contour(X, Y, P, colors='k',\n",
    "               levels=[-1, 0, 1], alpha=0.5,\n",
    "               linestyles=['--', '-', '--'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')\n",
    "plot_svc_decision_function(clf);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')\n",
    "plot_svc_decision_function(clf)\n",
    "plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],\n",
    "            s=200, facecolors='none');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.html.widgets import interact\n",
    "\n",
    "def plot_svm(N=10):\n",
    "    X, y = make_blobs(n_samples=200, centers=2,\n",
    "                      random_state=0, cluster_std=0.60)\n",
    "    X = X[:N]\n",
    "    y = y[:N]\n",
    "    clf = SVC(kernel='linear')\n",
    "    clf.fit(X, y)\n",
    "    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')\n",
    "    plt.xlim(-1, 4)\n",
    "    plt.ylim(-1, 6)\n",
    "    plot_svc_decision_function(clf, plt.gca())\n",
    "    plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],\n",
    "                s=200, facecolors='none')\n",
    "    \n",
    "interact(plot_svm, N=[10, 200], kernel='linear');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What if the data is not linear?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets.samples_generator import make_circles\n",
    "X, y = make_circles(100, factor=.1, noise=.1)\n",
    "\n",
    "clf = SVC(kernel='linear').fit(X, y)\n",
    "\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')\n",
    "plot_svc_decision_function(clf);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "widgets": {
   "state": {
    "59dece73c784433691328d95d18a2e4a": {
     "views": [
      {
       "cell_index": 44
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}