SKL-DimensionalityReduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Scikit-Learn (Python) Part 2 short\n",
    "\n",
    "Agenda:\n",
    "\n",
    "- Installation and a quick review of previous workshop\n",
    "- Scikit-Learn datasets\n",
    "- Unsupervised learning   Clustering algorithm (KMeans) and Dimensionality Reduction\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python version: 3.5.2 |Anaconda custom (64-bit)| (default, Jul  2 2016, 17:53:06) \n",
      "[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]\n",
      "scikit-learn version: 0.18.1\n"
     ]
    }
   ],
   "source": [
    "#check installed version \n",
    "import sys\n",
    "print(\"Python version: {}\".format(sys.version))\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import sklearn\n",
    "print(\"scikit-learn version: {}\".format(sklearn.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Dimensionality Reduction\n",
    "\n",
    "Dimension reduction is the process of reducing the number of random variables under consideration, via obtaining a set of principal variables. It can be divided into feature selection and feature extraction.   https://en.wikipedia.org/wiki/Dimensionality_reduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.kaggle.com/arthurtok/interactive-intro-to-dimensionality-reduction \n",
    "# https://www.kaggle.com/c/digit-recognizer  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Principal Component Analysis (PCA)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "cancer = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.79900000e+01,   1.03800000e+01,   1.22800000e+02, ...,\n",
       "          2.65400000e-01,   4.60100000e-01,   1.18900000e-01],\n",
       "       [  2.05700000e+01,   1.77700000e+01,   1.32900000e+02, ...,\n",
       "          1.86000000e-01,   2.75000000e-01,   8.90200000e-02],\n",
       "       [  1.96900000e+01,   2.12500000e+01,   1.30000000e+02, ...,\n",
       "          2.43000000e-01,   3.61300000e-01,   8.75800000e-02],\n",
       "       ..., \n",
       "       [  1.66000000e+01,   2.80800000e+01,   1.08300000e+02, ...,\n",
       "          1.41800000e-01,   2.21800000e-01,   7.82000000e-02],\n",
       "       [  2.06000000e+01,   2.93300000e+01,   1.40100000e+02, ...,\n",
       "          2.65000000e-01,   4.08700000e-01,   1.24000000e-01],\n",
       "       [  7.76000000e+00,   2.45400000e+01,   4.79200000e+01, ...,\n",
       "          0.00000000e+00,   2.87100000e-01,   7.03900000e-02]])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sklearn.datasets.base.Bunch"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(cancer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer['target'][:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.79900000e+01,   1.03800000e+01,   1.22800000e+02,\n",
       "          1.00100000e+03,   1.18400000e-01,   2.77600000e-01,\n",
       "          3.00100000e-01,   1.47100000e-01,   2.41900000e-01,\n",
       "          7.87100000e-02,   1.09500000e+00,   9.05300000e-01,\n",
       "          8.58900000e+00,   1.53400000e+02,   6.39900000e-03,\n",
       "          4.90400000e-02,   5.37300000e-02,   1.58700000e-02,\n",
       "          3.00300000e-02,   6.19300000e-03,   2.53800000e+01,\n",
       "          1.73300000e+01,   1.84600000e+02,   2.01900000e+03,\n",
       "          1.62200000e-01,   6.65600000e-01,   7.11900000e-01,\n",
       "          2.65400000e-01,   4.60100000e-01,   1.18900000e-01],\n",
       "       [  2.05700000e+01,   1.77700000e+01,   1.32900000e+02,\n",
       "          1.32600000e+03,   8.47400000e-02,   7.86400000e-02,\n",
       "          8.69000000e-02,   7.01700000e-02,   1.81200000e-01,\n",
       "          5.66700000e-02,   5.43500000e-01,   7.33900000e-01,\n",
       "          3.39800000e+00,   7.40800000e+01,   5.22500000e-03,\n",
       "          1.30800000e-02,   1.86000000e-02,   1.34000000e-02,\n",
       "          1.38900000e-02,   3.53200000e-03,   2.49900000e+01,\n",
       "          2.34100000e+01,   1.58800000e+02,   1.95600000e+03,\n",
       "          1.23800000e-01,   1.86600000e-01,   2.41600000e-01,\n",
       "          1.86000000e-01,   2.75000000e-01,   8.90200000e-02],\n",
       "       [  1.96900000e+01,   2.12500000e+01,   1.30000000e+02,\n",
       "          1.20300000e+03,   1.09600000e-01,   1.59900000e-01,\n",
       "          1.97400000e-01,   1.27900000e-01,   2.06900000e-01,\n",
       "          5.99900000e-02,   7.45600000e-01,   7.86900000e-01,\n",
       "          4.58500000e+00,   9.40300000e+01,   6.15000000e-03,\n",
       "          4.00600000e-02,   3.83200000e-02,   2.05800000e-02,\n",
       "          2.25000000e-02,   4.57100000e-03,   2.35700000e+01,\n",
       "          2.55300000e+01,   1.52500000e+02,   1.70900000e+03,\n",
       "          1.44400000e-01,   4.24500000e-01,   4.50400000e-01,\n",
       "          2.43000000e-01,   3.61300000e-01,   8.75800000e-02],\n",
       "       [  1.14200000e+01,   2.03800000e+01,   7.75800000e+01,\n",
       "          3.86100000e+02,   1.42500000e-01,   2.83900000e-01,\n",
       "          2.41400000e-01,   1.05200000e-01,   2.59700000e-01,\n",
       "          9.74400000e-02,   4.95600000e-01,   1.15600000e+00,\n",
       "          3.44500000e+00,   2.72300000e+01,   9.11000000e-03,\n",
       "          7.45800000e-02,   5.66100000e-02,   1.86700000e-02,\n",
       "          5.96300000e-02,   9.20800000e-03,   1.49100000e+01,\n",
       "          2.65000000e+01,   9.88700000e+01,   5.67700000e+02,\n",
       "          2.09800000e-01,   8.66300000e-01,   6.86900000e-01,\n",
       "          2.57500000e-01,   6.63800000e-01,   1.73000000e-01],\n",
       "       [  2.02900000e+01,   1.43400000e+01,   1.35100000e+02,\n",
       "          1.29700000e+03,   1.00300000e-01,   1.32800000e-01,\n",
       "          1.98000000e-01,   1.04300000e-01,   1.80900000e-01,\n",
       "          5.88300000e-02,   7.57200000e-01,   7.81300000e-01,\n",
       "          5.43800000e+00,   9.44400000e+01,   1.14900000e-02,\n",
       "          2.46100000e-02,   5.68800000e-02,   1.88500000e-02,\n",
       "          1.75600000e-02,   5.11500000e-03,   2.25400000e+01,\n",
       "          1.66700000e+01,   1.52200000e+02,   1.57500000e+03,\n",
       "          1.37400000e-01,   2.05000000e-01,   4.00000000e-01,\n",
       "          1.62500000e-01,   2.36400000e-01,   7.67800000e-02]])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer['data'][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['mean radius', 'mean texture', 'mean perimeter', 'mean area',\n",
       "       'mean smoothness', 'mean compactness', 'mean concavity',\n",
       "       'mean concave points', 'mean symmetry', 'mean fractal dimension',\n",
       "       'radius error', 'texture error', 'perimeter error', 'area error',\n",
       "       'smoothness error', 'compactness error', 'concavity error',\n",
       "       'concave points error', 'symmetry error', 'fractal dimension error',\n",
       "       'worst radius', 'worst texture', 'worst perimeter', 'worst area',\n",
       "       'worst smoothness', 'worst compactness', 'worst concavity',\n",
       "       'worst concave points', 'worst symmetry', 'worst fractal dimension'], \n",
       "      dtype='<U23')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer['feature_names'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(569, 30)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer.data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17070"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer.data.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer.data.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.79900000e+01,   1.03800000e+01,   1.22800000e+02,\n",
       "          1.00100000e+03,   1.18400000e-01,   2.77600000e-01,\n",
       "          3.00100000e-01,   1.47100000e-01,   2.41900000e-01,\n",
       "          7.87100000e-02,   1.09500000e+00,   9.05300000e-01,\n",
       "          8.58900000e+00,   1.53400000e+02,   6.39900000e-03,\n",
       "          4.90400000e-02,   5.37300000e-02,   1.58700000e-02,\n",
       "          3.00300000e-02,   6.19300000e-03,   2.53800000e+01,\n",
       "          1.73300000e+01,   1.84600000e+02,   2.01900000e+03,\n",
       "          1.62200000e-01,   6.65600000e-01,   7.11900000e-01,\n",
       "          2.65400000e-01,   4.60100000e-01,   1.18900000e-01]])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer.data[:1,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(cancer.data)\n",
    "X_scaled = scaler.transform(cancer.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.09706398, -2.07333501,  1.26993369,  0.9843749 ,  1.56846633,\n",
       "         3.28351467,  2.65287398,  2.53247522,  2.21751501,  2.25574689,\n",
       "         2.48973393, -0.56526506,  2.83303087,  2.48757756, -0.21400165,\n",
       "         1.31686157,  0.72402616,  0.66081994,  1.14875667,  0.90708308,\n",
       "         1.88668963, -1.35929347,  2.30360062,  2.00123749,  1.30768627,\n",
       "         2.61666502,  2.10952635,  2.29607613,  2.75062224,  1.93701461]])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_scaled[:1,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PCA(copy=True, iterated_power='auto', n_components=2, random_state=None,\n",
       "  svd_solver='auto', tol=0.0, whiten=False)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "# keep the first two principal components of the data\n",
    "pca = PCA(n_components=2)\n",
    "# fit PCA model to beast cancer data\n",
    "pca.fit(X_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 13.28160768,   5.69135461])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pca.explained_variance_ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.63243207651559441"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(pca.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original shape: (569, 30)\n",
      "Reduced shape: (569, 2)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "# transform data onto the first two principal components\n",
    "X_pca = pca.transform(X_scaled)\n",
    "print(\"Original shape: {}\".format(str(X_scaled.shape)))\n",
    "print(\"Reduced shape: {}\".format(str(X_pca.shape)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  9.19283683,   1.94858307],\n",
       "       [  2.3878018 ,  -3.76817174],\n",
       "       [  5.73389628,  -1.0751738 ],\n",
       "       ..., \n",
       "       [  1.25617928,  -1.90229671],\n",
       "       [ 10.37479406,   1.67201011],\n",
       "       [ -5.4752433 ,  -0.67063679]])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### Exercise: Try the same thing with the Iris Data Set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "iris = load_iris()\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(iris.data,\n",
    "                                                    iris.target,\n",
    "                                                    random_state=123,\n",
    "                                                    stratify=iris.target)\n",
    "\n",
    "sc = StandardScaler()\n",
    "sc.fit(X_train)\n",
    "pca = PCA(n_components=2)\n",
    "\n",
    "X_train_pca = pca.fit_transform(sc.transform(X_train))\n",
    "X_test_pca = pca.transform(sc.transform(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}