Module_2 Data Wrangling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 : Data Wrangling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data wrangling is one of the most important steps in make data clear and redable.\n",
    "\n",
    "In this section we are going to cover some important contents for data wrangling or data coleaning such as:\n",
    "\n",
    "- Identify missing values\n",
    "- Deal with missing values\n",
    "- Correct data format\n",
    "- Data standardization\n",
    "- Data Normalization (centring/scaling)\n",
    "- Binning\n",
    "- Indicator variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using above methods you can prepare you data to gain more and better information from dataset. Additionally, you \n",
    "prepare data for machine learning algorithms or recommendation systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.2.2 Import data\n",
    "You\n",
    "can\n",
    "find\n",
    "the\n",
    "\"Automobile\n",
    "Data\n",
    "Set\"\n",
    "from\n",
    "the\n",
    "following\n",
    "link:\n",
    "https://archive.ics.uci.edu/ml/machine-learning-databases/autos/imports-85.data. We will be\n",
    "using this data set throughout this course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, the first thing we should do to handle missing values at first we see many question marks (?) in dataset\n",
    "they are missing values\n",
    "\n",
    "Missing values can be in different forms, sometimes they show themselve as: *,? or NON hense we need to get rid of them with different methods which i try to cover them here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to handle missing values?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is diffent strategies to handle missing data but it depends on situation. We can calculate the mean of particular column and replace missing value with that. The second method is find the most frequent value in that column and replace the missing valie with that value or at last you can delete the row that contain the missing value.\n",
    "\n",
    "first we try to identify the missing values:\n",
    "\n",
    "### replace \"?\" with NaN In the car dataset, missing data comes with the question mark \"?\".\n",
    "We\n",
    "replace \"?\" with NaN (Not a Number), which is Python’s default missing value marker, for reasons\n",
    "of computational speed and convenience. Here we use the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "file_name = 'imports-85_csv'\n",
    "df = pd.read_csv(file_name)\n",
    "column_list= [\"symboling\",\"normalized-losses\",\"make\",\"fuel-type\",\"aspiration\", \"num-of-doors\",\"body-style\",\n",
    "         \"drive-wheels\",\"engine-location\",\"wheel-base\", \"length\",\"width\",\"height\",\"curb-weight\",\"engine-type\",\n",
    "         \"num-of-cylinders\", \"engine-size\",\"fuel-system\",\"bore\",\"stroke\",\"compression-ratio\",\"horsepower\",\n",
    "         \"peak-rpm\",\"city-mpg\",\"highway-mpg\",\"price\"]\n",
    "df.columns = column_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.replace('?', np.nan, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symboling</th>\n",
       "      <th>normalized-losses</th>\n",
       "      <th>make</th>\n",
       "      <th>fuel-type</th>\n",
       "      <th>aspiration</th>\n",
       "      <th>num-of-doors</th>\n",
       "      <th>body-style</th>\n",
       "      <th>drive-wheels</th>\n",
       "      <th>engine-location</th>\n",
       "      <th>wheel-base</th>\n",
       "      <th>...</th>\n",
       "      <th>engine-size</th>\n",
       "      <th>fuel-system</th>\n",
       "      <th>bore</th>\n",
       "      <th>stroke</th>\n",
       "      <th>compression-ratio</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>peak-rpm</th>\n",
       "      <th>city-mpg</th>\n",
       "      <th>highway-mpg</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>alfa-romero</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>convertible</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>88.6</td>\n",
       "      <td>...</td>\n",
       "      <td>130</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.47</td>\n",
       "      <td>2.68</td>\n",
       "      <td>9.0</td>\n",
       "      <td>111</td>\n",
       "      <td>5000</td>\n",
       "      <td>21</td>\n",
       "      <td>27</td>\n",
       "      <td>16500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>alfa-romero</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>hatchback</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>94.5</td>\n",
       "      <td>...</td>\n",
       "      <td>152</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>2.68</td>\n",
       "      <td>3.47</td>\n",
       "      <td>9.0</td>\n",
       "      <td>154</td>\n",
       "      <td>5000</td>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>16500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>164</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.8</td>\n",
       "      <td>...</td>\n",
       "      <td>109</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>10.0</td>\n",
       "      <td>102</td>\n",
       "      <td>5500</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>13950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>164</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>4wd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.4</td>\n",
       "      <td>...</td>\n",
       "      <td>136</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.0</td>\n",
       "      <td>115</td>\n",
       "      <td>5500</td>\n",
       "      <td>18</td>\n",
       "      <td>22</td>\n",
       "      <td>17450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.8</td>\n",
       "      <td>...</td>\n",
       "      <td>136</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.5</td>\n",
       "      <td>110</td>\n",
       "      <td>5500</td>\n",
       "      <td>19</td>\n",
       "      <td>25</td>\n",
       "      <td>15250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>158</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>105.8</td>\n",
       "      <td>...</td>\n",
       "      <td>136</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.5</td>\n",
       "      <td>110</td>\n",
       "      <td>5500</td>\n",
       "      <td>19</td>\n",
       "      <td>25</td>\n",
       "      <td>17710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>wagon</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>105.8</td>\n",
       "      <td>...</td>\n",
       "      <td>136</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.5</td>\n",
       "      <td>110</td>\n",
       "      <td>5500</td>\n",
       "      <td>19</td>\n",
       "      <td>25</td>\n",
       "      <td>18920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>158</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>turbo</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>105.8</td>\n",
       "      <td>...</td>\n",
       "      <td>131</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.13</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.3</td>\n",
       "      <td>140</td>\n",
       "      <td>5500</td>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>23875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>turbo</td>\n",
       "      <td>two</td>\n",
       "      <td>hatchback</td>\n",
       "      <td>4wd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.5</td>\n",
       "      <td>...</td>\n",
       "      <td>131</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.13</td>\n",
       "      <td>3.40</td>\n",
       "      <td>7.0</td>\n",
       "      <td>160</td>\n",
       "      <td>5500</td>\n",
       "      <td>16</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2</td>\n",
       "      <td>192</td>\n",
       "      <td>bmw</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>sedan</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>101.2</td>\n",
       "      <td>...</td>\n",
       "      <td>108</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.50</td>\n",
       "      <td>2.80</td>\n",
       "      <td>8.8</td>\n",
       "      <td>101</td>\n",
       "      <td>5800</td>\n",
       "      <td>23</td>\n",
       "      <td>29</td>\n",
       "      <td>16430</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   symboling normalized-losses         make fuel-type aspiration num-of-doors  \\\n",
       "0          3               NaN  alfa-romero       gas        std          two   \n",
       "1          1               NaN  alfa-romero       gas        std          two   \n",
       "2          2               164         audi       gas        std         four   \n",
       "3          2               164         audi       gas        std         four   \n",
       "4          2               NaN         audi       gas        std          two   \n",
       "5          1               158         audi       gas        std         four   \n",
       "6          1               NaN         audi       gas        std         four   \n",
       "7          1               158         audi       gas      turbo         four   \n",
       "8          0               NaN         audi       gas      turbo          two   \n",
       "9          2               192          bmw       gas        std          two   \n",
       "\n",
       "    body-style drive-wheels engine-location  wheel-base  ...    engine-size  \\\n",
       "0  convertible          rwd           front        88.6  ...            130   \n",
       "1    hatchback          rwd           front        94.5  ...            152   \n",
       "2        sedan          fwd           front        99.8  ...            109   \n",
       "3        sedan          4wd           front        99.4  ...            136   \n",
       "4        sedan          fwd           front        99.8  ...            136   \n",
       "5        sedan          fwd           front       105.8  ...            136   \n",
       "6        wagon          fwd           front       105.8  ...            136   \n",
       "7        sedan          fwd           front       105.8  ...            131   \n",
       "8    hatchback          4wd           front        99.5  ...            131   \n",
       "9        sedan          rwd           front       101.2  ...            108   \n",
       "\n",
       "   fuel-system  bore  stroke compression-ratio horsepower  peak-rpm city-mpg  \\\n",
       "0         mpfi  3.47    2.68               9.0        111      5000       21   \n",
       "1         mpfi  2.68    3.47               9.0        154      5000       19   \n",
       "2         mpfi  3.19    3.40              10.0        102      5500       24   \n",
       "3         mpfi  3.19    3.40               8.0        115      5500       18   \n",
       "4         mpfi  3.19    3.40               8.5        110      5500       19   \n",
       "5         mpfi  3.19    3.40               8.5        110      5500       19   \n",
       "6         mpfi  3.19    3.40               8.5        110      5500       19   \n",
       "7         mpfi  3.13    3.40               8.3        140      5500       17   \n",
       "8         mpfi  3.13    3.40               7.0        160      5500       16   \n",
       "9         mpfi  3.50    2.80               8.8        101      5800       23   \n",
       "\n",
       "  highway-mpg  price  \n",
       "0          27  16500  \n",
       "1          26  16500  \n",
       "2          30  13950  \n",
       "3          22  17450  \n",
       "4          25  15250  \n",
       "5          25  17710  \n",
       "6          25  18920  \n",
       "7          20  23875  \n",
       "8          22    NaN  \n",
       "9          29  16430  \n",
       "\n",
       "[10 rows x 26 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Evaluating for Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can use python to identify the missing value. There are 2 methods to classify missing data .isnull(),.notnull()\n",
    "True shows there is a missing value and false shows there is not missing value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symboling</th>\n",
       "      <th>normalized-losses</th>\n",
       "      <th>make</th>\n",
       "      <th>fuel-type</th>\n",
       "      <th>aspiration</th>\n",
       "      <th>num-of-doors</th>\n",
       "      <th>body-style</th>\n",
       "      <th>drive-wheels</th>\n",
       "      <th>engine-location</th>\n",
       "      <th>wheel-base</th>\n",
       "      <th>...</th>\n",
       "      <th>engine-size</th>\n",
       "      <th>fuel-system</th>\n",
       "      <th>bore</th>\n",
       "      <th>stroke</th>\n",
       "      <th>compression-ratio</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>peak-rpm</th>\n",
       "      <th>city-mpg</th>\n",
       "      <th>highway-mpg</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   symboling  normalized-losses   make  fuel-type  aspiration  num-of-doors  \\\n",
       "0      False               True  False      False       False         False   \n",
       "1      False               True  False      False       False         False   \n",
       "2      False              False  False      False       False         False   \n",
       "3      False              False  False      False       False         False   \n",
       "4      False               True  False      False       False         False   \n",
       "\n",
       "   body-style  drive-wheels  engine-location  wheel-base  ...    engine-size  \\\n",
       "0       False         False            False       False  ...          False   \n",
       "1       False         False            False       False  ...          False   \n",
       "2       False         False            False       False  ...          False   \n",
       "3       False         False            False       False  ...          False   \n",
       "4       False         False            False       False  ...          False   \n",
       "\n",
       "   fuel-system   bore  stroke  compression-ratio  horsepower  peak-rpm  \\\n",
       "0        False  False   False              False       False     False   \n",
       "1        False  False   False              False       False     False   \n",
       "2        False  False   False              False       False     False   \n",
       "3        False  False   False              False       False     False   \n",
       "4        False  False   False              False       False     False   \n",
       "\n",
       "   city-mpg  highway-mpg  price  \n",
       "0     False        False  False  \n",
       "1     False        False  False  \n",
       "2     False        False  False  \n",
       "3     False        False  False  \n",
       "4     False        False  False  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_data = df.isnull()\n",
    "missing_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3 Count missing values in each column\n",
    "Here i want to iterate in dataset and print how many missing value exists in each column as follow:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "symboling\n",
      "False    204\n",
      "Name: symboling, dtype: int64\n",
      "\n",
      "normalized-losses\n",
      "False    164\n",
      "True      40\n",
      "Name: normalized-losses, dtype: int64\n",
      "\n",
      "make\n",
      "False    204\n",
      "Name: make, dtype: int64\n",
      "\n",
      "fuel-type\n",
      "False    204\n",
      "Name: fuel-type, dtype: int64\n",
      "\n",
      "aspiration\n",
      "False    204\n",
      "Name: aspiration, dtype: int64\n",
      "\n",
      "num-of-doors\n",
      "False    202\n",
      "True       2\n",
      "Name: num-of-doors, dtype: int64\n",
      "\n",
      "body-style\n",
      "False    204\n",
      "Name: body-style, dtype: int64\n",
      "\n",
      "drive-wheels\n",
      "False    204\n",
      "Name: drive-wheels, dtype: int64\n",
      "\n",
      "engine-location\n",
      "False    204\n",
      "Name: engine-location, dtype: int64\n",
      "\n",
      "wheel-base\n",
      "False    204\n",
      "Name: wheel-base, dtype: int64\n",
      "\n",
      "length\n",
      "False    204\n",
      "Name: length, dtype: int64\n",
      "\n",
      "width\n",
      "False    204\n",
      "Name: width, dtype: int64\n",
      "\n",
      "height\n",
      "False    204\n",
      "Name: height, dtype: int64\n",
      "\n",
      "curb-weight\n",
      "False    204\n",
      "Name: curb-weight, dtype: int64\n",
      "\n",
      "engine-type\n",
      "False    204\n",
      "Name: engine-type, dtype: int64\n",
      "\n",
      "num-of-cylinders\n",
      "False    204\n",
      "Name: num-of-cylinders, dtype: int64\n",
      "\n",
      "engine-size\n",
      "False    204\n",
      "Name: engine-size, dtype: int64\n",
      "\n",
      "fuel-system\n",
      "False    204\n",
      "Name: fuel-system, dtype: int64\n",
      "\n",
      "bore\n",
      "False    200\n",
      "True       4\n",
      "Name: bore, dtype: int64\n",
      "\n",
      "stroke\n",
      "False    200\n",
      "True       4\n",
      "Name: stroke, dtype: int64\n",
      "\n",
      "compression-ratio\n",
      "False    204\n",
      "Name: compression-ratio, dtype: int64\n",
      "\n",
      "horsepower\n",
      "False    202\n",
      "True       2\n",
      "Name: horsepower, dtype: int64\n",
      "\n",
      "peak-rpm\n",
      "False    202\n",
      "True       2\n",
      "Name: peak-rpm, dtype: int64\n",
      "\n",
      "city-mpg\n",
      "False    204\n",
      "Name: city-mpg, dtype: int64\n",
      "\n",
      "highway-mpg\n",
      "False    204\n",
      "Name: highway-mpg, dtype: int64\n",
      "\n",
      "price\n",
      "False    200\n",
      "True       4\n",
      "Name: price, dtype: int64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for column in missing_data.columns.values.tolist():\n",
    "    print(column)\n",
    "    print(missing_data[column].value_counts())\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "- \"normalized-losses\": 41 missing data\n",
    "- \"num-of-doors\": 2 missing data\n",
    "- \"bore\": 4 missing data\n",
    "- \"stroke\" : 4 missing data\n",
    "- \"horsepower\": 2 missing data\n",
    "- \"peak-rpm\": 2 missing data\n",
    "- \"price\": 4 missing data\n",
    "## Deal with missing data How to deal with missing data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop \n",
    "We can drop the missing values row or column \n",
    "but when we drop data that there is alot of missing values in a column or data is very important for example here \n",
    "we will drop the missing values in price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dropna(subset=[\"price\"], axis=0, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repalce \n",
    "\n",
    "#### Replace by mean:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"normalized-losses\": 41 missing data, replace them with mean\n",
    "\"stroke\": 4 missing data, replace them with mean\n",
    "\"bore\": 4 missing data, replace them with mean\n",
    "\"horsepower\": 2 missing data, replace them with mean\n",
    "\"peak-rpm\": 2 missing data, replace them with mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the average of the column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_1 = df[\"normalized-losses\"].astype(\"float\").mean(axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replace \"NaN\" by mean value in \"normalized-losses\" column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"normalized-losses\"].replace(np.nan, avg_1, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the mean value for ’bore’ column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_2=df['bore'].astype('float').mean(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replace NaN by mean value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['bore'].replace(np.nan, avg_2, inplace= True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_3 = df[\"stroke\"].astype(\"float\").mean(axis = 0)\n",
    "df[\"stroke\"].replace(np.nan, avg_3, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_4=df['horsepower'].astype('float').mean(axis=0)\n",
    "df['horsepower'].replace(np.nan, avg_4, inplace= True)\n",
    "\n",
    "avg_5=df['peak-rpm'].astype('float').mean(axis=0)\n",
    "df['peak-rpm'].replace(np.nan, avg_5, inplace= True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see which values are present in a particular column, we can use the \".value_counts()\"\n",
    "method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "four    113\n",
       "two      85\n",
       "Name: num-of-doors, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['num-of-doors'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'four'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['num-of-doors'].value_counts().idxmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace\n",
    "\n",
    "df[\"num-of-doors\"].replace(np.nan, \"four\", inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symboling</th>\n",
       "      <th>normalized-losses</th>\n",
       "      <th>make</th>\n",
       "      <th>fuel-type</th>\n",
       "      <th>aspiration</th>\n",
       "      <th>num-of-doors</th>\n",
       "      <th>body-style</th>\n",
       "      <th>drive-wheels</th>\n",
       "      <th>engine-location</th>\n",
       "      <th>wheel-base</th>\n",
       "      <th>...</th>\n",
       "      <th>engine-size</th>\n",
       "      <th>fuel-system</th>\n",
       "      <th>bore</th>\n",
       "      <th>stroke</th>\n",
       "      <th>compression-ratio</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>peak-rpm</th>\n",
       "      <th>city-mpg</th>\n",
       "      <th>highway-mpg</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>122</td>\n",
       "      <td>alfa-romero</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>convertible</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>88.6</td>\n",
       "      <td>...</td>\n",
       "      <td>130</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.47</td>\n",
       "      <td>2.68</td>\n",
       "      <td>9.0</td>\n",
       "      <td>111</td>\n",
       "      <td>5000</td>\n",
       "      <td>21</td>\n",
       "      <td>27</td>\n",
       "      <td>16500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>122</td>\n",
       "      <td>alfa-romero</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>hatchback</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>94.5</td>\n",
       "      <td>...</td>\n",
       "      <td>152</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>2.68</td>\n",
       "      <td>3.47</td>\n",
       "      <td>9.0</td>\n",
       "      <td>154</td>\n",
       "      <td>5000</td>\n",
       "      <td>19</td>\n",
       "      <td>26</td>\n",
       "      <td>16500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>164</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.8</td>\n",
       "      <td>...</td>\n",
       "      <td>109</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>10.0</td>\n",
       "      <td>102</td>\n",
       "      <td>5500</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>13950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>164</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>4wd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.4</td>\n",
       "      <td>...</td>\n",
       "      <td>136</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.0</td>\n",
       "      <td>115</td>\n",
       "      <td>5500</td>\n",
       "      <td>18</td>\n",
       "      <td>22</td>\n",
       "      <td>17450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>122</td>\n",
       "      <td>audi</td>\n",
       "      <td>gas</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.8</td>\n",
       "      <td>...</td>\n",
       "      <td>136</td>\n",
       "      <td>mpfi</td>\n",
       "      <td>3.19</td>\n",
       "      <td>3.40</td>\n",
       "      <td>8.5</td>\n",
       "      <td>110</td>\n",
       "      <td>5500</td>\n",
       "      <td>19</td>\n",
       "      <td>25</td>\n",
       "      <td>15250</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   symboling normalized-losses         make fuel-type aspiration num-of-doors  \\\n",
       "0          3               122  alfa-romero       gas        std          two   \n",
       "1          1               122  alfa-romero       gas        std          two   \n",
       "2          2               164         audi       gas        std         four   \n",
       "3          2               164         audi       gas        std         four   \n",
       "4          2               122         audi       gas        std          two   \n",
       "\n",
       "    body-style drive-wheels engine-location  wheel-base  ...    engine-size  \\\n",
       "0  convertible          rwd           front        88.6  ...            130   \n",
       "1    hatchback          rwd           front        94.5  ...            152   \n",
       "2        sedan          fwd           front        99.8  ...            109   \n",
       "3        sedan          4wd           front        99.4  ...            136   \n",
       "4        sedan          fwd           front        99.8  ...            136   \n",
       "\n",
       "   fuel-system  bore  stroke compression-ratio horsepower  peak-rpm city-mpg  \\\n",
       "0         mpfi  3.47    2.68               9.0        111      5000       21   \n",
       "1         mpfi  2.68    3.47               9.0        154      5000       19   \n",
       "2         mpfi  3.19    3.40              10.0        102      5500       24   \n",
       "3         mpfi  3.19    3.40               8.0        115      5500       18   \n",
       "4         mpfi  3.19    3.40               8.5        110      5500       19   \n",
       "\n",
       "  highway-mpg  price  \n",
       "0          27  16500  \n",
       "1          26  16500  \n",
       "2          30  13950  \n",
       "3          22  17450  \n",
       "4          25  15250  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we obtain the dataset with no missing values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step in data wrangling is checking and making sure that all data is in the correct format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets list the data types for each column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "symboling              int64\n",
       "normalized-losses     object\n",
       "make                  object\n",
       "fuel-type             object\n",
       "aspiration            object\n",
       "num-of-doors          object\n",
       "body-style            object\n",
       "drive-wheels          object\n",
       "engine-location       object\n",
       "wheel-base           float64\n",
       "length               float64\n",
       "width                float64\n",
       "height               float64\n",
       "curb-weight            int64\n",
       "engine-type           object\n",
       "num-of-cylinders      object\n",
       "engine-size            int64\n",
       "fuel-system           object\n",
       "bore                  object\n",
       "stroke                object\n",
       "compression-ratio    float64\n",
       "horsepower            object\n",
       "peak-rpm              object\n",
       "city-mpg               int64\n",
       "highway-mpg            int64\n",
       "price                 object\n",
       "dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, as we see some of columns are not in correct dormat so we need to change them into a suitabgle format\n",
    "we can change the type by df.astype('specific type')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "df[[\"bore\", \"stroke\"]] = df[[\"bore\", \"stroke\"]].astype(\"float\")\n",
    "df[[\"normalized-losses\"]] = df[[\"normalized-losses\"]].astype(\"int\")\n",
    "df[[\"price\"]] = df[[\"price\"]].astype(\"float\")\n",
    "df[[\"peak-rpm\"]] = df[[\"peak-rpm\"]].astype(\"float\")\n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "symboling              int64\n",
       "normalized-losses      int64\n",
       "make                  object\n",
       "fuel-type             object\n",
       "aspiration            object\n",
       "num-of-doors          object\n",
       "body-style            object\n",
       "drive-wheels          object\n",
       "engine-location       object\n",
       "wheel-base           float64\n",
       "length               float64\n",
       "width                float64\n",
       "height               float64\n",
       "curb-weight            int64\n",
       "engine-type           object\n",
       "num-of-cylinders      object\n",
       "engine-size            int64\n",
       "fuel-system           object\n",
       "bore                 float64\n",
       "stroke               float64\n",
       "compression-ratio    float64\n",
       "horsepower            object\n",
       "peak-rpm             float64\n",
       "city-mpg               int64\n",
       "highway-mpg            int64\n",
       "price                float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, data set is clean and easy to undestand."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Standardization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "data Standardization is an eefective method. Data Standardization is also a term for a particular type of data normalization, where we subtract\n",
    "the mean and divide by the standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['city-L/100km'] = 235/df[\"city-mpg\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      11.190476\n",
       "1      12.368421\n",
       "2       9.791667\n",
       "3      13.055556\n",
       "4      12.368421\n",
       "5      12.368421\n",
       "6      12.368421\n",
       "7      13.823529\n",
       "9      10.217391\n",
       "10     10.217391\n",
       "11     11.190476\n",
       "12     11.190476\n",
       "13     11.750000\n",
       "14     14.687500\n",
       "15     14.687500\n",
       "16     15.666667\n",
       "17      5.000000\n",
       "18      6.184211\n",
       "19      6.184211\n",
       "20      6.351351\n",
       "21      7.580645\n",
       "22      9.791667\n",
       "23      7.580645\n",
       "24      7.580645\n",
       "25      7.580645\n",
       "26      9.791667\n",
       "27      9.791667\n",
       "28     12.368421\n",
       "29      4.795918\n",
       "30      7.580645\n",
       "         ...    \n",
       "174     8.703704\n",
       "175     8.703704\n",
       "176     8.703704\n",
       "177    11.750000\n",
       "178    12.368421\n",
       "179    11.750000\n",
       "180    12.368421\n",
       "181     6.351351\n",
       "182     8.703704\n",
       "183     6.351351\n",
       "184     8.703704\n",
       "185     8.703704\n",
       "186     6.351351\n",
       "187     9.038462\n",
       "188     9.791667\n",
       "189     9.791667\n",
       "190    12.368421\n",
       "191     7.121212\n",
       "192     9.400000\n",
       "193    10.217391\n",
       "194    10.217391\n",
       "195     9.791667\n",
       "196     9.791667\n",
       "197    13.823529\n",
       "198    13.823529\n",
       "199    10.217391\n",
       "200    12.368421\n",
       "201    13.055556\n",
       "202     9.038462\n",
       "203    12.368421\n",
       "Name: city-L/100km, Length: 200, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['city-L/100km']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalization is the process of impact data into a certain range for example between -1,1 or 0,1. In fact it is a part of data scaling process.\n",
    "\n",
    "But be careful if your data contains 0 and 1 scaling wont help you otherwise if one or many columns are very different from others you should scale them.\n",
    "\n",
    "for example we normalize 'length' and 'width' as follow:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['length'] = df['length']/df['length'].max()\n",
    "df['width'] = df['width']/df['width'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.040609515144035456"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['height'] = df['height']/df['height'].max()\n",
    "df['height'].head(3)\n",
    "df['height'].std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Binning is a process of transforming continuous numerical variables into discrete categorical\n",
    "’bins’, for grouped analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our dataset, \"horsepower\" is a real valued variable ranging from 48 to 288, it has 57 unique\n",
    "values. What if we only care about the price difference between cars with high horsepower,\n",
    "medium horsepower, and little horsepower (3 types)? Can we rearrange them into three ‘bins’\n",
    "to simplify analysis?\n",
    "We will use the Pandas method ’cut’ to segment the ’horsepower’ column into 3 bins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Binning Data In Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"horsepower\"]=df[\"horsepower\"].astype(float, copy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like four bins of equal size bandwidth,the forth is because the function \"cut\" include\n",
    "the rightmost edge:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "binwidth = (max(df[\"horsepower\"])-min(df[\"horsepower\"]))/4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 48. , 101.5, 155. , 208.5])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins = np.arange(min(df[\"horsepower\"]), max(df[\"horsepower\"]), binwidth)\n",
    "bins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we set the names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "group_names = ['Low', 'Medium', 'High']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>horsepower</th>\n",
       "      <th>horsepower-binned</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>111.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>154.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>102.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>115.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>110.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>110.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>110.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>140.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>101.0</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>101.0</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>121.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>121.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>121.0</td>\n",
       "      <td>Medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>182.0</td>\n",
       "      <td>High</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>182.0</td>\n",
       "      <td>High</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>182.0</td>\n",
       "      <td>High</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>48.0</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>70.0</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>70.0</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>68.0</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    horsepower horsepower-binned\n",
       "0        111.0            Medium\n",
       "1        154.0            Medium\n",
       "2        102.0            Medium\n",
       "3        115.0            Medium\n",
       "4        110.0            Medium\n",
       "5        110.0            Medium\n",
       "6        110.0            Medium\n",
       "7        140.0            Medium\n",
       "9        101.0               Low\n",
       "10       101.0               Low\n",
       "11       121.0            Medium\n",
       "12       121.0            Medium\n",
       "13       121.0            Medium\n",
       "14       182.0              High\n",
       "15       182.0              High\n",
       "16       182.0              High\n",
       "17        48.0               Low\n",
       "18        70.0               Low\n",
       "19        70.0               Low\n",
       "20        68.0               Low"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['horsepower-binned'] = pd.cut(df['horsepower'], bins, labels=group_names,include_lowest=True )\n",
    "df[['horsepower','horsepower-binned']].head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see the column \"fuel-type\" has two unique values, \"gas\" or \"diesel\". Regression doesn’t\n",
    "understand words, only numbers. To use this attribute in regression analysis, we convert \"fuel-\n",
    "type\" into indicator variables.\n",
    "We will use the panda’s method ’get_dummies’ to assign numerical values to different cate-\n",
    "gories of fuel type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>diesel</th>\n",
       "      <th>gas</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   diesel  gas\n",
       "0       0    1\n",
       "1       0    1\n",
       "2       0    1\n",
       "3       0    1\n",
       "4       0    1"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_variable_1 = pd.get_dummies(df[\"fuel-type\"])\n",
    "dummy_variable_1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>diesel</th>\n",
       "      <th>gas</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   diesel  gas\n",
       "0       0    1\n",
       "1       0    1\n",
       "2       0    1\n",
       "3       0    1\n",
       "4       0    1"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_variable_1.rename(columns={'fuel-type-diesel':'gas', 'fuel-type-diesel':'diesel'}, inplace=True)\n",
    "dummy_variable_1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([df, dummy_variable_1], axis=1)\n",
    "\n",
    "df.drop(\"fuel-type\", axis = 1, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symboling</th>\n",
       "      <th>normalized-losses</th>\n",
       "      <th>make</th>\n",
       "      <th>aspiration</th>\n",
       "      <th>num-of-doors</th>\n",
       "      <th>body-style</th>\n",
       "      <th>drive-wheels</th>\n",
       "      <th>engine-location</th>\n",
       "      <th>wheel-base</th>\n",
       "      <th>length</th>\n",
       "      <th>...</th>\n",
       "      <th>city-L/100km</th>\n",
       "      <th>horsepower-binned</th>\n",
       "      <th>diesel</th>\n",
       "      <th>gas</th>\n",
       "      <th>diesel</th>\n",
       "      <th>gas</th>\n",
       "      <th>diesel</th>\n",
       "      <th>gas</th>\n",
       "      <th>diesel</th>\n",
       "      <th>gas</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>122</td>\n",
       "      <td>alfa-romero</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>convertible</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>88.6</td>\n",
       "      <td>0.811148</td>\n",
       "      <td>...</td>\n",
       "      <td>11.190476</td>\n",
       "      <td>Medium</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>122</td>\n",
       "      <td>alfa-romero</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>hatchback</td>\n",
       "      <td>rwd</td>\n",
       "      <td>front</td>\n",
       "      <td>94.5</td>\n",
       "      <td>0.822681</td>\n",
       "      <td>...</td>\n",
       "      <td>12.368421</td>\n",
       "      <td>Medium</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>164</td>\n",
       "      <td>audi</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.8</td>\n",
       "      <td>0.848630</td>\n",
       "      <td>...</td>\n",
       "      <td>9.791667</td>\n",
       "      <td>Medium</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>164</td>\n",
       "      <td>audi</td>\n",
       "      <td>std</td>\n",
       "      <td>four</td>\n",
       "      <td>sedan</td>\n",
       "      <td>4wd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.4</td>\n",
       "      <td>0.848630</td>\n",
       "      <td>...</td>\n",
       "      <td>13.055556</td>\n",
       "      <td>Medium</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>122</td>\n",
       "      <td>audi</td>\n",
       "      <td>std</td>\n",
       "      <td>two</td>\n",
       "      <td>sedan</td>\n",
       "      <td>fwd</td>\n",
       "      <td>front</td>\n",
       "      <td>99.8</td>\n",
       "      <td>0.851994</td>\n",
       "      <td>...</td>\n",
       "      <td>12.368421</td>\n",
       "      <td>Medium</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   symboling  normalized-losses         make aspiration num-of-doors  \\\n",
       "0          3                122  alfa-romero        std          two   \n",
       "1          1                122  alfa-romero        std          two   \n",
       "2          2                164         audi        std         four   \n",
       "3          2                164         audi        std         four   \n",
       "4          2                122         audi        std          two   \n",
       "\n",
       "    body-style drive-wheels engine-location  wheel-base    length ...   \\\n",
       "0  convertible          rwd           front        88.6  0.811148 ...    \n",
       "1    hatchback          rwd           front        94.5  0.822681 ...    \n",
       "2        sedan          fwd           front        99.8  0.848630 ...    \n",
       "3        sedan          4wd           front        99.4  0.848630 ...    \n",
       "4        sedan          fwd           front        99.8  0.851994 ...    \n",
       "\n",
       "   city-L/100km  horsepower-binned  diesel  gas  diesel  gas  diesel  gas  \\\n",
       "0     11.190476             Medium       0    1       0    1       0    1   \n",
       "1     12.368421             Medium       0    1       0    1       0    1   \n",
       "2      9.791667             Medium       0    1       0    1       0    1   \n",
       "3     13.055556             Medium       0    1       0    1       0    1   \n",
       "4     12.368421             Medium       0    1       0    1       0    1   \n",
       "\n",
       "   diesel  gas  \n",
       "0       0    1  \n",
       "1       0    1  \n",
       "2       0    1  \n",
       "3       0    1  \n",
       "4       0    1  \n",
       "\n",
       "[5 rows x 35 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " save current data into a new csv file: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('cars.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}