notebooks/02_dataframes/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Tutorial 2"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "An introduction to data frames"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Loosely based on [this\n",
    "tutorial](https://juliaai.github.io/DataScienceTutorials.jl/data/dataframe/)\n",
    "from Data Science Tutorials."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "DataFrames.jl **cheatsheets**:\n",
    "[english](https://www.ahsmart.com/assets/pages/data-wrangling-with-data-frames-jl-cheat-sheet/DataFramesCheatSheet_v1.x_rev1.pdf),\n",
    "[中文](https://www.ahsmart.com/assets/pages/data-wrangling-with-data-frames-jl-cheat-sheet/DataFramesCheatSheet_v1.x_rev1_zh.pdf)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Setup"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Activating project at `~/GoogleDrive/Julia/HelloJulia`\n"
     ]
    }
   ],
   "cell_type": "code",
   "source": [
    "using Pkg\n",
    "Pkg.activate(joinpath(@__DIR__, \"..\", \"..\"))\n",
    "Pkg.instantiate()"
   ],
   "metadata": {},
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "source": [
    "## A simple handmade data frame"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "A `DataFrame` is one of [many kinds of\n",
    "objects](https://github.com/JuliaData/Tables.jl/blob/main/INTEGRATIONS.md) in the Julia\n",
    "ecosystem for representing tabular data. Here's a simple example of a table you can define\n",
    "using basic Julia (no libraries). It is a named tuple whose values are all vectors of the\n",
    "same length:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(age = [21, 25, 40], height = [1.89, 1.5, 1.4], married = Bool[1, 0, 0])"
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "cell_type": "code",
   "source": [
    "columntable = (\n",
    "    age = [21, 25, 40],\n",
    "    height = [1.89, 1.5, 1.4],\n",
    "    married = [true, false, false],\n",
    ")"
   ],
   "metadata": {},
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "source": [
    "One problem with such a table is that it's not a simple matter to grab a single row, or to\n",
    "filter rows (records) based on some criterion. For this we can convert our table to a\n",
    "`DataFrame` from the DataFrames.jl package:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×3 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m age   \u001b[0m\u001b[1m height  \u001b[0m\u001b[1m married \u001b[0m\n     │\u001b[90m Int64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Bool    \u001b[0m\n─────┼─────────────────────────\n   1 │    21     1.89     true\n   2 │    25     1.5     false\n   3 │    40     1.4     false",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×3 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">height</th><th style = \"text-align: left;\">married</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Bool\" style = \"text-align: left;\">Bool</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">21</td><td style = \"text-align: right;\">1.89</td><td style = \"text-align: right;\">true</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">25</td><td style = \"text-align: right;\">1.5</td><td style = \"text-align: right;\">false</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">40</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">false</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "cell_type": "code",
   "source": [
    "using DataFrames\n",
    "dataframe = DataFrame(columntable)"
   ],
   "metadata": {},
   "execution_count": 3
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we can do things like this:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m2×3 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m age   \u001b[0m\u001b[1m height  \u001b[0m\u001b[1m married \u001b[0m\n     │\u001b[90m Int64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Bool    \u001b[0m\n─────┼─────────────────────────\n   1 │    25      1.5    false\n   2 │    40      1.4    false",
      "text/html": [
       "<div><div style = \"float: left;\"><span>2×3 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">height</th><th style = \"text-align: left;\">married</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Bool\" style = \"text-align: left;\">Bool</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">25</td><td style = \"text-align: right;\">1.5</td><td style = \"text-align: right;\">false</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">40</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">false</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "cell_type": "code",
   "source": [
    "filter(dataframe) do row\n",
    "    row.married == false\n",
    "end"
   ],
   "metadata": {},
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "source": [
    "... and much more."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Grabbing the Titanic dataset as a DataFrame"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We'll be using [OpenML](https://www.openml.org/home) to grab datasets."
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "Tables.DictColumnTable"
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "cell_type": "code",
   "source": [
    "using OpenML\n",
    "\n",
    "table = OpenML.load(42638); # Titanic data set\n",
    "typeof(table)"
   ],
   "metadata": {},
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "source": [
    "This is not a `DataFrame`. However, it can be converted to one in the same way we\n",
    "converted our named-tuple table:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "df = DataFrame(table);"
   ],
   "metadata": {},
   "execution_count": 6
  },
  {
   "cell_type": "markdown",
   "source": [
    "Lets' look the first few rows (observations) of `df`:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m4×8 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m pclass \u001b[0m\u001b[1m sex    \u001b[0m\u001b[1m age     \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\u001b[1m cabin   \u001b[0m\u001b[1m embarked \u001b[0m\u001b[1m survived \u001b[0m\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Cat…   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Cat…?   \u001b[0m\u001b[90m Cat…?    \u001b[0m\u001b[90m Cat…     \u001b[0m\n─────┼────────────────────────────────────────────────────────────────────────\n   1 │ 3       male       22.0      1.0   7.25   \u001b[90m missing \u001b[0m S         0\n   2 │ 1       female     38.0      1.0  71.2833  C85      C         1\n   3 │ 3       female     26.0      0.0   7.925  \u001b[90m missing \u001b[0m S         1\n   4 │ 1       female     35.0      1.0  53.1     C123     S         1",
      "text/html": [
       "<div><div style = \"float: left;\"><span>4×8 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">pclass</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">cabin</th><th style = \"text-align: left;\">embarked</th><th style = \"text-align: left;\">survived</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">22.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">7.25</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">38.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">71.2833</td><td style = \"text-align: left;\">C85</td><td style = \"text-align: left;\">C</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">26.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">7.925</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">35.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">53.1</td><td style = \"text-align: left;\">C123</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "cell_type": "code",
   "source": [
    "first(df, 4)"
   ],
   "metadata": {},
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "source": [
    "Here's a summary of stats for each column:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m8×7 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m mean     \u001b[0m\u001b[1m min    \u001b[0m\u001b[1m median  \u001b[0m\u001b[1m max     \u001b[0m\u001b[1m nmissing \u001b[0m\u001b[1m eltype         \u001b[0m ⋯\n     │\u001b[90m Symbol   \u001b[0m\u001b[90m Union…   \u001b[0m\u001b[90m Any    \u001b[0m\u001b[90m Union…  \u001b[0m\u001b[90m Any     \u001b[0m\u001b[90m Int64    \u001b[0m\u001b[90m Type           \u001b[0m ⋯\n─────┼──────────────────────────────────────────────────────────────────────────\n   1 │ pclass   \u001b[90m          \u001b[0m 1      \u001b[90m         \u001b[0m 3               0  CategoricalValu ⋯\n   2 │ sex      \u001b[90m          \u001b[0m female \u001b[90m         \u001b[0m male            0  CategoricalValu\n   3 │ age       29.7589   0.42    30.0     80.0            0  Float64\n   4 │ sibsp     0.523008  0.0     0.0      8.0             0  Float64\n   5 │ fare      32.2042   0.0     14.4542  512.329         0  Float64         ⋯\n   6 │ cabin    \u001b[90m          \u001b[0m E31    \u001b[90m         \u001b[0m C148          687  Union{Missing,\n   7 │ embarked \u001b[90m          \u001b[0m C      \u001b[90m         \u001b[0m S               2  Union{Missing,\n   8 │ survived \u001b[90m          \u001b[0m 0      \u001b[90m         \u001b[0m 1               0  CategoricalValu\n\u001b[36m                                                                1 column omitted\u001b[0m",
      "text/html": [
       "<div><div style = \"float: left;\"><span>8×7 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">variable</th><th style = \"text-align: left;\">mean</th><th style = \"text-align: left;\">min</th><th style = \"text-align: left;\">median</th><th style = \"text-align: left;\">max</th><th style = \"text-align: left;\">nmissing</th><th style = \"text-align: left;\">eltype</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Symbol\" style = \"text-align: left;\">Symbol</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Type\" style = \"text-align: left;\">Type</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">pclass</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">1</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">3</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">CategoricalValue{String, UInt32}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">sex</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">female</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">CategoricalValue{String, UInt32}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">age</td><td style = \"text-align: left;\">29.7589</td><td style = \"text-align: left;\">0.42</td><td style = \"text-align: left;\">30.0</td><td style = \"text-align: left;\">80.0</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">Float64</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">sibsp</td><td style = \"text-align: left;\">0.523008</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">8.0</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">Float64</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">fare</td><td style = \"text-align: left;\">32.2042</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">14.4542</td><td style = \"text-align: left;\">512.329</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">Float64</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">6</td><td style = \"text-align: left;\">cabin</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">E31</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">C148</td><td style = \"text-align: right;\">687</td><td style = \"text-align: left;\">Union{Missing, CategoricalValue{String, UInt32}}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">7</td><td style = \"text-align: left;\">embarked</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">C</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">S</td><td style = \"text-align: right;\">2</td><td style = \"text-align: left;\">Union{Missing, CategoricalValue{String, UInt32}}</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">8</td><td style = \"text-align: left;\">survived</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">0</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">1</td><td style = \"text-align: right;\">0</td><td style = \"text-align: left;\">CategoricalValue{String, UInt32}</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "cell_type": "code",
   "source": [
    "describe(df)"
   ],
   "metadata": {},
   "execution_count": 8
  },
  {
   "cell_type": "markdown",
   "source": [
    "To get just the column names, use `names(df)`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Selecting a single entry:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "35.0"
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "cell_type": "code",
   "source": [
    "number = df[5, :age]   # or df[5, 4]"
   ],
   "metadata": {},
   "execution_count": 9
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Selecting a single row:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We use the wild-card `:`"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1mDataFrameRow\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m pclass \u001b[0m\u001b[1m sex  \u001b[0m\u001b[1m age     \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\u001b[1m cabin   \u001b[0m\u001b[1m embarked \u001b[0m\u001b[1m survived \u001b[0m\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Cat… \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Cat…?   \u001b[0m\u001b[90m Cat…?    \u001b[0m\u001b[90m Cat…     \u001b[0m\n─────┼──────────────────────────────────────────────────────────────────────\n   5 │ 3       male     35.0      0.0     8.05 \u001b[90m missing \u001b[0m S         0",
      "text/html": [
       "<div><div style = \"float: left;\"><span>DataFrameRow (8 columns)</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowLabel\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">pclass</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">cabin</th><th style = \"text-align: left;\">embarked</th><th style = \"text-align: left;\">survived</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowLabel\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowLabel\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">35.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">8.05</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">0</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 10
    }
   ],
   "cell_type": "code",
   "source": [
    "row = df[5, :]"
   ],
   "metadata": {},
   "execution_count": 10
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "35.0"
     },
     "metadata": {},
     "execution_count": 11
    }
   ],
   "cell_type": "code",
   "source": [
    "number2 = row.age"
   ],
   "metadata": {},
   "execution_count": 11
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Selecting multiple rows"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "By index:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m5×8 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m pclass \u001b[0m\u001b[1m sex    \u001b[0m\u001b[1m age     \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\u001b[1m cabin   \u001b[0m\u001b[1m embarked \u001b[0m\u001b[1m survived \u001b[0m\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Cat…   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Cat…?   \u001b[0m\u001b[90m Cat…?    \u001b[0m\u001b[90m Cat…     \u001b[0m\n─────┼────────────────────────────────────────────────────────────────────────\n   1 │ 3       female     26.0      0.0   7.925  \u001b[90m missing \u001b[0m S         1\n   2 │ 1       female     35.0      1.0  53.1     C123     S         1\n   3 │ 3       male       35.0      0.0   8.05   \u001b[90m missing \u001b[0m S         0\n   4 │ 3       male       30.0      0.0   8.4583 \u001b[90m missing \u001b[0m Q         0\n   5 │ 1       male       54.0      0.0  51.8625  E46      S         0",
      "text/html": [
       "<div><div style = \"float: left;\"><span>5×8 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">pclass</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">cabin</th><th style = \"text-align: left;\">embarked</th><th style = \"text-align: left;\">survived</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">26.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">7.925</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">35.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">53.1</td><td style = \"text-align: left;\">C123</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">35.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">8.05</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">30.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">8.4583</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">Q</td><td style = \"text-align: left;\">0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">54.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">51.8625</td><td style = \"text-align: left;\">E46</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">0</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 12
    }
   ],
   "cell_type": "code",
   "source": [
    "small_df = df[3:7, :]"
   ],
   "metadata": {},
   "execution_count": 12
  },
  {
   "cell_type": "markdown",
   "source": [
    "By applying some criterion:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m5×8 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m pclass \u001b[0m\u001b[1m sex    \u001b[0m\u001b[1m age     \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\u001b[1m cabin   \u001b[0m\u001b[1m embarked \u001b[0m\u001b[1m survived \u001b[0m\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Cat…   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Cat…?   \u001b[0m\u001b[90m Cat…?    \u001b[0m\u001b[90m Cat…     \u001b[0m\n─────┼────────────────────────────────────────────────────────────────────────\n   1 │ 1       female     63.0      1.0  77.9583  D7       S         1\n   2 │ 3       female     63.0      0.0   9.5875 \u001b[90m missing \u001b[0m S         1\n   3 │ 2       male       62.0      0.0  10.5    \u001b[90m missing \u001b[0m S         1\n   4 │ 1       male       80.0      0.0  30.0     A23      S         1\n   5 │ 1       female     62.0      0.0  80.0     B28     \u001b[90m missing  \u001b[0m 1",
      "text/html": [
       "<div><div style = \"float: left;\"><span>5×8 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">pclass</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">cabin</th><th style = \"text-align: left;\">embarked</th><th style = \"text-align: left;\">survived</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">63.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">77.9583</td><td style = \"text-align: left;\">D7</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">63.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">9.5875</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">2</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">62.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">10.5</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">80.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">30.0</td><td style = \"text-align: left;\">A23</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">62.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">80.0</td><td style = \"text-align: left;\">B28</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">1</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "cell_type": "code",
   "source": [
    "df2 = filter(df) do row\n",
    "    row.age > 60 && row.survived == \"1\"\n",
    "end"
   ],
   "metadata": {},
   "execution_count": 13
  },
  {
   "cell_type": "markdown",
   "source": [
    "(Alternatively, one can use `subset` or Boolean indexing; see the\n",
    "[cheatsheet](https://www.ahsmart.com/assets/pages/data-wrangling-with-data-frames-jl-cheat-sheet/DataFramesCheatSheet_v1.x_rev1.pdf).)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Selecting a single column"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Here's how I get the column named \"Age\":"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "5-element Vector{Float64}:\n 63.0\n 63.0\n 62.0\n 80.0\n 62.0"
     },
     "metadata": {},
     "execution_count": 14
    }
   ],
   "cell_type": "code",
   "source": [
    "df2.age"
   ],
   "metadata": {},
   "execution_count": 14
  },
  {
   "cell_type": "markdown",
   "source": [
    "Since this is just a vector, I can restrict the rows as usual:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "4-element Vector{Float64}:\n 63.0\n 62.0\n 80.0\n 62.0"
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "cell_type": "code",
   "source": [
    "df2.age[2:5]  # but same as df2[2:5, :Age]"
   ],
   "metadata": {},
   "execution_count": 15
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Selecting multiple columns - `select`"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The following wraps `df.age` as a single-column `DataFrame`:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m5×1 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m age     \u001b[0m\n     │\u001b[90m Float64 \u001b[0m\n─────┼─────────\n   1 │    63.0\n   2 │    63.0\n   3 │    62.0\n   4 │    80.0\n   5 │    62.0",
      "text/html": [
       "<div><div style = \"float: left;\"><span>5×1 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">age</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">63.0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">63.0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">62.0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: right;\">80.0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: right;\">62.0</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 16
    }
   ],
   "cell_type": "code",
   "source": [
    "select(df2, :age)"
   ],
   "metadata": {},
   "execution_count": 16
  },
  {
   "cell_type": "markdown",
   "source": [
    "... which generalizes to multiple columns:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m5×2 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m age     \u001b[0m\u001b[1m sex    \u001b[0m\n     │\u001b[90m Float64 \u001b[0m\u001b[90m Cat…   \u001b[0m\n─────┼─────────────────\n   1 │    63.0  female\n   2 │    63.0  female\n   3 │    62.0  male\n   4 │    80.0  male\n   5 │    62.0  female",
      "text/html": [
       "<div><div style = \"float: left;\"><span>5×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sex</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">63.0</td><td style = \"text-align: left;\">female</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">63.0</td><td style = \"text-align: left;\">female</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">62.0</td><td style = \"text-align: left;\">male</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: right;\">80.0</td><td style = \"text-align: left;\">male</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: right;\">62.0</td><td style = \"text-align: left;\">female</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 17
    }
   ],
   "cell_type": "code",
   "source": [
    "select(df2, [:age, :sex])"
   ],
   "metadata": {},
   "execution_count": 17
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m5×7 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m pclass \u001b[0m\u001b[1m sex    \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\u001b[1m cabin   \u001b[0m\u001b[1m embarked \u001b[0m\u001b[1m survived \u001b[0m\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Cat…   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Cat…?   \u001b[0m\u001b[90m Cat…?    \u001b[0m\u001b[90m Cat…     \u001b[0m\n─────┼───────────────────────────────────────────────────────────────\n   1 │ 1       female      1.0  77.9583  D7       S         1\n   2 │ 3       female      0.0   9.5875 \u001b[90m missing \u001b[0m S         1\n   3 │ 2       male        0.0  10.5    \u001b[90m missing \u001b[0m S         1\n   4 │ 1       male        0.0  30.0     A23      S         1\n   5 │ 1       female      0.0  80.0     B28     \u001b[90m missing  \u001b[0m 1",
      "text/html": [
       "<div><div style = \"float: left;\"><span>5×7 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">pclass</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">cabin</th><th style = \"text-align: left;\">embarked</th><th style = \"text-align: left;\">survived</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">77.9583</td><td style = \"text-align: left;\">D7</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">9.5875</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">2</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">10.5</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">30.0</td><td style = \"text-align: left;\">A23</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">80.0</td><td style = \"text-align: left;\">B28</td><td style = \"font-style: italic; text-align: left;\">missing</td><td style = \"text-align: left;\">1</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "cell_type": "code",
   "source": [
    "select(df2, Not(:age))"
   ],
   "metadata": {},
   "execution_count": 18
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m5×4 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m sex    \u001b[0m\u001b[1m age     \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\n─────┼───────────────────────────────────\n   1 │ female     63.0      1.0  77.9583\n   2 │ female     63.0      0.0   9.5875\n   3 │ male       62.0      0.0  10.5\n   4 │ male       80.0      0.0  30.0\n   5 │ female     62.0      0.0  80.0",
      "text/html": [
       "<div><div style = \"float: left;\"><span>5×4 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">63.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">77.9583</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">63.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">9.5875</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">62.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">10.5</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">80.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">30.0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">62.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">80.0</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 19
    }
   ],
   "cell_type": "code",
   "source": [
    "select(df2, Between(:sex, :fare))"
   ],
   "metadata": {},
   "execution_count": 19
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Copies or views of the data?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "So far, all the selection mechanisms discussed so far return a new\n",
    "`DataFrame` object, with these exceptions:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "* selecting a single row, as in `df[3,:]` - this returns a `DataFrameRow`.\n",
    "* selecting a single column as a vector, as in `df.age` - this returns a vector"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "These exceptions are *views* of the original `DataFrame`, in the\n",
    "sense that no data is copied and mutating the view mutates the\n",
    "original `DataFrame`:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "1000.0"
     },
     "metadata": {},
     "execution_count": 20
    }
   ],
   "cell_type": "code",
   "source": [
    "v = df.age;\n",
    "v[1] = 1000\n",
    "df[1, :age]"
   ],
   "metadata": {},
   "execution_count": 20
  },
  {
   "cell_type": "markdown",
   "source": [
    "Mutating the output of all the other selection mechanisms will not\n",
    "touch the original `DataFrame`, as data is copied in those cases."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Use `select!` to make \"in-place\" column selections which mutate the\n",
    "original `DataFrame`. For example, the following permanently drops a\n",
    "column from `df`:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m2×7 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m pclass \u001b[0m\u001b[1m sex    \u001b[0m\u001b[1m age     \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\u001b[1m embarked \u001b[0m\u001b[1m survived \u001b[0m\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Cat…   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Cat…?    \u001b[0m\u001b[90m Cat…     \u001b[0m\n─────┼───────────────────────────────────────────────────────────────\n   1 │ 3       male     1000.0      1.0   7.25    S         0\n   2 │ 1       female     38.0      1.0  71.2833  C         1",
      "text/html": [
       "<div><div style = \"float: left;\"><span>2×7 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">pclass</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">embarked</th><th style = \"text-align: left;\">survived</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">1000.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">7.25</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">38.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">71.2833</td><td style = \"text-align: left;\">C</td><td style = \"text-align: left;\">1</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 21
    }
   ],
   "cell_type": "code",
   "source": [
    "select!(df, Not(:cabin));\n",
    "first(df, 2)"
   ],
   "metadata": {},
   "execution_count": 21
  },
  {
   "cell_type": "markdown",
   "source": [
    "In addition to row and column views of a `DataFrame` we can construct\n",
    "larger views, of type `SubDataFrame`, which share most of the\n",
    "behviour of an ordinary `DataFrame`, except they only *point* to\n",
    "data in the parent `DataFrame`, rather than duplicate it:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×2 SubDataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m fare    \u001b[0m\u001b[1m age     \u001b[0m\n     │\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\n─────┼──────────────────\n   1 │  7.25     1000.0\n   2 │ 71.2833     38.0\n   3 │  7.925      26.0",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×2 SubDataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">age</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">7.25</td><td style = \"text-align: right;\">1000.0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">71.2833</td><td style = \"text-align: right;\">38.0</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">7.925</td><td style = \"text-align: right;\">26.0</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 22
    }
   ],
   "cell_type": "code",
   "source": [
    "df_copy = df[1:3, [:fare, :age]]       # this is copy\n",
    "df_view = @view df[1:3, [:fare, :age]] # this is a view"
   ],
   "metadata": {},
   "execution_count": 22
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "4000.0"
     },
     "metadata": {},
     "execution_count": 23
    }
   ],
   "cell_type": "code",
   "source": [
    "df_view[1, :age] = 4000\n",
    "df[1, :age]"
   ],
   "metadata": {},
   "execution_count": 23
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "1000.0"
     },
     "metadata": {},
     "execution_count": 24
    }
   ],
   "cell_type": "code",
   "source": [
    "df_copy[1, :age]"
   ],
   "metadata": {},
   "execution_count": 24
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Describing the data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "To broaden the summary provided by `describe(df)` we can pass a\n",
    "number of symbols to indicate the statistics we are after:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m7×6 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m min    \u001b[0m\u001b[1m max     \u001b[0m\u001b[1m mean     \u001b[0m\u001b[1m median  \u001b[0m\u001b[1m std     \u001b[0m\n     │\u001b[90m Symbol   \u001b[0m\u001b[90m Any    \u001b[0m\u001b[90m Any     \u001b[0m\u001b[90m Union…   \u001b[0m\u001b[90m Union…  \u001b[0m\u001b[90m Union…  \u001b[0m\n─────┼───────────────────────────────────────────────────────\n   1 │ pclass    1       3       \u001b[90m          \u001b[0m\u001b[90m         \u001b[0m\u001b[90m         \u001b[0m\n   2 │ sex       female  male    \u001b[90m          \u001b[0m\u001b[90m         \u001b[0m\u001b[90m         \u001b[0m\n   3 │ age       0.42    4000.0   34.2235   30.0     133.642\n   4 │ sibsp     0.0     8.0      0.523008  0.0      1.10274\n   5 │ fare      0.0     512.329  32.2042   14.4542  49.6934\n   6 │ embarked  C       S       \u001b[90m          \u001b[0m\u001b[90m         \u001b[0m\u001b[90m         \u001b[0m\n   7 │ survived  0       1       \u001b[90m          \u001b[0m\u001b[90m         \u001b[0m\u001b[90m         \u001b[0m",
      "text/html": [
       "<div><div style = \"float: left;\"><span>7×6 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">variable</th><th style = \"text-align: left;\">min</th><th style = \"text-align: left;\">max</th><th style = \"text-align: left;\">mean</th><th style = \"text-align: left;\">median</th><th style = \"text-align: left;\">std</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Symbol\" style = \"text-align: left;\">Symbol</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">pclass</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">3</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">sex</td><td style = \"text-align: left;\">female</td><td style = \"text-align: left;\">male</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">age</td><td style = \"text-align: left;\">0.42</td><td style = \"text-align: left;\">4000.0</td><td style = \"text-align: left;\">34.2235</td><td style = \"text-align: left;\">30.0</td><td style = \"text-align: left;\">133.642</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">sibsp</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">8.0</td><td style = \"text-align: left;\">0.523008</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">1.10274</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">fare</td><td style = \"text-align: left;\">0.0</td><td style = \"text-align: left;\">512.329</td><td style = \"text-align: left;\">32.2042</td><td style = \"text-align: left;\">14.4542</td><td style = \"text-align: left;\">49.6934</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">6</td><td style = \"text-align: left;\">embarked</td><td style = \"text-align: left;\">C</td><td style = \"text-align: left;\">S</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">7</td><td style = \"text-align: left;\">survived</td><td style = \"text-align: left;\">0</td><td style = \"text-align: left;\">1</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 25
    }
   ],
   "cell_type": "code",
   "source": [
    "describe(df, :min, :max, :mean, :median, :std)"
   ],
   "metadata": {},
   "execution_count": 25
  },
  {
   "cell_type": "markdown",
   "source": [
    "The following are all supported:\n",
    "* `mean`, `std`, `min`, `max`, `median`, `first`, `last` are all fairly self explanatory\n",
    "* `q25`, `q75` are respectively for the 25th and 75th percentile,\n",
    "* `eltype`, `nunique`, `nmissing` can also be used"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can also pass custom function, together  with a name for the generated column by passing a pair `function => :name`, as in"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×4 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m mean    \u001b[0m\u001b[1m median \u001b[0m\u001b[1m mae     \u001b[0m\n     │\u001b[90m Symbol   \u001b[0m\u001b[90m Union…  \u001b[0m\u001b[90m Union… \u001b[0m\u001b[90m Union…  \u001b[0m\n─────┼────────────────────────────────────\n   1 │ pclass   \u001b[90m         \u001b[0m\u001b[90m        \u001b[0m\u001b[90m         \u001b[0m\n   2 │ sex      \u001b[90m         \u001b[0m\u001b[90m        \u001b[0m\u001b[90m         \u001b[0m\n   3 │ age       34.2235  30.0    34.2235",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×4 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">variable</th><th style = \"text-align: left;\">mean</th><th style = \"text-align: left;\">median</th><th style = \"text-align: left;\">mae</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Symbol\" style = \"text-align: left;\">Symbol</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">pclass</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">sex</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">age</td><td style = \"text-align: left;\">34.2235</td><td style = \"text-align: left;\">30.0</td><td style = \"text-align: left;\">34.2235</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 26
    }
   ],
   "cell_type": "code",
   "source": [
    "using Statistics # to get functions like `mean` and `std`\n",
    "foo(v) = mean(abs.(v))\n",
    "d = describe(df, :mean, :median, foo => :mae)\n",
    "first(d, 3)"
   ],
   "metadata": {},
   "execution_count": 26
  },
  {
   "cell_type": "markdown",
   "source": [
    "Note that the object returned by `describe` is itself a `DataFrame`:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m7×2 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m mean     \u001b[0m\n     │\u001b[90m Symbol   \u001b[0m\u001b[90m Union…   \u001b[0m\n─────┼────────────────────\n   1 │ pclass   \u001b[90m          \u001b[0m\n   2 │ sex      \u001b[90m          \u001b[0m\n   3 │ age       34.2235\n   4 │ sibsp     0.523008\n   5 │ fare      32.2042\n   6 │ embarked \u001b[90m          \u001b[0m\n   7 │ survived \u001b[90m          \u001b[0m",
      "text/html": [
       "<div><div style = \"float: left;\"><span>7×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">variable</th><th style = \"text-align: left;\">mean</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Symbol\" style = \"text-align: left;\">Symbol</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">pclass</td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">sex</td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">age</td><td style = \"text-align: left;\">34.2235</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">sibsp</td><td style = \"text-align: left;\">0.523008</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">fare</td><td style = \"text-align: left;\">32.2042</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">6</td><td style = \"text-align: left;\">embarked</td><td style = \"font-style: italic; text-align: left;\"></td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">7</td><td style = \"text-align: left;\">survived</td><td style = \"font-style: italic; text-align: left;\"></td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 27
    }
   ],
   "cell_type": "code",
   "source": [
    "select(d, [:variable, :mean])"
   ],
   "metadata": {},
   "execution_count": 27
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Materializing as a matrix\n",
    "\n",
    "To convert the content of the dataframe as one big matrix do this:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "3×3 Matrix{Any}:\n \"3\"  \"male\"    4000.0\n \"1\"  \"female\"    38.0\n \"3\"  \"female\"    26.0"
     },
     "metadata": {},
     "execution_count": 28
    }
   ],
   "cell_type": "code",
   "source": [
    "mat = Matrix(df)\n",
    "mat[1:3, 1:3]"
   ],
   "metadata": {},
   "execution_count": 28
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Adding columns"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Adding a column to a dataframe is very easy:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×8 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m pclass \u001b[0m\u001b[1m sex    \u001b[0m\u001b[1m age     \u001b[0m\u001b[1m sibsp   \u001b[0m\u001b[1m fare    \u001b[0m\u001b[1m embarked \u001b[0m\u001b[1m survived \u001b[0m\u001b[1m weird   \u001b[0m ⋯\n     │\u001b[90m Cat…   \u001b[0m\u001b[90m Cat…   \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Cat…?    \u001b[0m\u001b[90m Cat…     \u001b[0m\u001b[90m Float64 \u001b[0m ⋯\n─────┼──────────────────────────────────────────────────────────────────────────\n   1 │ 3       male     4000.0      1.0   7.25    S         0         0.001812 ⋯\n   2 │ 1       female     38.0      1.0  71.2833  C         1         1.87588\n   3 │ 3       female     26.0      0.0   7.925   S         1         0.304808\n\u001b[36m                                                                1 column omitted\u001b[0m",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×8 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">pclass</th><th style = \"text-align: left;\">sex</th><th style = \"text-align: left;\">age</th><th style = \"text-align: left;\">sibsp</th><th style = \"text-align: left;\">fare</th><th style = \"text-align: left;\">embarked</th><th style = \"text-align: left;\">survived</th><th style = \"text-align: left;\">weird</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Union{Missing, CategoricalArrays.CategoricalValue{String, UInt32}}\" style = \"text-align: left;\">Cat…?</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">male</td><td style = \"text-align: right;\">4000.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">7.25</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">0</td><td style = \"text-align: right;\">0.0018125</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">1</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">38.0</td><td style = \"text-align: right;\">1.0</td><td style = \"text-align: right;\">71.2833</td><td style = \"text-align: left;\">C</td><td style = \"text-align: left;\">1</td><td style = \"text-align: right;\">1.87588</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">3</td><td style = \"text-align: left;\">female</td><td style = \"text-align: right;\">26.0</td><td style = \"text-align: right;\">0.0</td><td style = \"text-align: right;\">7.925</td><td style = \"text-align: left;\">S</td><td style = \"text-align: left;\">1</td><td style = \"text-align: right;\">0.304808</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 29
    }
   ],
   "cell_type": "code",
   "source": [
    "df.weird = df.fare ./ df.age;\n",
    "first(df, 3)"
   ],
   "metadata": {},
   "execution_count": 29
  },
  {
   "cell_type": "markdown",
   "source": [
    "That's it! Remember also that you can drop columns or make\n",
    "subselections with `select` and `select!`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The remainder of this tutorial is an abbreviated version of a\n",
    "[DataFrames.jl\n",
    "tutorial](http://juliadata.github.io/DataFrames.jl/latest/man/split_apply_combine/)."
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×5 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m sepallength \u001b[0m\u001b[1m sepalwidth \u001b[0m\u001b[1m petallength \u001b[0m\u001b[1m petalwidth \u001b[0m\u001b[1m class       \u001b[0m\n     │\u001b[90m Float64     \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Float64     \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Cat…        \u001b[0m\n─────┼───────────────────────────────────────────────────────────────\n   1 │         5.1         3.5          1.4         0.2  Iris-setosa\n   2 │         4.9         3.0          1.4         0.2  Iris-setosa\n   3 │         4.7         3.2          1.3         0.2  Iris-setosa",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×5 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">sepallength</th><th style = \"text-align: left;\">sepalwidth</th><th style = \"text-align: left;\">petallength</th><th style = \"text-align: left;\">petalwidth</th><th style = \"text-align: left;\">class</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">5.1</td><td style = \"text-align: right;\">3.5</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">Iris-setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">4.9</td><td style = \"text-align: right;\">3.0</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">Iris-setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">4.7</td><td style = \"text-align: right;\">3.2</td><td style = \"text-align: right;\">1.3</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">Iris-setosa</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 30
    }
   ],
   "cell_type": "code",
   "source": [
    "iris = OpenML.load(61) |> DataFrame;\n",
    "first(iris, 3)"
   ],
   "metadata": {},
   "execution_count": 30
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Split into sub-dataframes  according to values in a column using `groupby`\n",
    "\n",
    "The `groupby` function allows to form \"sub-dataframes\" corresponding\n",
    "to groups of rows.  This can be very convenient to run specific\n",
    "analyses for specific groups without copying the data."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The basic usage is `groupby(df, cols)` where `cols` specifies one or\n",
    "several columns to use for the grouping."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Consider a simple example: in `iris` there is a `Species` column\n",
    "with 3 species:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "3-element Vector{String}:\n \"Iris-setosa\"\n \"Iris-versicolor\"\n \"Iris-virginica\""
     },
     "metadata": {},
     "execution_count": 31
    }
   ],
   "cell_type": "code",
   "source": [
    "unique(iris.class)"
   ],
   "metadata": {},
   "execution_count": 31
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can form views for each of these:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "gdf = groupby(iris, :class);"
   ],
   "metadata": {},
   "execution_count": 32
  },
  {
   "cell_type": "markdown",
   "source": [
    "The `gdf` object now corresponds to **views** of the original\n",
    "dataframe for each of the 3 species; the first species\n",
    "is `\"Iris-setosa\"` with:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "DataFrames.SubDataFrame{DataFrames.DataFrame, DataFrames.Index, Vector{Int64}}"
     },
     "metadata": {},
     "execution_count": 33
    }
   ],
   "cell_type": "code",
   "source": [
    "subdf_setosa = gdf[1];\n",
    "typeof(subdf_setosa)"
   ],
   "metadata": {},
   "execution_count": 33
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m5×4 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m variable    \u001b[0m\u001b[1m min         \u001b[0m\u001b[1m mean   \u001b[0m\u001b[1m max         \u001b[0m\n     │\u001b[90m Symbol      \u001b[0m\u001b[90m Any         \u001b[0m\u001b[90m Union… \u001b[0m\u001b[90m Any         \u001b[0m\n─────┼───────────────────────────────────────────────\n   1 │ sepallength  4.3          5.006   5.8\n   2 │ sepalwidth   2.3          3.418   4.4\n   3 │ petallength  1.0          1.464   1.9\n   4 │ petalwidth   0.1          0.244   0.6\n   5 │ class        Iris-setosa \u001b[90m        \u001b[0m Iris-setosa",
      "text/html": [
       "<div><div style = \"float: left;\"><span>5×4 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">variable</th><th style = \"text-align: left;\">min</th><th style = \"text-align: left;\">mean</th><th style = \"text-align: left;\">max</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Symbol\" style = \"text-align: left;\">Symbol</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Union{Nothing, Float64}\" style = \"text-align: left;\">Union…</th><th title = \"Any\" style = \"text-align: left;\">Any</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">sepallength</td><td style = \"text-align: left;\">4.3</td><td style = \"text-align: left;\">5.006</td><td style = \"text-align: left;\">5.8</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">sepalwidth</td><td style = \"text-align: left;\">2.3</td><td style = \"text-align: left;\">3.418</td><td style = \"text-align: left;\">4.4</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">petallength</td><td style = \"text-align: left;\">1.0</td><td style = \"text-align: left;\">1.464</td><td style = \"text-align: left;\">1.9</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: left;\">petalwidth</td><td style = \"text-align: left;\">0.1</td><td style = \"text-align: left;\">0.244</td><td style = \"text-align: left;\">0.6</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: left;\">class</td><td style = \"text-align: left;\">Iris-setosa</td><td style = \"font-style: italic; text-align: left;\"></td><td style = \"text-align: left;\">Iris-setosa</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 34
    }
   ],
   "cell_type": "code",
   "source": [
    "describe(subdf_setosa, :min, :mean, :max)"
   ],
   "metadata": {},
   "execution_count": 34
  },
  {
   "cell_type": "markdown",
   "source": [
    "Recall that this means modifying `subdf_setosa` also modifies its\n",
    "parent `iris`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Do `?groupby` for more information."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## The `combine` function"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The `combine` function allows to derive a new dataframe out of\n",
    "transformations of an existing one.  Here's an example taken from\n",
    "the official doc (see `?combine`):"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×2 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m a     \u001b[0m\u001b[1m b     \u001b[0m\n     │\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n─────┼──────────────\n   1 │     1      4\n   2 │     2      5\n   3 │     3      6",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">a</th><th style = \"text-align: left;\">b</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Int64\" style = \"text-align: left;\">Int64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">1</td><td style = \"text-align: right;\">4</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">2</td><td style = \"text-align: right;\">5</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">3</td><td style = \"text-align: right;\">6</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 35
    }
   ],
   "cell_type": "code",
   "source": [
    "df3 = DataFrame(a=1:3, b=4:6)"
   ],
   "metadata": {},
   "execution_count": 35
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m1×2 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m a_sum \u001b[0m\u001b[1m nrow  \u001b[0m\n     │\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n─────┼──────────────\n   1 │     6      3",
      "text/html": [
       "<div><div style = \"float: left;\"><span>1×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">a_sum</th><th style = \"text-align: left;\">nrow</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Int64\" style = \"text-align: left;\">Int64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">6</td><td style = \"text-align: right;\">3</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 36
    }
   ],
   "cell_type": "code",
   "source": [
    "combine(df3, :a => sum, nrow)"
   ],
   "metadata": {},
   "execution_count": 36
  },
  {
   "cell_type": "markdown",
   "source": [
    "What happened here is that the derived DataFrame has two columns\n",
    "obtained respectively by (1) computing the sum of the first column\n",
    "and (2) applying the `nrow` function on the `df`.\n",
    "\n",
    "The transformation can produce one or several values, and `combine` will\n",
    "try to accomodate this with appropriate packing:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m2×2 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m a_maximum \u001b[0m\u001b[1m b_goo \u001b[0m\n     │\u001b[90m Int64     \u001b[0m\u001b[90m Int64 \u001b[0m\n─────┼──────────────────\n   1 │         3      4\n   2 │         3      5",
      "text/html": [
       "<div><div style = \"float: left;\"><span>2×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">a_maximum</th><th style = \"text-align: left;\">b_goo</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Int64\" style = \"text-align: left;\">Int64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">3</td><td style = \"text-align: right;\">4</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">3</td><td style = \"text-align: right;\">5</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 37
    }
   ],
   "cell_type": "code",
   "source": [
    "goo(v) = v[1:2]\n",
    "combine(df3, :a => maximum, :b => goo)"
   ],
   "metadata": {},
   "execution_count": 37
  },
  {
   "cell_type": "markdown",
   "source": [
    "Here the maximum value of `a` is copied twice so that the two\n",
    "columns have the same number of rows."
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m2×2 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m a_goo \u001b[0m\u001b[1m b_bar \u001b[0m\n     │\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n─────┼──────────────\n   1 │     1      5\n   2 │     2      6",
      "text/html": [
       "<div><div style = \"float: left;\"><span>2×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">a_goo</th><th style = \"text-align: left;\">b_bar</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Int64\" style = \"text-align: left;\">Int64</th><th title = \"Int64\" style = \"text-align: left;\">Int64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">1</td><td style = \"text-align: right;\">5</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">2</td><td style = \"text-align: right;\">6</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 38
    }
   ],
   "cell_type": "code",
   "source": [
    "bar(v) = v[end-1:end]\n",
    "combine(df3, :a => goo, :b => bar)"
   ],
   "metadata": {},
   "execution_count": 38
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Using `combine` with `groupby`\n",
    "\n",
    "Combining `groupby` with `combine` is very useful.  For instance you\n",
    "might want to compute statistics across groups for different\n",
    "variables:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×2 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m class           \u001b[0m\u001b[1m petallength_mean \u001b[0m\n     │\u001b[90m Cat…            \u001b[0m\u001b[90m Float64          \u001b[0m\n─────┼───────────────────────────────────\n   1 │ Iris-setosa                 1.464\n   2 │ Iris-versicolor             4.26\n   3 │ Iris-virginica              5.552",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">class</th><th style = \"text-align: left;\">petallength_mean</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">Iris-setosa</td><td style = \"text-align: right;\">1.464</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">Iris-versicolor</td><td style = \"text-align: right;\">4.26</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">Iris-virginica</td><td style = \"text-align: right;\">5.552</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 39
    }
   ],
   "cell_type": "code",
   "source": [
    "combine(groupby(iris, :class), :petallength => mean)"
   ],
   "metadata": {},
   "execution_count": 39
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's break this operatioin down:\n",
    "\n",
    "1. The `groupby(iris, :class)` creates groups using the `:class` column (which has values `Iris-setosa`, `Iris-versicolor`, `Iris-virginica`)\n",
    "2. The `combine` creates a derived dataframe by applying the `mean` function to the `:petallength` column\n",
    "3. Since there are three groups, we get one column (mean of `petallength`) and three rows (one per group).\n",
    "\n",
    "\n",
    "You can do this for several columns/statistics at the time and give\n",
    "new column names to the results:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×3 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m class           \u001b[0m\u001b[1m MPL     \u001b[0m\u001b[1m SPL      \u001b[0m\n     │\u001b[90m Cat…            \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64  \u001b[0m\n─────┼────────────────────────────────────\n   1 │ Iris-setosa        1.464  0.173511\n   2 │ Iris-versicolor    4.26   0.469911\n   3 │ Iris-virginica     5.552  0.551895",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×3 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">class</th><th style = \"text-align: left;\">MPL</th><th style = \"text-align: left;\">SPL</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">Iris-setosa</td><td style = \"text-align: right;\">1.464</td><td style = \"text-align: right;\">0.173511</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">Iris-versicolor</td><td style = \"text-align: right;\">4.26</td><td style = \"text-align: right;\">0.469911</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">Iris-virginica</td><td style = \"text-align: right;\">5.552</td><td style = \"text-align: right;\">0.551895</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 40
    }
   ],
   "cell_type": "code",
   "source": [
    "gdf3 = groupby(iris, :class)\n",
    "combine(gdf3, :petallength => mean => :MPL, :petallength => std => :SPL)"
   ],
   "metadata": {},
   "execution_count": 40
  },
  {
   "cell_type": "markdown",
   "source": [
    "So here we assign the names `:MPL` and `:SPL` to the derived\n",
    "columns.  If you want to apply something on all columns apart from\n",
    "the grouping one, using `names` and `Not` comes in handy:"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "\u001b[1m3×5 DataFrame\u001b[0m\n\u001b[1m Row \u001b[0m│\u001b[1m class           \u001b[0m\u001b[1m sepallength_std \u001b[0m\u001b[1m sepalwidth_std \u001b[0m\u001b[1m petallength_std \u001b[0m\u001b[1m peta\u001b[0m ⋯\n     │\u001b[90m Cat…            \u001b[0m\u001b[90m Float64         \u001b[0m\u001b[90m Float64        \u001b[0m\u001b[90m Float64         \u001b[0m\u001b[90m Floa\u001b[0m ⋯\n─────┼──────────────────────────────────────────────────────────────────────────\n   1 │ Iris-setosa             0.35249         0.381024         0.173511       ⋯\n   2 │ Iris-versicolor         0.516171        0.313798         0.469911\n   3 │ Iris-virginica          0.63588         0.322497         0.551895\n\u001b[36m                                                                1 column omitted\u001b[0m",
      "text/html": [
       "<div><div style = \"float: left;\"><span>3×5 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">class</th><th style = \"text-align: left;\">sepallength_std</th><th style = \"text-align: left;\">sepalwidth_std</th><th style = \"text-align: left;\">petallength_std</th><th style = \"text-align: left;\">petalwidth_std</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"CategoricalArrays.CategoricalValue{String, UInt32}\" style = \"text-align: left;\">Cat…</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: left;\">Iris-setosa</td><td style = \"text-align: right;\">0.35249</td><td style = \"text-align: right;\">0.381024</td><td style = \"text-align: right;\">0.173511</td><td style = \"text-align: right;\">0.10721</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: left;\">Iris-versicolor</td><td style = \"text-align: right;\">0.516171</td><td style = \"text-align: right;\">0.313798</td><td style = \"text-align: right;\">0.469911</td><td style = \"text-align: right;\">0.197753</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: left;\">Iris-virginica</td><td style = \"text-align: right;\">0.63588</td><td style = \"text-align: right;\">0.322497</td><td style = \"text-align: right;\">0.551895</td><td style = \"text-align: right;\">0.27465</td></tr></tbody></table></div>"
      ]
     },
     "metadata": {},
     "execution_count": 41
    }
   ],
   "cell_type": "code",
   "source": [
    "combine(gdf3, names(iris, Not(:class)) .=> std)"
   ],
   "metadata": {},
   "execution_count": 41
  },
  {
   "cell_type": "markdown",
   "source": [
    "where"
   ],
   "metadata": {}
  },
  {
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "4-element Vector{String}:\n \"sepallength\"\n \"sepalwidth\"\n \"petallength\"\n \"petalwidth\""
     },
     "metadata": {},
     "execution_count": 42
    }
   ],
   "cell_type": "code",
   "source": [
    "names(iris, Not(:class))"
   ],
   "metadata": {},
   "execution_count": 42
  },
  {
   "cell_type": "markdown",
   "source": [
    "and note the use of `.` in `.=>` to indicate that we broadcast the\n",
    "function over each column."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "---\n",
    "\n",
    "*This notebook was generated using [Literate.jl](https://github.com/fredrikekre/Literate.jl).*"
   ],
   "metadata": {}
  }
 ],
 "nbformat_minor": 3,
 "metadata": {
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.10.3"
  },
  "kernelspec": {
   "name": "julia-1.10",
   "display_name": "Julia 1.10.3",
   "language": "julia"
  }
 },
 "nbformat": 4
}