02-starting-with-data.Rmd

---
title: "Starting with data"
author: "Data Carpentry contributors"
minutes: 25
layout: topic
---


------------

> ## Learning Objectives
>
> * Load external tabular data from a .csv file into R.
> * Describe what an R data frame is.
> * Summarize the contents of a data frame in R.
> * Manipulate categorical data in R using factors.


------------

# Looking at Metadata

```{r, echo=FALSE, purl=TRUE}
# Looking at metadata
```

We are studying a population of Escherichia coli (designated Ara-3), which were propagated for more than 40,000 generations in a glucose-limited minimal medium. This medium was supplemented with citrate which E. coli cannot metabolize in the aerobic conditions of the experiment. Sequencing of the populations at regular time points reveals that spontaneous citrate-using mutants (Cit+) appeared at around 31,000 generations. This metadata describes information on the Ara-3 clones and the columns represent:

| Column           | Description                                |
|------------------|--------------------------------------------|
| sample           | clone name					|
| generation       | generation when sample frozen		|
| clade            | based on parsimony-based tree		|
| strain           | ancestral strain				|
| cit              | citrate-using mutant status		|
| run              | Sequence read archive sample ID		|
| genome_size      | size in Mbp (made up data for this lesson) |





The metadata file required for this lesson can be [downloaded directly here](https://raw.githubusercontent.com/datacarpentry/R-genomics/gh-pages/data/Ecoli_metadata.csv) or [viewed in Github](./data/Ecoli_metadata.csv).

> Tip:
> If you can't find the Ecoli_metadata.csv file, or have lost track of it,
> download the file directly using the R `download.file() function`

```{r, eval=TRUE,  purl=FALSE}
download.file("https://raw.githubusercontent.com/datacarpentry/R-genomics/gh-pages/data/Ecoli_metadata.csv", "data/Ecoli_metadata.csv")
```

You are now ready to load the data. We are going to use the R function  `read.csv()` to load the data file into memory (as a `data.frame`):

```{r, eval=TRUE,  purl=FALSE}
metadata <- read.csv('data/Ecoli_metadata.csv')
```

This statement doesn't produce any output because assignment doesn't display
anything. If we want to check that our data has been loaded, we can print the
variable's value: `metadata`

Alternatively, wrapping an assignment in parentheses will perform the assignment
and display it at the same time.

```{r, eval = TRUE, purl = FALSE}
(metadata <- read.csv('data/Ecoli_metadata.csv'))
```

Wow... that was a lot of output. At least it means the data loaded properly. Let's check the top (the first 6 lines) of this `data.frame` using the function `head()`:

```{r, results='show', purl=FALSE}
head(metadata)
```
> ### Note
>
> `read.csv` assumes that fields are delineated by commas, however, in several
> countries, the comma is used as a decimal separator and the semicolon (;) is
> used as a field delineator. If you want to read in this type of files in R,
> you can use the `read.csv2` function. It behaves exactly like `read.csv` but
> uses different parameters for the decimal and the field separators. If you are
> working with another format, they can be both specified by the user. Check out
> the help for `read.csv()` to learn more.

We've just done two very useful things.
1. We've read our data in to R, so now we can work with it in R
2. We've created a data frame (with the read.csv command) the
standard way R works with data.

# What are data frames?

`data.frame` is the _de facto_ data structure for most tabular data and what we
use for statistics and plotting.

A `data.frame` is a collection of vectors of identical lengths. Each vector
represents a column, and each vector can be of a different data type (e.g.,
characters, integers, factors). The `str()` function is useful to inspect the
data types of the columns.

A `data.frame` can be created by the functions `read.csv()` or `read.table()`, in
other words, when importing spreadsheets from your hard drive (or the web).

By default, `data.frame` converts (= coerces) columns that contain characters
(i.e., text) into the `factor` data type. Depending on what you want to do with
the data, you may want to keep these columns as `character`. To do so,
`read.csv()` and `read.table()` have an argument called `stringsAsFactors` which
can be set to `FALSE`:

Let's now check the __str__ucture of this `data.frame` in more details with the
function `str()`:

```{r, purl=FALSE}
str(metadata)
```

# Inspecting `data.frame` objects

We already saw how the functions `head()` and `str()` can be useful to check the
content and the structure of a `data.frame`. Here is a non-exhaustive list of
functions to get a sense of the content/structure of the data.

* Size:
    + `dim()` - returns a vector with the number of rows in the first element, and
    the number of columns as the second element (the __dim__ensions of the object)
    + `nrow()` - returns the number of rows
    + `ncol()` - returns the number of columns
* Content:
    + `head()` - shows the first 6 rows
    + `tail()` - shows the last 6 rows
* Names:
    + `names()` - returns the column names (synonym of `colnames()` for `data.frame`
	objects)
    + `rownames()` - returns the row names
* Summary:
    + `str()` - structure of the object and information about the class, length and
	content of  each column
    + `summary()` - summary statistics for each column

Note: most of these functions are "generic", they can be used on other types of
objects besides `data.frame`.

> ### Challenge
>
> Based on the output of `str(metadata)`, can you answer the following questions?
>
> * What is the class of the object `metadata`?
> * How many rows and how many columns are in this object?
> * How many citrate+ mutants have been recorded in this population?
>

<!--
> ```{r, answer=TRUE, results="markup", purl=FALSE}
>
> str(metadata)
>
> ## * class: data frame
> ## * how many rows: 30,  how many columns: 7
> ## * how many citrate+ mutants: 9
> ```
--->


As you can see, many of the columns in our data frame are of a special class called
`factor`. Before we learn more about the `data.frame` class, we are going to
talk about factors. They are very useful but not necessarily intuitive, and
therefore require some attention.


# Factors
```{r, echo=FALSE, purl=TRUE}
### Factors
```

When we did `str(metadata)` we saw that several of the columns consist of
integers, however, the columns `clade`, `strain`, `cit`, `run`, ... are
of a special class called a `factor`. Factors are very useful and are actually
something that make R particularly well suited to working with data, so we're
going to spend a little time introducing them.

Factors are used to represent categorical data. Factors can be ordered or
unordered and are an important class for statistical analysis and for plotting.

Factors are stored as integers, and have labels associated with these unique
integers. While factors look (and often behave) like character vectors, they are
actually integers under the hood, and you need to be careful when treating them
like strings.

In the data frame we just imported, let's do
```{r, purl=TRUE}
str(metadata)
```

We can see the names of the multiple columns. And, we see that
some say things like `Factor w/ 3 levels`

You can learn what these level are by using the function `levels()`, 
```{r, results = TRUE, purl=FALSE}
levels(metadata$cit)
```
and check the number of levels using `nlevels()`:

```{r, results = TRUE, purl=FALSE}
nlevels(metadata$cit)
```

When we read in a file, any column that contains text is automatically
assumed to be a factor. Once created, factors can only contain a pre-defined set values, known as
*levels*. By default, R always sorts *levels* in alphabetical order.

For instance, we see that `cit` is a Factor w/ 3 levels, `minus`, `plus` and `unknown`.

Sometimes, the order of the factors does not matter, other times you might want
to specify the order because it is meaningful (e.g., "low", "medium", "high"),
it improves your visualization, or it is required by a particular type of
analysis. Here, one way to reorder our levels in the `cit` vector would be:

```{r, results=TRUE, purl=FALSE}
cit <-metadata$cit
cit # current order
cit <- factor(cit, levels = c("plus", "minus", "unknown"))
cit # after re-ordering
```

>### Challenge
>
>The function `table()` tabulates observations and can be used to create bar plots quickly. For instance:
>
>* Question: How can you recreate this plot but by having "control" being listed last instead of first?
>```{r, purl=TRUE}
> exprmt <- factor(c("treat1", "treat2", "treat1", "treat3", "treat1", "control",
                   "treat1", "treat2", "treat3"))
> table(exprmt)
> barplot(table(exprmt))
>```

<!--
> ```{r, answer=TRUE, results="markup", purl=TRUE}
>
>
> exprmt <- factor(exprmt, levels = c("treat1", "treat2", "treat3", "control"))
>
> barplot(table(exprmt))
> ```
--->

In R's memory, these factors are represented by integers (1, 2, 3), but are more
informative than integers because factors that are self describing: `"plus"`,
`"minus"` is more descriptive than `1`, `2`. Which one is "plus"?  You wouldn't
be able to tell just from the integer data. Factors, on the other hand, have
this information built in. It is particularly helpful when there are many levels
(like the strains in our example dataset).

### Converting factors

If you need to convert a factor to a character vector, you use
`as.character(x)`.

```{r, results=TRUE,purl=FALSE}
as.character(cit)
```
Converting factors where the levels appear as numbers (such as concentration
levels, generations or years) to a numeric vector is a little trickier. 

Lets simulate an error in importing the dataset where generation was misidentified as a factor rather than a integer
```{r, results =TRUE, purl=TRUE}
generation <- factor(metadata$generation) 
```

The `as.numeric()` function returns the index values of the factor, not its levels, so it will 
result in an entirely new (and unwanted in this case) set of numbers. 
One method to avoid this is to convert factors to characters and then numbers.  
Another method is to use the `levels()` function. Compare:

```{r, results =TRUE, purl=TRUE}
as.numeric(generation)               # Wrong! And there is no warning...
as.numeric(as.character(generation)) # Works...
as.numeric(levels(generation))[generation]    # The recommended way.
```

Notice that in the `levels()` approach, three important steps occur:

* We obtain all the factor levels using `levels(generation)`
* We convert these levels to numeric values using `as.numeric(levels(generation))`
* We then access these numeric values using the underlying integers of the
  vector `generation` inside the square brackets

The automatic conversion of data type is sometimes a blessing, sometimes an
annoyance. Be aware that it exists, learn the rules, and double check that data
you import in R are of the correct type within your data frame. If not, use it
to your advantage to detect mistakes that might have been introduced during data
entry (a letter in a column that should only contain numbers for instance).


### Renaming factors

When your data is stored as a factor, you can use the `plot()` function to get a
quick glance at the number of observations represented by each factor
level. Let's look at the number of citrate-using mutants (Cit+) over the course of
the experiment:

```{r, purl=TRUE}
## bar plot of the number of clade in the samples:
plot(metadata$cit)
```

In addition to minus and plus, there are about 12 samples for which the
cit information hasn't been recorded. Additionally, for these individuals,
there is no label to indicate that the information is missing. Let's rename this
label to something more meaningful. Before doing that, we're going to pull out
the data on cit mutant status and work with that data, so we're not modifying the working copy
of the data frame

```{r, results=TRUE, purl=FALSE}
cit <- metadata$cit
head(cit)
levels(cit)
head(cit)
```

> ### Challenge
>
> * Rename "minus" and "plus" to "negative" and "postive" respectively.
>

<!--
### Challenge

The function `table()` tabulates observations and can be used to create
bar plots quickly. For instance:

```{r wrong-order, results='show', purl=TRUE}
## Question: How can you recreate this plot but by having "control"
## being listed last instead of first?
exprmt <- factor(c("treat1", "treat2", "treat1", "treat3", "treat1", "control",
                   "treat1", "treat2", "treat3"))
table(exprmt)
barplot(table(exprmt))
```


```{r correct-order, purl=FALSE}
exprmt <- factor(exprmt, levels=c("treat1", "treat2", "treat3", "control"))
barplot(table(exprmt))
```
--->