Today

  • News items:

    • Two formative assessments
      • Light data analysis and interpretation
      • Due Friday 11:59 PM the following week
    • Reminder: Office hours

  • Topics for today:

    • Pipes
    • Subsetting with {dplyr}
    • Creating and modifying variables
    • Distributions
      • Tabulations and Cross-Tabulations
      • Summarizing data
      • Correlations

{tidyverse}

 

Installing {tidyverse}

 

We’re going to practice loading files and manipulating data.

We will use a packages called {readr} and {dplyr} to do this neatly.

These packages are part of the Tidyverse family of R packages

  • These packages make using R much easier

If you have not already installed the tidyverse, type, in the console: install.packages("tidyverse")

 

This will install a large number of R packages we will use throughout the term, including {readr}, {ggplot2}, and {dplyr}.

Loading packages

 

library(readr)
library(ggplot2)
library(dplyr)

Attaching package: 'dplyr'
The following objects are masked from 'package:stats':

    filter, lag
The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

Wait, was that an error?

 

When you load packages in R that have functions sharing the same name as functions you already have, the more recently loaded functions overwrite the previous ones (“masks them”).

This message is just letting you know that.

 

Sometimes you may get a warning message when loading packages—usually because you aren’t running the latest version of R:

Warning message:
package `dplyr' was built under R version 4.6.0

Update R to get rid of these!

Importing and exporting data

 

Delimited text files

Data are commonly stored in a delimited text files, e.g. comma-separated values (.csv) or tab-separated values (.tsv). Here is .csv data:

"Id","Offense","Sex","Month"
101,"Battery","Male",1,
101,"Battery","Male",1,
101,"Robbery","Male",1,
101,"Battery","Male",2,
101,"Robbery","Male",2,
101,"Homicide","Male",3,
103,"Robbery","Female",1,
103,"Robbery","Female",3,
103,"Battery","Female",4,

Values are comma-separated and observations are separated by line breaks

{readr}

R has a variety of built-in functions for importing delimited text, like read.table() and read.csv().

 

I recommend using the versions in the {readr} package instead: read_csv(), read_tsv(), and read_delim().

 

{readr} function features:

  • Faster—with loading bars for large files
  • Smarter about dates and times
  • Handy function problems() you can run if there are errors

{readr} importing example

 

Let’s use read_csv() from {readr} to import some community crime data based on those in yesterday’s CRM lecture

communities <- 
  read_csv(
    "https://clanfear.github.io/ioc_iqa/_data/communities.csv"
    )
Rows: 300 Columns: 5
── Column specification ────────────────────────────────────────────
Delimiter: ","
chr (3): area, disadvantage, incarceration
dbl (2): pop_density, crime_rate

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

{dplyr}

 

A glimpse() of communities

 

{dplyr} gives us access to the handy glimpse() for inspecting dataframes.

glimpse(communities)
Rows: 300
Columns: 5
$ area          <chr> "Urban", "Urban", "Rural", "Rural", "Urban",…
$ pop_density   <dbl> 18.166008, 21.356727, 10.023975, 14.926138, …
$ crime_rate    <dbl> 28.63123, 58.60800, 14.19840, 22.29075, 79.7…
$ disadvantage  <chr> "Low", "Medium", "Low", "Medium", "High", "H…
$ incarceration <chr> "High", "Medium", "High", "Medium", "High", …

 

This is like a slightly fancier str()

Pipes!

 

{dplyr} and rest of the Tidyverse are built around using pipe operators (|>)

Instead of nesting functions like this:

proportions(table(communities$disadvantage))

     High       Low    Medium 
0.3366667 0.3400000 0.3233333

We can pipe them like this:

communities |> pull(disadvantage) |> table() |> proportions()

     High       Low    Medium 
0.3366667 0.3400000 0.3233333

Read this as, “take communities, and then pull out the incarceration column, and then make a table(), and then calculate proportions().”

filter() data frames

 

communities |> filter(incarceration == "High") |> head()
# A tibble: 6 × 5
  area  pop_density crime_rate disadvantage incarceration
  <chr>       <dbl>      <dbl> <chr>        <chr>        
1 Urban       18.2       28.6  Low          High         
2 Rural       10.0       14.2  Low          High         
3 Urban       21.9       79.8  High         High         
4 Urban       18.5       31.1  High         High         
5 Urban       22.5       28.6  Low          High         
6 Rural        8.95       6.53 Medium       High

 

What is this doing?

 

filter() is a {dplyr} function for indexing dataframe rows

It takes only logical vectors (the result of expressions) as an argument

Multiple conditions

 

And: &

communities |>
  filter(disadvantage == "Low" & 
         crime_rate > 40)

Or: |

communities |>
  filter(disadvantage == "Low" | 
         crime_rate > 40)

%in% operator

Common use case: Filter rows to things in some set.

We can use %in% like == but for matching any element in a vector

communities |>
  filter(disadvantage %in% c("High", "Low")) |>
  tail()
# A tibble: 6 × 5
  area  pop_density crime_rate disadvantage incarceration
  <chr>       <dbl>      <dbl> <chr>        <chr>        
1 Urban       13.2       10.2  Low          Medium       
2 Urban       18.5       43.9  Low          Medium       
3 Rural       11.0       10.2  Low          Medium       
4 Rural        6.59       2.27 Low          Medium       
5 Urban       16.3       29.5  High         High         
6 Rural       10.7       12.0  Low          Medium

Read as: “filter() to rows where disadvantage is "High" or "Low"”

Sorting: arrange()

 

Along with filtering the data to see certain rows, we might want to sort it:

communities |>
  arrange(disadvantage, desc(crime_rate)) |> 
  head()
# A tibble: 6 × 5
  area  pop_density crime_rate disadvantage incarceration
  <chr>       <dbl>      <dbl> <chr>        <chr>        
1 Urban        27.9       98.2 High         High         
2 Urban        22.4       98.1 High         High         
3 Urban        22.9       88.7 High         High         
4 Urban        21.1       85.6 High         High         
5 Urban        21.9       79.8 High         High         
6 Urban        22.9       77.7 High         High

The data are sorted by ascending disadvantage and descending crime_rate.

Keeping columns: select()

 

Not only can we subset rows, but we can include specific columns (and put them in the order listed) using select()

communities |> select(area, pop_density, crime_rate) |> head()
# A tibble: 6 × 3
  area  pop_density crime_rate
  <chr>       <dbl>      <dbl>
1 Urban        18.2       28.6
2 Urban        21.4       58.6
3 Rural        10.0       14.2
4 Rural        14.9       22.3
5 Urban        21.9       79.8
6 Urban        18.5       31.1

Dropping columns: select()

 

We can instead drop only specific columns with select() using - signs:

communities |> select(-area, -pop_density, -crime_rate) |> head()
# A tibble: 6 × 2
  disadvantage incarceration
  <chr>        <chr>        
1 Low          High         
2 Medium       Medium       
3 Low          High         
4 Medium       Medium       
5 High         High         
6 High         High

Renaming with select()

 

We can rename columns using select(), but that drops everything that isn’t mentioned:

communities |> select(Area = area) |> head()
# A tibble: 6 × 1
  Area 
  <chr>
1 Urban
2 Urban
3 Rural
4 Rural
5 Urban
6 Urban

Safer: rename with rename()

 

rename() renames variables using the same syntax as select() without dropping unmentioned variables

communities |> rename(Area = area) |> head()
# A tibble: 6 × 5
  Area  pop_density crime_rate disadvantage incarceration
  <chr>       <dbl>      <dbl> <chr>        <chr>        
1 Urban        18.2       28.6 Low          High         
2 Urban        21.4       58.6 Medium       Medium       
3 Rural        10.0       14.2 Low          High         
4 Rural        14.9       22.3 Medium       Medium       
5 Urban        21.9       79.8 High         High         
6 Urban        18.5       31.1 High         High

Creating columns

 

disadvantage looks like an ordinal variable (incarceration too) but R doesn’t know this—it just puts them in alphabetical order

communities |> pull(disadvantage) |> table()

  High    Low Medium 
   101    102     97

To fix this, we need to know how to create or modify variables in our data

Creating columns

 

dplyr uses the mutate() function to create or modify variables:

communities |>
  mutate(high_crime = crime_rate > mean(crime_rate)) |>
  head()
# A tibble: 6 × 6
  area  pop_density crime_rate disadvantage incarceration high_crime
  <chr>       <dbl>      <dbl> <chr>        <chr>         <lgl>     
1 Urban        18.2       28.6 Low          High          TRUE      
2 Urban        21.4       58.6 Medium       Medium        TRUE      
3 Rural        10.0       14.2 Low          High          FALSE     
4 Rural        14.9       22.3 Medium       Medium        FALSE     
5 Urban        21.9       79.8 High         High          TRUE      
6 Urban        18.5       31.1 High         High          TRUE

This created a logical (TRUE/FALSE) variable because we used a logical expression

Modifying columns

In R, we “modify” objects—including columns in our data—by replacing them with new versions

We saw before that our disadvantage and incarceration variables are ordinal but not being recognized that way

We can give them an order by making them factors and specifying their levels

communities <- communities |>  # Assigning back to overwrite!
  mutate(disadvantage = 
           factor(disadvantage, levels = c("Low", "Medium", "High")),
         incarceration = 
           factor(incarceration, levels = c("Low", "Medium", "High"))
         )

To modify the original dataset, we just assign back to it—overwriting it with our changes!

Fixed!

 

communities |> pull(disadvantage) |> table()

   Low Medium   High 
   102     97    101 
communities |> pull(incarceration) |> table()

   Low Medium   High 
    95    106     99

Distributions

 

Numbers today (boo!)

 

Pictures next week (fun!)

Tabulations

Let’s look at tabulations first. They’re useful for summarizing categorical data.

count() is a {dplyr} function for tabulating one or more columns

communities |> count(incarceration)
# A tibble: 3 × 2
  incarceration     n
  <fct>         <int>
1 Low              95
2 Medium          106
3 High             99

I creates a new column, n with the count of each level

communities |> count(incarceration) |> mutate(proportion = n/sum(n))
# A tibble: 3 × 3
  incarceration     n proportion
  <fct>         <int>      <dbl>
1 Low              95      0.317
2 Medium          106      0.353
3 High             99      0.33

{janitor}

{janitor} is a data cleaning package, but it makes tabulations easier too

install.packages("janitor")
library(janitor)

Attaching package: 'janitor'
The following objects are masked from 'package:stats':

    chisq.test, fisher.test

 

communities |> tabyl(incarceration)
 incarceration   n   percent
           Low  95 0.3166667
        Medium 106 0.3533333
          High  99 0.3300000

 

It comes in really handy for cross-tabs—we’ll see them soon!

summarize()

{dplyr}’s summarize() takes your column(s) of data and computes something using every row:

  • Calculate the mean (mean())
  • Calculate the standard deviation (sd())
  • Obtain a sample size (n())
communities |>
  summarize(mean_crime_rate = mean(crime_rate),
            sd_crime_rate   = sd(crime_rate), 
            n               = n())
# A tibble: 1 × 3
  mean_crime_rate sd_crime_rate     n
            <dbl>         <dbl> <int>
1            25.2          21.8   300

You can use any function in summarize() that aggregates multiple values into a single value (like sum(), median(), or max()).

Associations

 

Which are really about joint distributions

Cross-tabs

Let’s look at cross-tabs first. They’re used for associations between categorical variables.

This is where {janitor}’s tabyl() begins to shine:

communities |> 
  tabyl(disadvantage, incarceration)
 disadvantage Low Medium High
          Low  40     45   17
       Medium  31     30   36
         High  24     31   46

 

communities |> 
  tabyl(disadvantage, incarceration) |>
  adorn_percentages() # converts to cell percentages 
 disadvantage       Low    Medium      High
          Low 0.3921569 0.4411765 0.1666667
       Medium 0.3195876 0.3092784 0.3711340
         High 0.2376238 0.3069307 0.4554455

Fancy cross-tabs

We can assemble fancy tables bit-by-bit with {janitor}

communities |> 
  tabyl(disadvantage, incarceration) # make table
 disadvantage Low Medium High
          Low  40     45   17
       Medium  31     30   36
         High  24     31   46

Fancy cross-tabs

Add row and column totals!

communities |> 
  tabyl(disadvantage, incarceration) |> # make table
  adorn_totals(c("row", "col")) # add row/col totals
 disadvantage Low Medium High Total
          Low  40     45   17   102
       Medium  31     30   36    97
         High  24     31   46   101
        Total  95    106   99   300

Fancy cross-tabs

Turn cells into (row) percentages instead of counts

communities |> 
  tabyl(disadvantage, incarceration) |> # make table
  adorn_totals(c("row", "col")) |> # add row/col totals
  adorn_percentages()# make cells proportions
 disadvantage       Low    Medium      High Total
          Low 0.3921569 0.4411765 0.1666667     1
       Medium 0.3195876 0.3092784 0.3711340     1
         High 0.2376238 0.3069307 0.4554455     1
        Total 0.3166667 0.3533333 0.3300000     1

Fancy cross-tabs

Round those percentages to two decimal places!

communities |> 
  tabyl(disadvantage, incarceration) |> # make table
  adorn_totals(c("row", "col")) |> # add row/col totals
  adorn_percentages() |> # make cells percentages
  adorn_pct_formatting(digits = 1) # percents with 1 digit
 disadvantage   Low Medium  High  Total
          Low 39.2%  44.1% 16.7% 100.0%
       Medium 32.0%  30.9% 37.1% 100.0%
         High 23.8%  30.7% 45.5% 100.0%
        Total 31.7%  35.3% 33.0% 100.0%

Fancy cross-tabs

Add counts back in parentheses!

communities |> 
  tabyl(disadvantage, incarceration) |> # make table
  adorn_totals(c("row", "col")) |> # add row/col totals
  adorn_percentages() |> # make cells percentages
  adorn_pct_formatting(digits = 1) |> # percents with 1 digit
  adorn_ns() # add counts in parentheses
 disadvantage        Low      Medium       High        Total
          Low 39.2% (40) 44.1%  (45) 16.7% (17) 100.0% (102)
       Medium 32.0% (31) 30.9%  (30) 37.1% (36) 100.0%  (97)
         High 23.8% (24) 30.7%  (31) 45.5% (46) 100.0% (101)
        Total 31.7% (95) 35.3% (106) 33.0% (99) 100.0% (300)

Fancy cross-tabs

Add column variable name!

communities |> 
  tabyl(disadvantage, incarceration) |> # make table
  adorn_totals(c("row", "col")) |> # add row/col totals
  adorn_percentages() |> # make cells percentages
  adorn_pct_formatting(digits = 1) |> # percents with 1 digit
  adorn_ns() |> # add counts in parentheses
  adorn_title() # add col variable name
              incarceration                                    
 disadvantage           Low      Medium       High        Total
          Low    39.2% (40) 44.1%  (45) 16.7% (17) 100.0% (102)
       Medium    32.0% (31) 30.9%  (30) 37.1% (36) 100.0%  (97)
         High    23.8% (24) 30.7%  (31) 45.5% (46) 100.0% (101)
        Total    31.7% (95) 35.3% (106) 33.0% (99) 100.0% (300)

Not bad! That’s not far from paper ready!

Grouped measures

If we wanted to calculated means for different groups, one easy way is to just subset the data

communities |>
  filter(disadvantage=="High") |> 
  summarize(
   mean_crime = mean(crime_rate),
   sd_crime   = sd(crime_rate), 
   n          = n())
# A tibble: 1 × 3
  mean_crime sd_crime     n
       <dbl>    <dbl> <int>
1       31.9     24.4   101
communities |>
  filter(disadvantage=="Low") |> 
  summarize(
   mean_crime = mean(crime_rate),
   sd_crime   = sd(crime_rate), 
   n          = n())
# A tibble: 1 × 3
  mean_crime sd_crime     n
       <dbl>    <dbl> <int>
1       18.7     17.4   102

Imagine if you had many groups, though! There’s a better way.

group_by()

The special function group_by() changes how functions operate on the data, most importantly summarize().

Functions after group_by() are computed within each group as defined by variables given, rather than over all rows at once.

Excel analogue: pivot tables

group_by() example

 

communities |>
  group_by(disadvantage) |> 
  summarize(mean_crime = mean(crime_rate),
            sd_crime   = sd(crime_rate), 
            n          = n())
# A tibble: 3 × 4
  disadvantage mean_crime sd_crime     n
  <fct>             <dbl>    <dbl> <int>
1 Low                18.7     17.4   102
2 Medium             25.2     21.2    97
3 High               31.9     24.4   101

 

Because we did group_by() with disadvantage then used summarize(), we get one row per value of disadvantage!

Each value of disadvantage is its own group!

.by = example

 

You can also do grouping within summarize() or mutate() using .by =

communities |>
  summarize(mean_crime = mean(crime_rate),
            sd_crime   = sd(crime_rate), 
            n          = n(),
            .by = disadvantage) 
# A tibble: 3 × 4
  disadvantage mean_crime sd_crime     n
  <fct>             <dbl>    <dbl> <int>
1 Low                18.7     17.4   102
2 Medium             25.2     21.2    97
3 High               31.9     24.4   101

 

I recommend using this method when possible as it automatically ungroups afterward

Correlations

Correlations are mathematically complicated but simple to code

cor(communities$pop_density, communities$crime_rate)
[1] 0.8064513

 

Alternatively, you can use the with() command for a bit less typing

with(communities, cor(pop_density, crime_rate))
[1] 0.8064513

 

These are a bit easier than dplyr if you just want one correlation

communities |> summarize(R = cor(pop_density, crime_rate))
# A tibble: 1 × 1
      R
  <dbl>
1 0.806

Multiple correlations

 

If you want to do correlations within groups, {dplyr} is king again

communities |>
  summarize(R = cor(pop_density, crime_rate), .by = disadvantage)
# A tibble: 3 × 2
  disadvantage     R
  <fct>        <dbl>
1 Low          0.743
2 Medium       0.822
3 High         0.826

In this case, the correlation between pop_density and crime_rate is similar at all levels of disadvantage

Correlation matrices

 

cor() produces correlation matrices when given multiple continuous variables

anscombe |> # Anscombe's quartet from yesterday!
  select(x1, y1, y2, y3, y4) |> 
  cor()
           x1         y1         y2         y3         y4
x1  1.0000000  0.8164205  0.8162365  0.8162867 -0.3140467
y1  0.8164205  1.0000000  0.7500054  0.4687167 -0.4891162
y2  0.8162365  0.7500054  1.0000000  0.5879193 -0.4780949
y3  0.8162867  0.4687167  0.5879193  1.0000000 -0.1554718
y4 -0.3140467 -0.4891162 -0.4780949 -0.1554718  1.0000000

Wrap-up

  • Recommended reading for next week
    • Kaplan, chapter 10 is review of today’s content
      • Chapter 14 (graphing) is what we’ll get into next week
    • You’re pretty busy, so make it low priority!
  • {swirl} units if you want practice
    • 5 covers missing values
    • 6 covers subsetting
  • Next time:
    • More on relationships
    • Distributions
    • Maybe a start on inference