Today

  • What is causality anyway?

  • Causal graphs

    • Types of paths
    • An example

  • Theory testing

    • Turning theories into graphs
    • Turning graphs into estimators

Causality

Theory and causes

Criminological theories are causal: They explain why things happen

  • “Increasing the perceived risk of being caught reduces the likelihood of a person committing a particular crime”
  • “Learning violent definitions of situations increases a person’s likelihood of using violence”
  • “Strong social institutions in neighbourhoods reduce crime”

These are all testable, causal propositions about how the world works

Theory testing is causal inference: Drawing causal conclusions from data

So what does that even mean?

Perspectives

Different philosophical perspectives

  • Probabilistic
  • Manipulation
  • Mechanistic
  • Counterfactual

For a primer, consider Illari & Russo

Counterfactual causality

\(X\) causes \(Y\) if changing only \(X\) would change \(Y\)

  • \(X\) need not be the only cause of \(Y\)
  • \(X\) may cause \(Y\) for multiple reasons
  • \(X\) may cause \(Y\) only under certain conditions

All that matters for the relationship to be causal is that if the distribution of \(X\) was different, it would result in a different distribution of \(Y\)

 

We call the process of estimating a causal effect identification

 

We’ll focus on two-valued \(X\): a binary treatment

Two worlds

The treatment effect

Potential outcomes

 

We can define a causal effect—an estimand of interest—as the difference (\(\delta\)) between two worlds or potential outcomes:

\[\delta = Y^1 - Y^0\]

  • \(Y^1\): The outcome if treated
  • \(Y^0\): The outcome if not treated

 

This raises a fundamental problem

World where A is treated

World where A is untreated

The fundamental problem

 

We want to know the difference between the factual (what happened) and the counterfactual (what did not)

This is a missing data problem

Let us assume:

  • Potential outcomes are fixed for each unit
  • Potential outcomes vary across units
  • Treatment determines which potential outcome is observed

We can estimate \(\delta\) if we can estimate the value of the unobserved outcome(s)

Causal inference is about estimating these counterfactuals we cannot see

An example

Clearing lots

 

What is the effect of clearing vacant lots on nearby violent crime?1

We can look at vacant lots and see what happens

But we only see one outcome per lot:

  • If we clear a lot, we don’t see what happens if we had not
  • If we don’t clear it, we don’t see what happens if we had

What could we do?

Between comparison

We could compare different lots:

  • One cleared (\(T=1\))
  • One not cleared (\(T=0\))

Use the difference (\(\delta\)) in observed crime as the causal effect

 

What if the difference is because the lots are different?

We could look at identical looking lots

 

But if those lots are identical, why did one get cleared and not the other?

We can’t be sure they’re similar without knowing why it got cleared

A substitution

Before and after comparison

Could compare the same lots:

  • After clearing (\(T=1\))
  • Before clearing (\(T=0\))

Use the before-and-after difference (\(\delta\)) as the causal effect

 

But what if the change was going to happen anyway?

Maybe the crime change even caused the clearing!

 

What if the lot was cleared because it would make a difference?

Maybe there would have been no difference in other lots!

Different potential outcomes

These are really the same problem:


Differences in potential outcomes between cleared and uncleared lots

Put in two equivalent ways:

  • The treatment effect is associated with the likelihood of being treated
  • Treated lots respond differently to treatment than untreated ones would

 

How do we make sure the cleared and uncleared lots are the same?

Randomization

We can do this with a randomized controlled trial:

  • Take a relatively large number of lots
  • Randomly pick some lots to clear
  • Cleared and uncleared lots will have similar potential outcomes on average

The probability of being treated is now independent of the outcomes:

\[(Y^0,Y^1) \perp T \]

While all the units may have different potential outcomes, the outcomes will not differ systematically between treatment groups

This means, essentially, we can ignore the treatment assignment process

Ignorability enables substitution

Average treatment effects

 

If we randomly assign a treatment, we can estimate a causal effect as the average difference in outcomes between the treated units and the untreated units

This is the average treatment effect:

\[ATE = E[Y^1]-E[Y^0]\]

You can calculate this with a cross-tab1

 

But what if we’re interested in things we can’t randomly assign?

Back to counterfactuals

 

Causal inference hinges on substituting in plausible counterfactuals in place of the potential outcomes we never observe

Substituting in counterfactuals requires knowing why units got treated

 

This is the same as with randomization:

  • With randomization, we know it happened randomly!
  • Without randomization, we need to know the assignment process!

Theory strikes back

When we can’t randomize, we need to theorize

We need to figure out the data generating process for our treatment and outcome

  • What makes some units get treated?
  • What makes treatment more (or less) effective for some units?

This requires making assumptions, which are part of your model

  • \(Z\) is unrelated to \(X\)
  • \(B\) only causes \(Y\) through \(C\)

If our model is correct, it tells us exactly how to identify the causal effect

So how do we make the model?

Directed acyclic graphs

 

Causal graphs

Classic statistics is mathy, using things like this:

\[Y = f(X,Z)\] … to say \(Y\) is caused by a combination of \(X\) and \(Z\)

We’re going to work with directed acyclic graphs (DAGs) instead:

Directed acyclic graphs have pretty simple rules:

  • Arrows between variables point from causes to outcomes (directed)
  • No arrow means no direct causal relationship
  • Something can’t cause itself… even through other things (acyclic)

Paths

A path is any set of arrows linking two variables together

There are three paths from \(X\) to \(Y\):

  • \(X \rightarrow Y\)
  • \(\color{#003C71}{X \leftarrow Z \rightarrow Y}\)
  • \(\color{#8A1538}{X \rightarrow W \leftarrow Y}\)

Let’s say we want to know the effect of \(X\) on \(Y\)

  • Paths where all arrows point from \(X\) to \(Y\) are called front doors
  • Paths where arrows point to both \(X\) to \(Y\) are back doors
  • Paths where arrows point at each other are colliders

DAGs and identification

 

DAGs are a way to write out a model that tells you how to identify effects

 

We estimate effects through front doors by closing every back door and avoiding colliders

 

Let’s see how that works

Closing back doors

Variables that open back doors are called confounders

  • Confounders cause dependence between treatment and potential outcomes

We close doors by adjusting, controlling, or holding constant variables along the path—either by design or in our statistical estimator1

[1] We’ll see this in a bit

 

Adjusting for Z closes the back door

The only remaining path is the front door we’re interested in

The effect of \(X\) on \(Y\) is identified

More front doors

Very often your model has multiple front doors

You don’t adjust for these—they’re ways \(X\) impacts \(Y\)!

  • We call these mechanisms or mediators

There are two front doors

  • \(X \rightarrow Y\)
  • \(X \rightarrow A \rightarrow Y\)

We’ve adjusted for \(Z\) but not \(A\)

The effect of \(X\) on \(Y\) is identified!

The total estimated effect of \(X\) on \(Y\) will be the sum of all front door paths leading between them.

Colliders

Colliders are the opposite of back door paths: They block associations from going down their path unless we adjust for them

If you control for a collider on the path between \(X\) and \(Y\), it opens the association back up and introduces bias

Adjusting for Z closes the back door

But here we’ve adjusted for \(W\) too

Now our estimate of \(X \rightarrow Y\) contains parts of \(Y \rightarrow W\) and \(X \rightarrow W\)

Controlling for a collider is sometimes conditioning on a post-treatment variable or Berkson’s paradox, and is one form of selection bias.

Collider bias

Do people commit more crime when they’re good at it?

Unobservabed variables

A DAG must accurately represent what we think the DGP is to be useful

Sometimes we know a variable is important but we can’t measure it

Unobserved variables are often indicated with circles

Because they’re unobserved, we can’t adjust for them directly

Here we can still block the path through \(W\) by adjusting for \(Z\)

Most of the time we can’t observe everything that matters, but rigorous research will acknowledge this

This is all very abstract, so let’s put them to work

DAGs at work

 

Measuring bias in policing

 

Use of force

 

Question: Do US police use more force against black civilians?

 

Imagine you sample police encounters identical except for race

But suppose bias leads police to:

  • Stop white civilians only for crimes
  • Stop black civilians with or without a crime

Then, discard data on anyone police observed but did not stop.

 

This is what police data actually show: we only see the stopped people!

A bad comparison

 

You are now comparing use of force between two groups:

  • White people committing crime
  • Black people committing no crime or committing a crime

 

If use of force were the same, we’d have a serious problem

What we want

 

 

Question: How does perceived race impact officer decisions to use force?

We could compare how often force is used on white vs. black subjects

We could even get sophisticated:

  • Compare similar encounters
  • Compare similar subjects
  • Compare similar officers

Good ideas, but none of them will help

Oh no

 

 

‍Problem: Race impacts likelihood of being stopped in the first place, but we never see unstopped people

  • We’ve conditioned on being stopped—a mediator!
  • We’ve blocked part of our front door!

We can’t identify the causal effect!

Oh noes

 

 

More problems

  • Racial composition varies by neighborhood
  • Police deployments and strategy vary by neighborhood
  • Reasonable suspicion predicts both stops and use of force—and can’t be observed

Consequences

  • Detecting bias in the decision to stop is difficult

  • If any stop bias exists, it is difficult to measure bias in…1

    • Arrests
    • Use of force
    • Frisks
    • Shootings

[1] Clever assumptions allow bounded identification showing substantial use of force bias by US police

  • Raw numbers can easily show opposite patterns from underlying reality

  • Studies showing no bias—or anti-white bias—get lots of media, social media, and political traction

Theory testing

 

Recursive models

Criminological theories tell us how to construct DAGs

Often this is fairly explicit, such as when authors provide a causal diagram… and it already obeys DAG rules

 

Source: Matsueda (1982)

Nonrecursive models

This is less clear when the diagram provided doesn’t conform to DAG rules

Converting a model with cycles into an acyclic one can take some thought

All causal relationships really are acyclic though

Broken windows DAG

Usually the trick to converting a cyclic graph to an acyclic one is realizing effects occur over time

  • This requires assumptions about (relative) timing

Another way is through intervening or cross-level mechanisms

  • Many macro theories are circular without individual-level mechanisms

No graphical model

Making a DAG is least clear when the authors provide no graphical representation of the model at all1

[1] Or when you’re developing something new yourself!

  • Record all the core variables they use

    • e.g., self control

  • Make note of causal statements

    • e.g., “self control influences crime by…”
    • Make these into arrows!

  • Try to reconcile circular paths

    • Consider temporal order
    • Consider cross-level mechanisms

From DAG to estimator

 

Estimating Causal Effects

Once you have a DAG, it is remarkably easy to proceed to an estimator

Figure out what you can and cannot measure

If your can close all back doors…

  • Adjust for back doors
  • Don’t adjust for colliders or mechanisms
  • Choose appropriate method
  • Estimate the causal effect

If you can’t close all back doors…

  • Close what you can and estimate an association
    • Might be close to identified!
  • Give up
  • Get clever

Remember: clever estimators are a last resort for research design

It is always better to solve hard problems with design

Controlling for variables

 

Regression again

So far we’ve just fit a simple straight line using one variable predicting another

Term Estimate
Pop Density 3.09
(Intercept) -20.78

This doesn’t get us very far for testing theories

 

But there’s no reason to have only one predictor

Multiple predictors

When we add predictors, we are adjusting or controlling for them to identify the effect we’re interested in

Term Estimate
Urban -3.39
Pop Density 3.36
(Intercept) -23.04

  • “The association between density and crime, adjusting for area type”
  • “The average crime rate difference between urban and rural areas, adjusting for population density”

We could argue this is causal if we think the diagram is complete

How it works

More variables

The same basic principle applies when more variables are introduced

A verbal interpretation can sometimes get difficult…

Term Estimate
Urban -2.19
Pop Density 3.11
Incarc.: Medium -6.45
Incarc.: Low -5.42
(Intercept) -15.97

But a graph always fully encodes our expectations about relationships1

[1] If we’re not sure how two variables are related, sometimes we just assume there is an unobserved (U) confounder connecting them

What paths are actually identified if this model is true?

Don’t interpret controls

DAGs highlight one of the most common mistakes made in empirical research: Researchers often write as if every path is identified

  • Interpreting the main “effect” as adjusted for covariates
  • Interpreting all covariates as adjusted for everything else

Usually, these interpretations are logically impossible

The purpose of controls is to close back doors, not to add additional interpretable causal paths

  • If a variable closes a back door, its effect on the outcome is not identified
  • Control variable associations are typically not worth interpreting1

[1] Reviewers or editors may make authors do this. Savvy authors will describe them appropriately.

Other assumptions

 

DAGs do not make parametric assumptions about relationships, such as:

  • Straight lines (versus curves)
  • Additive effects (versus multiplicative)
  • Error distributions

In contrast, all estimation techniques make some sort of assumptions about how relationships should be calculated

Other assumptions

 

Our estimators have been straight lines with additive independent effects:

  • \(E[y_i] = b_0 + b_1x_i + b_2z_i\)

Common modifications to these methods include:

  • Polynomials or splines (curved lines)
  • Interaction / multiplicative terms (group-specific slopes)
  • Different error distributions (e.g., Poisson)

These are important but beyond the level of this course

Causal qualitative research

DAGs aren’t just non-parametric—they are not necessarily quantitative

DAGs can be useful for thinking through qualitative research too

In particular, propositional (hypothesis testing) qualitative research:

  • Event studies
  • Case-comparisons

Qualitative research can cover the blind spots of quantitative research

Quantitative data are sometimes not up to the task of identification

  • Difficult to measure concepts
  • Confounding colliders
  • Very small samples / populations

Wrap-up

 

  • Know and state counterfactuals

    • Counterfactuals are often unstated, sometimes implausible
    • Counterfactuals come from area knowledge

  • Practice drawing DAGs of theories you learn in your other courses!

  • Use DAGs in your dissertation if applicable