\(R^2\), the coefficient of determination, indicates the proportion of variation in one variable linearly predicted by another

• Range from 0 to 1
  • 0: Neither explains each other (linearly)
  • 1: They perfectly explain (or determine) each other

The distributions of data we observe are the result of a real-world process we cannot directly observe

We call this the data generating process (DGP)

Models aim to approximate this data generating process mathematically to allow us to make quantitative statements about how the world works

Most of the time, in any given project, we’re really interested in an estimate of just one part of the DGP or one characteristic of the population:

• A distribution or measure of central tendency
• A correlation or other association
• A causal effect (next week!)

Any given estimate consists also of three components:

\[Estimate = Estimand + Bias + Noise\]

Noise comes from randomness in the real world or sampling (today)

Bias comes from using the wrong estimator (next week)

Regression just summarizes a relationship between variables using a line

• e.g., predicting \(Y\) using \(X\)

• The variable being predicted is the dependent or outcome variable

• The variables used for prediction are independent or explanatory variables, sometimes called covariates or regressors

The regression line is the conditional expectation of the dependent variable

• e.g., the expected value (mean) of \(Y\) for each value of \(X\)

The estimator used for fitting this line has two parts:

1. The model—the shape of the line (e.g., straight) and our choice of covariates

2. The estimation method—the maths to fit that shape to the data, e.g., ordinary least squares (OLS)

OLS draws a line through the data that minimizes the sum of squared residuals

• Residuals are the \(Y\) distances between the line and each point
• OLS draws the line where the sum of squared residuals is smallest

Regression line equations look like this:

\[y_i = a + b x_i + e_i\]

\(a\) is the intercept, the expected value of \(y\) when \(x = 0\)

\(b\) is a slope, how the mean of \(y\) rises for each increase in \(x\)

\(e_i\) is the residual

• The difference between prediction ( \(a + bx\) ) and each value of \(y\)
• The square of this is what OLS is trying to minimize

The regression formula tells us those predictions:

\[CrimeRate = -20.8 + 3.09 PopDensity\]

We expect \(CrimeRate\) to be 3.09 higher for every additional unit of \(PopDensity\)

When \(PopDensity\) is zero, the expected \(CrimeRate\) is \(-20.8\)

If \(PopDensity\) is \(20\), then we expect \(CrimeRate\) to be:

\[41 = -20.8 + 3.09*20\]

But once we estimate those parameters from data, i.e., show what occurs in the sample, we typically use Latin for those estimates (they’re observed!):

\[y_i = a + b x_i + e_i\]

Sometimes, though, you’ll see Greek wearing hats for estimates:

\[y_i = \hat{\alpha} + \hat{\beta} x_i + \hat{\epsilon}_i\]

We cannot observe it directly, but we can use data to infer things about it

Hypothesis testing is one approach to learning about DGPs

The basic idea:

• Make an informed guess—a hypothesis—about what the DGP looks like
  • Theory is about DGPs!
• Test if it is likely observed data were created by the hypothetical DGP
• If it seems unlikely, we reject our hypothesis

In short, we get closer to the truth by rejecting false explanations

• via negativa: Knowledge by subtracting what is wrong

1. State hypotheses about estimand:

   • Null Hypothesis: There is no relationship
   • Alternative Hypothesis: There is a relationship (at all or directional)

2. Make an estimate

3. Test how likely this estimate would be if our null hypothesis is true

   • We choose a threshold for what we consider “unlikely”, e.g., less than 5% chance

4. Either reject or fail to reject the null; we do not accept the null

Frequentism is based on the logic of infinite random sampling

• For a given DGP, imagine randomly sampling from the population forever

• The distribution of values in the samples (e.g., the means) would follow a known theoretical distribution

  • i.e., we know how likely it is to obtain any particular estimate if the theoretical DGP is true

• Based on these theoretical distributions, we assess how (un)likely observed results are under a given hypothesis

• Importantly, all of these rely on properties of random sampling

  • Non-random samples lead to bias

What about 15 heads?

If the coin is truly fair, it should have a 0.5 probability of coming up heads

• Our null hypothesis is the coin is fair and has a heads probability of 0.5
• Our alternative is it is unfair—likely to give more or fewer heads than 0.5

We can calculate how likely it is to observe any particular number of heads flips if our null hypothesis is true

• e.g., how likely is it to get 15 heads on 20 flips with a fair coin
• We can do this just by tabulating every possible outcome of 20 coin flips

We’re 1,000 times more likely to get 10 heads than 2 heads!

The probability we get any particular number of heads is can be described by a theoretical distribution: the binomial distribution

• The p-value for 15 out of 20 (75%) flips being heads is 0.041
• A fair coin would give us a result as far from 50% as this 1 in 25 times
• p-values are typically calculated using a test statistic derived from a theoretical distribution (e.g., \(t\) or \(\chi^2\) tests)

The p-value is not the probability our null hypothesis is true

All a p-value tells us is how frequently the theoretical distribution we chose would generate the data we observe

• p-values are only valid if we’ve chosen the right theoretical distribution

\(\alpha\) is the rejection threshold, corresponding to a confidence level of \(1-\alpha\)

• The threshold for rejecting the null distribution as a source of our data
• e.g., 95% confidence corresponds to \(\alpha=.05\)

\(\alpha\) indicates our “comfort” in rejecting the null when it is true

• e.g., \(\alpha=.05\) means we reject a true null 5% of the time

Higher levels of confidence reduce false positives, but lead to more false negatives or Type II errors

• Failing to reject the null when it is false

Rural and urban areas equally probable to be in each level of disadvantage

But we could get different probabilities just from randomness in sampling.

Disadvantage and Area are more strongly related in this sample than is likely to occur due only to sampling noise.

• i.e. this is evidence for an association in the population

This is because means have a useful sampling property, described by the central limit theorem:

No matter what the distribution* of the variable in the population you sample from, the distribution of the means of all samples will be normally distributed around the population’s mean

This means we can use the normal distribution (or something related to it) as a theoretical distribution in many different situations

• Population means
• Differences in means
• Correlations
• Regression parameters
• Categorical associations (\(\chi^2\)!)

In particular, we can assess the precision and statistical significance of estimates

Smaller standard errors indicate more precise estimates

Larger sample sizes mean smaller standard errors

Intuitively, confidence intervals (CI) are a range of reasonably likely values for the estimand, given our observed sample values

• Based on a confidence level, e.g. 95%
• 95% CI of (5.2, 7.5) is compatible with estimands between 5.2 and 7.5

Less intuitively, if you re-estimated the quantity with new data infinitely, 95% of the 95% CIs would overlap the true estimand

Confidence intervals are closely related to p-values:

• The CI will not include zero if the p-value is significant

Confidence interval example

Do we believe this is a good model, however?

If for some reason we’re only interested in the (unconditional, linear) association between density and the crime rate, this might be just fine!

It is trivial to perform many analyses, cherry-pick significant ones, then submit only those for review

• The scientific community observes only what is submitted

Similarly, one can invest enormous time and money into a project and receive null results

• Inability to publish these is frustrating and damaging to careers
  • publish or perish

Journal have historically declined my submissions with null findings

• Articles are rejected for being “not novel”
• Articles are never submitted because authors know they will be rejected

Conversely, journals (and the media) reward “novel” significant results

This leads to a number of problems:

• Literatures filled with false positives and missing true negatives
• Absence of replication studies (likely null findings)
• Average estimates greater in magnitude than the real estimands

These all work best to address researchers behaving ethically

It can be very hard to detect unethical researchers

• As is the case for many types of white-collar crime, it is mainly the incompetent who get caught