Complex Numbers in R

R provides basic functionality for working with complex numbers, documented in the helpful ?complex. The imaginary unit in R is denoted by 1i. Don’t try i by itself because that won’t work:

i # this doesn't work

Error:
! object 'i' not found

1i # this works

[1] 0+1i

To write a complex number, simply use + to separate the real and imaginary parts, e.g.

3 + 4i

[1] 3+4i

A common error when learning R is to write code with “implied multiplication” e.g. 4x (wrong) rather than 4 * x (right). But don’t be tempted to try 4 * i since, again, i is not the imaginary unit in R:

4 * i # this doesn't work 

Error:
! object 'i' not found

4 * 1i # this works

[1] 0+4i

To demonstrate R’s basic functionality for working with complex numbers, let’s specify a complex number in Cartesian coordinates: . The real part is , the imaginary part is , the modulus is , and the complex conjugate is . All of these operations are available in R:

z <- 3 + 4i
Re(z)

[1] 3

Im(z)

[1] 4

Mod(z)

[1] 5

abs(z) # same as Mod(z) for a complex number z

[1] 5

Conj(z)

[1] 3-4i

All of the basic operations +, -, *, /, and ^ work on complex numbers and are vectorized

w <- (-3) - 1i
z + w

[1] 0+3i

z - w

[1] 6+5i

z * w

[1] -5-15i

z / w

[1] -1.3-0.9i

z^3

[1] -117+44i

as are functions such as log(), exp(), sin(), cos(), etc. If you supply them a complex input, they will return a complex output; if you supply them a real input, they will return a real output. This behavior explains a common gotcha: sqrt(-1) doesn’t work because there’s no real number whose square root is -1 but sqrt(-1 + 0i) does work:

sqrt(-1) # this doesn't work

Warning in sqrt(-1): NaNs produced

[1] NaN

sqrt(-1 + 0i)  # this works

[1] 0+1i

Another common gotcha is trying to apply functions like max() and min() or operators like < and > to complex numbers. Since complex numbers aren’t ordered, this doesn’t work:

min(c(z, w))

Error in `min()`:
! invalid 'type' (complex) of argument

z < w

Error in `z < w`:
! invalid comparison with complex values

but testing for equality / non-equality does work:

z == w

[1] FALSE

z != w

[1] TRUE

Complex Step Differentiation in Action

Let’s test out complex step differentiation with a simple example. Let and suppose we want to compute . This one is easy to compute analytically: so we can check how accurate different numerical approximations turn out to be. Computing the derivative directly gives

direct <- 4.5 * (1.5)^(3.5)
direct

[1] 18.60081

Now we’ll compare this value against three numerical derivatives: the “simple difference” approach, the “symmetric difference” approach, and the complex step approach.

f <- function(x) x^(4.5) # the function to differentiate

x <- 1.5 # the point where we'll evaluate f'(x)

Since R functions are vectorized, we can evaluate the quality of each approximation over many values of at once by setting up a vector of progressively smaller positive values:

delta <- 1 / 10^(1:15)

simple <- (f(x + delta) - f(x)) / delta

symmetric <- (f(x + delta) - f(x - delta)) / (2 * delta)

complex_step <- Im(f(x + delta * 1i)) / delta

Now we’ll make two tables: the first containing raw results—the approximate value of —and the second containing the relative error of the approximation in percentage points. When reading the first table, recall from above that direct calculation gives .

cbind(delta, simple, symmetric, complex_step) |> 
  knitr::kable(digits = 15)

get_rel_error <- function(x, truth) 100 * abs(x - truth) / abs(truth)

cbind(
  delta,
  simple = get_rel_error(simple, direct),
  symmetric = get_rel_error(symmetric, direct),
  complex_step = get_rel_error(complex_step, direct)
) |> 
  knitr::kable(digits = c(15, rep(4, 3)))

The simple difference approach is clearly less accurate than both the symmetric difference and complex step approach, particularly at larger values of . And while it may seem that there’s not much to choose between these latter two approximations, look carefully at what happens as gets smaller. Eventually the relative error of the simple and symmetric difference approaches starts to increase! For example, the symmetric difference approximation is better with (relative error ) than it is at (relative error ).

How can this be? The derivative is defined as the limit of the simple difference approximation as . So how can the approximation get worse if becomes smaller?

Catastrophic Cancellation

The culprit is catastrophic cancellation. As explained in an earlier post, computers cannot represent all real numbers exactly. Instead they use floating point numbers as an approximation. So when we compute a difference like what we’re really computing is the difference of two approximations. The problem is that even when is a very good approximation to and is a very good approximation to , the difference may be a poor approximation to . This is an unfortunate property of subtraction, nicely illustrated in this post by John D. Cook.

A double in R is accurate to around 16 decimal places. For small values of , and agree in nearly all of those digits. Subtracting them eliminates the digits that match, leaving behind what is effectively rounding noise. Dividing by then magnifies this noise. For example, since in our example is around when we compute we should obtain a result of approximately but instead we get

sprintf("%.20f", f(x + 1e-15) - f(x))

[1] "0.00000000000002042810"

The first couple of digits are in the ballpark, but the remaining ones are just noise.
The symmetric difference formula has the same problem because it too relies on a subtraction of two approximate values in the numerator:

sprintf("%.20f", f(x + 1e-15) - f(x - 1e-15))

[1] "0.00000000000004085621"

Here we’d expect a result of around and, again, every digit after the first two is just noise.

In contrast, complex step differentiation has no subtraction in the numerator. This means that it is not subject to catastrophic cancellation. As becomes smaller, the approximation just keeps improving—down to roughly machine precision. The cost is that must be analytic at and evaluable at complex inputs. Most smooth functions you’ll encounter qualify, but those involving , , , or indicator functions generally don’t.

Don’t Roll Your Own

In this post I computed the complex step derivative by hand. That’s useful for understanding the method, but it’s a bad idea in practice. Whenever you can, you should rely on high-quality existing libraries to implement numerical methods. Numerical analysis is a deeply complicated subject and we are but lowly econometricians! Fortunately for us, the grad() function from the numDeriv R package implements complex step differentiation as one of its three methods:

library(numDeriv)
grad(f, 1.5, method = 'complex')

[1] 18.60081

So the next time you need to differentiate something numerically, remember the value in making things more complex than they need to be!

A Word About Notation

While I phrased my first post about overlapping intervals in terms of sample and population means for two groups, the idea is general. Nothing substantive changes if we replace the parameters with , the estimators with , and the standard errors with . As long as the two estimators are uncorrelated and (approximately) normally distributed, the results from Part I apply to the individual intervals for and versus the interval for the difference . Today we will ask what happens when the estimators are potentially correlated. To make it clear that our results are general, I’ll use the more agnostic notation throughout.

A Motivating Example

Consider a randomized controlled trial with two active treatments. In our paper on pawn lending, for example, my co-authors and I compare default rates between borrowers assigned to the status quo pawn contract (control), a new structured contract (Treatment A), and a choice arm (Treatment B) in which they were free to choose whichever contract they preferred. To learn the causal effect of the structured contract, we compare the mean default rates of borrowers who received Treatment A against the corresponding rate in the control group. Call this difference of means . Similarly, to learn the effect of choice, we make the analogous comparison between Treatment B and the control group. Call this difference of means . Now suppose you’re reading a paper that reports both of these estimators and their standard errors. To find out which treatment is more effective, you need to compare and , but these estimators must be correlated because they both involve the control group average: Granted, if you had access to the raw data, you could easily solve this problem without worrying about the correlation. In the difference , the control group mean cancels out So if we had the raw data for treatments A and B we’d be back to a familiar independent samples comparison of means, as in Part I. But if you’re reading a paper that only reports and their standard errors, you cannot directly calculate the standard error for the difference. The common variation from the control group mean is baked into the way both and are computed, even though this variation is irrelevant for .

Allowing Correlation

So how do we compute if the two estimators are correlated? Recall that the standard error is the standard deviation of that estimator’s sampling distribution, and a standard deviation is merely the square root of the corresponding variance.¹ Using the properties of variance and covariance, Defining , it follows that If this reduces to our formula from Part I: so we can equate the LHS with the length of the hypotenuse of a right triangle whose legs have lengths and . If , however, this connection to the Pythagorean Theorem no longer holds. Nevertheless, there are still triangles hiding in this standard error formula! To reveal them, we need a more general theorem about triangles.

The Law of Cosines

Consider a triangle whose sides have lengths and . Let be the angle between the sides whose lengths are and . Then by the Law of Cosines
This equality holds for any triangle. For a right triangle whose legs have lengths and , we have so the Law of Cosines reduces to the Pythagorean Theorem When , the “correction term” shows how the length of differs from that of a right triangle with legs of lengths and . When the cosine is positive so the correction term shortens ; when the cosine is negative so the correction term lengthens . Regardless of the angle , however, the Triangle Inequality still holds: because the shortest distance between two points is a straight line.²

From Geometry to Statistics

The cosine of an angle is always between negative one and one. Can you think of anything else that shares this property? That’s right: correlation! So let’s put the Law of Cosines and our standard error formula from above side-by-side: The analogy is perfect. We can view and as the lengths of two sides of a triangle and as the cosine of the angle between these sides. This makes the length of the third side, indicated in blue in the following diagram. When is positive, the standard error of the difference is smaller than it would be under independence; if is negative, the standard error of the difference is larger than it would be under independence.

The Grand Finale

Since we’ve equated , and with the sides of a triangle, the Triangle Inequality gives assuming that . And now we’re on familiar ground. Let be the appropriate quantile of a normal distribution, i.e.  for a 95% confidence interval. Just as we argued in Part I, the individual confidence intervals for and overlap precisely when and there is a significant difference between the two parameters when . The condition for overlap and a significant difference is which holds by the Triangle Inequality. So we’re back to exactly the same situation we were in when ! As long as the same results concerning confidence interval overlap apply when and are correlated as when they are uncorrelated:

1. Overlap doesn’t tell us anything about whether there is a significant difference, but
2. a lack of overlap implies that there must be a significant difference.

Setting the Stage

Case I: Overlapping Intervals for A and B

Case II: Intervals for A and B that Don’t Overlap

Now for the easy one. Since holds if and only if the intervals overlap, we merely reverse the inequality to get a condition for intervals that do not overlap, namely Appending what we learned above from our right triangle diagram gives So if the intervals do not overlap, then must be greater than , which is precisely the condition for a significant difference between the two groups.

In Summary

In an independent samples problem where the two individual confidence intervals overlap, there may or may not be a significant difference between the groups. Even if the two intervals both contain zero, there could still be a significant difference between them. If the two intervals do not overlap then we can conclude that there is a significant difference. But if you only take one lesson away from this post it should be this one: the difference between significant and not significant is not itself significant. If you want to carry out inference for a difference, you need to construct the standard error for that difference.

Recap of Part I

Suppose that is our endogenous regressor of interest in the linear causal model where but , and where is an instrumental variable that is correlated with . Now consider the population linear regression of on both and , namely where the error term satisfies by construction. Further define the population linear regression of on , namely where the error term satisfies by construction. Finally, define the population linear regression of on as Using this notation, the result from my earlier post can be written as To understand what this tells us, notice that, using the “first-stage” regression of on , we can write This shows that whenever is a relevant instrument , we must have . It follows that is more biased than : adding as a control regressor only makes our estimate of the effect of worse!²

What about ?

So if soaks up the good variation in , what about the coefficient on the instrument ? Perhaps this coefficient contains some useful information about the causal effect of on ? To find out, we’ll use the FWL Theorem as follows: where is the population linear regression of on . This is the reverse of the first-stage regression of on described above. Here the error term satisfies by construction. Substituting the causal model gives since by construction. Now, substituting the definition of , since by assumption. We can already see that is not going to help us learn about . First of all, the term containing vanished; second of all, the term that remained is polluted by the endogeneity of , namely .

Still, let’s see if we can get a clean expression for . So far we have calculated the numerator of the FWL expression, showing that . The next step is to calculate : Since , our expression for simplifies to so we have discovered that:

Call me old-fashioned, but I really don’t like having in that expression. I’d feel much happier if we could find a way to re-write this in terms of the more familiar IV first-stage coefficient . Let’s give it a try! Let’s use my favorite trick of multiplying by one: Substituting for gives We can simplify this even further by substituting from above to obtain And now we recognize something from above: was the bias of relative to the true causal effect ! This means we can also write .

A Little Simulation

We seem to be doing an awful lot of algebra on this blog lately. To make sure that we haven’t made any silly mistakes, let’s check our work using a little simulation experiment taken from my earlier post. Spoiler alert: everything checks out!

set.seed(1234)
n <- 1e5

# Simulate instrument (z)
z <- rnorm(n)

# Simulate error terms (u, v)
library(mvtnorm)
Rho <- matrix(c(1, 0.5, 
                0.5, 1), 2, 2, byrow = TRUE)
errors <- rmvnorm(n, sigma = Rho)

# Simulate linear causal model
u <- errors[, 1]
v <- errors[, 2]
x <- 0.5 + 0.8 * z + v
y <- -0.3 + x + u

# Regression of y on x and z
gamma <- lm(y ~ x + z) |> 
  coefficients()

gamma

(Intercept)           x           z 
 -0.5471213   1.5018705  -0.3981116 

# First-stage regression of x on z
pi <- lm(x ~ z) |> 
  coefficients()

pi

(Intercept)           z 
  0.5020338   0.7963889 

# Compare two different expressions for gamma_Z to the estimate itself
c(gamma_z = unname(gamma[3]),
  version1 = unname(-0.8 * cov(x, u) / var(v)),
  version2 = unname(-pi[2] * (gamma[2] - 1))
)

   gamma_z   version1   version2 
-0.3981116 -0.4024918 -0.3996841 

Making Sense of This Result

So far all we’ve done is horrible, tedious algebra and a little simulation to check that it’s correct. But in fact there’s some very interesting intuition for the results we’ve obtained, intuition that is deeply connected to the idea of a bad control in a directed acyclic graph (DAG).

In the model we’ve described above, has a causal effect on . This is because causes which in turn causes . Because is an instrument, its only effect on goes through . The unobserved confounder is a common cause of and but is unrelated to . Even if you’re not familiar with DAGs, you will probably find this diagram relatively intuitive:

library(ggdag)
library(ggplot2)

iv_dag <- dagify(
  Y ~ X + U,
  X ~ Z + U,
  coords = list(
    x = c(Z = 1, X = 3, U = 4, Y = 5),
    y = c(Z = 1, X = 1, U = 2, Y = 1)
  )
)

iv_dag |>
  ggdag() +
  theme_dag()

In the figure, an arrow from to means that is a cause of . A causal path is a sequence of arrows that “obeys one-way signs” and leads from to . Because there is a directed path from to , we say that is a cause of . To see this using our regression equations from above, substitute the IV first-stage into the linear causal model to obtain This gives us a linear equation with on the left-hand side and alone on the right-hand side. This is called the “reduced-form” regression. Since by assumption and by construction, the reduced-form is a bona fide population linear regression. That means that regressing on will indeed give us a slope that equals . To see why the slope is a product, recall that is the causal effect of on , the arrow in the diagram, while is the causal effect of on , the arrow in the diagram. Because the only way can influence is through , it makes sense that the causal effect of on is the product of these two effects.

So now we see that the reduced-form coefficient is indeed a causal effect. How does this relate to ? Remember that was the coefficient on in a regression of on and , in other words a regression that adjusted for . So is adjusting for the right call? Absolutely not! There are no back-door paths between and .³ This means that we don’t have to adjust for anything to learn the causal effect of on . In fact adjusting for is a mistake for two different reasons.

First, is a mediator on the path . If there were no confounding, i.e. if so there is no arrow, adjusting for would block the only causal path from to . We can see this in our equations from above. Suppose that . Then we have but ! There was a dead giveaway in our derivation: the formula for doesn’t depend on at all.

Second, because there is confounding, adjusting for creates a spurious association between and through the back-door path . Because is a collider on the path , this path starts out closed. Adjusting for opens this back-door path, creating a spurious association between and . To see why this is the case, suppose that . In this case there is no causal effect of on and hence no causal effect of on . But if , then we have !

So if you want to learn the causal effect of on , it’s not just that is a bad control; it’s a doubly bad control! Without adjusting for , everything is fine: the reduced-form regression of on gives us exactly what we’re after.⁴

Epilogue

When I showed this post to another colleague he asked me whether there is any way to learn about by combining and . The answer is no: the regression of on and alone doesn’t contain enough information. Since we can rearrange to obtain the following expression for : which we can verify in our little simulation example as follows:

gamma[2] + gamma[3]/pi[2]

       x 
1.001975 

Thus, in order to solve for , we need to run the first-stage regression to learn .

Simulation Example

Let’s start with a little simulation. First we’ll generate 5000 observations of predictors and from a joint normal distribution with standard deviations of one, means of zero, and a correlation of 0.5.

set.seed(1066)
library(mvtnorm)

# Simulate linear regression with two predictors: X and W
covariance_matrix <- matrix(
  c(1, 0.5, 0.5, 1), 
  nrow = 2
)

n_sims <- 5000

x_w <- rmvnorm(
  n = n_sims,  
  mean = c(0, 0), 
  sigma = covariance_matrix
)

x <- x_w[, 1]
w <- x_w[, 2]

Next we’ll simulate the outcome variable where the true coefficient on is one and the true coefficient on is -1, adding standard normal errors.

y <- 0.5 + x - w + rnorm(n_sims)

Now we’ll run the “auxiliary regressions”. The first one regresses on and saves the residuals. Call these residuals x_tilde.

# Residuals from regression of X on W
x_tilde <- lm(x ~ w) |> 
  residuals()

The next one regresses on and saves the residuals. Call these residuals y_tilde.

# Residuals from regression of Y on W
y_tilde <- lm(y ~ w) |>
  residuals()

To make the code that follows a little simpler, I’ll also create a helper function that runs a linear regression and returns the coefficients after stripping away any variable names.

get_coef <- function(formula) {
  formula |>  
    lm() |> 
    coef() |> 
    unname() 
}

Now we’re ready to compare some regressions! The “long regression” is a standard linear regression of on and . The “FWL Standard” is a regression of y_tilde on x_tilde. In other words, it regresses the residuals of on the residuals of . The FWL as it is usually encountered in textbooks implies that we should recover the same coefficient on in “Long Regression” and in “FWL Standard”, and indeed the simulation bears this out.

c(
  "Long Regression" = get_coef(y ~ x + w)[2],
  "FWL Standard" = get_coef(y_tilde ~ x_tilde - 1)[1], 
  "FWL Alternative" = get_coef(y ~ x_tilde)[2]
)

Long Regression    FWL Standard FWL Alternative 
      0.9937046       0.9937046       0.9937046 

But now take a look at “FWL” alternative: this is a regression of on x_tilde. Compared to the standard FWL approach, this version does not residualize with respect to . But it still gives us exactly the same coefficient on as the other two regressions. That leaves us with two unanswered questions:

1. Why does the “alternative” FWL approach work?
2. Given that the alternative approach works, why does anyone ever teach the “standard” version?

In the rest of this post we’ll answer both questions using simple algebra and the properties of linear regression. There are lots of deep ideas here, but there’s no need to bring out the big matrix algebra guns to explain them.

A Bit of Notation

First we need a bit of notation. I find it a bit simpler to work with population linear regressions rather than sample regressions, but the ideas are the same either way. So if you prefer to put “hats” on everything and work with sums rather than expectations and covariances, be my guest!

First we’ll define the “Long Regression” as a population linear regression of on and , namely Next I’ll define two additional population linear regressions: first the regression of on and second the regression of on I’ve already linked to a post making this point, but it bears repeating: all of the properties of the error terms , and that I’ve stated here hold by construction. They are not assumptions; they are merely what defines an error term in a population linear regression.

Why does the “alternative” FWL approach work?

As mentioned in the discussion of our simulation experiment from above, the standard FWL theorem says that a regression of on with no intercept gives us , while the alternative version says that a regression of on with an intercept also gives us . It is the second claim that we’ll prove now.¹

The alternative FWL theorem claims that . Since is uncorrelated with by construction, we can expand the numerator as follows: But since we also have since and are uncorrelated with by construction. So to prove our original claim it suffices to show that . To see why this holds, first write using . Next, expand as follows: and then subtract from : This shows that and are equal if and only if . But since is the coefficient from the regression of on , we already know that ! With a bit of algebra using the properties of covariance and the definition of a population linear regression, we’ve shown that the alternative FWL theorem holds.

What’s different about the “usual” FWL theorem?

At this point you may be wondering why anyone teaches the “usual” version of the FWL theorem at all. If that extra short regression of on isn’t needed to learn , why bother?

To answer this question, we’ll start by re-writing the long regression two different ways. First, we’ll substitute into the long regression and re-arrange, yielding Next we’ll substitute on the left-hand side of the preceding equation and rearrange to isolate . This leaves us with Now we have two expressions, each with as one of the terms on the right-hand side and as another. Notice that both expressions have an intercept and a term in which is multiplied by a constant. What’s more, the intercepts are closely related across the two equations, as are the coefficients. I’m now going to make a bold assertion: the intercept and coefficient in the second expression, the one, are both equal to zero Perhaps you don’t believe me, but just for the moment suppose that I’m correct. In this case it would immediately follow that leaving us with two simple linear regressions, namely We’re tantalizingly close to unraveling the mystery of why the “usual” FWL theorem is so popular. But first we need to verify my bold claim from the previous paragraph. To do so, we’ll fall back on our old friend: the omitted variable bias formula, also known as the regression anatomy formula: Thus, as claimed. One down, one more to go. By definition, . Substituting the long regression for , we have by the linearity of expectation and the fact that by construction. Now, we’re trying to show that . Substituting for in this expression gives Inspecting our work so far, we see that the two alternative expressions for will be equal precisely when . But re-arranging this gives , which we already proved above using the omitted variables bias formula!

Taking Stock

That was a lot of algebra, so let’s spend some time thinking about the results. We showed that Now, if you’ll permit me, I’d like to re-write that first equality as Since is uncorrelated with , as explained above, and since by construction, it follows that is a bona fide population linear regression model. If we regress on the slope coefficient will be and the error term will be . This regression corresponds to the standard FWL theorem. Notice that it has an intercept of zero and an error term that is identical to that of the long regression. We can verify this using our simulation experiment from above as follows:

# Standard FWL has same residuals as long regression
u_hat <- resid(lm(y ~ x + w))
u_tilde <- resid(lm(y_tilde ~ x_tilde - 1))
all.equal(u_hat, u_tilde)

[1] TRUE

# Standard FWL has an intercept of zero (to machine precision!)
coef(lm(y_tilde ~ x_tilde))[1] # fit with intercept; check it's (numerically) 0

 (Intercept) 
8.273433e-17 

So what about ? This is the regression that corresponds to the alternative FWL theorem. Since and is uncorrelated with both and , this too is a population regression. But unless , it has a different error term. In other words, . Moreover, this regression includes an intercept that is not in general zero. Again we can verify this using our simulation example from above:

# Alternative FWL has different residuals than long regression
v_hat <- resid(lm(y ~ x_tilde))
all.equal(u_hat, v_hat)

[1] "Mean relative difference: 0.4905107"

# Alternative FWL has a non-zero intercept
coef(lm(y ~ x_tilde))[1]

(Intercept) 
  0.4878453 

The Punchline

If your goal is merely to learn , then either version of the FWL theorem will do the trick and the alternative version is simpler because it only involves one auxiliary regression instead of two. But if you want to ensure that you end up with the same error term as in the original long regression, then you need to use the standard version of the FWL theorem. This is crucial for the purposes of inference because the properties of the error term determine the standard errors of your estimates.

Example: Height and Handspan

Here’s a simple example: a regression of height, measured in inches, on handspan, measured in centimeters.¹

library(tidyverse)
library(broom)
dat <- read_csv('https://ditraglia.com/data/height-handspan.csv')

ggplot(dat, aes(y = height, x = handspan)) +
  geom_point(alpha = 0.2) +
  geom_smooth(method = "lm", color = "red") +
  labs(y = "Height (in)", x = "Handspan (cm)")

# Fit the regression
reg1 <- lm(height ~ handspan, data = dat)
tidy(reg1)

# A tibble: 2 × 5
  term        estimate std.error statistic  p.value
  <chr>          <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)    40.9     1.67        24.5 9.19e-76
2 handspan        1.27    0.0775      16.3 3.37e-44

As expected, bigger people are bigger in all dimensions, on average, so we see a positive relationship between handspan and height. Now let’s save the fitted values from this regression and run a second regression of height on the fitted values:

dat <- reg1 |> 
  augment(dat)
reg2 <- lm(height ~ .fitted, data = dat)
tidy(reg2)

# A tibble: 2 × 5
  term         estimate std.error statistic   p.value
  <chr>           <dbl>     <dbl>     <dbl>     <dbl>
1 (Intercept) -1.51e-13    4.17   -3.62e-14 1.000e+ 0
2 .fitted      1.00e+ 0    0.0612  1.63e+ 1 3.37 e-44

The intercept isn’t quite zero, but it’s about as close as we can reasonably expect to get on a computer and the slope is exactly one. Now how about the R-squared? Let’s check:

glance(reg1)

# A tibble: 1 × 12
  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>
1     0.452         0.450  3.02      267. 3.37e-44     1  -822. 1650. 1661.
# ℹ 3 more variables: deviance <dbl>, df.residual <int>, nobs <int>

glance(reg2)

# A tibble: 1 × 12
  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>
1     0.452         0.450  3.02      267. 3.37e-44     1  -822. 1650. 1661.
# ℹ 3 more variables: deviance <dbl>, df.residual <int>, nobs <int>

The R-squared values from the two regressions are identical! Surprised? Now’s your last chance to think it through on your own before I give my solution.

Solution

Suppose we wanted to choose and to minimize where . This is equivalent to minimizing By construction and minimize , so unless and we’d have a contradiction!

Similar reasoning explains why the R-squared values for the two regressions are the same. The R-squared of a regression equals The total sum of squares is the same for both regressions because they have the same outcome variable. The residual sum of squares is the same because and together imply that both regressions have the same fitted values.

Here I focused on the case of a simple linear regression, one with a single predictor variable, but the same basic idea holds in general.

To Instrument or Not to Instrument?

Suppose I want to predict someone’s wage as accurately as possible using a linear model–that is, I want my predictions to be as close as they can be to the actual wages. (In fact we will predict the log of wage.) I observe a representative sample of workers that includes their log wage and years of schooling . I could use an OLS regression of on to make my predictions, but years of schooling are the classic example of an endogenous regressor; they’re correlated with myriad unobserved causes of wages, like “ability” and family background. Fortunately, I also have a valid and relevant instrument: quarter of birth is correlated with years of schooling and (supposedly) uncorrelated with unobserved causes of wage.²

So here’s the question: to get the best possible predictions of wage from the information I have, should I run OLS or IV? More specifically, let’s use mean squared error (MSE) as our measure of “best”. To borrow a term from Grant Sanderson, “pause and ponder” before reading further.

Taking it to the Data

The Angrist & Krueger (1991) dataset is available from Michal Kolesár’s ManyIV R package.³ Here I’ll restrict attention to people born in the first or fourth quarter of the year. The instrument is a dummy variable for being born in the fourth quarter, relative to being born in the first quarter:

# remotes::install_github("kolesarm/ManyIV") # if needed

library(ManyIV) # Contains Angrist & Krueger (1991) dataset

# For information about the dataset, see the package documentation:
# ?ManyIV::ak80

library(dplyr)

dat <- ak80 |> 
  as_tibble() |> 
  filter(qob %in% c('Q1', 'Q4')) |> 
  mutate(z = (qob == 'Q4')) |> 
  select(x = education, y = lwage, z)

To test how well OLS and IV perform as predictors, we’ll carry out a “pseudo-out-of-sample” experiment. First we’ll randomly split dat into a “training” sample containing 80% of the observations and a “test” sample containing the remaining 20%:

set.seed(1693) # For reproducibility

n_total <- nrow(dat) 
n_train <- round(0.8 * n_total) 
n_test <- n_total - n_train 

train_indices <- sample(n_total, n_train, replace = FALSE) 

dat_train <- dat[train_indices, ] 
dat_test <- dat[-train_indices, ] 

Now we’ll use dat_train to fit IV and OLS:

ols_fit <- lm(y ~ x, data = dat_train) 
ols_coefs <- coef(ols_fit)

library(ivreg) # install with `install.packages("ivreg")` if needed 
iv_fit <- ivreg(y ~ x | z, data = dat_train)
iv_coefs <- coef(iv_fit)

rbind(OLS = ols_coefs, IV = iv_coefs)

    (Intercept)          x
OLS    5.004283 0.07008633
IV     4.749959 0.09000644

Now we’re ready to make our predictive comparison! We’ll “pretend” that we don’t know the wages of the people in our test sample and use the OLS and IV coefficients from above to predict the “missing” wages:

dat_test <- dat_test |> 
  mutate(ols_pred = ols_coefs[1] + ols_coefs[2] * x,
         iv_pred = iv_coefs[1] + iv_coefs[2] * x) 

Of course we actually do know the wages of everyone in dat_test; this is the column y. So we can now compare our predictions against the truth.⁴ A common measure of predictive quality is mean squared error (MSE), the average squared difference between the truth and our predictions. Because it squares the difference between the truth and our prediction, MSE penalizes larger errors more than smaller ones. While there are other ways to measure prediction error, MSE is a common choice and one that will play a key role in the rest of this post. And the winner is … OLS! Because it has a lower MSE, the predictions from the OLS model are, on average, closer to the true wages than the predictions from the IV model:

dat_test |> 
  summarize(ols_mse = mean((y - ols_pred)^2),
            iv_mse = mean((y - iv_pred)^2))

# A tibble: 1 × 2
  ols_mse iv_mse
    <dbl>  <dbl>
1   0.407  0.411

OLS beats IV by a small but appreciable margin. (The relatively small difference in this case reflects the fact that IV and OLS estimates are fairly similar in this example.) It turns out that this isn’t a fluke. The same will be true in any example. Unless the instrument is perfectly correlated with the endogenous regressor, OLS will always have a lower predictive MSE than IV.

What’s really going on here?

I ask this question of my introductory econometric students every year and most of them are surprised by the answer. If we have an endogenous regressor OLS is biased and inconsistent; why would we ever pass up the opportunity to use a valid and relevant instrument! The answer is surprisingly simple: by definition the OLS estimand gives the best linear predictor of , the one that minimizes MSE: . This is true regardless of whether is endogenous. Indeed, from a predictive perspective, endogeneity is a feature not a bug! The fact that years of schooling “smuggles in” information about ability and family background is exactly why it gives better predictions than IV. Remember: the whole point of IV is to remove the part of that is related to unobserved causes of . This is exactly what we want if our goal is to understand cause-and-effect, but it’s the opposite of what would make sense in a prediction problem, where we’d like to use as much information as possible.

A Red Herring: The Bias-Variance Tradeoff

Students sometimes answer this question by invoking the bias-variance tradeoff, pointing out that “OLS is biased but has a lower variance than IV, so it could have a lower MSE.” This is correct, but misses the deeper point. They’re thinking about bias in estimating the causal parameter.⁵ But, again, the point here is that this isn’t relevant when prediction is our goal. When ML researchers discuss the bias-variance tradeoff in predictive settings, they mean something entirely different: bias of a linear predictive model relative to the true conditional mean function. OLS gives the best linear approximation to , so it’s what we want in this example, since I stipulated we’d be working with linear models.

Take Home Message

Causal inference and prediction are different goals. Causality is about counterfactuals: what would happen if we intervened to change someone’s years of education? Prediction answers a different question: if I observe that someone has eight years of schooling, what is my best guess of their wage? If you want to predict, use OLS; if you want to estimate a causal effect, use IV.

Warm-up Exercise

This section provides some important background that we’ll need to understand Stein’s Paradox later in the post reviewing the ideas of bias, variance and mean-squared error along with introducing a very simple shrinkage estimator. To make these ideas as transparent as possible we’ll start with a ridiculously simple problem. Suppose that you observe , a single draw from a normal distribution with variance one and unknown mean . Your task is to estimate . This may strike you as a very silly problem: it only involves a single datapoint and we assume the variance of is one! But in fact there’s nothing special about and a variance of one: these merely make the notation simpler. If you prefer, you can think of as the sample mean of iid draws from a population with unknown mean where we’ve rescaled everything to have variance one. So how should we estimate ? A natural and reasonable idea is to use the sample mean, in this case itself. This is in fact the maximum likelihood estimator for , so I’ll define . But is this estimator any good? And can we find something better?

# Range of values for the unknown parameter mu
mu <- seq(-4, 4, length = 100)
# Try three different values of lambda
lambda1 <- 0.1
lambda2 <- 0.2
lambda3 <- 0.3
# Plot the MSE of the shrinkage estimator as a function of mu for all 
# three values of lambda at once
matplot(mu, cbind((1 - lambda1)^2 + lambda1^2 * mu^2, 
                  (1 - lambda2)^2 + lambda2^2 * mu^2, 
                  (1 - lambda3)^2 + lambda3^2 * mu^2), 
        type = 'l', lty = 1, lwd = 2, 
        col = c('red', 'blue', 'green'), 
        xlab = expression(mu), ylab = 'MSE', 
        main = 'MSE of Shrinkage Estimator')
# Add legend
legend('topright', legend = c(expression(lambda == 0.1), 
                              expression(lambda == 0.2), 
                              expression(lambda == 0.3)), 
       col = c('red', 'blue', 'green'), lty = 1, lwd = 2)
# Add dashed line for MSE of ML estimator
abline(h = 1, lty = 2, lwd = 2)

# Plot the MSE of the shrinkage estimator as a function of mu for lambda = 0.3
lambda <- 0.3
plot(mu, (1 - lambda)^2 + lambda^2 * mu^2, type = 'l', lty = 1, lwd = 2, 
     col = 'blue', xlab = expression(mu), ylab = 'MSE', 
     main = 'Boundary of Region Where Shrinkage is Better than ML')
# Add dashed line for MSE of ML estimator
abline(h = 1, lty = 2, lwd = 2)
# Add boundaries of region where shrinkage is better than ML estimator
abline(v = c(sqrt(2/lambda - 1), -sqrt(2/lambda - 1)), lty = 3, lwd = 2,
       col = 'red')

Stein’s Paradox

Where does the James-Stein Estimator Come From?

set.seed(1983)

nreps <- 50
mu1 <- mu2 <- 1
x1 <- mu1 + rnorm(nreps)
x2 <- mu2 + rnorm(nreps)

csq <- mu1^2 + mu2^2
lambda <- csq / (2 + csq)

par(mfrow = c(1, 2))

# Left panel: ML Estimator
plot(x1, x2, main = 'MLE', pch = 20, col = 'black', cex = 2, 
     xlab = expression(mu[1]), ylab = expression(mu[2]))
abline(v = mu1, lty = 1, col = 'red', lwd = 2)
abline(h = mu2, lty = 1, col = 'red', lwd = 2)

# Add MSE to the plot
text(x = 2, y = 3, labels = paste("MSE =", 
                                  round(mean((x1 - mu1)^2 + (x2 - mu2)^2), 2)))

# Right panel: Shrinkage Estimator
plot(x1, x2, main = 'Shrinkage', xlab = expression(mu[1]), 
     ylab = expression(mu[2]))
points(lambda * x1, lambda * x2, pch = 20, col = 'blue', cex = 2)
segments(x0 = x1, y0 = x2, x1 = lambda * x1, y1 = lambda * x2, lty = 2)
abline(v = mu1, lty = 1, col = 'red', lwd = 2)
abline(h = mu2, lty = 1, col = 'red', lwd = 2)
abline(v = 0, lty = 1, lwd = 2)
abline(h = 0, lty = 1, lwd = 2)

# Add MSE to the plot
text(x = 2, y = 3, labels = paste("MSE =", 
                                  round(mean((lambda * x1 - mu1)^2 + 
                                               (lambda * x2 - mu2)^2), 2)))

Conclusion

Before we conclude, there’s one important caveat to bear in mind. In addition to the qualifications that NQ isn’t quite JS, and that JS only dominates the MLE when , there’s one more fundamental issue that could be easily missed. Our decision to minimize composite MSE is absolutely crucial to the reasoning given above. The magic of shrinkage depends on our willingness to accept a trade-off in which we do a worse job estimating one mean in exchange for doing a better job estimating another, as composite MSE imposes. Whether this makes sense in practice depends on the context.

If we’re searching for a lost submarine in the ocean (a 3-dimensional problem), it makes perfect sense to be willing to be farther from the submarine in one dimension in exchange for being closer in another. That’s because Euclidean distance is obviously what we’re after here. But if instead we’re estimating teacher value-added and the results of our estimation exercise will be used to determine which teachers lose their jobs, it’s less clear that we should be willing to be farther from one teacher in exchange for being closer to another. Certainly that would be no consolation to someone who had been wrongly dismissed! If we were merely using this information to identify teachers who might need extra help, it’s another story. But the point I’m trying to make here is that our choice of which criterion to minimize necessarily encodes our values in a particular problem.

But with that said, I hope you’re satisfied that this extremely long post was worth the effort. Without using any fancy mathematics or statistical theory, we’ve managed to invent something that is nearly identical to the James-Stein estimator and thus to resolve Stein’s paradox. We started by pretending what we knew and showed that this would allow us to derive a shrinkage estimator with a lower composite MSE than the ML estimator. Then we simply plugged in an unbiased estimator of the key unknown quantity: . Because all the observations contain information about , it makes sense that we should decide how much to shrink one component by using all of the others. At this point, I hope that the James-Stein estimator seems not only plausible but practically obvious, excepting of course that pesky in the numerator.

A Quick Review

Consider a binary treatment and an observed outcome . Let be the potential outcomes corresponding to the treatment . Our goal is to learn the average treatment effect but, unless is randomly assigned, using the difference of observed means to estimate the ATE in general won’t work. The idea of selection-on-observables is that might be “as good as randomly assigned” after we adjust for a collection of observed covariates .

Regression adjustment relies on two assumptions: selection-on-observables and overlap. The selection-on-observables assumption says that learning provides no additional information about the average values of and , provided that we already know . This implies that we can learn the conditional average treatment effect (CATE) by comparing observed outcomes of the treated and untreated holding fixed: For example: older people might be more likely to take a new medication but also more likely to die without it. If so, perhaps by comparing average outcomes holding age fixed we can learn the causal effect of the medication. The overlap assumption says that, for any fixed value of the covariates, there are some treated and some untreated people. This allows us to learn for every value of in the population and average it using the law of iterated expectations to recover the ATE:
In the medication example, this would correspond to computing the difference of means for each age group separately, and then averaging them using the share of people in each age group. Notice that this is only possible if there are some people who took the medication and some who didn’t in each age group. That’s exactly what the overlap assumption buys us. For example, if there were no senior citizens who didn’t take the medication, we wouldn’t be able to learn the effect of the medication for senior citizens.

Which regression should we run?

So suppose that we’ve found a set of covariates that satisfy the required assumptions. How should we actually carry out regression adjustment? To answer this question, let’s start by making things a bit simpler. Suppose that is a single binary covariate. At the end of the post, we’ll return to the general case. Since and are both binary, we can write the conditional mean function of given as Since the true conditional mean function is linear, a linear regression of on , , and an intercept will recover . But what on earth do these coefficients actually mean?! Substituting all possible values of , And so, after a bit of re-arranging, What a mess! Alas, we’ll need a few more steps of algebra to figure out how these relate to the ATE. Notice that equals the CATE when since Proceeding similarly for the CATE when , we find that Now that we have expressions for each of the two conditional average treatment effects, corresponding to each of the values that can take, we’re finally ready to compute the ATE: where we define the shorthand . So to compute the ATE, we need to know the coefficients and from the regression of on , , and , in addition to the share of people with . Needless to say, your favorite regression package will not spit out the ATE for you if you run the regression from above. And it certainly won’t spit out the standard error! So what can we do besides computing everything by hand?

Two Simple Alternatives

It turns out that there are two simple ways to get the your favorite software package to spit out the ATE for you and associated standard error. Each involves a slight re-parameterization of the conditional mean expression from above. The first one replaces with where and . To see why this works, notice that This works perfectly well, but there’s something about it that offends my sense of order: why subtract the mean from in one place but not in another? If you share my aesthetic sensibilities, then you can feel free to replace that offending with another since where we define . Notice that the only coefficient that changes is the intercept, and we’re typically not interested in this anyway!

What if we ignore the interaction?

Wait a minute, you may be ready to object, when researchers claim to be “adjusting” or “controlling” for in practice, they very rarely include an interaction term between and in their regression! Instead, they just regress on and . What can we say about this approach? To answer this question, let’s continue with our example from above and define the following population linear regression model: where is the population linear regression error term so that, by construction, . Notice that I’ve called the coefficients in this regression rather than . That’s because they will not in general coincide with the conditional mean function from above, namely . In particular, the regression of on and without an interaction will only coincide with the true conditional mean function if .

So what, if anything, can we say about in relation to the ATE? By Yule’s Rule² we have where is the error term from a population linear regression of on . In words, the way that a regression of on and “adjusts” for is by first regressing on , taking the part of that is not correlated with , namely , and regressing on this alone.³ As shown in the appendix to this post, in this example. And since it follows that The only thing that’s random in this expression is . Both expectations involve averaging over its distribution. To make this clearer, define the propensity score . Using this notation, since is binary. Defining , we see that where we introduce the shorthand In other words, the coefficient on in a regression of on and excluding the interaction term gives a weighted average of the conditional average treatment effects for the different values of . The weights are between zero and one and sum to one. Because is increasing in , values of that are more common are given more weight just as they are in the ATE. But since is also increasing in , values of for which is closer to 0.5 are given more weight, unlike in the ATE. As such, we could describe as a variance-weighted average of the conditional average treatment effects.

In general, the weighted average will not coincide with the ATE, although there are two special cases where it will. The first case is when does not depend on , i.e. treatment effects are homogeneous. In this case so there is no interaction term in the conditional mean function! The second is when does not depend on , in which case the probability of treatment does not depend on , so we don’t need to adjust for in the first place!

What about the general case?

All of the above derivations assumed that is one-dimensional and binary. So how much of this still applies more generally? First, if is a vector of binary variables representing categories like sex, race etc., everything goes through exactly as above.⁴ All that changes is that , and become vectors. The coefficient on in a regression of on , and the interaction is still the ATE, and the coefficient on in a regression that excludes the interaction term is still a weighted average of CATEs that does not in general equal the ATE.

So whenever the covariates you need to adjust for are categorical, this post has you covered.⁵ But what if some of our covariates are continuous? In this case things are a bit more complicated, but all of the results from above still go through if we’re willing to assume that the conditional mean functions , and are linear in . This is undoubtedly a strong assumption, but not perhaps as strong as it seems. For example, could include logs, squares or other functions of some underlying continuous covariates, e.g. age or years of experience. In this case, the weighted average interpretation of the coefficient on in a regression that excludes the interaction term still holds but now involves an integral rather than a sum.

Does it really work? An Empirical Example

But perhaps you don’t trust my algebra.⁶ To assuage your fears, let’s take this to the data! The following example is based on Peisakhin & Rozenas (2018) - Electoral Effects of Biased Media: Russian Television in Ukraine. I’ve adapted it from Llaudet and Imai’s fantastic book Data Analysis for Social Science, the perfect holiday or birthday gift for the budding social scientist in your life.

Here’s a bit of background. In the lead-up to Ukraine’s 2014 parliamentary election, Russian state-controlled TV mounted a fierce media campaign against the Ukrainian government. Ukrainians who lived near the border with Russia could potentially receive Russian TV signals. Did receiving these signals cause them to support pro-Russia parties in the election? To answer this question, we’ll use a dataset called precincts that contains aggregate election results in precincts close to the Russian border:

library(tidyverse)
precincts <- read_csv('https://ditraglia.com/data/UA_precincts.csv')

Each row of precincts is an electoral precinct in Ukraine that is near the Russian border. The columns pro_russian and prior_pro_russian give the vote share (in percentage points) of pro-Russian parties in the 2014 and 2012 Ukrainian elections, respectively. Our outcome of interest will be the change in pro-Russian vote share between the two elections, so we first need to construct this:

precincts <- precincts |>
  mutate(change = pro_russian - prior_pro_russian) |> 
  select(-pro_russian, -prior_pro_russian)
precincts

# A tibble: 3,589 × 3
   russian_tv within_25km change
        <dbl>       <dbl>  <dbl>
 1          0           1  -22.4
 2          0           0  -34.5
 3          1           1  -18.8
 4          0           1  -12.2
 5          0           0  -27.7
 6          1           0  -44.2
 7          0           0  -34.5
 8          0           0  -29.5
 9          0           0  -24.1
10          0           0  -25.4
# ℹ 3,579 more rows

The column russian_tv equals 1 if the precinct has Russian TV reception. This is our treatment variable: . But crucially, this is not randomly assigned. While it’s true that there is some natural variation in signal strength that is plausibly independent of other factors related to voting behavior, on average precincts closer to Russia are more likely to receive a signal. So suppose for the sake of argument that conditional on proximity to the Russian border, russian_tv is as good as randomly assigned. This is the selection on observables assumption. There’s no way to check this using our data alone. It’s something we need to justify based on our understanding of the world and the substantive problem at hand.

As our measure of proximity, we’ll use the dummy variable within_25km which equals 1 if the precinct is within 25km of the Russian border. This our -variable. The overlap assumption requires that there are some precincts with Russian TV reception and some without in each distance category. This is an assumption that we can check using the data, so let’s do so before proceeding:

precincts |> 
  group_by(within_25km) |>
  summarize(`share with Russian tv` = mean(russian_tv))

# A tibble: 2 × 2
  within_25km `share with Russian tv`
        <dbl>                   <dbl>
1           0                   0.105
2           1                   0.692

We see that just over 10% of that are not within 25km of the border have Russian TV reception while just under 70% of those within 25km have reception, so overlap is satisfied in this example. Neither of these values is close to 0% or 100%, so this dataset comfortably satisfies the overlap assumption.

To avoid taxing your memory about which variable is which, for the rest of this exercise, I’ll create a new dataset that renames the columns of precincts to D, X, and Y for the treatment, covariate, and outcome, respectively.

dat <- precincts |> 
  rename(D = russian_tv, X = within_25km, Y = change)

# Step 1: compute the mean Y for each combination of (D, X)
means <- dat |> 
  group_by(D, X) |> 
  summarize(Ybar = mean(Y))
means # display the results

# A tibble: 4 × 3
# Groups:   D [2]
      D     X  Ybar
  <dbl> <dbl> <dbl>
1     0     0 -24.6
2     0     1 -34.2
3     1     0 -13.0
4     1     1 -32.2

# Step 2: reshape so the means of Y|D=0,X and Y|D=1,X are in separate cols
means <- means |>
  pivot_wider(names_from = D, 
              values_from = Ybar, 
              names_prefix = 'Ybar')
means # display the results

# A tibble: 2 × 3
      X Ybar0 Ybar1
  <dbl> <dbl> <dbl>
1     0 -24.6 -13.0
2     1 -34.2 -32.2

# Step 3: attach a column with the proportion of X = 0 and X = 1
regression_adjustment <- dat |> 
  group_by(X) |> 
  summarize(count = n()) |> 
  mutate(p = count / sum(count)) |> 
  select(-count) |> 
  left_join(means, by = "X") |> 
  mutate(CATE = Ybar1 - Ybar0) # compute the CATEs
regression_adjustment # display the results

# A tibble: 2 × 5
      X     p Ybar0 Ybar1  CATE
  <dbl> <dbl> <dbl> <dbl> <dbl>
1     0 0.849 -24.6 -13.0 11.6 
2     1 0.151 -34.2 -32.2  2.01

# Step 4: at long last, compute the ATE!
ATE <- regression_adjustment |> 
  mutate(out = (Ybar1 - Ybar0) * p) |> 
  pull(out) |> 
  sum()
ATE

[1] 10.12062

# Construct Xtilde = X - mean(X) 
dat <- dat |> 
  mutate(Xtilde = X - mean(X))

# Regression of Y on D, X, and D*Xtilde
lm(Y ~ D + X + D:Xtilde, dat)

Call:
lm(formula = Y ~ D + X + D:Xtilde, data = dat)

Coefficients:
(Intercept)            D            X     D:Xtilde  
    -24.591       10.121       -9.604       -9.562  

# Regression of Y on D, Xtilde, and D*Xtilde
lm(Y ~ D * Xtilde, dat)

Call:
lm(formula = Y ~ D * Xtilde, data = dat)

Coefficients:
(Intercept)            D       Xtilde     D:Xtilde  
    -26.045       10.121       -9.604       -9.562  

library(estimatr)
library(broom)
lm_robust(Y ~ D * Xtilde, dat) |> 
  tidy() |> 
  filter(term == 'D') |> 
  select(-df, -outcome)

  term estimate std.error statistic      p.value conf.low conf.high
1    D 10.12062 0.4838613  20.91636 9.315921e-92 9.171946  11.06929

# Compute the propensity score pi(X)
pscore <- dat |> 
  group_by(X) |>
  summarize(pi = mean(D))

# Compute the weights w 
regression_adjustment <- left_join(regression_adjustment, pscore, by = "X") |> 
  mutate(w = p * pi * (1 - pi) / sum(p * pi * (1 - pi))) 

regression_adjustment # display the results

# A tibble: 2 × 7
      X     p Ybar0 Ybar1  CATE    pi     w
  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1     0 0.849 -24.6 -13.0 11.6  0.105 0.713
2     1 0.151 -34.2 -32.2  2.01 0.692 0.287

# Compute the variance weighted average of the CATEs
wCATE <- regression_adjustment |> 
  summarize(wCATE = sum(w * CATE)) |> 
  pull(wCATE)

c(wCATE = wCATE, ATE = ATE)

    wCATE       ATE 
 8.822285 10.120617 

lm(Y ~ D + X, dat)

Call:
lm(formula = Y ~ D + X, data = dat)

Coefficients:
(Intercept)            D            X  
    -24.302        8.822      -14.614  

Conclusion

I hope this post has convinced you that regression adjustment isn’t simply a matter of tossing a collection of covariates into your regression! In general, the coefficient on in a regression of on and will not equal the ATE of . Instead it will be a weighted average of CATEs. To obtain the ATE we need to include an interaction between and . The simplest way to get your favorite statistical software package to calculate this for you, along with an appropriate standard error, is by de-meaning before including the interaction. And don’t forget that causal inference always requires untestable assumptions, in this case the selection-on-observables assumption. While implementation details are important, getting them right won’t make any difference if you’re not adjusting for the right covariates in the first place.

Appendix: The Missing Algebra

This section provides the algebra needed to justify the expression for from a regression that omits the interaction between and . In particular, we will show that where is the error term from a population linear regression of on , namely so that by construction. The proof isn’t too difficult, but it’s a bit tedious so I thought you might prefer to skip it on a first reading. Still here? Great! Let’s dive into the algebra.

We need to calculate and . A nice way to carry out this calculation is by applying the law of total covariance. You may have heard of the law of total variance, but in my view the law of total covariance is more useful. Just as you can deduce all the properties of variance from the properties of covariance, using , you can deduce the law of total variance from the law of covariance! In the present example, the law of total covariance allows us to write If this looks intimidating, don’t worry: we’ll break it down piece by piece. The second term on the RHS is a covariance between two random variables: and .⁸ We already have an equation for , namely the population linear regression of on , so let’s use it to simplify : Here’s the key thing to note: since is binary, the population linear regression of on is identical to the conditional mean of given .⁹ This tells us that . Since the covariance of anything with a constant is zero, the second term on the RHS of the law of total covariance drops out, leaving us with Now let’s deal with the conditional covariance inside the expectation. Remember that conditioning on is equivalent to saying “suppose that were known”. Anything that’s known is constant, not random. So we can treat both and as constants and apply the usual rules for covariance to obtain Therefore, . A very similar calculation using the law of total variance gives since and the variance of any constant is simply zero. So, with the help of the laws of total covariance and variance, we’ve established that
in this example. Note that this does not hold in general: it relies on the fact that , which holds in our example because given that is binary.

We’re very nearly finished. All that remains is to simplify the numerator. To do this, we’ll use the equality where satisfies by construction. This allows us to write So what about that pesky term? By the law of iterated expectations this turns out to equal zero, since and, again, by construction. So we’re left with

An Open Invitation

If you’re still hungry for more Bayes’ Theorem after reading this post, then why not join the Summer of Bayes 2024 online reading group? If you’d like to be added to the mailing list, just send an email to bayes [at] user.sent.as. Recordings of past sessions along with slides and other materials are available to group members via the Summer of Bayes discussion board. And now back to your regularly-scheduled blog content…

Odds aren’t so odd!

While I give you a few minutes to pause and ponder this question, here’s a brief rant on the topic of odds. If you’re anything like me, the first time you encountered odds, you thought to yourself

What is this $*@%^!? Why would anyone want to spoil a perfectly good probability by dividing it by one minus itself?“¹

But it’s time to take the red pill and see the world as it really is: the only reason you prefer to think in terms of probabilities rather than odds is because you’ve been brainwashed by the educational system. Of course I exaggerate slightly, but the point is that odds are just as natural as probabilities; we’re just not as accustomed to working with them. In many situations in probability, statistics, and econometrics, it turns out that working with odds (or their logarithm) makes life much simpler, as I will try to convince you with a simple example.

First we need to define odds. Consider some event with probability of occurring. Then we say that the odds of are . For example, if then the event is equivalent to drawing a red ball from an urn that contains one red and two blue balls: the probability gives the ratio of red balls to total balls. The odds of , on the other hand, equal : odds give the ratio of red balls to blue balls. Since probabilities are between 0 and 1, odds are between 0 and . Odds of 0 mean that the event is impossible, while odds of mean that the event is certain. Odds of 1 mean that the event is just as likely to occur as not to occur.

Now here’s an example that you’ve surely seen before:

One in a hundred women has breast cancer . If you have breast cancer, there is a 95% chance that you will test positive ; if you do not have breast cancer , there is a 2% chance that you will nonetheless test positive . We know nothing about Alice other than the fact that she tested positive. How likely is it that she has breast cancer?

It’s easy enough to solve this problem using Bayes’ Theorem, as long as you have pen and paper handy: But what if I asked you how the result would change if only one in a thousand women had breast cancer? What if I changed the sensitivity of the test from 95% to 99% or the specificity from 98% to 95%? If you’re anything like me, you would struggle to do these calculations in your head. That’s because is a highly non-linear function of , , and .

In contrast, working with odds makes this problem a snap. The key point is that and have the same denominator, namely : Notice that was the “complicated” term in ; the numerator was simple. Since the odds of given is defined as the ratio of to , the denominator cancels and we’re left with In other words, the posterior odds of equal the likelihood ratio, , multiplied by the prior odds of , : Now we can easily solve the original problem in our head. The prior odds are 1/99 while the likelihood ratio is 95/2. Rounding these to 0.01 and 50 respectively, we find that the posterior odds are around 1/2. This means that Alice’s chance of having breast cancer is roughly equivalent to the chance of drawing a red ball from an urn with one red and two blue balls. There’s no need to convert this back to a probability since we can already answer the question: it’s considerably more likely that Alice does not have breast cancer. But if you insist, odds of 1/2 give a probability of 1/3, so in spite of rounding and calculating in our heads we’re within 0.3% of the exact answer!

Repeat after me: odds are on a multiplicative scale. This is their key virtue and the reason why they make it so easy to explore variations on the original problem. If one in a thousand women has breast cancer, the prior odds become 1/999 so we simply divide our previous result by 10, giving posterior odds of around 1/20. If we instead changed the sensitivity from 95% to 99% and the specificity from 98% to 95%, then the likelihood ratio would change from to .

The Solution

Have I given you enough time to come up with your own solution? Fantastic! In case you hadn’t already guessed, that little digression about odds served an important purpose: my solution will use odds rather than probabilities. Our goal is to increase the positive predictive value (PPV) of the test, namely by as much as possible, either by improving the test’s sensitivity or its specificity To answer this question, we’ll start by substituting these definitions into the odds form of Bayes’ Theorem introduced above, yielding This expression makes it clear that increasing either the sensitivity or specificity of the test increases the posterior odds. And because the PPV is a strictly increasing function of the posterior odds, namely this also increases the PPV. So now the question is: which of these two possibilities gives us the most bang for our buck? A natural idea would be to compare the marginal effect of increasing sensitivity by a small amount to the marginal effect of increasing specificity by the same amount. We can do this by comparing the partial derivatives of the PPV with respect to sensitivity and specificity. But, again, the PPV is an increasing function of the posterior odds, so we can simplify our task by comparing the derivatives of the posterior odds with respect to sensitivity and specificity. By the chain rule, any claim about the relative magnitudes of these derivatives computed for the odds will also hold for the PPV.

But why stop with the odds? We can simplify our task even further by comparing the derivatives of the logarithm of the posterior odds with respect to sensitivity and specificity. This is because the logarithm is, again, an increasing transformation of the odds. Since our required derivatives are Now for the punchline: the ratio of the derivative with respect to specificity divided by that with respect to sensitivity is and this is precisely the likelihood ratio from the odds form of Bayes’ Theorem! Hence, whenever the likelihood ratio is greater than one we’d prefer to increase the test’s specificity; whenever it’s less than one we’d prefer to increase the sensitivity. If the likelihood ratio is equal to one, then it doesn’t matter which we choose.

Case closed, right? Well not quite. We can say a bit more by thinking about what it means for the likelihood ratio to be greater than or less than one. Examining the odds form of Bayes’ Theorem from above, we see that a likelihood ratio less than one means that our posterior probability that a person is sick falls when she tests positive. In other words, this corresponds to a test that is worse than useless: it’s actually misleading. In contrast, a likelihood ratio greater than one means that the test is informative: a positive test result increases our belief that the person is sick. Any real-world diagnostic test will have a likelihood ratio greater than one. Indeed, if we had such an actively mis-leading test, we could easily convert it into an informative one by simply reversing the test’s outcome: if someone tests positive, we tell them they’re negative, and vice versa. This reversal would result in a likelihood ratio greater than one. Therefore, in all cases–whether we start with an informative test or reverse a misleading one–we should prefer to increase the test’s specificity.

Epilogue

Of course, this exercise is predicated upon the assumption that we want to maximize the PPV and that we can freely adjust both the test’s sensitivity and its specificity. In practice, one or more of these assumptions might not hold. Indeed, PPV is not the be all and end all of diagnostic testing. A full accounting would need to consider the relative costs of false positives and false negatives along with the prevalence of the disease. Still, I hope this exercise gives you a flavor of the power of odds for simplifying complex problems in probability and statistics.

Read Something Else Instead

The first question to ask yourself is whether you should even be reading this paper in the first place. Just because White’s (1980) paper on heteroskedasticity-robust standard errors is a “classic” in econometrics, that doesn’t mean that you should read it. In fact, as a graduate student just starting out, you probably shouldn’t! The paper that introduces a new idea or procedure is rarely the paper that gives the clearest explanation. Reading a good textbook explanation is a much more effective way to get to grips with a new idea. You might, for example, try reading the relevant chapters in White’s textbook Asymptotic Theory for Econometricians instead.

But sometimes you have to read a particular paper. Maybe it’s the paper you’ve been assigned to present in a reading group, or maybe it’s highly relevant to your own research. In that case you may still want to start by reading something else. For example, there might be a more recent paper or review article that gives a good summary of the idea or method in question. Reading this paper first can make it much easier for you to tackle the original paper.

So to all those professors out there who keep telling their students to “read the papers!” I say: “read the papers, but only after you’ve read something else first!”

A Simple Example

To set the stage for Sims & Uhlig (1991), consider the following simple example: where is unknown but is known to equal . Let be the sample mean. Then is an approximate 95% Frequentist confidence interval for . In words: among 95% of the possible datasets that we could potentially observe, the interval will cover the true, unknown value of ; in the remaining of datasets, the interval will not cover .

The Frequentist interval conditions on and treats as random. In contrast, a Bayesian credible interval conditions on and treats as random. This doesn’t require us to believe that is “really” random. Bayesian reasoning simply uses the language of probability to express uncertainty about any quantity that we cannot observe. Let be the observed value of . Under a vague prior for , e.g. a Normal(0, 100) distribution, the 95% Bayesian highest posterior density interval for is approximately . In words: given that we have observed , there is a 95% probability that lies in the interval .

The comforting thing about this example is that, regardless of whether we choose a Bayesian or Frequentist perspective, our inference remains the same: compute the sample mean, then add and subtract . This means that the Frequentist interval inherits all the nice properties of Bayesian inferences, and the Bayesian interval has correct Frequentist coverage. This equivalence between Bayesian and Frequentist methods crops up in many simple examples, especially in situations where the sample size is large. But in more complex settings, the two approaches can give radically different answers. And to head off a common mis-understanding, this isn’t because Bayesians use priors. In the limit as we accumulate more and more data, the influence of the prior wanes. The key difference is that Bayesian inference adheres to the likelihood principle, whereas common Frequentist methods do not.¹

A Not-so-simple Example

Sims & Uhlig consider the AR(1) model and the conditional maximum likelihood estimator given the initial , namely Their simulation contrasts the Frequentist sampling distribution of with the Bayesian posterior distribution of under a flat prior on . When is near one, these two distributions differ markedly: while the Bayesian posterior is always symmetric and centered at , the Frequentist sampling distribution is highly skewed when is close to one. This shows that the Bayesian-Frequentist equivalence we found in our simple population mean example from above breaks down completely in this more complex example.

Sims & Uhlig argue that the Bayesian posterior provides a much more sensible and useful characterization of the information contained in the data and after reading the paper, I’m inclined to agree. My replication code follows below, along with plots of the joint distribution of under a uniform prior for and the conditional distributions (Frequentist Sampling Distribution) and (Bayesian Posterior).²

The Replication

#-------------------------------------------------------------------------------
# Sims, C. A., & Uhlig, H. (1991). Understanding unit rooters: A helicopter tour
#
# (See also: Example 6.10.6 from Poirier "Intermediate Statistics and 'Metrics")
#-------------------------------------------------------------------------------
# In the next section we will proceed to construct, by Monte Carlo, an estimated
# joint pdf for \rho and \hat{\rho} under a uniform prior pdf on \rho. We choose
# 31 values of \rho, from 0.8 to 1.1 at intervals of 0.01. We draw 10000 100 x 1
# iid N(0,1) vectors of random variables to use as realizations of \epsilon. For
# each of the 10000 \epsilon vectors and each of the 31 \rho values, we
# construct a y vector with y(0) = 0, y(t) generated by equation (1).
#
# Equation (1): y(t) = \rho y(t-1) + \epsilon(t), t = 0, ..., T
#
# For each of these y vectors, we construct \hat{\rho}. Using as bins the
# intervals [-\infty, 0.795), [0.795, 0.805), [0.805, 0.815), etc. we construct
# a histogram that estimates the pdf of \hat{rho} for each fixed \rho value.
# When these histograms are lined up side by side, they form a surface that is
# the joint pdf for \rho and \hat{\rho} under a flat prior on \rho.
#-------------------------------------------------------------------------------
set.seed(1693)

library(tidyverse)
library(tictoc)
library(patchwork)

draw_rho_hat <- function(rho) {
# Carry out the simulation once for a fixed value of rho; return rho_hat
  nT <- 100
  y <- rep(0, nT + 1)
  for (t in 2:(nT + 1)) {
    y[t] <- rho * y[t - 1] + rnorm(1)
  }
  y_t <- y[-1]
  y_tminus1 <- y[-length(y)]
  sum(y_t * y_tminus1) / sum(y_tminus1^2)
}

# Function to run the simulation for a fixed value of rho (10000 times)
run_sim <- \(rho) map_dbl(1:1e4, \(i) draw_rho_hat(rho))

tic()
foo <- run_sim(0.9)
toc() # ~0.6 seconds on my machine

0.368 sec elapsed

# Full sequence of rho values from Sims & Uhlig (1991)
rho <- seq(from = 0.8, to = 1.1, by = 0.01)

tic()
results <- tibble(rho = rho,
                  rho_hat = map(rho, run_sim)) # List columns
toc() # ~17 seconds on my machine (1991 was a long time ago!)

11.287 sec elapsed

# The results tibble uses a list column for rho_hat. This is convenient for
# making histograms of the frequentist sampling distribution (rho fixed) but
# not for making histograms of the Bayesian posterior (rho_hat) fixed. For the
# latter, we will use the unnest() function to "expand" the list column rho_hat
# into a regular column. This is the "joint" distribution of rho and rho_hat.
joint <- results |>
  unnest(rho_hat)

joint |>
  ggplot(aes(x = rho, y = rho_hat)) +
  geom_density2d_filled() +
  coord_cartesian(ylim = c(0.8, 1.1)) + # Restrict rho_hat axis
  labs(title = "Joint Distribution",
       x = expression(rho),
       y = expression(hat(rho))) +
  theme_minimal()

joint |>
  filter(rho_hat >= 0.995 & rho_hat < 1.005) |>
  ggplot(aes(x = rho)) +
  geom_histogram(binwidth = 0.01, fill = "skyblue", color = "black") +
  labs(title = expression(hat(rho) == 1),
       x = expression(rho),
       y = "Frequency") +
  theme_minimal()

joint |>
  filter(rho == 1) |>
  ggplot(aes(x = rho_hat)) +
  geom_histogram(binwidth = 0.01, fill = "skyblue", color = "black") +
  labs(title = expression(rho == 1),
       x = expression(hat(rho)),
       y = "Frequency") +
  theme_minimal()

# Function that makes the preceding two plots, puts them side-by-side and lets
# the user specify the value of rho/rho_hat that we condition on:
plot_Bayes_vs_Freq <- \(r) {
  p1 <- joint |>
    filter(rho_hat >= r - 0.005 & rho_hat < r + 0.005) |>
    ggplot(aes(x = rho)) +
    geom_histogram(aes(y = after_stat(density)),
                   binwidth = 0.01, fill = "skyblue", color = "black") +
    geom_vline(xintercept = r, color = "red", linetype = "dashed", linewidth = 1) +
    labs(title = bquote(hat(rho) == .(round(r, 3))),
         x = expression(rho)) +
    theme_minimal()

  p2 <- joint |>
    filter(rho >= r - 0.005 & rho < r + 0.005) |>
    ggplot(aes(x = rho_hat)) +
    geom_histogram(aes(y = after_stat(density)),
                   binwidth = 0.01, fill = "skyblue", color = "black") +
    geom_vline(xintercept = r, color = "red", linetype = "dashed", linewidth = 1) +
    labs(title = bquote(rho == .(round(r, 3))),
         x = expression(hat(rho))) +
    theme_minimal()

  p1 + p2
}

plot_Bayes_vs_Freq(0.98)

plot_Bayes_vs_Freq(0.99)

plot_Bayes_vs_Freq(1.0)

plot_Bayes_vs_Freq(1.01)

plot_Bayes_vs_Freq(1.02)

The Model

We’ll start by being a bit more precise about the setup. Suppose that is related to according to the following linear causal model where is the causal effect of interest and represents unobserved causes of that may be related to . Now, for any observed random variable , we can define This is the population linear regression of on . By construction it satisfies .² Thus we can write, for any random variables and , simply by constructing as described above. If , we say that is relevant. If , we say that is exogenous. If is both relevant and exogenous, we say that it is a valid instrument for .

As we’ve defined it above, is simply a regression residual. But if is a valid instrument, it turns out that we can think of as the “endogenous part” of . To see why, expand as follows: since we have assumed that . In words, the endogeneity of is precisely the same thing as the covariance between and .

Here’s a helpful way of thinking about this. If is exogenous then our regression of on partitions the overall variation in into two components: the “good” (exogenous) variation is uncorrelated with , while the “bad” (endogenous) variation is correlated with . The logic of two-stage least squares is that regressing on the “good” variation, allows us to recover , the causal effect of interest.³

A Simulation Example

Using the model and derivations from above, let’s run a little simulation. To simulate a valid instrument and an endogenous regressor we can proceed as follows. First generate independent standard normal draws . Next independently generate pairs of correlated standard normal draws with . Finally, set for each value of between and .⁴ The following chunk of R code runs this simulation with , , , , and :

set.seed(1234)
n <- 5000
z <- rnorm(n)

library(mvtnorm)
Rho <- matrix(c(1, 0.5, 
                0.5, 1), 2, 2, byrow = TRUE)
errors <- rmvnorm(n, sigma = Rho)

u <- errors[, 1]
v <- errors[, 2]
x <- 0.5 + 0.8 * z + v
y <- -0.3 + x + u

In the simulation is a valid instrument, is an endogenous regressor, and the true causal effect of interest equals one. Using our simulation data, let’s test out three possible estimators:

• the slope coefficient from an OLS regression of on .
• slope coefficient from an IV regression of on with as an instrument.
• the coefficient on in an OLS regression of on and .

c(truth = 1,
  b_OLS = cov(x, y) / var(x), 
  b_IV = cov(z, y) / cov(z, x), 
  b_x.z = unname(coef(lm(y ~ x + z))[2])) |> # unname() makes the names prettier!
  round(2)

truth b_OLS  b_IV b_x.z 
 1.00  1.31  1.01  1.49 

As expected, OLS is far from the truth while IV pretty much nails it. Interestingly, the regression of y on x and z gives the worst performance of all! Is this just a fluke? Perhaps it’s an artifact of the simulation parameters I chose, or just bad luck arising from some unusual simulation draws. To find out, we’ll need a bit more algebra. But stay with me: the payoff is worth it, and there’s not too much extra math required!

The General Result

Some Intuition

In our simulation, gave a worse estimate of than . The derivations from above show that this wasn’t a fluke: adding a valid instrument as an additional control regressor only makes the bias in our estimated causal effect worse than it was to begin with. This holds for any valid instrument and any endogenous regressor in a linear causal model. I hope you found the derivations from above convincing. But even so, you may be wondering if there’s an intuitive explanation for this phenomenon. I am pleased to inform you that the answer is yes!

In an earlier post I described the control function approach to instrumental variables regression. That post showed that the coefficient on in a regression of on and gives the correct causal effect. We don’t know , but we can estimate it by regressing on and saving the residuals. The logic of multiple regression shows that including as a control regressor “soaks up” the portion of that is explained by . Because represents the “bad” (endogenous) variation in , this solves our endogeneity problem. In effect, captures the unobserved “omitted variables” that play havoc with a naive regression of on .

Now, contrast this with a regression of on and . In this case, we soak up the variation in that is explained by . But represents the good (exogenous) variation in ! Soaking up this variation leaves only the bad variation behind, making our endogeneity problem worse than it was to begin with. In this example, is what is known as a bad control, a control regressor that makes things worse rather than better. A common piece of advice for avoiding bad controls is to only include control regressors that are correlated with and but are not themselves caused by . The example in this post shows that this advice is wrong. Here is not caused by , and is correlated with both and . Nevertheless, it is a bad control. In short, a valid instrument provides a powerful way to carry out causal inference from observational data, but only if you use it in the right way. A good instrument is a bad control!

Prerequisites

While written at an introductory level, this post assumes basic familiarity with calculations involving discrete and continuous random variables. In particular, I assume that:

• You know the definitions of expected value, variance, covariance, and correlation.
• You are comfortable working with joint, marginal, and conditional distributions of a pair of discrete random variables.
• You understand the uniform distribution and how to compute its moments (mean, variance, etc.).
• You’ve encountered the notion of conditional expectation and the law of iterated expectations.

If you’re a bit rusty on this material, lectures 7-11 from these slides should be helpful. For bivariate, discrete distributions I also suggest watching this video from 1:07:00 to the end and this other video from 0:00:00 up to the one hour mark.

Two Examples

Example #1 - Discrete RVs

My first example involves two discrete random variables and with joint probability mass function given by

Even without doing any math, we see that knowing conveys information about , and vice-versa. For example, if then we know that must equal zero. Similarly, if then must equal zero. Spend a bit of time thinking about this joint distribution before reading further. We’ll have plenty of time for mathematics below, but it’s always worth seeing where our intuition takes us before calculating everything.

To streamline our discussion below, it will be helpful to work out a few basic results about and . A quick calculation with shows that Calculating the marginal pmfs for we see that Similarly, calculating the marginal pmf of , we obtain We’ll use these results as ingredients below as we explain and relate three key notions of dependence: correlation, conditional mean independence, and statistical independence.

Uncorrelatedness

Recall that the correlation between two random variables and is defined as where and . We say that and are uncorrelated if . Unless and are both constants their variances must be positive. This means that the denominator of our expression for is likewise positive. It follows that zero correlation is the same thing as zero covariance. Correlation is simply covariance rescaled so that the units of and cancel out and the result always lies between and .

Correlation and covariance are both measures of linear dependence. If is, on average, above its mean when is above its mean, then and are both positive. If is, on average, below its mean when is above its mean, then and are both negative. If there is, on average, no linear relationship between and , then both the correlation and covariance between them are zero. Using the “shortcut formula” for covariance, namely it follows that uncorrelatedness is equivalent to Rendering this in English rather than mathematics,

Two random variables and are uncorrelated if and only if the expectation of their product equals the product of their expectations.

set.seed(1983)
n_sims <- 250
w <- runif(n_sims, -1, 1)
z <- w^2
cor(w, z)

[1] 0.008379101

plot(w, z)
abline(lm(z ~ w))

Conditional Mean Independence

We say that is mean independent of if . In words,

is mean independent of if the conditional mean of given equals the unconditional mean of .

Just to make things confusing, this property is sometimes called “conditional mean independence” and sometimes called simply “mean independence.” The terms are completely interchangeable. Reversing the roles of and , we say that is mean independent of if the conditional mean of given is the same as the unconditional mean of . Spoiler alert: it is possible for to be mean independent of while is not mean independent of . We’ll discuss this further below.

To better understand the concept of mean independence, let’s quickly review the difference between an unconditional mean and a conditional mean. The unconditional mean , also known as the “expected value” or “expectation” of , is a constant number.³ If is discrete, this is simply the probability-weighted average of all possible realizations of , namely If is continuous, it’s the same idea but with an integral replacing the sum and a probability density multiplied by replacing the probability mass function . Either way, we’re simply multiplying numbers together and adding up the result. Despite the similarity in notation, the conditional expectation is a function of that tells us how the mean of varies with . Since is a random variable, so is . If is conditionally mean independent of then equals . In words, the mean of does not vary with . Regardless of the value that takes on, the mean of is the same: .

There’s another way to think about this property in terms of prediction. With a bit of calculus, we can show that solves the following optimization problem: In other words, is the constant number that is as close as possible to on average, where “close” is measured by squared euclidean distance. In this sense, we can think of as our “best guess” of the value that will take. Again using a bit of calculus, it turns out that solves the following optimization problem: (See this video for a proof.) Thus, is the function of that is as close as possible to on average, where “close” is measured using squared Euclidean distance. Thus, is our “best guess” of after observing . We have seen that and are the solutions to two related but distinct optimization problems; the former is a constant number that doesn’t depend on the realization of whereas the latter is a function of . Mean independence is the special case in which the solutions to the two optimization problems coincide: . Therefore,

is mean independent of if our best guess of taking into account is the same as our best guess of ignoring , where “best” is defined by “minimizes average squared distance to .”

Statistical Independence

When you see the word “independent” without any qualification, this means “statistically independent.” In keeping with this usage, I often write “independent” rather than “statistically independent.” Whichever terminology you prefer, there are three equivalent ways of defining this idea:

and are statistically independent if and only if:

1. their joint distribution equals the product of their marginals, or
2. the conditional distribution of equals the unconditional distribution of , or
3. the conditional distribution of equals the unconditional distribution of .

The link between these three alternatives is the definition of conditional probability. Suppose that and are discrete random variables with joint pmf , marginal pmfs and , and conditional pmfs and . Version 1 requires that for all realizations . But by the definition of conditional probability, If , these expressions simplify to so 1 implies 2 and 3. Similarly, if then by the definition of conditional probability Re-arranging, this shows that , so 3 implies 1. An almost identical argument shows that 2 implies 1, completing our proof that these three seemingly different definitions of statistical independence are equivalent. If and are continuous, the idea is the same but with densities replacing probability mass functions, e.g.  and so on.

In most examples, it’s easier to show independence (or the lack thereof) using 2 or 3 rather than 1. These latter two definitions are also more intuitively appealing. To say that the conditional distribution of is the same as the unconditional distribution of is the same thing as saying that

provides absolutely no information about whatsoever.

If learning tells us anything at all about , then and are not independent. Similarly, if tells us anything about at all, then and are not independent.

Relating the Three Properties

Now that we’ve described uncorrelatedness, mean independence, and statistical independence, we’re ready to see how these properties relate to one another. Let’s start by reviewing what we learned from the examples given above. In example #1:

• and are uncorrelated
• is mean independent of
• is not mean independent of
• and are not independent.

In example #2, we found that

• and are uncorrelated
• is mean independent of .
• is not mean independent of .
• and are not independent.

These are worth remembering, because they are relatively simple and provide a source of counterexamples to help you avoid making tempting but incorrect statements about correlation, mean independence, and statistical independence. For example:

1. Uncorrelatedness does NOT IMPLY statistical independence: and are not independent, but they are uncorrelated. (Ditto for and .)
2. Mean independence does NOT IMPLY statistical independence: is mean independent of but these random variables are not independent.
3. Mean independence is NOT SYMMETRIC: is mean independent of , but is not mean independent of .

Now that we have a handle on what’s not true, let’s see what can be said about correlation, mean independence, and statistical independence.

Summary

In this post we have shown that:

• Statistical Independence Mean Independence Uncorrelatedness.
• Uncorrelatedness does not imply mean independence or statistical independence.
• Mean independence does not imply statistical independence.
• Statistical independence and correlation are symmetric; mean independence is not.

Reading the figure from the very beginning of this post from top to bottom: statistical independence is the strongest notion, followed by mean independence, followed by uncorrelatedness.

Numerical problems? Try taking logs.

Now, suppose that we wanted to plot the pmf of a Poisson RV with rate . The R function for the pmf of a Poisson RV is dpois(), so we can make our plot as follows (indicating the rate parameter as a vertical line)

library(tidyverse)
tibble(x = 0:300) |>
  mutate(p = dpois(x, 171)) |>
  ggplot(aes(x, p)) +
  geom_point() +
  geom_vline(xintercept = 171) +
  ylab('Poisson(171) pmf')

For such a large value of , this distribution looks decidedly bell-shaped. And indeed, it turns out to be extremely well-approximated by a normal distribution, as we’ll see below. It’s also clear that is most likely to take on a value relatively close to 171. We can use dpois() to calculate the exact probability that as follows: the answer is just over 3%.

dpois(171, 171)

[1] 0.03049301

Now let’s try to calculate exactly the same probability by hand, that is by using the formula for the Poisson pmf from above.

my_dpois <- function(x, mu) {
  exp(-mu) * mu^x / factorial(x)
}
my_dpois(171, 171)

[1] NaN

What gives?! The abbreviation NaN stands for “not a number.” The problem in this case is that both the numerator and denominator of the fraction inside of my_dpois() evaluate to infinity when mu and x are 171, and the ratio is undefined.¹

c(numerator = exp(-171) * 171^171, denominator = factorial(171))

  numerator denominator 
        Inf         Inf 

As I discussed in an earlier post, computers can only store a finite number of distinct numeric values. It’s not literally true that factorial(171) equals . What’s really going on here is that factorial(171) is such a large number that it can’t be stored as a floating-point number. In this case there’s a very simple fix. If you haven’t seen this trick before, it’s a helpful one to keep up your sleeves: if you run into numerical problems with very large or very small values, try taking logs.² The log of the Poisson pmf is simply R even has a convenient, built-in function for evaluating the natural log of a factorial: lfactorial(). Now we can compute the log of our desired probability as follows:

-171 + 171 * log(171) - lfactorial(171)

[1] -3.490258

To obtain the probability, simply exponentiate:

exp(-171 + 171 * log(171) - lfactorial(171))

[1] 0.03049301

Of course this just passes the buck to lfactorial(). So how does this mysterious function work? The bad news is that I’m not going to tell you; the good news is that I’m going to show you something even better, namely Stirling’s approximation: a way to understand how behaves qualitatively that turns out to give a pretty darned good approximation to lfactorial(). This may seem like an odd topic for a blog devoted to econometrics and statistics, so allow me to offer a few words of justification. First, computations involving come up all the time in applied work. Second, it can be extremely helpful for certain theoretical arguments to have good approximations to for large values of . Finally, and most importantly from my perspective, the heuristic argument I’ll use below relies on none other than the central limit theorem. So even if you’ve seen a more traditional proof of Stirling’s approximation, I hope you’ll enjoy this alternative approach.³

Stirling’s Approximation

The key step in our argument is to show that the pmf of a random variable is well-approximated by the density. This explains the bell-shaped curve that we plotted above. To obtain this result, we’ll use the central limit theorem. But there is one fact that you will need to take on faith if you don’t already know it: if is independent of then . Proceeding by induction we can view a Poisson(171) random variable as the sum of 171 independent Poisson(1) random variables. More generally, we can view a Poisson RV with rate parameter as the sum of iid Poisson(1) random variables. By the central limit theorem, it follows that since the mean and variance of a Poisson(1) RV are both equal to one. From a practical perspective, this means that is approximately equal to , a standard normal random variable. Re-arranging, and is simply a random variable! This is a quick way of seeing why the distribution is well-approximated by the distribution when is large.

Now let’s run with this. As we just saw, for large the Poisson pmf is well-approximated by the Normal density: This approximation is particularly accurate for near the mean. This is convenient, because substituting for considerably simplifies the right hand side: Re-arranging, we obtain Taking logs of both sides gives: Writing this with in place of gives the following: This is called Stirling’s Approximation. The usual way of writing this excludes the term, yielding , which is fairly easy to remember. Including the extra term, however, gives increased accuracy for smaller values of . While I haven’t formally proved this, it turns out that as . In other words, the ratio of the LHS and RHS tends to one in the large limit. Perhaps surprisingly, this approximation is extremely accurate even for fairly small values of , as we can see by comparing it against lfactorial().

stirling1 <- function(n) n * log(n) - n
stirling2 <- function(n) n * log(n) - n + 0.5 * log(2 * pi * n)
tibble(n = 1:20) |>
  mutate(Stirling1 = stirling1(n),
         Stirling2 = stirling2(n),
         R = lfactorial(n)) |>
  knitr::kable(digits = 3)

Epilogue

I have a bad habit of trying to add a “moral” or “lesson” to the end of my posts, but I suppose there’s no point trying to break the habit today! While there are easier ways to derive Stirling’s approximation, there are two things I enjoy about this one. First, we get a more accurate approximation than with practically no effort. Second, making unexpected connections between facts that we already know both deepens our understanding and helps us “compress” information. If you ever forget Stirling’s approximation, now you know how to very quickly re-derive it on the spot!