1 Complex Numbers in R

R provides basic functionality for working with complex numbers, documented in the helpful ?complex. The imaginary unit in R is denoted by 1i. Don’t try i by itself because that won’t work:

i # this doesn't work

Error:
! object 'i' not found

1i # this works

[1] 0+1i

To write a complex number, simply use + to separate the real and imaginary parts, e.g.

3 + 4i

[1] 3+4i

A common error when learning R is to write code with “implied multiplication” e.g. 4x (wrong) rather than 4 * x (right). But don’t be tempted to try 4 * i since, again, i is not the imaginary unit in R:

4 * i # this doesn't work 

Error:
! object 'i' not found

4 * 1i # this works

[1] 0+4i

To demonstrate R’s basic functionality for working with complex numbers, let’s specify a complex number in Cartesian coordinates: . The real part is , the imaginary part is , the modulus is , and the complex conjugate is . All of these operations are available in R:

z <- 3 + 4i
Re(z)

[1] 3

Im(z)

[1] 4

Mod(z)

[1] 5

abs(z) # same as Mod(z) for a complex number z

[1] 5

Conj(z)

[1] 3-4i

All of the basic operations +, -, *, /, and ^ work on complex numbers and are vectorized

w <- (-3) - 1i
z + w

[1] 0+3i

z - w

[1] 6+5i

z * w

[1] -5-15i

z / w

[1] -1.3-0.9i

z^3

[1] -117+44i

as are functions such as log(), exp(), sin(), cos(), etc. If you supply them a complex input, they will return a complex output; if you supply them a real input, they will return a real output. This behavior explains a common gotcha: sqrt(-1) doesn’t work because there’s no real number whose square root is -1 but sqrt(-1 + 0i) does work:

sqrt(-1) # this doesn't work

Warning in sqrt(-1): NaNs produced

[1] NaN

sqrt(-1 + 0i)  # this works

[1] 0+1i

Another common gotcha is trying to apply functions like max() and min() or operators like < and > to complex numbers. Since complex numbers aren’t ordered, this doesn’t work:

min(c(z, w))

Error in `min()`:
! invalid 'type' (complex) of argument

z < w

Error in `z < w`:
! invalid comparison with complex values

but testing for equality / non-equality does work:

z == w

[1] FALSE

z != w

[1] TRUE

2 Complex Step Differentiation in Action

Let’s test out complex step differentiation with a simple example. Let and suppose we want to compute . This one is easy to compute analytically: so we can check how accurate different numerical approximations turn out to be. Computing the derivative directly gives

direct <- 4.5 * (1.5)^(3.5)
direct

[1] 18.60081

Now we’ll compare this value against three numerical derivatives: the “simple difference” approach, the “symmetric difference” approach, and the complex step approach.

f <- function(x) x^(4.5) # the function to differentiate

x <- 1.5 # the point where we'll evaluate f'(x)

Since R functions are vectorized, we can evaluate the quality of each approximation over many values of at once by setting up a vector of progressively smaller positive values:

delta <- 1 / 10^(1:15)

simple <- (f(x + delta) - f(x)) / delta

symmetric <- (f(x + delta) - f(x - delta)) / (2 * delta)

complex_step <- Im(f(x + delta * 1i)) / delta

Now we’ll make two tables: the first containing raw results—the approximate value of —and the second containing the relative error of the approximation in percentage points. When reading the first table, recall from above that direct calculation gives .

cbind(delta, simple, symmetric, complex_step) |> 
  knitr::kable(digits = 15)

get_rel_error <- function(x, truth) 100 * abs(x - truth) / abs(truth)

cbind(
  delta,
  simple = get_rel_error(simple, direct),
  symmetric = get_rel_error(symmetric, direct),
  complex_step = get_rel_error(complex_step, direct)
) |> 
  knitr::kable(digits = c(15, rep(4, 3)))

The simple difference approach is clearly less accurate than both the symmetric difference and complex step approach, particularly at larger values of . And while it may seem that there’s not much to choose between these latter two approximations, look carefully at what happens as gets smaller. Eventually the relative error of the simple and symmetric difference approaches starts to increase! For example, the symmetric difference approximation is better with (relative error ) than it is at (relative error ).

How can this be? The derivative is defined as the limit of the simple difference approximation as . So how can the approximation get worse if becomes smaller?

3 Catastrophic Cancellation

The culprit is catastrophic cancellation. As explained in an earlier post, computers cannot represent all real numbers exactly. Instead they use floating point numbers as an approximation. So when we compute a difference like what we’re really computing is the difference of two approximations. The problem is that even when is a very good approximation to and is a very good approximation to , the difference may be a poor approximation to . This is an unfortunate property of subtraction, nicely illustrated in this post by John D. Cook.

A double in R is accurate to around 16 decimal places. For small values of , and agree in nearly all of those digits. Subtracting them eliminates the digits that match, leaving behind what is effectively rounding noise. Dividing by then magnifies this noise. For example, since in our example is around when we compute we should obtain a result of approximately but instead we get

sprintf("%.20f", f(x + 1e-15) - f(x))

[1] "0.00000000000002042810"

The first couple of digits are in the ballpark, but the remaining ones are just noise.
The symmetric difference formula has the same problem because it too relies on a subtraction of two approximate values in the numerator:

sprintf("%.20f", f(x + 1e-15) - f(x - 1e-15))

[1] "0.00000000000004085621"

Here we’d expect a result of around and, again, every digit after the first two is just noise.

In contrast, complex step differentiation has no subtraction in the numerator. This means that it is not subject to catastrophic cancellation. As becomes smaller, the approximation just keeps improving—down to roughly machine precision. The cost is that must be analytic at and evaluable at complex inputs. Most smooth functions you’ll encounter qualify, but those involving , , , or indicator functions generally don’t.

4 Don’t Roll Your Own

In this post I computed the complex step derivative by hand. That’s useful for understanding the method, but it’s a bad idea in practice. Whenever you can, you should rely on high-quality existing libraries to implement numerical methods. Numerical analysis is a deeply complicated subject and we are but lowly econometricians! Fortunately for us, the grad() function from the numDeriv R package implements complex step differentiation as one of its three methods:

library(numDeriv)
grad(f, 1.5, method = 'complex')

[1] 18.60081

So the next time you need to differentiate something numerically, remember the value in making things more complex than they need to be!

1 A Word About Notation

While I phrased my first post about overlapping intervals in terms of sample and population means for two groups, the idea is general. Nothing substantive changes if we replace the parameters with , the estimators with , and the standard errors with . As long as the two estimators are uncorrelated and (approximately) normally distributed, the results from Part I apply to the individual intervals for and versus the interval for the difference . Today we will ask what happens when the estimators are potentially correlated. To make it clear that our results are general, I’ll use the more agnostic notation throughout.

2 A Motivating Example

Consider a randomized controlled trial with two active treatments. In our paper on pawn lending, for example, my co-authors and I compare default rates between borrowers assigned to the status quo pawn contract (control), a new structured contract (Treatment A), and a choice arm (Treatment B) in which they were free to choose whichever contract they preferred. To learn the causal effect of the structured contract, we compare the mean default rates of borrowers who received Treatment A against the corresponding rate in the control group. Call this difference of means . Similarly, to learn the effect of choice, we make the analogous comparison between Treatment B and the control group. Call this difference of means . Now suppose you’re reading a paper that reports both of these estimators and their standard errors. To find out which treatment is more effective, you need to compare and , but these estimators must be correlated because they both involve the control group average: Granted, if you had access to the raw data, you could easily solve this problem without worrying about the correlation. In the difference , the control group mean cancels out So if we had the raw data for treatments A and B we’d be back to a familiar independent samples comparison of means, as in Part I. But if you’re reading a paper that only reports and their standard errors, you cannot directly calculate the standard error for the difference. The common variation from the control group mean is baked into the way both and are computed, even though this variation is irrelevant for .

6 The Grand Finale

Since we’ve equated , and with the sides of a triangle, the Triangle Inequality gives assuming that . And now we’re on familiar ground. Let be the appropriate quantile of a normal distribution, i.e.  for a 95% confidence interval. Just as we argued in Part I, the individual confidence intervals for and overlap precisely when and there is a significant difference between the two parameters when . The condition for overlap and a significant difference is which holds by the Triangle Inequality. So we’re back to exactly the same situation we were in when ! As long as the same results concerning confidence interval overlap apply when and are correlated as when they are uncorrelated:

1. Overlap doesn’t tell us anything about whether there is a significant difference, but
2. a lack of overlap implies that there must be a significant difference.

3 Case II: Intervals for A and B that Don’t Overlap

Now for the easy one. Since holds if and only if the intervals overlap, we merely reverse the inequality to get a condition for intervals that do not overlap, namely Appending what we learned above from our right triangle diagram gives So if the intervals do not overlap, then must be greater than , which is precisely the condition for a significant difference between the two groups.

4 In Summary

In an independent samples problem where the two individual confidence intervals overlap, there may or may not be a significant difference between the groups. Even if the two intervals both contain zero, there could still be a significant difference between them. If the two intervals do not overlap then we can conclude that there is a significant difference. But if you only take one lesson away from this post it should be this one: the difference between significant and not significant is not itself significant. If you want to carry out inference for a difference, you need to construct the standard error for that difference.

2 What about ?

So if soaks up the good variation in , what about the coefficient on the instrument ? Perhaps this coefficient contains some useful information about the causal effect of on ? To find out, we’ll use the FWL Theorem as follows: where is the population linear regression of on . This is the reverse of the first-stage regression of on described above. Here the error term satisfies by construction. Substituting the causal model gives since by construction. Now, substituting the definition of , since by assumption. We can already see that is not going to help us learn about . First of all, the term containing vanished; second of all, the term that remained is polluted by the endogeneity of , namely .

Still, let’s see if we can get a clean expression for . So far we have calculated the numerator of the FWL expression, showing that . The next step is to calculate : Since , our expression for simplifies to so we have discovered that:

Call me old-fashioned, but I really don’t like having in that expression. I’d feel much happier if we could find a way to re-write this in terms of the more familiar IV first-stage coefficient . Let’s give it a try! Let’s use my favorite trick of multiplying by one: Substituting for gives We can simplify this even further by substituting from above to obtain And now we recognize something from above: was the bias of relative to the true causal effect ! This means we can also write .

3 A Little Simulation

We seem to be doing an awful lot of algebra on this blog lately. To make sure that we haven’t made any silly mistakes, let’s check our work using a little simulation experiment taken from my earlier post. Spoiler alert: everything checks out!

set.seed(1234)
n <- 1e5

# Simulate instrument (z)
z <- rnorm(n)

# Simulate error terms (u, v)
library(mvtnorm)
Rho <- matrix(c(1, 0.5, 
                0.5, 1), 2, 2, byrow = TRUE)
errors <- rmvnorm(n, sigma = Rho)

# Simulate linear causal model
u <- errors[, 1]
v <- errors[, 2]
x <- 0.5 + 0.8 * z + v
y <- -0.3 + x + u

# Regression of y on x and z
gamma <- lm(y ~ x + z) |> 
  coefficients()

gamma

(Intercept)           x           z 
 -0.5471213   1.5018705  -0.3981116 

# First-stage regression of x on z
pi <- lm(x ~ z) |> 
  coefficients()

pi

(Intercept)           z 
  0.5020338   0.7963889 

# Compare two different expressions for gamma_Z to the estimate itself
c(gamma_z = unname(gamma[3]),
  version1 = unname(-0.8 * cov(x, u) / var(v)),
  version2 = unname(-pi[2] * (gamma[2] - 1))
)

   gamma_z   version1   version2 
-0.3981116 -0.4024918 -0.3996841 

5 Epilogue

When I showed this post to another colleague he asked me whether there is any way to learn about by combining and . The answer is no: the regression of on and alone doesn’t contain enough information. Since we can rearrange to obtain the following expression for : which we can verify in our little simulation example as follows:

gamma[2] + gamma[3]/pi[2]

       x 
1.001975 

Thus, in order to solve for , we need to run the first-stage regression to learn .

1 Simulation Example

Let’s start with a little simulation. First we’ll generate 5000 observations of predictors and from a joint normal distribution with standard deviations of one, means of zero, and a correlation of 0.5.

set.seed(1066)
library(mvtnorm)

# Simulate linear regression with two predictors: X and W
covariance_matrix <- matrix(
  c(1, 0.5, 0.5, 1), 
  nrow = 2
)

n_sims <- 5000

x_w <- rmvnorm(
  n = n_sims,  
  mean = c(0, 0), 
  sigma = covariance_matrix
)

x <- x_w[, 1]
w <- x_w[, 2]

Next we’ll simulate the outcome variable where the true coefficient on is one and the true coefficient on is -1, adding standard normal errors.

y <- 0.5 + x - w + rnorm(n_sims)

Now we’ll run the “auxiliary regressions”. The first one regresses on and saves the residuals. Call these residuals x_tilde.

# Residuals from regression of X on W
x_tilde <- lm(x ~ w) |> 
  residuals()

The next one regresses on and saves the residuals. Call these residuals y_tilde.

# Residuals from regression of Y on W
y_tilde <- lm(y ~ w) |>
  residuals()

To make the code that follows a little simpler, I’ll also create a helper function that runs a linear regression and returns the coefficients after stripping away any variable names.

get_coef <- function(formula) {
  formula |>  
    lm() |> 
    coef() |> 
    unname() 
}

Now we’re ready to compare some regressions! The “long regression” is a standard linear regression of on and . The “FWL Standard” is a regression of y_tilde on x_tilde. In other words, it regresses the residuals of on the residuals of . The FWL as it is usually encountered in textbooks implies that we should recover the same coefficient on in “Long Regression” and in “FWL Standard”, and indeed the simulation bears this out.

c(
  "Long Regression" = get_coef(y ~ x + w)[2],
  "FWL Standard" = get_coef(y_tilde ~ x_tilde - 1)[1], 
  "FWL Alternative" = get_coef(y ~ x_tilde)[2]
)

Long Regression    FWL Standard FWL Alternative 
      0.9937046       0.9937046       0.9937046 

But now take a look at “FWL” alternative: this is a regression of on x_tilde. Compared to the standard FWL approach, this version does not residualize with respect to . But it still gives us exactly the same coefficient on as the other two regressions. That leaves us with two unanswered questions:

1. Why does the “alternative” FWL approach work?
2. Given that the alternative approach works, why does anyone ever teach the “standard” version?

In the rest of this post we’ll answer both questions using simple algebra and the properties of linear regression. There are lots of deep ideas here, but there’s no need to bring out the big matrix algebra guns to explain them.

2 A Bit of Notation

First we need a bit of notation. I find it a bit simpler to work with population linear regressions rather than sample regressions, but the ideas are the same either way. So if you prefer to put “hats” on everything and work with sums rather than expectations and covariances, be my guest!

First we’ll define the “Long Regression” as a population linear regression of on and , namely Next I’ll define two additional population linear regressions: first the regression of on and second the regression of on I’ve already linked to a post making this point, but it bears repeating: all of the properties of the error terms , and that I’ve stated here hold by construction. They are not assumptions; they are merely what defines an error term in a population linear regression.

4 What’s different about the “usual” FWL theorem?

At this point you may be wondering why anyone teaches the “usual” version of the FWL theorem at all. If that extra short regression of on isn’t needed to learn , why bother?

To answer this question, we’ll start by re-writing the long regression two different ways. First, we’ll substitute into the long regression and re-arrange, yielding Next we’ll substitute on the left-hand side of the preceding equation and rearrange to isolate . This leaves us with Now we have two expressions, each with as one of the terms on the right-hand side and as another. Notice that both expressions have an intercept and a term in which is multiplied by a constant. What’s more, the intercepts are closely related across the two equations, as are the coefficients. I’m now going to make a bold assertion: the intercept and coefficient in the second expression, the one, are both equal to zero Perhaps you don’t believe me, but just for the moment suppose that I’m correct. In this case it would immediately follow that leaving us with two simple linear regressions, namely We’re tantalizingly close to unraveling the mystery of why the “usual” FWL theorem is so popular. But first we need to verify my bold claim from the previous paragraph. To do so, we’ll fall back on our old friend: the omitted variable bias formula, also known as the regression anatomy formula: Thus, as claimed. One down, one more to go. By definition, . Substituting the long regression for , we have by the linearity of expectation and the fact that by construction. Now, we’re trying to show that . Substituting for in this expression gives Inspecting our work so far, we see that the two alternative expressions for will be equal precisely when . But re-arranging this gives , which we already proved above using the omitted variables bias formula!

5 Taking Stock

That was a lot of algebra, so let’s spend some time thinking about the results. We showed that Now, if you’ll permit me, I’d like to re-write that first equality as Since is uncorrelated with , as explained above, and since by construction, it follows that is a bona fide population linear regression model. If we regress on the slope coefficient will be and the error term will be . This regression corresponds to the standard FWL theorem. Notice that it has an intercept of zero and an error term that is identical to that of the long regression. We can verify this using our simulation experiment from above as follows:

# Standard FWL has same residuals as long regression
u_hat <- resid(lm(y ~ x + w))
u_tilde <- resid(lm(y_tilde ~ x_tilde - 1))
all.equal(u_hat, u_tilde)

[1] TRUE

# Standard FWL has an intercept of zero (to machine precision!)
coef(lm(y_tilde ~ x_tilde))[1] # fit with intercept; check it's (numerically) 0

 (Intercept) 
8.273433e-17 

So what about ? This is the regression that corresponds to the alternative FWL theorem. Since and is uncorrelated with both and , this too is a population regression. But unless , it has a different error term. In other words, . Moreover, this regression includes an intercept that is not in general zero. Again we can verify this using our simulation example from above:

# Alternative FWL has different residuals than long regression
v_hat <- resid(lm(y ~ x_tilde))
all.equal(u_hat, v_hat)

[1] "Mean relative difference: 0.4905107"

# Alternative FWL has a non-zero intercept
coef(lm(y ~ x_tilde))[1]

(Intercept) 
  0.4878453 

6 The Punchline

If your goal is merely to learn , then either version of the FWL theorem will do the trick and the alternative version is simpler because it only involves one auxiliary regression instead of two. But if you want to ensure that you end up with the same error term as in the original long regression, then you need to use the standard version of the FWL theorem. This is crucial for the purposes of inference because the properties of the error term determine the standard errors of your estimates.

2 Solution

4 Conclusion

Before we conclude, there’s one important caveat to bear in mind. In addition to the qualifications that NQ isn’t quite JS, and that JS only dominates the MLE when , there’s one more fundamental issue that could be easily missed. Our decision to minimize composite MSE is absolutely crucial to the reasoning given above. The magic of shrinkage depends on our willingness to accept a trade-off in which we do a worse job estimating one mean in exchange for doing a better job estimating another, as composite MSE imposes. Whether this makes sense in practice depends on the context.

If we’re searching for a lost submarine in the ocean (a 3-dimensional problem), it makes perfect sense to be willing to be farther from the submarine in one dimension in exchange for being closer in another. That’s because Euclidean distance is obviously what we’re after here. But if instead we’re estimating teacher value-added and the results of our estimation exercise will be used to determine which teachers lose their jobs, it’s less clear that we should be willing to be farther from one teacher in exchange for being closer to another. Certainly that would be no consolation to someone who had been wrongly dismissed! If we were merely using this information to identify teachers who might need extra help, it’s another story. But the point I’m trying to make here is that our choice of which criterion to minimize necessarily encodes our values in a particular problem.

But with that said, I hope you’re satisfied that this extremely long post was worth the effort. Without using any fancy mathematics or statistical theory, we’ve managed to invent something that is nearly identical to the James-Stein estimator and thus to resolve Stein’s paradox. We started by pretending what we knew and showed that this would allow us to derive a shrinkage estimator with a lower composite MSE than the ML estimator. Then we simply plugged in an unbiased estimator of the key unknown quantity: . Because all the observations contain information about , it makes sense that we should decide how much to shrink one component by using all of the others. At this point, I hope that the James-Stein estimator seems not only plausible but practically obvious, excepting of course that pesky in the numerator.

1 A Quick Review

Consider a binary treatment and an observed outcome . Let be the potential outcomes corresponding to the treatment . Our goal is to learn the average treatment effect but, unless is randomly assigned, using the difference of observed means to estimate the ATE in general won’t work. The idea of selection-on-observables is that might be “as good as randomly assigned” after we adjust for a collection of observed covariates .

Regression adjustment relies on two assumptions: selection-on-observables and overlap. The selection-on-observables assumption says that learning provides no additional information about the average values of and , provided that we already know . This implies that we can learn the conditional average treatment effect (CATE) by comparing observed outcomes of the treated and untreated holding fixed: For example: older people might be more likely to take a new medication but also more likely to die without it. If so, perhaps by comparing average outcomes holding age fixed we can learn the causal effect of the medication. The overlap assumption says that, for any fixed value of the covariates, there are some treated and some untreated people. This allows us to learn for every value of in the population and average it using the law of iterated expectations to recover the ATE:
In the medication example, this would correspond to computing the difference of means for each age group separately, and then averaging them using the share of people in each age group. Notice that this is only possible if there are some people who took the medication and some who didn’t in each age group. That’s exactly what the overlap assumption buys us. For example, if there were no senior citizens who didn’t take the medication, we wouldn’t be able to learn the effect of the medication for senior citizens.

2 Which regression should we run?

So suppose that we’ve found a set of covariates that satisfy the required assumptions. How should we actually carry out regression adjustment? To answer this question, let’s start by making things a bit simpler. Suppose that is a single binary covariate. At the end of the post, we’ll return to the general case. Since and are both binary, we can write the conditional mean function of given as Since the true conditional mean function is linear, a linear regression of on , , and an intercept will recover . But what on earth do these coefficients actually mean?! Substituting all possible values of , And so, after a bit of re-arranging, What a mess! Alas, we’ll need a few more steps of algebra to figure out how these relate to the ATE. Notice that equals the CATE when since Proceeding similarly for the CATE when , we find that Now that we have expressions for each of the two conditional average treatment effects, corresponding to each of the values that can take, we’re finally ready to compute the ATE: where we define the shorthand . So to compute the ATE, we need to know the coefficients and from the regression of on , , and , in addition to the share of people with . Needless to say, your favorite regression package will not spit out the ATE for you if you run the regression from above. And it certainly won’t spit out the standard error! So what can we do besides computing everything by hand?

3 Two Simple Alternatives

It turns out that there are two simple ways to get the your favorite software package to spit out the ATE for you and associated standard error. Each involves a slight re-parameterization of the conditional mean expression from above. The first one replaces with where and . To see why this works, notice that This works perfectly well, but there’s something about it that offends my sense of order: why subtract the mean from in one place but not in another? If you share my aesthetic sensibilities, then you can feel free to replace that offending with another since where we define . Notice that the only coefficient that changes is the intercept, and we’re typically not interested in this anyway!

7 Conclusion

I hope this post has convinced you that regression adjustment isn’t simply a matter of tossing a collection of covariates into your regression! In general, the coefficient on in a regression of on and will not equal the ATE of . Instead it will be a weighted average of CATEs. To obtain the ATE we need to include an interaction between and . The simplest way to get your favorite statistical software package to calculate this for you, along with an appropriate standard error, is by de-meaning before including the interaction. And don’t forget that causal inference always requires untestable assumptions, in this case the selection-on-observables assumption. While implementation details are important, getting them right won’t make any difference if you’re not adjusting for the right covariates in the first place.

2 Solution

1 Read Something Else Instead

The first question to ask yourself is whether you should even be reading this paper in the first place. Just because White’s (1980) paper on heteroskedasticity-robust standard errors is a “classic” in econometrics, that doesn’t mean that you should read it. In fact, as a graduate student just starting out, you probably shouldn’t! The paper that introduces a new idea or procedure is rarely the paper that gives the clearest explanation. Reading a good textbook explanation is a much more effective way to get to grips with a new idea. You might, for example, try reading the relevant chapters in White’s textbook Asymptotic Theory for Econometricians instead.

But sometimes you have to read a particular paper. Maybe it’s the paper you’ve been assigned to present in a reading group, or maybe it’s highly relevant to your own research. In that case you may still want to start by reading something else. For example, there might be a more recent paper or review article that gives a good summary of the idea or method in question. Reading this paper first can make it much easier for you to tackle the original paper.

So to all those professors out there who keep telling their students to “read the papers!” I say: “read the papers, but only after you’ve read something else first!”

3 The Replication

#-------------------------------------------------------------------------------
# Sims, C. A., & Uhlig, H. (1991). Understanding unit rooters: A helicopter tour
#
# (See also: Example 6.10.6 from Poirier "Intermediate Statistics and 'Metrics")
#-------------------------------------------------------------------------------
# In the next section we will proceed to construct, by Monte Carlo, an estimated
# joint pdf for \rho and \hat{\rho} under a uniform prior pdf on \rho. We choose
# 31 values of \rho, from 0.8 to 1.1 at intervals of 0.01. We draw 10000 100 x 1
# iid N(0,1) vectors of random variables to use as realizations of \epsilon. For
# each of the 10000 \epsilon vectors and each of the 31 \rho values, we
# construct a y vector with y(0) = 0, y(t) generated by equation (1).
#
# Equation (1): y(t) = \rho y(t-1) + \epsilon(t), t = 0, ..., T
#
# For each of these y vectors, we construct \hat{\rho}. Using as bins the
# intervals [-\infty, 0.795), [0.795, 0.805), [0.805, 0.815), etc. we construct
# a histogram that estimates the pdf of \hat{rho} for each fixed \rho value.
# When these histograms are lined up side by side, they form a surface that is
# the joint pdf for \rho and \hat{\rho} under a flat prior on \rho.
#-------------------------------------------------------------------------------
set.seed(1693)

library(tidyverse)
library(tictoc)
library(patchwork)

draw_rho_hat <- function(rho) {
# Carry out the simulation once for a fixed value of rho; return rho_hat
  nT <- 100
  y <- rep(0, nT + 1)
  for (t in 2:(nT + 1)) {
    y[t] <- rho * y[t - 1] + rnorm(1)
  }
  y_t <- y[-1]
  y_tminus1 <- y[-length(y)]
  sum(y_t * y_tminus1) / sum(y_tminus1^2)
}

# Function to run the simulation for a fixed value of rho (10000 times)
run_sim <- \(rho) map_dbl(1:1e4, \(i) draw_rho_hat(rho))

tic()
foo <- run_sim(0.9)
toc() # ~0.6 seconds on my machine

0.38 sec elapsed

# Full sequence of rho values from Sims & Uhlig (1991)
rho <- seq(from = 0.8, to = 1.1, by = 0.01)

tic()
results <- tibble(rho = rho,
                  rho_hat = map(rho, run_sim)) # List columns
toc() # ~17 seconds on my machine (1991 was a long time ago!)

11.574 sec elapsed

# The results tibble uses a list column for rho_hat. This is convenient for
# making histograms of the frequentist sampling distribution (rho fixed) but
# not for making histograms of the Bayesian posterior (rho_hat) fixed. For the
# latter, we will use the unnest() function to "expand" the list column rho_hat
# into a regular column. This is the "joint" distribution of rho and rho_hat.
joint <- results |>
  unnest(rho_hat)

joint |>
  ggplot(aes(x = rho, y = rho_hat)) +
  geom_density2d_filled() +
  coord_cartesian(ylim = c(0.8, 1.1)) + # Restrict rho_hat axis
  labs(title = "Joint Distribution",
       x = expression(rho),
       y = expression(hat(rho)))

joint |>
  filter(rho_hat >= 0.995 & rho_hat < 1.005) |>
  ggplot(aes(x = rho)) +
  geom_histogram(binwidth = 0.01, fill = "skyblue", color = "black") +
  labs(title = expression(hat(rho) == 1),
       x = expression(rho),
       y = "Frequency")

joint |>
  filter(rho == 1) |>
  ggplot(aes(x = rho_hat)) +
  geom_histogram(binwidth = 0.01, fill = "skyblue", color = "black") +
  labs(title = expression(rho == 1),
       x = expression(hat(rho)),
       y = "Frequency")

# Function that makes the preceding two plots, puts them side-by-side and lets
# the user specify the value of rho/rho_hat that we condition on:
plot_Bayes_vs_Freq <- \(r) {
  p1 <- joint |>
    filter(rho_hat >= r - 0.005 & rho_hat < r + 0.005) |>
    ggplot(aes(x = rho)) +
    geom_histogram(aes(y = after_stat(density)),
                   binwidth = 0.01, fill = "skyblue", color = "black") +
    geom_vline(xintercept = r, color = "red", linetype = "dashed", linewidth = 1) +
    labs(title = bquote(hat(rho) == .(round(r, 3))),
         x = expression(rho))

  p2 <- joint |>
    filter(rho >= r - 0.005 & rho < r + 0.005) |>
    ggplot(aes(x = rho_hat)) +
    geom_histogram(aes(y = after_stat(density)),
                   binwidth = 0.01, fill = "skyblue", color = "black") +
    geom_vline(xintercept = r, color = "red", linetype = "dashed", linewidth = 1) +
    labs(title = bquote(rho == .(round(r, 3))),
         x = expression(hat(rho)))

  p1 + p2
}

plot_Bayes_vs_Freq(0.98)

plot_Bayes_vs_Freq(0.99)

plot_Bayes_vs_Freq(1.0)

plot_Bayes_vs_Freq(1.01)

plot_Bayes_vs_Freq(1.02)

4 Some Intuition

In our simulation, gave a worse estimate of than . The derivations from above show that this wasn’t a fluke: adding a valid instrument as an additional control regressor only makes the bias in our estimated causal effect worse than it was to begin with. This holds for any valid instrument and any endogenous regressor in a linear causal model. I hope you found the derivations from above convincing. But even so, you may be wondering if there’s an intuitive explanation for this phenomenon. I am pleased to inform you that the answer is yes!

In an earlier post I described the control function approach to instrumental variables regression. That post showed that the coefficient on in a regression of on and gives the correct causal effect. We don’t know , but we can estimate it by regressing on and saving the residuals. The logic of multiple regression shows that including as a control regressor “soaks up” the portion of that is explained by . Because represents the “bad” (endogenous) variation in , this solves our endogeneity problem. In effect, captures the unobserved “omitted variables” that play havoc with a naive regression of on .

Now, contrast this with a regression of on and . In this case, we soak up the variation in that is explained by . But represents the good (exogenous) variation in ! Soaking up this variation leaves only the bad variation behind, making our endogeneity problem worse than it was to begin with. In this example, is what is known as a bad control, a control regressor that makes things worse rather than better. A common piece of advice for avoiding bad controls is to only include control regressors that are correlated with and but are not themselves caused by . The example in this post shows that this advice is wrong. Here is not caused by , and is correlated with both and . Nevertheless, it is a bad control. In short, a valid instrument provides a powerful way to carry out causal inference from observational data, but only if you use it in the right way. A good instrument is a bad control!

1 Prerequisites

While written at an introductory level, this post assumes basic familiarity with calculations involving discrete and continuous random variables. In particular, I assume that:

• You know the definitions of expected value, variance, covariance, and correlation.
• You are comfortable working with joint, marginal, and conditional distributions of a pair of discrete random variables.
• You understand the uniform distribution and how to compute its moments (mean, variance, etc.).
• You’ve encountered the notion of conditional expectation and the law of iterated expectations.

If you’re a bit rusty on this material, lectures 7-11 from these slides should be helpful. For bivariate, discrete distributions I also suggest watching this video from 1:07:00 to the end and this other video from 0:00:00 up to the one hour mark.

2 Two Examples

Example #1 - Discrete RVs

My first example involves two discrete random variables and with joint probability mass function given by

Even without doing any math, we see that knowing conveys information about , and vice-versa. For example, if then we know that must equal zero. Similarly, if then must equal zero. Spend a bit of time thinking about this joint distribution before reading further. We’ll have plenty of time for mathematics below, but it’s always worth seeing where our intuition takes us before calculating everything.

To streamline our discussion below, it will be helpful to work out a few basic results about and . A quick calculation with shows that Calculating the marginal pmfs for we see that Similarly, calculating the marginal pmf of , we obtain We’ll use these results as ingredients below as we explain and relate three key notions of dependence: correlation, conditional mean independence, and statistical independence.

5 Statistical Independence

When you see the word “independent” without any qualification, this means “statistically independent.” In keeping with this usage, I often write “independent” rather than “statistically independent.” Whichever terminology you prefer, there are three equivalent ways of defining this idea:

and are statistically independent if and only if:

1. their joint distribution equals the product of their marginals, or
2. the conditional distribution of equals the unconditional distribution of , or
3. the conditional distribution of equals the unconditional distribution of .

The link between these three alternatives is the definition of conditional probability. Suppose that and are discrete random variables with joint pmf , marginal pmfs and , and conditional pmfs and . Version 1 requires that for all realizations . But by the definition of conditional probability, If , these expressions simplify to so 1 implies 2 and 3. Similarly, if then by the definition of conditional probability Re-arranging, this shows that , so 3 implies 1. An almost identical argument shows that 2 implies 1, completing our proof that these three seemingly different definitions of statistical independence are equivalent. If and are continuous, the idea is the same but with densities replacing probability mass functions, e.g.  and so on.

In most examples, it’s easier to show independence (or the lack thereof) using 2 or 3 rather than 1. These latter two definitions are also more intuitively appealing. To say that the conditional distribution of is the same as the unconditional distribution of is the same thing as saying that

provides absolutely no information about whatsoever.

If learning tells us anything at all about , then and are not independent. Similarly, if tells us anything about at all, then and are not independent.

7 Summary

In this post we have shown that:

• Statistical Independence Mean Independence Uncorrelatedness.
• Uncorrelatedness does not imply mean independence or statistical independence.
• Mean independence does not imply statistical independence.
• Statistical independence and correlation are symmetric; mean independence is not.

Reading the figure from the very beginning of this post from top to bottom: statistical independence is the strongest notion, followed by mean independence, followed by uncorrelatedness.

2 Stirling’s Approximation

The key step in our argument is to show that the pmf of a random variable is well-approximated by the density. This explains the bell-shaped curve that we plotted above. To obtain this result, we’ll use the central limit theorem. But there is one fact that you will need to take on faith if you don’t already know it: if is independent of then . Proceeding by induction we can view a Poisson(171) random variable as the sum of 171 independent Poisson(1) random variables. More generally, we can view a Poisson RV with rate parameter as the sum of iid Poisson(1) random variables. By the central limit theorem, it follows that since the mean and variance of a Poisson(1) RV are both equal to one. From a practical perspective, this means that is approximately equal to , a standard normal random variable. Re-arranging, and is simply a random variable! This is a quick way of seeing why the distribution is well-approximated by the distribution when is large.

Now let’s run with this. As we just saw, for large the Poisson pmf is well-approximated by the Normal density: This approximation is particularly accurate for near the mean. This is convenient, because substituting for considerably simplifies the right hand side: Re-arranging, we obtain Taking logs of both sides gives: Writing this with in place of gives the following: This is called Stirling’s Approximation. The usual way of writing this excludes the term, yielding , which is fairly easy to remember. Including the extra term, however, gives increased accuracy for smaller values of . While I haven’t formally proved this, it turns out that as . In other words, the ratio of the LHS and RHS tends to one in the large limit. Perhaps surprisingly, this approximation is extremely accurate even for fairly small values of , as we can see by comparing it against lfactorial().

stirling1 <- function(n) n * log(n) - n
stirling2 <- function(n) n * log(n) - n + 0.5 * log(2 * pi * n)
tibble(n = 1:20) |>
  mutate(Stirling1 = stirling1(n),
         Stirling2 = stirling2(n),
         R = lfactorial(n)) |>
  knitr::kable(digits = 3)

3 Epilogue

I have a bad habit of trying to add a “moral” or “lesson” to the end of my posts, but I suppose there’s no point trying to break the habit today! While there are easier ways to derive Stirling’s approximation, there are two things I enjoy about this one. First, we get a more accurate approximation than with practically no effort. Second, making unexpected connections between facts that we already know both deepens our understanding and helps us “compress” information. If you ever forget Stirling’s approximation, now you know how to very quickly re-derive it on the spot!

1 The Setup

Consider a simple linear causal model of the form where is endogenous, i.e. related to the unobserved random variable . Our goal is to learn , the causal effect of on . To take a simple example, suppose that is wage and is years of schooling. Then is the causal effect of one additional year of schooling on a person’s wage. The random variable is a catchall, representing all unobserved causes of wage, such as ability, family background, and so on. A linear regression of on will not allow us to learn . For example, if you’re very smart, you will probably find school easier and stay in school longer. But being smarter likely has its own effect on your wage, separate from years of education. Ability is a confounder because it causes both years of schooling and wage.

Now suppose that is an instrumental variable: something that is uncorrelated with (exogenous) but correlated with (relevant). For example, a very famous paper pointed out that quarter of birth is correlated with years of schooling in the US and argued that it is unrelated to other causes of wages. Finding a good instrumental variable is very hard in practice. Indeed, I remain skeptical that quarter of birth is really unrelated to . But that’s a conversation for another day. For the moment, suppose we have a bona fide exogenous and relevant instrument at our disposal. To make things even simpler, suppose that the true causal effect is homogeneous, i.e. the same for everyone.

2 1st Perspective: The IV Approach

Regress on to find the causal effect of on . Rescale it to obtain the causal effect of on .

If is a valid and relevant instrument, then which is precisely the causal effect we’re after! The ratio of to is called the instrumental variables (IV) estimand, but it seems to come out of nowhere. A more intuitive way to write this quantity multiplies the numerator and denominator by to yield We see that is the ratio of two linear regression slopes: the slope from a regression of on divided by the slope from a regression of on . This makes intuitive sense if we think about units. Because is unrelated to , gives the causal effect of on . If is measured in dollars and is measured in miles (e.g. distance to college), then is measured in dollars per mile. If is years of schooling, then should be measured in dollars per year. To convert from dollars/mile to dollars/year, we need to multiply by miles/year or equivalently to divide by years/mile. And indeed, is measured in years/mile as required! This is yet another example of my favorite maxim: most formulas in statistics and econometrics are obvious if you keep track of the units.

3 2nd Perspective: The TSLS Approach

Construct by using to “clean out” the part of that is correlated with . Then regress on .

Let be the intercept and be the slope from a population linear regression of on . Defining , we can write By definition is the best linear predictor of based on , in that and solve the optimization problem What’s more, by construction since: And since is uncorrelated with , so is : So now we have a variable that is a good predictor of but is uncorrelated with . In essence, we’ve used to “clean out” the endogeneity from and we did this using a first stage regression of on . Two-stage least squares (TSLS) combines this with a second stage regression of on to recover . To see why this works, substitute for in the causal model, yielding where we define . Finally, since a regression of on recovers the causal effect of on .

4 3rd Perspective: The Control Function Approach

Use to solve for , the part of that is correlated with . Then regress on controlling for .

I’m willing to bet that you haven’t seen this approach before! The so-called control function approach starts from the same place as TSLS: the first-stage regression of on from above, namely Like the error term from the causal model , the first stage regression error is unobserved. But as strange as it sounds, imagine running a regression of on . Then we would obtain where by construction. Now, since the causal model for includes an intercept, . And since the first-stage linear regression model that defines likewise includes an intercept, as well. This means that so the regression of on becomes Now, substituting for in the causal model gives By construction . And since , it follows that Therefore, if only we could observe , a regression of on that controls for would allow us to recover the causal effect of interest, namely . Such a regression would also give us . To see why this is interesting, notice that Since , tell us the direction of endogeneity in . If then is positively correlated with , if then is negatively correlated with , and if then is exogenous. If is ability and ability has a positive effect on years of schooling, for example, then will be positive.

Now it’s time to address the elephant in the room: is unobserved! It’s all fine and well to say that if were observed our problems would be solved, but given that it is not in fact observed what are we supposed to do? Here’s where the TSLS first stage regression comes to the rescue. Both and are observed, so we can learn and by regressing on . Given these coefficients, we can simply solve for the unobserved error: . Like TSLS, the control function approach relies crucially on the first stage regression. But whereas TSLS uses it to construct , the control function approach uses it to construct . We don’t replace with its exogenous component ; instead we “pull out” the component of that is correlated with , namely . In effect we control for the “omitted variable” , hence the name control function.

5 Simulating the Three Approaches

Perhaps that was all a bit abstract. Let’s make it concrete by simulating some data and actually calculating estimates of using each of the three approaches described above. Because this exercise relies on a sample of data rather than a population, estimates will replace parameters and residuals will replace error terms.

To begin, we need to simulate independently of . For simplicity I’ll make these standard normal and set the correlation between and to 0.5.

set.seed(1983) # for replicability of pseudo-random draws
n <- 1000
Z <- rnorm(n)
library(mvtnorm)
cor_mat <- matrix(c(1, 0.5,
                    0.5, 1), 2, 2, byrow = TRUE)
errors <- rmvnorm(n, sigma = cor_mat)
head(errors)

           [,1]        [,2]
[1,]  0.1612255 -0.96692422
[2,]  1.4020130  1.55818062
[3,]  1.7212525 -0.01997204
[4,] -0.6972637 -0.68551762
[5,]  1.3471669 -0.01766333
[6,] -1.0441467 -0.23113677

U <- errors[,1]
V <- errors[,2]
rm(errors)

Since this is a simulation we actually can observe and and hence could regress the one on the other. Since I set the standard deviation of both of them equal to one, will simply equal the correlation between them, namely 0.5

coef(lm(U ~ V - 1)) # exclude an intercept

        V 
0.5047334 

Excellent! Everything is working as it should. The next step is to generate and . Again to keep things simple, in my simulation I’ll set .

pi <- 0.3
beta <- 1.1
X <- pi * Z + V
Y <- beta * X + U

Now we’re ready to run some regressions! We’ll start with an OLS regression of on . This substantially overestimates because is in fact positively correlated with .

OLS <- coef(lm(Y ~ X))[2]
OLS

       X 
1.567642 

In contrast, the IV approach works well.

IV <- cov(Y, Z) / cov(X, Z)
IV

[1] 1.049043

For the TSLS and control function approaches we need to run the first-stage regression of on and store the results.

first_stage <- lm(X ~ Z)

The TSLS approach uses the fitted values of this regression as .

Xtilde <- predict(first_stage)
TSLS <- coef(lm(Y ~ Xtilde))[2] # drop the intercept since we're not interested in it
TSLS

  Xtilde 
1.049043 

In contrast, the control function approach uses the residuals from the first stage regression. It also gives us in addition to .

Vhat <- residuals(first_stage) 
CF <- coef(lm(Y ~ X + Vhat))[-1] # drop the intercept since we're not interested in it
CF # The coefficient on Vhat is lambda

        X      Vhat 
1.0490432 0.5558904 

Notice that we obtain precisely the same estimates for using each of the three approaches.

c(IV, TSLS, CF[1])

           Xtilde        X 
1.049043 1.049043 1.049043 

It turns out that in this simple linear model with a single endogenous regressor and a single instrument, the three approaches are numerically equivalent. In other words, they give exactly the same answer. This will not necessarily be true in more complicated models, so be careful!

6 Epilogue

It’s time to admit that this post had a secret agenda: to introduce the idea of a control function in the simplest way possible! If you’re interested in learning more about control functions, a canonical example that does not turn out to be identical to IV is the so-called Heckman Selection Model, which you can learn more about here. (Scroll down until you see the heading “Heckman Selection Model.”) The basic logic is similar: to solve an endogeneity problem, use a first-stage regression to estimate an unobserved quantity that “soaks up” the part of the error term that is correlated with your endogenous regressor of interest. If these videos whet your appetite for more control function fun, Wooldridge (2015) provides a helpful overview along with many references to the econometrics literature.

3 Causal Model

Both interpretations of from above are purely predictive; they say nothing about whether causes . To indicate that a linear model is meant to be causal, it is traditional to write something like “suppose that where may be endogenous.” Often “may be endogenous” is replaced by “where may be correlated with .” What on earth is this supposed to mean? The language is vague, evasive, and imprecise. It also stretches the meaning of “” beyond all reason. Here’s my suggested improvement:

Consider the causal model where is unobserved and may be dependent.

Causality is intrinsically directional: cigarettes cause lung cancer; lung cancer doesn’t cause cigarettes. The notation “” makes this clear. In stark contrast, the notion of mathematical equality is symmetric. If , it is just as true to say that . Of course this is nonsensical when applied to cigarettes and cancer.

In a causal model, does have a life of its own; it represents the causes of that we cannot observe. Perhaps is wage, is years of schooling and is “family background” plus “ability.” For this reason I do not write “define .” We aren’t defining a residual in a prediction problem. We are taking a stand on how the world works by writing down a particular causal model. In a randomized controlled trial, any unobserved causes would be independent of . Here we have not made this assumption. We have, however, assumed a particular form for the causal relationship: linear with constant coefficients. Each additional year of schooling causes the same increase (or decrease) in wage regardless of who you are or how many years of schooling you already have. This model could be wrong. But right or wrong, it is fundamentally distinct from the population linear regression and conditional mean models described above. Let’s endeavour to make this clear in our notation.

1 Where does the regression line come from?

If you’ve taken an introductory statistics or econometrics course, you most likely learned that the least squares regression line minimizes the sum of squared vertical deviations by solving the optimization problem You probably also learned that the solution is given by In words: the regression slope equals the ratio of the covariance between height and handspan to the variance of handspan, and the regression line passes through the sample average values of height and handspan. We can check that all of these formulas agree with what we calculated above using lm()

b <- with(dat, cov(height, handspan) / var(handspan))
a <- with(dat, mean(height) - b * mean(handspan))
c(a, b)

[1] 40.943127  1.266961

coef(lm(height ~ handspan, dat))

(Intercept)    handspan 
  40.943127    1.266961 

This is all perfectly correct, and an entirely reasonable way of looking at the problem. But I’d now like to suggest a completely different way of looking at regression. Why bother? The more different ways we have of understanding an idea or method, the more deeply we understand how it works, when it works, and when it is likely to fail. So bear with me while I take you on what might at first appear to be a poorly-motivated computational detour. I promise that there’s a payoff at the end!

2 A Crazy Idea

There is a unique line that passes through any two distinct points in a plane. So here’s a crazy idea: let’s form every unique pair of students from my height and handspan dataset. To understand what I have in mind, consider a small subset of the data, call it test

test <- dat[1:3,]
test

# A tibble: 3 × 2
  height handspan
   <dbl>    <dbl>
1     73     22.5
2     65     17  
3     69     21  

With three students, there are three unique unordered pairs: . Corresponding to these three pairs are three line segments, one through each pair:

plot(height ~ handspan, test, pch = 20)
segments(x0 = c(17, 17, 21),     # FROM: x-coordinates
         y0 = c(65, 65, 69),     # FROM: y-coordinates
         x1 = c(21, 22.5, 22.5), # TO:   x-coordinates
         y1 = c(69, 73, 73),     # TO:   y-coordinates
         lty = 2)

And associated with each line segment is a slope

y_differences <- c(69 - 65, 73 - 65, 73 - 69)
x_differences <- c(21 - 17, 22.5 - 17, 22.5 - 21) 
slopes <- y_differences / x_differences
slopes

[1] 1.000000 1.454545 2.666667

And here’s my question for you: what, if anything, is the relationship between these three slopes and the slope of the regression line? While it’s a bit silly to run a regression with only three observations, the results are as follows:

coef(lm(height ~ handspan, test))

(Intercept)    handspan 
  41.556701    1.360825 

The slope of the regression line doesn’t equal any of the three slopes we calculated above, but it does lie between them. This makes sense: if the regression line were steeper or less steep than all three line segments from above, it couldn’t possibly minimize the sum of squared vertical deviations. Perhaps the regression slope is the arithmetic mean of slopes? No such luck:

mean(slopes)

[1] 1.707071

Something interesting is going on here, but it’s not clear what. To learn more, it would be helpful to play with more than three points. But doing this by hand would be extremely tedious. Time to write a function!

3 All Pairs of Students

The following function generates all unique pairs of elements taken from a vector x and stores them in a matrix:

make_pairs <- function(x) {
# Returns a data frame whose rows contain each unordered pair of elements of x
# i.e. all committees of two with members drawn from x
  n <- length(x)
  pair_indices <- combn(1:n, 2)
  matrix(x[c(pair_indices)], ncol = 2, byrow = TRUE)
}

For example, applying make_pairs() to the vector c(1, 2, 3, 4, 5) gives

make_pairs(1:5)

      [,1] [,2]
 [1,]    1    2
 [2,]    1    3
 [3,]    1    4
 [4,]    1    5
 [5,]    2    3
 [6,]    2    4
 [7,]    2    5
 [8,]    3    4
 [9,]    3    5
[10,]    4    5

Notice that make_pairs() is constructed such that order doesn’t matter: we don’t distinguish between and , for example. This makes sense for our example: Alice and Bob denotes the same pair of students as Bob and Alice.

To generate all possible pairs of students from dat, we apply make_pairs() to dat$handspan and dat$height separately and then combine the result into a single dataframe:

regression_pairs <- data.frame(make_pairs(dat$handspan), 
                               make_pairs(dat$height))
head(regression_pairs)