1 Complex Numbers in R

R provides basic functionality for working with complex numbers, documented in the helpful ?complex. The imaginary unit in R is denoted by 1i. Don’t try i by itself because that won’t work:

i # this doesn't work

Error:
! object 'i' not found

1i # this works

[1] 0+1i

To write a complex number, simply use + to separate the real and imaginary parts, e.g.

3 + 4i

[1] 3+4i

A common error when learning R is to write code with “implied multiplication” e.g. 4x (wrong) rather than 4 * x (right). But don’t be tempted to try 4 * i since, again, i is not the imaginary unit in R:

4 * i # this doesn't work 

Error:
! object 'i' not found

4 * 1i # this works

[1] 0+4i

To demonstrate R’s basic functionality for working with complex numbers, let’s specify a complex number in Cartesian coordinates: . The real part is , the imaginary part is , the modulus is , and the complex conjugate is . All of these operations are available in R:

z <- 3 + 4i
Re(z)

[1] 3

Im(z)

[1] 4

Mod(z)

[1] 5

abs(z) # same as Mod(z) for a complex number z

[1] 5

Conj(z)

[1] 3-4i

All of the basic operations +, -, *, /, and ^ work on complex numbers and are vectorized

w <- (-3) - 1i
z + w

[1] 0+3i

z - w

[1] 6+5i

z * w

[1] -5-15i

z / w

[1] -1.3-0.9i

z^3

[1] -117+44i

as are functions such as log(), exp(), sin(), cos(), etc. If you supply them a complex input, they will return a complex output; if you supply them a real input, they will return a real output. This behavior explains a common gotcha: sqrt(-1) doesn’t work because there’s no real number whose square root is -1 but sqrt(-1 + 0i) does work:

sqrt(-1) # this doesn't work

Warning in sqrt(-1): NaNs produced

[1] NaN

sqrt(-1 + 0i)  # this works

[1] 0+1i

Another common gotcha is trying to apply functions like max() and min() or operators like < and > to complex numbers. Since complex numbers aren’t ordered, this doesn’t work:

min(c(z, w))

Error in `min()`:
! invalid 'type' (complex) of argument

z < w

Error in `z < w`:
! invalid comparison with complex values

but testing for equality / non-equality does work:

z == w

[1] FALSE

z != w

[1] TRUE

2 Complex Step Differentiation in Action

Let’s test out complex step differentiation with a simple example. Let and suppose we want to compute . This one is easy to compute analytically: so we can check how accurate different numerical approximations turn out to be. Computing the derivative directly gives

direct <- 4.5 * (1.5)^(3.5)
direct

[1] 18.60081

Now we’ll compare this value against three numerical derivatives: the “simple difference” approach, the “symmetric difference” approach, and the complex step approach.

f <- function(x) x^(4.5) # the function to differentiate

x <- 1.5 # the point where we'll evaluate f'(x)

Since R functions are vectorized, we can evaluate the quality of each approximation over many values of at once by setting up a vector of progressively smaller positive values:

delta <- 1 / 10^(1:15)

simple <- (f(x + delta) - f(x)) / delta

symmetric <- (f(x + delta) - f(x - delta)) / (2 * delta)

complex_step <- Im(f(x + delta * 1i)) / delta

Now we’ll make two tables: the first containing raw results—the approximate value of —and the second containing the relative error of the approximation in percentage points. When reading the first table, recall from above that direct calculation gives .

cbind(delta, simple, symmetric, complex_step) |> 
  knitr::kable(digits = 15)

get_rel_error <- function(x, truth) 100 * abs(x - truth) / abs(truth)

cbind(
  delta,
  simple = get_rel_error(simple, direct),
  symmetric = get_rel_error(symmetric, direct),
  complex_step = get_rel_error(complex_step, direct)
) |> 
  knitr::kable(digits = c(15, rep(4, 3)))

The simple difference approach is clearly less accurate than both the symmetric difference and complex step approach, particularly at larger values of . And while it may seem that there’s not much to choose between these latter two approximations, look carefully at what happens as gets smaller. Eventually the relative error of the simple and symmetric difference approaches starts to increase! For example, the symmetric difference approximation is better with (relative error ) than it is at (relative error ).

How can this be? The derivative is defined as the limit of the simple difference approximation as . So how can the approximation get worse if becomes smaller?

3 Catastrophic Cancellation

The culprit is catastrophic cancellation. As explained in an earlier post, computers cannot represent all real numbers exactly. Instead they use floating point numbers as an approximation. So when we compute a difference like what we’re really computing is the difference of two approximations. The problem is that even when is a very good approximation to and is a very good approximation to , the difference may be a poor approximation to . This is an unfortunate property of subtraction, nicely illustrated in this post by John D. Cook.

A double in R is accurate to around 16 decimal places. For small values of , and agree in nearly all of those digits. Subtracting them eliminates the digits that match, leaving behind what is effectively rounding noise. Dividing by then magnifies this noise. For example, since in our example is around when we compute we should obtain a result of approximately but instead we get

sprintf("%.20f", f(x + 1e-15) - f(x))

[1] "0.00000000000002042810"

The first couple of digits are in the ballpark, but the remaining ones are just noise.
The symmetric difference formula has the same problem because it too relies on a subtraction of two approximate values in the numerator:

sprintf("%.20f", f(x + 1e-15) - f(x - 1e-15))

[1] "0.00000000000004085621"

Here we’d expect a result of around and, again, every digit after the first two is just noise.

In contrast, complex step differentiation has no subtraction in the numerator. This means that it is not subject to catastrophic cancellation. As becomes smaller, the approximation just keeps improving—down to roughly machine precision. The cost is that must be analytic at and evaluable at complex inputs. Most smooth functions you’ll encounter qualify, but those involving , , , or indicator functions generally don’t.

4 Don’t Roll Your Own

In this post I computed the complex step derivative by hand. That’s useful for understanding the method, but it’s a bad idea in practice. Whenever you can, you should rely on high-quality existing libraries to implement numerical methods. Numerical analysis is a deeply complicated subject and we are but lowly econometricians! Fortunately for us, the grad() function from the numDeriv R package implements complex step differentiation as one of its three methods:

library(numDeriv)
grad(f, 1.5, method = 'complex')

[1] 18.60081

So the next time you need to differentiate something numerically, remember the value in making things more complex than they need to be!

2 What about ?

So if soaks up the good variation in , what about the coefficient on the instrument ? Perhaps this coefficient contains some useful information about the causal effect of on ? To find out, we’ll use the FWL Theorem as follows: where is the population linear regression of on . This is the reverse of the first-stage regression of on described above. Here the error term satisfies by construction. Substituting the causal model gives since by construction. Now, substituting the definition of , since by assumption. We can already see that is not going to help us learn about . First of all, the term containing vanished; second of all, the term that remained is polluted by the endogeneity of , namely .

Still, let’s see if we can get a clean expression for . So far we have calculated the numerator of the FWL expression, showing that . The next step is to calculate : Since , our expression for simplifies to so we have discovered that:

Call me old-fashioned, but I really don’t like having in that expression. I’d feel much happier if we could find a way to re-write this in terms of the more familiar IV first-stage coefficient . Let’s give it a try! Let’s use my favorite trick of multiplying by one: Substituting for gives We can simplify this even further by substituting from above to obtain And now we recognize something from above: was the bias of relative to the true causal effect ! This means we can also write .

3 A Little Simulation

We seem to be doing an awful lot of algebra on this blog lately. To make sure that we haven’t made any silly mistakes, let’s check our work using a little simulation experiment taken from my earlier post. Spoiler alert: everything checks out!

set.seed(1234)
n <- 1e5

# Simulate instrument (z)
z <- rnorm(n)

# Simulate error terms (u, v)
library(mvtnorm)
Rho <- matrix(c(1, 0.5, 
                0.5, 1), 2, 2, byrow = TRUE)
errors <- rmvnorm(n, sigma = Rho)

# Simulate linear causal model
u <- errors[, 1]
v <- errors[, 2]
x <- 0.5 + 0.8 * z + v
y <- -0.3 + x + u

# Regression of y on x and z
gamma <- lm(y ~ x + z) |> 
  coefficients()

gamma

(Intercept)           x           z 
 -0.5471213   1.5018705  -0.3981116 

# First-stage regression of x on z
pi <- lm(x ~ z) |> 
  coefficients()

pi

(Intercept)           z 
  0.5020338   0.7963889 

# Compare two different expressions for gamma_Z to the estimate itself
c(gamma_z = unname(gamma[3]),
  version1 = unname(-0.8 * cov(x, u) / var(v)),
  version2 = unname(-pi[2] * (gamma[2] - 1))
)

   gamma_z   version1   version2 
-0.3981116 -0.4024918 -0.3996841 

5 Epilogue

When I showed this post to another colleague he asked me whether there is any way to learn about by combining and . The answer is no: the regression of on and alone doesn’t contain enough information. Since we can rearrange to obtain the following expression for : which we can verify in our little simulation example as follows:

gamma[2] + gamma[3]/pi[2]

       x 
1.001975 

Thus, in order to solve for , we need to run the first-stage regression to learn .

1 Simulation Example

Let’s start with a little simulation. First we’ll generate 5000 observations of predictors and from a joint normal distribution with standard deviations of one, means of zero, and a correlation of 0.5.

set.seed(1066)
library(mvtnorm)

# Simulate linear regression with two predictors: X and W
covariance_matrix <- matrix(
  c(1, 0.5, 0.5, 1), 
  nrow = 2
)

n_sims <- 5000

x_w <- rmvnorm(
  n = n_sims,  
  mean = c(0, 0), 
  sigma = covariance_matrix
)

x <- x_w[, 1]
w <- x_w[, 2]

Next we’ll simulate the outcome variable where the true coefficient on is one and the true coefficient on is -1, adding standard normal errors.

y <- 0.5 + x - w + rnorm(n_sims)

Now we’ll run the “auxiliary regressions”. The first one regresses on and saves the residuals. Call these residuals x_tilde.

# Residuals from regression of X on W
x_tilde <- lm(x ~ w) |> 
  residuals()

The next one regresses on and saves the residuals. Call these residuals y_tilde.

# Residuals from regression of Y on W
y_tilde <- lm(y ~ w) |>
  residuals()

To make the code that follows a little simpler, I’ll also create a helper function that runs a linear regression and returns the coefficients after stripping away any variable names.

get_coef <- function(formula) {
  formula |>  
    lm() |> 
    coef() |> 
    unname() 
}

Now we’re ready to compare some regressions! The “long regression” is a standard linear regression of on and . The “FWL Standard” is a regression of y_tilde on x_tilde. In other words, it regresses the residuals of on the residuals of . The FWL as it is usually encountered in textbooks implies that we should recover the same coefficient on in “Long Regression” and in “FWL Standard”, and indeed the simulation bears this out.

c(
  "Long Regression" = get_coef(y ~ x + w)[2],
  "FWL Standard" = get_coef(y_tilde ~ x_tilde - 1)[1], 
  "FWL Alternative" = get_coef(y ~ x_tilde)[2]
)

Long Regression    FWL Standard FWL Alternative 
      0.9937046       0.9937046       0.9937046 

But now take a look at “FWL” alternative: this is a regression of on x_tilde. Compared to the standard FWL approach, this version does not residualize with respect to . But it still gives us exactly the same coefficient on as the other two regressions. That leaves us with two unanswered questions:

1. Why does the “alternative” FWL approach work?
2. Given that the alternative approach works, why does anyone ever teach the “standard” version?

In the rest of this post we’ll answer both questions using simple algebra and the properties of linear regression. There are lots of deep ideas here, but there’s no need to bring out the big matrix algebra guns to explain them.

2 A Bit of Notation

First we need a bit of notation. I find it a bit simpler to work with population linear regressions rather than sample regressions, but the ideas are the same either way. So if you prefer to put “hats” on everything and work with sums rather than expectations and covariances, be my guest!

First we’ll define the “Long Regression” as a population linear regression of on and , namely Next I’ll define two additional population linear regressions: first the regression of on and second the regression of on I’ve already linked to a post making this point, but it bears repeating: all of the properties of the error terms , and that I’ve stated here hold by construction. They are not assumptions; they are merely what defines an error term in a population linear regression.

4 What’s different about the “usual” FWL theorem?

At this point you may be wondering why anyone teaches the “usual” version of the FWL theorem at all. If that extra short regression of on isn’t needed to learn , why bother?

To answer this question, we’ll start by re-writing the long regression two different ways. First, we’ll substitute into the long regression and re-arrange, yielding Next we’ll substitute on the left-hand side of the preceding equation and rearrange to isolate . This leaves us with Now we have two expressions, each with as one of the terms on the right-hand side and as another. Notice that both expressions have an intercept and a term in which is multiplied by a constant. What’s more, the intercepts are closely related across the two equations, as are the coefficients. I’m now going to make a bold assertion: the intercept and coefficient in the second expression, the one, are both equal to zero Perhaps you don’t believe me, but just for the moment suppose that I’m correct. In this case it would immediately follow that leaving us with two simple linear regressions, namely We’re tantalizingly close to unraveling the mystery of why the “usual” FWL theorem is so popular. But first we need to verify my bold claim from the previous paragraph. To do so, we’ll fall back on our old friend: the omitted variable bias formula, also known as the regression anatomy formula: Thus, as claimed. One down, one more to go. By definition, . Substituting the long regression for , we have by the linearity of expectation and the fact that by construction. Now, we’re trying to show that . Substituting for in this expression gives Inspecting our work so far, we see that the two alternative expressions for will be equal precisely when . But re-arranging this gives , which we already proved above using the omitted variables bias formula!

5 Taking Stock

That was a lot of algebra, so let’s spend some time thinking about the results. We showed that Now, if you’ll permit me, I’d like to re-write that first equality as Since is uncorrelated with , as explained above, and since by construction, it follows that is a bona fide population linear regression model. If we regress on the slope coefficient will be and the error term will be . This regression corresponds to the standard FWL theorem. Notice that it has an intercept of zero and an error term that is identical to that of the long regression. We can verify this using our simulation experiment from above as follows:

# Standard FWL has same residuals as long regression
u_hat <- resid(lm(y ~ x + w))
u_tilde <- resid(lm(y_tilde ~ x_tilde - 1))
all.equal(u_hat, u_tilde)

[1] TRUE

# Standard FWL has an intercept of zero (to machine precision!)
coef(lm(y_tilde ~ x_tilde))[1] # fit with intercept; check it's (numerically) 0

 (Intercept) 
8.273433e-17 

So what about ? This is the regression that corresponds to the alternative FWL theorem. Since and is uncorrelated with both and , this too is a population regression. But unless , it has a different error term. In other words, . Moreover, this regression includes an intercept that is not in general zero. Again we can verify this using our simulation example from above:

# Alternative FWL has different residuals than long regression
v_hat <- resid(lm(y ~ x_tilde))
all.equal(u_hat, v_hat)

[1] "Mean relative difference: 0.4905107"

# Alternative FWL has a non-zero intercept
coef(lm(y ~ x_tilde))[1]

(Intercept) 
  0.4878453 

6 The Punchline

If your goal is merely to learn , then either version of the FWL theorem will do the trick and the alternative version is simpler because it only involves one auxiliary regression instead of two. But if you want to ensure that you end up with the same error term as in the original long regression, then you need to use the standard version of the FWL theorem. This is crucial for the purposes of inference because the properties of the error term determine the standard errors of your estimates.

2 Solution

4 Conclusion

Before we conclude, there’s one important caveat to bear in mind. In addition to the qualifications that NQ isn’t quite JS, and that JS only dominates the MLE when , there’s one more fundamental issue that could be easily missed. Our decision to minimize composite MSE is absolutely crucial to the reasoning given above. The magic of shrinkage depends on our willingness to accept a trade-off in which we do a worse job estimating one mean in exchange for doing a better job estimating another, as composite MSE imposes. Whether this makes sense in practice depends on the context.

If we’re searching for a lost submarine in the ocean (a 3-dimensional problem), it makes perfect sense to be willing to be farther from the submarine in one dimension in exchange for being closer in another. That’s because Euclidean distance is obviously what we’re after here. But if instead we’re estimating teacher value-added and the results of our estimation exercise will be used to determine which teachers lose their jobs, it’s less clear that we should be willing to be farther from one teacher in exchange for being closer to another. Certainly that would be no consolation to someone who had been wrongly dismissed! If we were merely using this information to identify teachers who might need extra help, it’s another story. But the point I’m trying to make here is that our choice of which criterion to minimize necessarily encodes our values in a particular problem.

But with that said, I hope you’re satisfied that this extremely long post was worth the effort. Without using any fancy mathematics or statistical theory, we’ve managed to invent something that is nearly identical to the James-Stein estimator and thus to resolve Stein’s paradox. We started by pretending what we knew and showed that this would allow us to derive a shrinkage estimator with a lower composite MSE than the ML estimator. Then we simply plugged in an unbiased estimator of the key unknown quantity: . Because all the observations contain information about , it makes sense that we should decide how much to shrink one component by using all of the others. At this point, I hope that the James-Stein estimator seems not only plausible but practically obvious, excepting of course that pesky in the numerator.

1 A Quick Review

Consider a binary treatment and an observed outcome . Let be the potential outcomes corresponding to the treatment . Our goal is to learn the average treatment effect but, unless is randomly assigned, using the difference of observed means to estimate the ATE in general won’t work. The idea of selection-on-observables is that might be “as good as randomly assigned” after we adjust for a collection of observed covariates .

Regression adjustment relies on two assumptions: selection-on-observables and overlap. The selection-on-observables assumption says that learning provides no additional information about the average values of and , provided that we already know . This implies that we can learn the conditional average treatment effect (CATE) by comparing observed outcomes of the treated and untreated holding fixed: For example: older people might be more likely to take a new medication but also more likely to die without it. If so, perhaps by comparing average outcomes holding age fixed we can learn the causal effect of the medication. The overlap assumption says that, for any fixed value of the covariates, there are some treated and some untreated people. This allows us to learn for every value of in the population and average it using the law of iterated expectations to recover the ATE:
In the medication example, this would correspond to computing the difference of means for each age group separately, and then averaging them using the share of people in each age group. Notice that this is only possible if there are some people who took the medication and some who didn’t in each age group. That’s exactly what the overlap assumption buys us. For example, if there were no senior citizens who didn’t take the medication, we wouldn’t be able to learn the effect of the medication for senior citizens.

2 Which regression should we run?

So suppose that we’ve found a set of covariates that satisfy the required assumptions. How should we actually carry out regression adjustment? To answer this question, let’s start by making things a bit simpler. Suppose that is a single binary covariate. At the end of the post, we’ll return to the general case. Since and are both binary, we can write the conditional mean function of given as Since the true conditional mean function is linear, a linear regression of on , , and an intercept will recover . But what on earth do these coefficients actually mean?! Substituting all possible values of , And so, after a bit of re-arranging, What a mess! Alas, we’ll need a few more steps of algebra to figure out how these relate to the ATE. Notice that equals the CATE when since Proceeding similarly for the CATE when , we find that Now that we have expressions for each of the two conditional average treatment effects, corresponding to each of the values that can take, we’re finally ready to compute the ATE: where we define the shorthand . So to compute the ATE, we need to know the coefficients and from the regression of on , , and , in addition to the share of people with . Needless to say, your favorite regression package will not spit out the ATE for you if you run the regression from above. And it certainly won’t spit out the standard error! So what can we do besides computing everything by hand?

3 Two Simple Alternatives

It turns out that there are two simple ways to get the your favorite software package to spit out the ATE for you and associated standard error. Each involves a slight re-parameterization of the conditional mean expression from above. The first one replaces with where and . To see why this works, notice that This works perfectly well, but there’s something about it that offends my sense of order: why subtract the mean from in one place but not in another? If you share my aesthetic sensibilities, then you can feel free to replace that offending with another since where we define . Notice that the only coefficient that changes is the intercept, and we’re typically not interested in this anyway!

7 Conclusion

I hope this post has convinced you that regression adjustment isn’t simply a matter of tossing a collection of covariates into your regression! In general, the coefficient on in a regression of on and will not equal the ATE of . Instead it will be a weighted average of CATEs. To obtain the ATE we need to include an interaction between and . The simplest way to get your favorite statistical software package to calculate this for you, along with an appropriate standard error, is by de-meaning before including the interaction. And don’t forget that causal inference always requires untestable assumptions, in this case the selection-on-observables assumption. While implementation details are important, getting them right won’t make any difference if you’re not adjusting for the right covariates in the first place.

3 The Replication

#-------------------------------------------------------------------------------
# Sims, C. A., & Uhlig, H. (1991). Understanding unit rooters: A helicopter tour
#
# (See also: Example 6.10.6 from Poirier "Intermediate Statistics and 'Metrics")
#-------------------------------------------------------------------------------
# In the next section we will proceed to construct, by Monte Carlo, an estimated
# joint pdf for \rho and \hat{\rho} under a uniform prior pdf on \rho. We choose
# 31 values of \rho, from 0.8 to 1.1 at intervals of 0.01. We draw 10000 100 x 1
# iid N(0,1) vectors of random variables to use as realizations of \epsilon. For
# each of the 10000 \epsilon vectors and each of the 31 \rho values, we
# construct a y vector with y(0) = 0, y(t) generated by equation (1).
#
# Equation (1): y(t) = \rho y(t-1) + \epsilon(t), t = 0, ..., T
#
# For each of these y vectors, we construct \hat{\rho}. Using as bins the
# intervals [-\infty, 0.795), [0.795, 0.805), [0.805, 0.815), etc. we construct
# a histogram that estimates the pdf of \hat{rho} for each fixed \rho value.
# When these histograms are lined up side by side, they form a surface that is
# the joint pdf for \rho and \hat{\rho} under a flat prior on \rho.
#-------------------------------------------------------------------------------
set.seed(1693)

library(tidyverse)
library(tictoc)
library(patchwork)

draw_rho_hat <- function(rho) {
# Carry out the simulation once for a fixed value of rho; return rho_hat
  nT <- 100
  y <- rep(0, nT + 1)
  for (t in 2:(nT + 1)) {
    y[t] <- rho * y[t - 1] + rnorm(1)
  }
  y_t <- y[-1]
  y_tminus1 <- y[-length(y)]
  sum(y_t * y_tminus1) / sum(y_tminus1^2)
}

# Function to run the simulation for a fixed value of rho (10000 times)
run_sim <- \(rho) map_dbl(1:1e4, \(i) draw_rho_hat(rho))

tic()
foo <- run_sim(0.9)
toc() # ~0.6 seconds on my machine

0.38 sec elapsed

# Full sequence of rho values from Sims & Uhlig (1991)
rho <- seq(from = 0.8, to = 1.1, by = 0.01)

tic()
results <- tibble(rho = rho,
                  rho_hat = map(rho, run_sim)) # List columns
toc() # ~17 seconds on my machine (1991 was a long time ago!)

11.574 sec elapsed

# The results tibble uses a list column for rho_hat. This is convenient for
# making histograms of the frequentist sampling distribution (rho fixed) but
# not for making histograms of the Bayesian posterior (rho_hat) fixed. For the
# latter, we will use the unnest() function to "expand" the list column rho_hat
# into a regular column. This is the "joint" distribution of rho and rho_hat.
joint <- results |>
  unnest(rho_hat)

joint |>
  ggplot(aes(x = rho, y = rho_hat)) +
  geom_density2d_filled() +
  coord_cartesian(ylim = c(0.8, 1.1)) + # Restrict rho_hat axis
  labs(title = "Joint Distribution",
       x = expression(rho),
       y = expression(hat(rho)))

joint |>
  filter(rho_hat >= 0.995 & rho_hat < 1.005) |>
  ggplot(aes(x = rho)) +
  geom_histogram(binwidth = 0.01, fill = "skyblue", color = "black") +
  labs(title = expression(hat(rho) == 1),
       x = expression(rho),
       y = "Frequency")

joint |>
  filter(rho == 1) |>
  ggplot(aes(x = rho_hat)) +
  geom_histogram(binwidth = 0.01, fill = "skyblue", color = "black") +
  labs(title = expression(rho == 1),
       x = expression(hat(rho)),
       y = "Frequency")

# Function that makes the preceding two plots, puts them side-by-side and lets
# the user specify the value of rho/rho_hat that we condition on:
plot_Bayes_vs_Freq <- \(r) {
  p1 <- joint |>
    filter(rho_hat >= r - 0.005 & rho_hat < r + 0.005) |>
    ggplot(aes(x = rho)) +
    geom_histogram(aes(y = after_stat(density)),
                   binwidth = 0.01, fill = "skyblue", color = "black") +
    geom_vline(xintercept = r, color = "red", linetype = "dashed", linewidth = 1) +
    labs(title = bquote(hat(rho) == .(round(r, 3))),
         x = expression(rho))

  p2 <- joint |>
    filter(rho >= r - 0.005 & rho < r + 0.005) |>
    ggplot(aes(x = rho_hat)) +
    geom_histogram(aes(y = after_stat(density)),
                   binwidth = 0.01, fill = "skyblue", color = "black") +
    geom_vline(xintercept = r, color = "red", linetype = "dashed", linewidth = 1) +
    labs(title = bquote(rho == .(round(r, 3))),
         x = expression(hat(rho)))

  p1 + p2
}

plot_Bayes_vs_Freq(0.98)

plot_Bayes_vs_Freq(0.99)

plot_Bayes_vs_Freq(1.0)

plot_Bayes_vs_Freq(1.01)

plot_Bayes_vs_Freq(1.02)

4 Some Intuition

In our simulation, gave a worse estimate of than . The derivations from above show that this wasn’t a fluke: adding a valid instrument as an additional control regressor only makes the bias in our estimated causal effect worse than it was to begin with. This holds for any valid instrument and any endogenous regressor in a linear causal model. I hope you found the derivations from above convincing. But even so, you may be wondering if there’s an intuitive explanation for this phenomenon. I am pleased to inform you that the answer is yes!

In an earlier post I described the control function approach to instrumental variables regression. That post showed that the coefficient on in a regression of on and gives the correct causal effect. We don’t know , but we can estimate it by regressing on and saving the residuals. The logic of multiple regression shows that including as a control regressor “soaks up” the portion of that is explained by . Because represents the “bad” (endogenous) variation in , this solves our endogeneity problem. In effect, captures the unobserved “omitted variables” that play havoc with a naive regression of on .

Now, contrast this with a regression of on and . In this case, we soak up the variation in that is explained by . But represents the good (exogenous) variation in ! Soaking up this variation leaves only the bad variation behind, making our endogeneity problem worse than it was to begin with. In this example, is what is known as a bad control, a control regressor that makes things worse rather than better. A common piece of advice for avoiding bad controls is to only include control regressors that are correlated with and but are not themselves caused by . The example in this post shows that this advice is wrong. Here is not caused by , and is correlated with both and . Nevertheless, it is a bad control. In short, a valid instrument provides a powerful way to carry out causal inference from observational data, but only if you use it in the right way. A good instrument is a bad control!

2 Stirling’s Approximation

The key step in our argument is to show that the pmf of a random variable is well-approximated by the density. This explains the bell-shaped curve that we plotted above. To obtain this result, we’ll use the central limit theorem. But there is one fact that you will need to take on faith if you don’t already know it: if is independent of then . Proceeding by induction we can view a Poisson(171) random variable as the sum of 171 independent Poisson(1) random variables. More generally, we can view a Poisson RV with rate parameter as the sum of iid Poisson(1) random variables. By the central limit theorem, it follows that since the mean and variance of a Poisson(1) RV are both equal to one. From a practical perspective, this means that is approximately equal to , a standard normal random variable. Re-arranging, and is simply a random variable! This is a quick way of seeing why the distribution is well-approximated by the distribution when is large.

Now let’s run with this. As we just saw, for large the Poisson pmf is well-approximated by the Normal density: This approximation is particularly accurate for near the mean. This is convenient, because substituting for considerably simplifies the right hand side: Re-arranging, we obtain Taking logs of both sides gives: Writing this with in place of gives the following: This is called Stirling’s Approximation. The usual way of writing this excludes the term, yielding , which is fairly easy to remember. Including the extra term, however, gives increased accuracy for smaller values of . While I haven’t formally proved this, it turns out that as . In other words, the ratio of the LHS and RHS tends to one in the large limit. Perhaps surprisingly, this approximation is extremely accurate even for fairly small values of , as we can see by comparing it against lfactorial().

stirling1 <- function(n) n * log(n) - n
stirling2 <- function(n) n * log(n) - n + 0.5 * log(2 * pi * n)
tibble(n = 1:20) |>
  mutate(Stirling1 = stirling1(n),
         Stirling2 = stirling2(n),
         R = lfactorial(n)) |>
  knitr::kable(digits = 3)

3 Epilogue

I have a bad habit of trying to add a “moral” or “lesson” to the end of my posts, but I suppose there’s no point trying to break the habit today! While there are easier ways to derive Stirling’s approximation, there are two things I enjoy about this one. First, we get a more accurate approximation than with practically no effort. Second, making unexpected connections between facts that we already know both deepens our understanding and helps us “compress” information. If you ever forget Stirling’s approximation, now you know how to very quickly re-derive it on the spot!

1 The Setup

Consider a simple linear causal model of the form where is endogenous, i.e. related to the unobserved random variable . Our goal is to learn , the causal effect of on . To take a simple example, suppose that is wage and is years of schooling. Then is the causal effect of one additional year of schooling on a person’s wage. The random variable is a catchall, representing all unobserved causes of wage, such as ability, family background, and so on. A linear regression of on will not allow us to learn . For example, if you’re very smart, you will probably find school easier and stay in school longer. But being smarter likely has its own effect on your wage, separate from years of education. Ability is a confounder because it causes both years of schooling and wage.

Now suppose that is an instrumental variable: something that is uncorrelated with (exogenous) but correlated with (relevant). For example, a very famous paper pointed out that quarter of birth is correlated with years of schooling in the US and argued that it is unrelated to other causes of wages. Finding a good instrumental variable is very hard in practice. Indeed, I remain skeptical that quarter of birth is really unrelated to . But that’s a conversation for another day. For the moment, suppose we have a bona fide exogenous and relevant instrument at our disposal. To make things even simpler, suppose that the true causal effect is homogeneous, i.e. the same for everyone.

2 1st Perspective: The IV Approach

Regress on to find the causal effect of on . Rescale it to obtain the causal effect of on .

If is a valid and relevant instrument, then which is precisely the causal effect we’re after! The ratio of to is called the instrumental variables (IV) estimand, but it seems to come out of nowhere. A more intuitive way to write this quantity multiplies the numerator and denominator by to yield We see that is the ratio of two linear regression slopes: the slope from a regression of on divided by the slope from a regression of on . This makes intuitive sense if we think about units. Because is unrelated to , gives the causal effect of on . If is measured in dollars and is measured in miles (e.g. distance to college), then is measured in dollars per mile. If is years of schooling, then should be measured in dollars per year. To convert from dollars/mile to dollars/year, we need to multiply by miles/year or equivalently to divide by years/mile. And indeed, is measured in years/mile as required! This is yet another example of my favorite maxim: most formulas in statistics and econometrics are obvious if you keep track of the units.

3 2nd Perspective: The TSLS Approach

Construct by using to “clean out” the part of that is correlated with . Then regress on .

Let be the intercept and be the slope from a population linear regression of on . Defining , we can write By definition is the best linear predictor of based on , in that and solve the optimization problem What’s more, by construction since: And since is uncorrelated with , so is : So now we have a variable that is a good predictor of but is uncorrelated with . In essence, we’ve used to “clean out” the endogeneity from and we did this using a first stage regression of on . Two-stage least squares (TSLS) combines this with a second stage regression of on to recover . To see why this works, substitute for in the causal model, yielding where we define . Finally, since a regression of on recovers the causal effect of on .

4 3rd Perspective: The Control Function Approach

Use to solve for , the part of that is correlated with . Then regress on controlling for .

I’m willing to bet that you haven’t seen this approach before! The so-called control function approach starts from the same place as TSLS: the first-stage regression of on from above, namely Like the error term from the causal model , the first stage regression error is unobserved. But as strange as it sounds, imagine running a regression of on . Then we would obtain where by construction. Now, since the causal model for includes an intercept, . And since the first-stage linear regression model that defines likewise includes an intercept, as well. This means that so the regression of on becomes Now, substituting for in the causal model gives By construction . And since , it follows that Therefore, if only we could observe , a regression of on that controls for would allow us to recover the causal effect of interest, namely . Such a regression would also give us . To see why this is interesting, notice that Since , tell us the direction of endogeneity in . If then is positively correlated with , if then is negatively correlated with , and if then is exogenous. If is ability and ability has a positive effect on years of schooling, for example, then will be positive.

Now it’s time to address the elephant in the room: is unobserved! It’s all fine and well to say that if were observed our problems would be solved, but given that it is not in fact observed what are we supposed to do? Here’s where the TSLS first stage regression comes to the rescue. Both and are observed, so we can learn and by regressing on . Given these coefficients, we can simply solve for the unobserved error: . Like TSLS, the control function approach relies crucially on the first stage regression. But whereas TSLS uses it to construct , the control function approach uses it to construct . We don’t replace with its exogenous component ; instead we “pull out” the component of that is correlated with , namely . In effect we control for the “omitted variable” , hence the name control function.

5 Simulating the Three Approaches

Perhaps that was all a bit abstract. Let’s make it concrete by simulating some data and actually calculating estimates of using each of the three approaches described above. Because this exercise relies on a sample of data rather than a population, estimates will replace parameters and residuals will replace error terms.

To begin, we need to simulate independently of . For simplicity I’ll make these standard normal and set the correlation between and to 0.5.

set.seed(1983) # for replicability of pseudo-random draws
n <- 1000
Z <- rnorm(n)
library(mvtnorm)
cor_mat <- matrix(c(1, 0.5,
                    0.5, 1), 2, 2, byrow = TRUE)
errors <- rmvnorm(n, sigma = cor_mat)
head(errors)

           [,1]        [,2]
[1,]  0.1612255 -0.96692422
[2,]  1.4020130  1.55818062
[3,]  1.7212525 -0.01997204
[4,] -0.6972637 -0.68551762
[5,]  1.3471669 -0.01766333
[6,] -1.0441467 -0.23113677

U <- errors[,1]
V <- errors[,2]
rm(errors)

Since this is a simulation we actually can observe and and hence could regress the one on the other. Since I set the standard deviation of both of them equal to one, will simply equal the correlation between them, namely 0.5

coef(lm(U ~ V - 1)) # exclude an intercept

        V 
0.5047334 

Excellent! Everything is working as it should. The next step is to generate and . Again to keep things simple, in my simulation I’ll set .

pi <- 0.3
beta <- 1.1
X <- pi * Z + V
Y <- beta * X + U

Now we’re ready to run some regressions! We’ll start with an OLS regression of on . This substantially overestimates because is in fact positively correlated with .

OLS <- coef(lm(Y ~ X))[2]
OLS

       X 
1.567642 

In contrast, the IV approach works well.

IV <- cov(Y, Z) / cov(X, Z)
IV

[1] 1.049043

For the TSLS and control function approaches we need to run the first-stage regression of on and store the results.

first_stage <- lm(X ~ Z)

The TSLS approach uses the fitted values of this regression as .

Xtilde <- predict(first_stage)
TSLS <- coef(lm(Y ~ Xtilde))[2] # drop the intercept since we're not interested in it
TSLS

  Xtilde 
1.049043 

In contrast, the control function approach uses the residuals from the first stage regression. It also gives us in addition to .

Vhat <- residuals(first_stage) 
CF <- coef(lm(Y ~ X + Vhat))[-1] # drop the intercept since we're not interested in it
CF # The coefficient on Vhat is lambda

        X      Vhat 
1.0490432 0.5558904 

Notice that we obtain precisely the same estimates for using each of the three approaches.

c(IV, TSLS, CF[1])

           Xtilde        X 
1.049043 1.049043 1.049043 

It turns out that in this simple linear model with a single endogenous regressor and a single instrument, the three approaches are numerically equivalent. In other words, they give exactly the same answer. This will not necessarily be true in more complicated models, so be careful!

6 Epilogue

It’s time to admit that this post had a secret agenda: to introduce the idea of a control function in the simplest way possible! If you’re interested in learning more about control functions, a canonical example that does not turn out to be identical to IV is the so-called Heckman Selection Model, which you can learn more about here. (Scroll down until you see the heading “Heckman Selection Model.”) The basic logic is similar: to solve an endogeneity problem, use a first-stage regression to estimate an unobserved quantity that “soaks up” the part of the error term that is correlated with your endogenous regressor of interest. If these videos whet your appetite for more control function fun, Wooldridge (2015) provides a helpful overview along with many references to the econometrics literature.

1 Where does the regression line come from?

If you’ve taken an introductory statistics or econometrics course, you most likely learned that the least squares regression line minimizes the sum of squared vertical deviations by solving the optimization problem You probably also learned that the solution is given by In words: the regression slope equals the ratio of the covariance between height and handspan to the variance of handspan, and the regression line passes through the sample average values of height and handspan. We can check that all of these formulas agree with what we calculated above using lm()

b <- with(dat, cov(height, handspan) / var(handspan))
a <- with(dat, mean(height) - b * mean(handspan))
c(a, b)

[1] 40.943127  1.266961

coef(lm(height ~ handspan, dat))

(Intercept)    handspan 
  40.943127    1.266961 

This is all perfectly correct, and an entirely reasonable way of looking at the problem. But I’d now like to suggest a completely different way of looking at regression. Why bother? The more different ways we have of understanding an idea or method, the more deeply we understand how it works, when it works, and when it is likely to fail. So bear with me while I take you on what might at first appear to be a poorly-motivated computational detour. I promise that there’s a payoff at the end!

2 A Crazy Idea

There is a unique line that passes through any two distinct points in a plane. So here’s a crazy idea: let’s form every unique pair of students from my height and handspan dataset. To understand what I have in mind, consider a small subset of the data, call it test

test <- dat[1:3,]
test

# A tibble: 3 × 2
  height handspan
   <dbl>    <dbl>
1     73     22.5
2     65     17  
3     69     21  

With three students, there are three unique unordered pairs: . Corresponding to these three pairs are three line segments, one through each pair:

plot(height ~ handspan, test, pch = 20)
segments(x0 = c(17, 17, 21),     # FROM: x-coordinates
         y0 = c(65, 65, 69),     # FROM: y-coordinates
         x1 = c(21, 22.5, 22.5), # TO:   x-coordinates
         y1 = c(69, 73, 73),     # TO:   y-coordinates
         lty = 2)

And associated with each line segment is a slope

y_differences <- c(69 - 65, 73 - 65, 73 - 69)
x_differences <- c(21 - 17, 22.5 - 17, 22.5 - 21) 
slopes <- y_differences / x_differences
slopes

[1] 1.000000 1.454545 2.666667

And here’s my question for you: what, if anything, is the relationship between these three slopes and the slope of the regression line? While it’s a bit silly to run a regression with only three observations, the results are as follows:

coef(lm(height ~ handspan, test))

(Intercept)    handspan 
  41.556701    1.360825 

The slope of the regression line doesn’t equal any of the three slopes we calculated above, but it does lie between them. This makes sense: if the regression line were steeper or less steep than all three line segments from above, it couldn’t possibly minimize the sum of squared vertical deviations. Perhaps the regression slope is the arithmetic mean of slopes? No such luck:

mean(slopes)

[1] 1.707071

Something interesting is going on here, but it’s not clear what. To learn more, it would be helpful to play with more than three points. But doing this by hand would be extremely tedious. Time to write a function!

3 All Pairs of Students

The following function generates all unique pairs of elements taken from a vector x and stores them in a matrix:

make_pairs <- function(x) {
# Returns a data frame whose rows contain each unordered pair of elements of x
# i.e. all committees of two with members drawn from x
  n <- length(x)
  pair_indices <- combn(1:n, 2)
  matrix(x[c(pair_indices)], ncol = 2, byrow = TRUE)
}

For example, applying make_pairs() to the vector c(1, 2, 3, 4, 5) gives

make_pairs(1:5)

      [,1] [,2]
 [1,]    1    2
 [2,]    1    3
 [3,]    1    4
 [4,]    1    5
 [5,]    2    3
 [6,]    2    4
 [7,]    2    5
 [8,]    3    4
 [9,]    3    5
[10,]    4    5

Notice that make_pairs() is constructed such that order doesn’t matter: we don’t distinguish between and , for example. This makes sense for our example: Alice and Bob denotes the same pair of students as Bob and Alice.

To generate all possible pairs of students from dat, we apply make_pairs() to dat$handspan and dat$height separately and then combine the result into a single dataframe:

regression_pairs <- data.frame(make_pairs(dat$handspan), 
                               make_pairs(dat$height))
head(regression_pairs)

    X1   X2 X1.1 X2.1
1 22.5 17.0   73   65
2 22.5 21.0   73   69
3 22.5 25.5   73   71
4 22.5 25.0   73   78
5 22.5 20.0   73   68
6 22.5 20.5   73   75

The names are ugly, so let’s clean them up a bit. Handspan is our “x” variable and height is our “y” variable, so we’ll refer to the measurements from each pair as x1, x2, y1, y2

names(regression_pairs) <- c('x1', 'x2', 'y1', 'y2')
head(regression_pairs)

    x1   x2 y1 y2
1 22.5 17.0 73 65
2 22.5 21.0 73 69
3 22.5 25.5 73 71
4 22.5 25.0 73 78
5 22.5 20.0 73 68
6 22.5 20.5 73 75

The 1 and 2 indices indicate a particular student: in a given row x1 and y1 are the handspan and height of the “first student” in the pair while x2 and y2 are the handspan and height of the “second student.” Each student appears many times in the regression_pairs dataframe. This is because there are many pairs of students that include Alice: we can pair her up with any other student in the class. For this reason, regression_pairs has a tremendous number of rows, 52975 to be precise. This is the number of ways to form a committee of size 2 from a collection of 326 people when order doesn’t matter:

choose(326, 2)

[1] 52975

4 I just calculated 52,975 slopes.

Corresponding to each row of regression_pairs is a slope. We can calculate and summarize them as follows, using the dplyr package from the tidyverse to make things easier to read:

library(dplyr)
regression_pairs <- regression_pairs |>
  mutate(slope = (y2 - y1) / (x2 - x1))
regression_pairs |>
  pull(slope) |>
  summary()

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.     NAs 
   -Inf   0.000   1.333     NaN   2.667     Inf     419 

The sample mean slope is NaN, the minimum is -Inf, the maximum is Inf, and there are 419 missing values. So what on earth does this mean? First things first: the abbreviation NaN stands for “not a number.” This is R’s way of expressing , and an NaN “counts” as a missing value:

0/0

[1] NaN

is.na(0/0)

[1] TRUE

In contrast, Inf and -Inf are R’s way of expressing . These do not count as missing values, and they also arise when a number is too big or too small for your computer to represent:

c(-1/0, 1/0, exp(9999999), -exp(9999999))

[1] -Inf  Inf  Inf -Inf

is.na(c(Inf, -Inf))

[1] FALSE FALSE

So where do these Inf and NaN values come from? Our slope calculation from above was (y2 - y1) / (x2 - x1). If x2 == x1, the denominator is zero. This occurs when the two students in a given pair have the same handspan. Because we only measured handspan to the nearest 0.5cm, there are many such pairs. Indeed, handspan only takes on 23 distinct values in our dataset but there are 326 students:

sort(unique(dat$handspan))

 [1] 16.5 17.0 17.5 18.0 18.5 19.0 19.5 20.0 20.5 21.0 21.5 22.0 22.5 23.0 23.5
[16] 24.0 24.5 25.0 25.5 26.0 26.5 27.0 28.0

If y1 and y2 are different but x1 and x2 are the same, the slope will either be Inf or -Inf, depending on whether y1 > y2 or the reverse. When y1 == y2 and x1 == x2 the slope is NaN.

This isn’t an arcane numerical problem. When y1 == y2 and x1 == x2, our pair contains only a single point, so there’s no way to draw a line segment. With no line to draw, there’s clearly no slope to calculate. When y1 != y2 but x1 == x2 we can draw a line segment, but it will be vertical. Should we call the slope of this vertical line ? Or should we call it ? Because the labels 1 and 2 for the students in a given pair were arbitrary–order doesn’t matter–there’s no way to choose between Inf and -Inf. From the perspective of calculating a slope, it simply doesn’t make sense to construct pairs of students with the same handspan.

With this in mind, let’s see what happens if we average all of the slopes that are not -Inf, Inf, or NaN

regression_pairs |>
  filter(!is.na(slope) & !is.infinite(slope)) |>
  pull(slope) |>
  mean()

[1] 1.259927

This is tantalizingly close to the slope of the regression line from above: 1.266961. But it’s still slightly off.

5 Not All Slopes are Created Equal

The median handspan in my dataset is 21.5. Let’s take a closer look at the heights of students whose handspans are close to this value:

boxplot(height ~ handspan, dat, subset = handspan %in% c(21, 21.5, 22))

There is a large amount of variation in height for a given value of handspan. Indeed, from this boxplot alone you might not even guess that there is a strong positive relationship between height and handspan in the dataset as a whole! The “boxes” in the figure, representing the middle 50% of heights for a given handspan, overlap substantially. If we were to randomly choose one student with a handspan of 21 and another with a handspan of 21.5, it’s quite likely that the slope between them would be negative. It’s true that students with bigger hands are taller on average. But the difference in height that we’d predict for two students who differed by 0.5cm in handspan is very small: 0.6 inches according to the linear regression from the beginning of this post. In contrast, the standard deviation of height among students with the median handspan is more than five times as large:

sd(subset(dat, handspan == 21.5)$height)

[1] 3.172248

The 25th percentile of handspan is 20 while the 75th percentile is 23. Comparing the heights of students with these handspans rather than those close to the median, gives a very different picture:

boxplot(height ~ handspan, dat, subset = handspan %in% c(20, 23))

Now there’s much less overlap in the distributions of height. This accords with the predictions of the linear regression from above: for two students whose handspan differs by 3cm, we would predict a difference of 3.8 inches in height. This difference is big enough to discern in spite of the variation in height for students with the same handspan. If we were to choose one student with a handspan of 20cm and another with a handspan of 23cm, it’s fairly unlikely that the slope between these points would be negative.

So where does this leave us? Above we saw that forming a pair of students with the same handspan does not allow us to calculate a slope. Now we’ve seen that the slope for a pair of students with a very similar handspan can give a misleading impression about the overall relationship. This turns out to be the key to our puzzle from above. The ordinary least squares slope estimate does equal an average of the slopes for each pair of students, but this average gives more weight to pairs with a larger difference in handspan. As I’ll derive in my next post: The notation is shorthand for the set , in other words the set of all unique pairs where order doesn’t matter. The weights are between zero and one and sum to one over all pairs. Pairs with are given zero weight; pairs in which is far from are given more weight than pairs where these values are closer. And you don’t have to wait for my next post to see that it works:

regression_pairs <- regression_pairs |>
  mutate(x_dist = abs(x2 - x1),
         weight = x_dist^2 / sum(x_dist^2))

regression_pairs |>
  filter(!is.infinite(slope) & !is.na(slope)) |>
  summarize(sum(weight * slope)) |>
  pull()

[1] 1.266961

coef(lm(height ~ handspan, dat))[2]

handspan 
1.266961 

So there you have it: in a simple linear regression, the OLS slope estimate is a weighted average of the slopes of the line segments between all pairs of observations. The weights are proportional to the squared Euclidean distance between -coordinates. I’ll leave things here for today, but there’s much more to say on this topic. Stay tuned for the next installment!

1 Tests and CIs – Two Sides of the Same Coin

Suppose that we observe a random sample from a normal population with unknown mean and known variance . Under these assumptions, the sample mean follows a distribution. Centering and standardizing, Now, suppose we want to test against the two-sided alternative at the 5% significance level. If , then the test statistic follows a standard normal distribution. If , then does not follow a standard normal distribution. To carry out the test, we reject if is greater than , the quantile of a standard normal distribution for . To put it another way, we fail to reject if . So for what values of will we fail to reject? By the definition of absolute value and the definition of from above, is equivalent to Re-arranging, this in turn is equivalent to This tells us that the values of we will fail to reject are precisely those that lie in the interval . Does this look familiar? It should: it’s the usual 95% confidence interval for the mean of a normal population with known variance. The 95% confidence interval corresponds exactly to the set of values that we fail to reject at the 5% level.

This example is a special case a more general result. If you give me a confidence interval for a parameter , I can use it to test against . All I have to do is check whether lies inside the confidence interval, in which case I fail to reject, or outside, in which case I reject. Conversely, if you give me a two-sided test of with significance level , I can use it to construct a confidence interval for . All I have to do is collect the values of that are not rejected. This procedure is called inverting a test.

2 How to Confuse Your Introductory Statistics Students

Around the same time as we teach students the duality between testing and confidence intervals–you can use a confidence interval to carry out a test or a test to construct a confidence interval–we throw a wrench into the works. The most commonly-presented test for a population proportion does not coincide with the most commonly-presented confidence interval for . To quote from page 355 of Kosuke Imai’s fantastic textbook Quantitative Social Science: An Introduction

the standard error used for confidence intervals is different from the standard error used for hypothesis testing. This is because the latter standard error is derived under the null hypothesis … whereas the standard error for confidence intervals is computed using the estimated proportion.

Let’s translate this into mathematics. Suppose that and let . The two standard errors that Imai describes are Following the advice of our introductory textbook, we test against at the level by checking whether exceeds . This is called the score test for a proportion. Again following the advice of our introductory textbook, we report as our 95% confidence interval for . As you may recall from my earlier post, this is the so-called Wald confidence interval for . Because the two standard error formulas in general disagree, the relationship between tests and confidence intervals breaks down.

To make this more concrete, let’s plug in some numbers. Suppose that and our observed sample contains 5 ones and 20 zeros. Then and we can calculate and the Wald confidence interval as follows

n <- 25
n1 <- 5
p_hat <- n1 / n
alpha <- 0.05
SE_hat <- sqrt(p_hat * (1 - p_hat) / n)
p_hat + c(-1, 1) * qnorm(1 - alpha / 2) * SE_hat

[1] 0.04320288 0.35679712

The value 0.07 is well within this interval. This suggests that we should fail to reject against the two-sided alternative. But when we compute the score test statistic we obtain a value well above 1.96, so that is soundly rejected:

p0 <- 0.07
SE0 <- sqrt(p0 * (1 - p0) / n)
abs((p_hat - p0) / SE0)

[1] 2.547551

The test says reject and the confidence interval says don’t. Upon encountering this example, your students decide that statistics is a tangled mess of contradictions, despair of ever making sense of it, and resign themselves to simply memorizing the requisite formulas for the exam.

4 Inverting the Score Test

Suppose we collect all values that the score test does not reject at the 5% level. If the score test is working well–if its nominal type I error rate is close to 5%–the resulting set of values will be an approximate confidence interval for . Why is this so? Suppose that is the true population proportion. Then an interval constructed in this way will cover precisely when the score test does not reject . This occurs with probability . Because the score test is much more accurate than the Wald test, the confidence interval that we obtain by inverting it way will be much more accurate than the Wald interval. This interval is called the score interval or the Wilson interval.

So let’s do it: let’s invert the score test. Our goal is to find all values such that where is the normal critical value for a two-sided test with significance level . Squaring both sides of the inequality and substituting the definition of from above gives Multiplying both sides of the inequality by , expanding, and re-arranging leaves us with a quadratic inequality in , namely Remember: we are trying to find the values of that satisfy the inequality. The terms along with and are constants. Once we choose , the critical value is known. Once we observe the data, and are known. Since , the left-hand side of the inequality is a parabola in that opens upwards. This means that the values of that satisfy the inequality must lie between the roots of the quadratic equation By the quadratic formula, these roots are Factoring out of the numerator and denominator of the right-hand side and simplifying, we can re-write this as using our definition of from above. And there you have it: the right-hand side of the final equality is the Wilson confidence interval for a proportion, where is the normal critical value for a two-sided test with significance level , and .

Compared to the Wald interval, , the Wilson interval is certainly more complicated. But it is constructed from exactly the same information: the sample proportion , two-sided critical value and sample size . Computing it by hand is tedious, but programming it in R is a snap:

get_wilson_CI <- function(x, alpha = 0.05) {
  #-----------------------------------------------------------------------------
  # Compute the Wilson (aka Score) confidence interval for a popn. proportion
  #-----------------------------------------------------------------------------
  # x        vector of data (zeros and ones)
  # alpha    1 - (confidence level)
  #-----------------------------------------------------------------------------
  n <- length(x)
  p_hat <- mean(x)
  SE_hat_sq <- p_hat * (1 - p_hat) / n
  crit <- qnorm(1 - alpha / 2)
  omega <- n / (n + crit^2)
  A <- p_hat + crit^2 / (2 * n)
  B <- crit * sqrt(SE_hat_sq + crit^2 / (4 * n^2))
  CI <- c('lower' = omega * (A - B), 
          'upper' = omega * (A + B))
  return(CI)
}

Notice that this is only slightly more complicated to implement than the Wald confidence interval:

get_wald_CI <- function(x, alpha = 0.05) {
  #-----------------------------------------------------------------------------
  # Compute the Wald confidence interval for a popn. proportion
  #-----------------------------------------------------------------------------
  # x        vector of data (zeros and ones)
  # alpha    1 - (confidence level)
  #-----------------------------------------------------------------------------
  n <- length(x)
  p_hat <- mean(x)
  SE_hat <- sqrt(p_hat * (1 - p_hat) / n)
  ME <- qnorm(1 - alpha / 2) * SE_hat
  CI <- c('lower' = p_hat - ME, 
          'upper' = p_hat + ME)
  return(CI)
}

With a computer rather than pen and paper there’s very little cost using the more accurate interval. Indeed, the built-in R function prop.test() reports the Wilson confidence interval rather than the Wald interval:

set.seed(1234)
x <- rbinom(20, 1, 0.5)
prop.test(sum(x), length(x), correct = FALSE) # no continuity correction

    1-sample proportions test without continuity correction

data:  sum(x) out of length(x), null probability 0.5
X-squared = 0.2, df = 1, p-value = 0.6547
alternative hypothesis: true p is not equal to 0.5
95 percent confidence interval:
 0.3420853 0.7418021
sample estimates:
   p 
0.55 

get_wilson_CI(x)

    lower     upper 
0.3420853 0.7418021 

7 More to Come on Inference for a Proportion!

This has been a post of epic proportions, pun very much intended. Amazingly, we have yet to fully exhaust this seemingly trivial problem. In a future post I will explore yet another approach to inference: the likelihood ratio test and its corresponding confidence interval. This will complete the classical “trinity” of tests for maximum likelihood estimation: Wald, Score (Lagrange Multiplier), and Likelihood Ratio. In yet another future post, I will revisit this problem from a Bayesian perspective, uncovering many unexpected connections along the way. Until then, be sure to maintain a sense of proportion in all your inferences and never use the Wald confidence interval for a proportion.

8 Appendix: R Code

get_test_size <- function(p_true, n, test, alpha = 0.05) {
# Compute the size of a hypothesis test for a population proportion  
#   p_true    true population proportion
#   n         sample size
#   test      function of p_hat, n, and p_0 that computes test stat 
#   alpha     nominal size of the test
  x <- 0:n
  p_x <- dbinom(x, n, p_true)
  test_stats <- test(p_hat = x / n, sample_size = n, p0 = p_true)
  reject <- abs(test_stats) > qnorm(1 - alpha / 2)
  sum(reject * p_x)
}
get_score_test_stat <- function(p_hat, sample_size, p0) {
  SE_0 <- sqrt(p0 * (1 - p0) / sample_size)
  return((p_hat - p0) / SE_0)
}
get_wald_test_stat <- function(p_hat, sample_size, p0) {
  SE_hat <- sqrt(p_hat * (1 - p_hat) / sample_size)
  return((p_hat - p0) / SE_hat)
}

plot_size <- function(n, test, nominal = 0.05, title = '') {
  p_seq <- seq(from = 0.01, to = 0.99, by = 0.001)
  size <- sapply(p_seq, function(p) get_test_size(p, n, test, nominal))
  plot(p_seq, size, type = 'l', xlab = 'p', 
       ylab = 'Type I Error Rate', 
       main = title)
  text(0.5, 0.98 * max(size), bquote(n == .(n)))
  abline(h = nominal, lty = 2, col = 'red', lwd = 2)
}
plot_size_comparison <- function(n, nominal = 0.05) {
  par(mfrow = c(1, 2))
  plot_size(n, get_score_test_stat, nominal, title = 'Score Test')
  plot_size(n, get_wald_test_stat, nominal, title = 'Wald Test')
  par(mfrow = c(1, 1))
}

2 Numeric Types in R

Because R is a dynamically-typed programming language, we can almost always ignore the question of precisely how it stores numeric values “under the hood.” In fact R has two numeric types: integer and double. Integers are rare in practice. The operator : returns an integer vector

y <- 1:5
typeof(y)

[1] "integer"

and the length of a vector is always an integer

n <- length(y)
typeof(n)

[1] "integer"

but nearly every other numeric value you encounter in R will be stored as a double, i.e. a double precision floating point number:

typeof(4)

[1] "double"

typeof(4.0)

[1] "double"

typeof(cos(0))

[1] "double"

To force R to store a value as an integer rather than double, we can either append an L

z <- 4L
typeof(z)

[1] "integer"

or coerce, i.e. convert, a double to an integer using as.integer()

a <- 4
typeof(a)

[1] "double"

b <- as.integer(a)
typeof(b)

[1] "integer"

The trade-off between integers and doubles is between precision and range. Calculations carried out with integers are always exact, but integers can only be used to represent a fairly limited number of values. Calculations with doubles, on the other hand, are not always exact, but doubles can store a much larger range of values, including decimals.

This post isn’t the right place to delve into the details of floating point numbers, of which doubles are an instance, but there are two points worth emphasizing. First, it’s generally safe to store a value that is “truly” an integer, e.g. 4, as double. As explained in the help file integer {base}

current implementations of R use 32-bit integers for integer vectors, so the range of representable integers is restricted to about +/-2*10^9: doubles can hold much larger integers exactly.

This explains why you may never have encountered the L suffix in the wild. Because doubles can represent very large integers exactly, calculations with whole numbers stored as doubles will also be exact. Notice that R automatically converts integers that are “too big” into doubles:

x <- 999999999L
typeof(x)

[1] "integer"

y <- 9999999999L
typeof(y)

[1] "double"

While converting integers to doubles in innocuous, you need to be careful when converting doubles to integers. This turns out to be the root of our problem with rep() from above. Both 0.2 * 20 and (1 - 0.8) * 20 are doubles, and both appear to equal 4

x <- 0.2 * 20
x

[1] 4

typeof(x)

[1] "double"

y <- (1 - 0.8) * 20
y

[1] 4

But x and y are coerced to different integer values:

as.integer(x)

[1] 4

as.integer(y)

[1] 3

The function rep() expects its second argument times to be an integer. If we supply a double instead, then rep() makes a conversion in the same way as the function as.integer(), namely by truncating. Far down in the help file for rep() we find this crucial caveat:

Non-integer values of times will be truncated towards zero. If times is a computed quantity it is prudent to add a small fuzz or use round().

But wait: aren’t x and y precisely equal to each other? How can one truncate to 4 while the other truncates to 3? As it turns out, appearances can be deceiving:

identical(x, y)

[1] FALSE

To find out why these values aren’t equal, we need to learn a bit more about how computers approximate real numbers using doubles.

3 What you see isn’t always what you get.

R has various handy built-in constants, including

pi

[1] 3.141593

Notwithstanding bill number 246 of the 1897 sitting of the Indiana General Assembly, is an irrational number. By default, however, R only shows us a small number of its digits. To display twenty digits of , we can specify the argument digits to the function print() like so

print(pi, digits = 20)

[1] 3.141592653589793116

To see even more digits, we can use the function sprintf(). Let’s try to display 60 digits of :

sprintf("%.60f", pi)

[1] "3.141592653589793115997963468544185161590576171875000000000000"

Why do the last twelve decimal points display as zero? The answer is that computers cannot represent real numbers to infinite precision. At some point, the remaining digits of get chopped off, and we’re left with an approximation that’s more than sufficient for any practical application.

At first glance, the number 0.8 seems nothing like . It is, after all, a rational number: 4/5. But sprintf() reveals that there’s more here than meets the eye:

sprintf("%.54f", 0.8)

[1] "0.800000000000000044408920985006261616945266723632812500"

The fraction 4/5 cannot be represented exactly as a double; it can only be approximated. The same is true of 1/5. Because 0.2 and 0.8 can only be represented approximately, 1 - 0.8 and 0.2 turn out not to be equal from the computer’s perspective:

identical(1 - 0.8, 0.2)

[1] FALSE

This in turn explains why (1 - 0.8) * 20 and 0.2 * 20 truncate to different integer values:

sprintf("%.54f", 20 * c(1 - 0.8, 0.2))