Mostly Harmless Econometrics: An Empiricist’s Companion

Serial Correlation in Panels and Difference-in-Difference Models

Serial correlation - the tendency for one observation to be correlated with those that have gone before - used to be Somebody Else’s Problem, specifically, the unfortunate souls who make their living out of time series data (macroeconomists, for example). Applied microeconometricians have therefore long ignored it.[126] But our data often have a time dimension too, especially in differences-in-differences models. This fact combined with clustering can have a major impact on statistical inference.

Suppose, as in Section 5.2, that we are interested in the effects of a state minimum wage. In this context, the regression version of differences-in-differences includes additive state and time effects. We therefore we get an equation like (5.2.3), repeated below:

y ist = 7s + Dst + "ist; (8.2.9)

As before, Уist is the outcome for individual i in state s in year t and Dst is a dummy variable that indicates treatment states in post-treatment periods.

The error term in (8.2.9) reflects the idiosyncratic variation in potential outcomes that varies across people, states, and time. Some of this variation is likely to be common to individuals in the same state and year, for example, a regional business cycle. We can model this common component by thinking of £ist as the sum of a state-year shock, vst, and an idiosyncratic individual component, ^st. So we have:

y ist = 7s + ^t + fiDst + vst + Vist. (8.2.10)

We assume that in repeated draws across states and over time, E[vst] = 0, while E[^st] =0 by definition.

State-year shocks are bad news for differences-in-differences models. As with the Moulton problem, state - and time-specific random effects generate a clustering problem that affects statistical inference. But that might be the least of our problems in this case. To see why, suppose we have only two periods and two states, as in the Card and Krueger (1994) New Jersey/Pennsylvania study. The empirical difference-in-differences is

fiCK = (y s=NJ;t=Nov y s=NJ;t=Feb) (ys=PA, t=Nov ys=PA, t=Feb).

This estimator is unbiased since E[vst] = E[^st] = 0. On the other hand, assuming we think of probability limits as increasing group size while keeping the choice of states and periods fixed, state-year shocks render fck inconsistent:

plim'f3CK = f + {(vs=Nj;t=Nov — Vs=NJ, t=Feb) — (v. s=PA, t=Nov ~ Vs=PA, t=Feb)}-

Averaging larger and larger samples within New Jersey or Pennsylvania in a given period does nothing to eliminate the regional shocks specific to a given location and period. With only two states and years, we have no way to distinguish the differences-in-differences generated by a policy change from the difference-in- dfferences due to the fact that, say, the New Jersey economy was holding steady in 1992 while Pennsylvania was experiencing a mild cyclical downturn. We can think of the presence of vst as a failure of the common trends assumption discussed in Section 5.2.

The solution to the inconsistency induced by random shocks in differences in differences models is to have either multiple time periods or many states (or both). For example, Card (1992) uses 51 states to study minimum wage changes while Card and Krueger (2000) take another look at the New Jersey-Pennsylvania experiment with a longer monthly time series of payroll data. With multiple states and/or periods, we can hope that the vst average out to zero. As in the first part of this chapter on the Moulton problem, the inference framework in this context relies on asymptotic distribution theory with many groups and not on group size (or, at least, not on group size alone). The most important inference issue then becomes the behavior of vst. In particular, if we are prepared to assume that shocks are independent across states and over time - i. e., they are serially uncorrelated - we are back to the plain-vanilla Moulton problem in Section

8.2.1, in which case we would cluster by state x year. But in most cases, the assumption that vst is serially uncorrelated is hard to defend. Almost certainly, for example, regional shocks are highly serially correlated: if things are bad in Pennsylvania in one month, they are likely to be just about as bad in the next.

The consequences of serial correlation for clustered panels are highlighted by Bertrand, Duflo, and Mul - lainathan (2004) and Kezdi (2004). Any research design with a group structure where the group means are correlated can be said to have the serial correlation problem. The upshot of recent work on serial correlation in data with a group structure is that, just as we must adjust our standard errors for the correlation within groups induced by the presence of vst, we must further adjust for serial correlation in the vst themselves. There are a number of ways to do this, not all equally effective in all situations. It seems fair to say that the question of how best to approach the serial correlation problem is currently under study and a consensus has not yet emerged. We try here to give a flavor of the approaches and summarize the emerging findings.

The simplest and most widely applied approach is simply to pass the clustering buck one level higher. So in the state-year example, we can report Liang and Zeger (1986) standard errors clustered by state instead of by state and year (e. g., using Stata cluster). This might seem odd at first blush, since the model controls for state effects. The state effect, js, in (8.2.10) removes the time mean of vst, which we denote by vs. Nevertheless, vst — vs is probably still serially correlated. Clustering at the state level takes account of this since the one-level-up clustered covariance estimator allows for completely non-parametric residual correlation within clusters - including the time series correlation in vst — vs. This is a quick and easy fix. The problem here, as you might have guessed, is that passing the buck up one level reduces the number of clusters. And asymptotic inference supposes we have a large number of clusters because we need a lot of states or periods to estimate the correlation between vst — vs and vst-i — vs reasonably well. Few clusters means biased standard errors and misleading inferences.