What Are We Estimating When We Estimate Difference-in-Differences?

|

Difference-in-differences estimation is one of the most widely used quasi-experimental tools for measuring the impacts of development policies.  In 2018, I calculate that more than 5 percent of articles published in the Journal of Development Economics used a difference-in-differences (or “DD”) methodology.  In DD estimation, a researcher compares the change in outcomes in a (non-random) treatment group before vs. after treatment to the change in outcomes in a comparison group over the same time period (even though the comparison group never received treatment).  The figure below illustrates the basic idea.

The DD estimate of the treatment effect is: (B – A) – (D – C).  Intuitively, pre-treatment differences between the treatment group and the comparison group reflect selection bias, while pre-period vs. post-period changes in outcomes within the comparison group reflect time trends.  The DD approach removes these confounds (under certain assumptions) by differencing them out, leaving us with a credible quasi-experimental estimate of the treatment effect of interest.  A more detailed overview of DD methods is available here.

The DD approach is often used to estimate the impacts of policies that are implemented at different times in different regions, for example policies like Medicaid and food stamps that were implemented in different times in different U.S. states.  In development, this approach was recently used by Jesse Antilla-Hughes, Lia Fernald, Paul Gertler, Patrick Krause, and Bruce Wydick to study the impacts of infant formula on child mortality in low- and middle-income countries.  In such settings, researchers typically implement the DD using two-way fixed effects models controlling for both period-specific and unit-specific shocks.

Though the two-way fixed effects approach to DD is widely used, the formal justification for treating it as a DD estimator is often fairly ad hoc.  A recent working paper by Andrew Goodman-Bacon, an Assistant Professor of Economics at Vanderbilt University, looks under the hood of the two-way fixed effects approach to DD estimation.  Goodman-Bacon shows that any two-way fixed effects estimate of DD relying on variation in treatment timing can be decomposed into a weighted average of all possible two-by-two difference-in-differences estimators that can be constructed from the panel data set.

Consider a (hypothetical) data set comprising two types of countries:  one group that made primary education free in 2000, and another group that made primary education free in 2005.  Call the first group the “early” countries and the second group the “late” countries. Suppose you had data on primary school completion rates every year from 1990 to 2010.  Such a data set allows for two distinct DD comparisons.  First, we could focus on the period from 1990 to 2004 (the left panel in the figure below).  In that time frame, the late adopter countries are “never treated” in the sense that they do not implement free primary education – so they can be used as a comparison group to estimate the impact of free primary on test scores in the early adopter countries.  However, we can also construct a second DD estimate of the treatment effect of free primary school by focusing on the years 2001 to 2010.  During that period, the treatment status of the early adopters never changes – they remain treated throughout – so they can be used as a comparison group to estimate the impact of free primary on test scores in the late adopting countries.

Goodman-Bacon shows that any two-way fixed effects estimate of DD with variation in treatment timing can be decomposed in this way.  It is a weighted average of (1) comparisons between (relatively) early adopters and later adopters over the periods when the later adopters are not yet treated, (2) comparisons between early adopters and later adopters over the periods when the early adopters are treated – so that they can be used as a comparison group for the later adopters, and (3) comparisons between different timing groups (e.g., early adopters or later adopters) and the never-treated group, if there is one.

Though interesting in its own right, this decomposition has several important implications.  First, two-way fixed effects estimates of DD that rely on variation in treatment timing only recover the average treatment effect when treatment effects are homogeneous.  When treatment effects are heterogeneous across units, OLS over-weights units with more variance in treatment status in order to achieve a more precise estimate of the treatment effect.  Hence, units that are treated near the middle of the evaluation window receive relatively more weight.  If one wishes to estimate the average treatment effect on the treated, some sort of re-weighting is required.

Second, and more troublingly, DD estimates are biased when treatment effects change over time within unit.  Intuitively, this occurs because already treated units serve as controls in some of the two-by-two DDs underlying the weighted average.  When treatment effects are not constant over time (so, for example, the treatment effect in the first year after treatment differs from the treatment effect five years after treatment), using already treated units as controls necessarily biases estimates of the treatment effect (by introducing a term representing the change in the treatment effect on the already treated units).  In such situations, Goodman-Bacon’s analysis shows that two-way fixed effects estimators are not appropriate, and alternative approaches (e.g., event study estimation) should be used.

In light of this, it is important to check whether common trends are satisfied in any panel data set being used for DD analysis.  When treatment turns on at different times, one way to do this graphically is to re-center and stack all the possible two-by-two DD comparisons.  Critically, identification relies on common trends both before and after treatment – as discussed above.  The good news is that, because of the re-weighting of the underlying two-by-two DD estimators, some violations of common trends are worse than others.  Goodman-Bacon provides instructions for testing “variance-weighted common trends” to assess the severity of any observed deviations.

If this all sounds like it raises the bar for DD analysis: it does.  Fortunately, there’s already a Stata command to help you implement the Goodman-Bacon’s DD decomposition.  Search for “bacondecomp” and you will find Goodman-Bacon, Goldring, and Nichols’ (2019) helpful tool for decomposing and plotting the underlying variation in your two-by-two DD estimates.

Join the Conversation

Irene
September 30, 2019

Thank you for writing this so concisely. I have been seeing snippet conversations of this on Twitter and was having a hard time following along.

Kelly Jones
September 30, 2019

Can you point me to background on how event study estimators differ from a D-in-D with two-way fixed effects? I think folks often refer to the latter (perhaps incorrectly) as an "event study D-in-D".

Pam Jakiela
October 01, 2019

Hi Kelly!

Yes, people sometimes use the terms interchangeably, but I think of the event study specification as a diff-in-diff that includes lagged treatment dummies (and potentially leads as well) - in other words, separate dummies for the 1, 2, 3, etc. years/periods after treatment starts. A recent paper that focuses more on the event study specification is here: https://arxiv.org/pdf/1804.05785.pdf.

Pam

BT
September 30, 2019

"When treatment effects are not constant over time (so, for example, the treatment effect in the first year after treatment differs from the treatment effect five years after treatment), using already treated units as controls necessarily biases estimates of the treatment effect (by introducing a term representing the change in the treatment effect on the already treated units)."

I do not think 'biases estimates is a good enough criticism. In what way does it bias the estimate. If the treatment effects are positive and increase over time, using already treated as control will, as far as I can see it, bias the estimates downwards. If the effects are still significant, one can always add the qualifier that the `true effects', if such a thing exists, must be higher.

October 01, 2019

You can see Andrew’s discussion of the bias on pages 11-13 of the paper: https://cdn.vanderbilt.edu/vu-my/wp-content/uploads/sites/2318/2019/07/….

If the treatment effect were monotonically increasing in magnitude, it’s true that you might be able to argue that the two-way FE estimator is a lower bound of some sort – though if this rate of increase were large enough, some of your two-way DD estimators might be signed incorrectly. And, of course, if treatment effects are initially large and then diminish as people adapt or equilibrium conditions change, then estimates might be biased in either direction.

Guy Grossman
September 30, 2019

Thanks Pamela! This is such great service. Question: what if the treatment is continuous (intensity rather than a binary on/off)? Is there a good solution out there (bacondecomp seems to only work with binary treatment)?

October 01, 2019

Have you seen this paper?

Don’t know of a good blog summary of that one, though...

Emmanuel Maliti
October 01, 2019

Can these pages from the world bank blog offer printable versions

Lauren Solis
October 08, 2019

Hi Pam. Thanks for this great summary of Goodman-Bacon's paper. One thing that is not clear to me is What happen when we estimate a Two Way Fixed Effect Triple DD? I mean you mention that "First, two-way fixed effects estimates of DD that rely on variation in treatment timing only recover the average treatment effect when treatment effects are homogeneous. When treatment effects are heterogeneous across units, OLS over-weights units with more variance in treatment status in order to achieve a more precise estimate of the treatment effect." So, if we include another interaction (some X) with the aim to account for the treatment heterogeneity, may be able to correct this kind of bias?