Syndicate content

Jed Friedman's blog

The often (unspoken) assumptions behind the difference-in-difference estimator in practice

Jed Friedman's picture
This post is co-written with Ricardo Mora and Iliana Reggio
 
The difference-in-difference (DID) evaluation method should be very familiar to our readers – a method that infers program impact by comparing the pre- to post-intervention change in the outcome of interest for the treated group relative to a comparison group. The key assumption here is what is known as the “Parallel Paths” assumption, which posits that the average change in the comparison group represents the counterfactual change in the treatment group if there were no treatment. It is a popular method in part because the data requirements are not particularly onerous – it requires data from only two points in time – and the results are robust to any possible confounder as long as it doesn’t violate the Parallel Paths assumption. When data on several pre-treatment periods exist, researchers like to check the Parallel Paths assumption by testing for differences in the pre-treatment trends of the treatment and comparison groups. Equality of pre-treatment trends may lend confidence but this can’t directly test the identifying assumption; by construction that is untestable. Researchers also tend to explicitly model the “natural dynamics” of the outcome variable by including flexible time dummies for the control group and a parametric time trend differential between the control and the treated in the estimating specification.
 
Typically, the applied researcher’s practice of DID ends at this point. Yet a very recent working paper by Ricardo Mora and Iliana Reggio (two co-authors of this post) points out that DID-as-commonly-practiced implicitly involves other assumptions instead of  Parallel Paths, assumptions perhaps unknown to the researcher, which may influence the estimate of the treatment effect. These assumptions concern the dynamics of the outcome of interest, both before and after the introduction of treatment, and the implications of the particular dynamic specification for the Parallel Paths assumption.

Policy learning with impact evaluation and the “science of delivery”

Jed Friedman's picture
The “science of delivery”, a relatively new term among development practitioners, refers to the focused study of the processes, contexts, and general determinants of the delivery of public services and goods. Or to paraphrase my colleague Adam Wagstaff, the term represents a broadening of inquiry towards an understanding of the “how to deliver” and not simply a focus on the “what to deliver”.
 

Measuring the rate at which we discount the future: a comparison of two new field-based approaches

Jed Friedman's picture
Many key economic decisions involve implicit trade-offs over time: how much to save or invest today affects how much to spend both today and tomorrow, and individuals will differ in their preferences for satisfaction today versus delayed satisfaction tomorrow. Economists call the relative preference (or disfavor) for the present over the future a discount rate (i.e. the rate at which we discount the future for the present), and the discount rate is a core parameter in economic models of choice and behavior.

Behind low rates of participation in micro-insurance: a misunderstanding of the insurance concept?

Jed Friedman's picture
Micro-insurance pilot programs begin with grand hopes that the target population will enroll and obtain program benefits, but many are disappointed that after much planning and effort so few actually take up the program. Apparently take-up rates greater than 30% are rare and often do not exceed 15%. Furthermore, only a fraction of beneficiaries choose to renew their participation after the initial enrollment period.

Tools of the trade: recent tests of matching estimators through the evaluation of job-training programs

Jed Friedman's picture
Of all the impact evaluation methods, the one that consistently (and justifiably) comes last in the methods courses we teach is matching. We de-emphasize this method because it requires the strongest assumptions to yield a valid estimate of causal impact. Most importantly this concerns the assumption of unconfoundedness, namely that selection into treatment can be accurately captured solely as a function of observable covariates in the data.

Do financial incentives undermine the motivation of public sector workers? Maybe, but where is the evidence from the field?

Jed Friedman's picture
These past weeks I’ve visited several southern African nations to assist on-going evaluations of health sector pay-for-performance reforms. It’s been a whirlwind of government meetings, field trips, and periods of data crunching. We’ve made good progress and also discovered roadblocks – in other words business as usual in this line of work. One qualitative data point has stayed with me throughout these weeks, the paraphrased words of one clinic worker: “I like this new program because it makes me feel that the people in charge of the system care about us.”

Using spatial variation in program performance to identify causal impact

Jed Friedman's picture
I’ve read several research proposals in the past few months, as well engaged in discussions, that touch on the same question: how to use the spatial variation in a program’s intensity to evaluate its causal impact. Since these proposals and conversations all mentioned the same fairly recent paper by Markus Frolich and Michael Lechner, I eagerly sat down to read it.

Caution when applying impact evaluation lessons across contexts: the case of financial incentives for health workers

Jed Friedman's picture

These past few weeks I’ve been immersed in reviews of health systems research proposals and it’s fascinating to see the common themes that emerge from each round of proposals as well as the literature cited to justify these themes as worthy of funding.

Tools of the trade: when to use those sample weights

Jed Friedman's picture

In numerous discussions with colleagues I am struck by the varied views and confusion around whether to use sample weights in regression analysis (a confusion that I share at times). A recent working paper by Gary Solon, Steven Haider, and Jeffrey Wooldridge aims at the heart of this topic. It is short and comprehensive, and I recommend it to all practitioners confronted by this question.

Pages