Tools of the Trade
http://blogs.worldbank.org/impactevaluations/taxonomy/term/3844/all
enCurves in all the wrong places: Gelman and Imbens on why not to use higher-order polynomials in RD
http://blogs.worldbank.org/impactevaluations/curves-all-wrong-places-gelman-and-imbens-why-not-use-higher-order-polynomials-rd
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even">A good regression-discontinuity can be a beautiful thing, as Dave Evans illustrates in a <a href="http://blogs.worldbank.org/impactevaluations/regression-discontinuity-porn" rel="nofollow">previous post</a>. The typical RD consists of controlling for a smooth function of the forcing variable (i.e. the score that has a cut-off where people on one side of the cut-off get the treatment, and those on the other side do not), and then looking for a discontinuity in the outcome of interest at this cut-off. A key practical problem is then how exactly to control for the forcing variable.<br /><br /></div></div></div>Mon, 08 Sep 2014 15:10:00 +0000David McKenzie1154 at http://blogs.worldbank.org/impactevaluationsTools of the Trade: Graphing Impacts with Standard Error Bars
http://blogs.worldbank.org/impactevaluations/tools-trade-graphing-impacts-standard-error-bars
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even">This week I finally got around to learning how to make a graph which displays the means of different treatment groups for a range of outcomes, along with standard error bars to show whether there is a significant difference between groups. Here is an example:<br /><img alt="" src="http://blogs.worldbank.org/impactevaluations/files/impactevaluations/GraphingImpactsStata.jpg" style="height:375px; width:500px" /><br /></div></div></div>Sat, 08 Feb 2014 01:29:00 +0000David McKenzie1082 at http://blogs.worldbank.org/impactevaluationsTools of the trade: recent tests of matching estimators through the evaluation of job-training programs
http://blogs.worldbank.org/impactevaluations/tools-trade-recent-tests-matching-estimators-through-evaluation-job-training-programs
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even">Of all the impact evaluation methods, the one that consistently (and justifiably) comes last in the methods courses we teach is <em>matching</em>. We de-emphasize this method because it requires the strongest assumptions to yield a valid estimate of causal impact. Most importantly this concerns the assumption of <em>unconfoundedness</em>, namely that selection into treatment can be accurately captured solely as a function of observable covariates in the data.</div></div></div>Wed, 05 Jun 2013 13:44:00 +0000Jed Friedman997 at http://blogs.worldbank.org/impactevaluationsTools of the trade: when to use those sample weights
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-when-to-use-those-sample-weights
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p><span style="LINE-HEIGHT: 115%; FONT-SIZE: 11pt">In numerous discussions with colleagues I am struck by the varied views and confusion around whether to use sample weights in regression analysis (a confusion that I share at times). A <a href="http://www.nber.org/papers/w18859"><font color="#0000ff">recent working paper</font></a> by Gary Solon, Steven Haider, and Jeffrey Wooldridge aims at the heart of this topic. It is short and comprehensive, and I recommend it to all practitioners confronted by this question.</span></p></div></div></div>Wed, 13 Mar 2013 13:01:04 +0000Jed Friedman960 at http://blogs.worldbank.org/impactevaluations“Oops! Did I just ruin this impact evaluation?” Top 5 of mistakes and how the new Impact Evaluation Toolkit can help.
http://blogs.worldbank.org/impactevaluations/oops-did-i-just-ruin-this-impact-evaluation-top-5-of-mistakes-and-how-the-new-impact-evaluation-tool
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>On October 3rd, I sent out a survey asking people what was the biggest, most embarrassing, dramatic, funny, or other oops mistake they made in an impact evaluation. Within a few hours, a former manager came into my office to warn me: “Christel, I tried this 10 years ago, and I got exactly two responses.” </p></div></div></div>Wed, 12 Dec 2012 14:19:02 +0000Christel Vermeersch924 at http://blogs.worldbank.org/impactevaluationsTools of the Trade: Intra-cluster correlations
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-intra-cluster-correlations
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>In clustered randomized experiments, random assignment occurs at the group level, with multiple units observed within each group. For example, education interventions might be assigned at the school level, with outcomes measured at the student level, or microfinance interventions might be assigned at the savings group level, with outcomes measured for individual clients.</p></div></div></div>Sun, 02 Dec 2012 21:03:18 +0000David McKenzie919 at http://blogs.worldbank.org/impactevaluationsTools of the Trade: A quick adjustment for multiple hypothesis testing
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-a-quick-adjustment-for-multiple-hypothesis-testing
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>As our impact evaluations broaden to consider more and more possible outcomes of economic interventions (an extreme example being the 334 unique outcome variables considered by <a href="http://elsa.berkeley.edu/~emiguel/pdfs/miguel_gbf.pdf"><font color="#0000ff">Casey et al.</font></a> in their CDD evaluation) and increasingly investigate the channels of impact through subgroup heterogeneity analysis, the <b>issue of multiple hypothesis testing </b>is gaining increasing prominence.</p></div></div></div>Mon, 22 Oct 2012 01:40:15 +0000David McKenzie893 at http://blogs.worldbank.org/impactevaluationsTools of the trade: The covariate balanced propensity score
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-the-covariate-balanced-propensity-score
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p><span style="FONT-SIZE: 10pt">The primary goal of an impact evaluation study is to estimate the causal effect of a program, policy, or intervention. Randomized assignment of treatment enables the researcher to draw causal inference in a relatively assumption free manner. If randomization is not feasible there are more assumption driven methods, termed quasi-experimental, such as regression discontinuity or propensity score matching. For many of our readers this summary is nothing new. But fortunately in our “community of practice” new statistical tools are developed at a rapid rate.</span></p></div></div></div>Wed, 03 Oct 2012 12:53:24 +0000Jed Friedman878 at http://blogs.worldbank.org/impactevaluationsHelp for attrition is just a phone call away – a new bounding approach to help deal with non-response
http://blogs.worldbank.org/impactevaluations/help-for-attrition-is-just-a-phone-call-away-a-new-bounding-approach-to-help-deal-with-non-response
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>Attrition is a bugbear for most impact evaluations, and can cause even the best designed experiments to be subject to potential bias. In a <a href="http://www.iza.org/en/webcontent/publications/papers/viewAbstract?dp_id=6751"><font color="#0000ff">new paper</font></a>, Luc Behaghel, Bruno Crépon, Marc Gurgand and Thomas Le Barbanchon describe a clever new way to deal with this problem using information on the number of attempts it takes to get someone to respond to a survey.</p></div></div></div>Mon, 24 Sep 2012 01:40:32 +0000David McKenzie871 at http://blogs.worldbank.org/impactevaluationsWhether to probit or to probe it: in defense of the Linear Probability Model
http://blogs.worldbank.org/impactevaluations/whether-to-probit-or-to-probe-it-in-defense-of-the-linear-probability-model
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>Last week David linked to a virtual discussion involving Dave Giles and Steffen Pischke on the merits or demerits of the Linear Probability Model (LPM).</p></div></div></div>Wed, 18 Jul 2012 12:54:39 +0000Jed Friedman846 at http://blogs.worldbank.org/impactevaluationsGerber and Green’s new textbook on Field Experiments – should you read it, and what should they add for version 2.0?
http://blogs.worldbank.org/impactevaluations/gerber-and-green-s-new-textbook-on-field-experiments-should-you-read-it-and-what-should-they-add-for
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>Alan Gerber and Don Green, political scientists at Yale and Columbia respectively, and authors of a large number of voting experiments, have a new textbook out titled <b>Field Experiments: Design, Analysis, and Interpretation. </b>This is noteworthy because despite the massive growth in field experiments, to date there hasn’t been an accessible and modern textbook for social scientists looking to work in, or better understand, this area. The new book is very good, and I definitely recommend anyone working in this area to read at least key chapters.</p></div></div></div>Mon, 02 Jul 2012 12:05:31 +0000David McKenzie834 at http://blogs.worldbank.org/impactevaluationsTools of the Trade: Beyond mean decompositions (with an application to the gender wage gap in China)
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-beyond-mean-decompositions-with-an-application-to-the-gender-wage-gap-in-china
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>Suppose you were investigating the observed wage gap in urban China, where men are paid approximately 30% more than women. The first thing you would like to know is whether the higher wages paid to men are a result of the greater average years of schooling and years in the labor force that men have or whether, instead, men are paid more even after accounting for education and experience. If the latter situation is the case then the difference in wages may at least in part be due to labor market discrimination.</p></div></div></div>Wed, 22 Feb 2012 14:17:30 +0000Jed Friedman750 at http://blogs.worldbank.org/impactevaluationsTools of the Trade: estimating correct standard errors in small sample cluster studies, another take
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-estimating-correct-standard-errors-in-small-sample-cluster-studies-another-take
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>For many years, researchers have recognized the need to correct standard error estimates for observational dependence within clusters. An earlier post <a href="http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-getting-those-standard-errors-correct-in-small-sample-cluster-studies">contrasted the typical approach to this matter, the cluster robust standard error (CRSE), and various methods to cluster bootstrap the standard error</a>.</p></div></div></div>Wed, 25 Jan 2012 14:46:18 +0000Jed Friedman730 at http://blogs.worldbank.org/impactevaluationsTools of the Trade: Getting those standard errors correct in small sample cluster studies
http://blogs.worldbank.org/impactevaluations/annals-of-good-ie-practice-getting-those-standard-errors-correct-in-small-sample-clustered-studies
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>Some of the earliest posts on this blog concerned the inferential challenges of cluster randomized trials when clusters are few in number (see <a href="http://blogs.worldbank.org/impactevaluations/on-experimental-evaluations-of-systems-interventions"><font color="#0000ff">here</font></a> and <a href="http://blogs.worldbank.org/impactevaluations/on-improving-power-in-small-sample-studies"><font color="#0000ff">here</font></a> for two examples of discussion). Today’s post continues this theme with a focus on better practice in the treatment of standard errors.</p></div></div></div>Wed, 16 Nov 2011 14:09:36 +0000Jed Friedman690 at http://blogs.worldbank.org/impactevaluationsTools of the Trade: Dealing with Multiple Lotteries
http://blogs.worldbank.org/impactevaluations/tools-of-the-trade-dealing-with-multiple-lotteries
<div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even"><p>Random lotteries to allocate scarce slots for an oversubscribed program provide a useful tool for estimating impacts of such a program. However, an issue which can arise in practice is that there may be multiple lotteries that an individual can apply for. For example,</p></div></div></div>Mon, 14 Nov 2011 14:22:00 +0000David McKenzie689 at http://blogs.worldbank.org/impactevaluations