Syndicate content

David McKenzie's blog

Weekly links March 11: Defining a large effect size, helping job-seekers, a field research guide, Papua New Guineans, and more…

David McKenzie's picture
  • What is a large effect size? In the Huffington Post, Robert Slavin educational research and finds average effect sizes differ depending on whether the sample size is small or large, and non-experimental (matching) or randomized – and comes up with the table below. The average effect size for a randomized evaluation on a large sample is 0.11 S.D. compared to 0.32 S.D. for a matching-based evaluation on a small sample. He suggests effect sizes therefore need to be “graded on a curve”, with what constitutes big depending on the method of evaluation and the size of the sample.(Although also recall our posts on the problems of using S.D. to compare effect sizes in the first place).

What does Alwyn Young’s paper mean for analysis of experiments?

David McKenzie's picture

I’ve been asked several times what I think of Alwyn Young’s recent working paper “Channelling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results”. After reading the paper several times and reflecting on it, I thought I would share some thoughts, with a particular emphasis on what I think it means for people analyzing experimental data going forward.

Weekly links March 4: all measures suck, make your work group thrive, lean research, and more…

David McKenzie's picture

Weekly links February 19: field experiments in accounting, and the legal profession’s resistance to RCTs, when won’t scientists move?, and more…

David McKenzie's picture
  •  On the 3ie blog, Manny Jimenez notes the glaring omission of evaluations of private sector programs during last year’s “Year of Evaluation”
  • On the Conversation, ideas42 shares what insights from behavioral economics tell us about how to help people with their finances
  • Floyd and List on using field experiments in accounting and finance: with a recommendation to work in developing countries because the potential for randomization is higher, the firms aren’t as big, and the setting is less complex.
  • Greiner and Matthews on the (limited) use of RCTs in the legal profession in the U.S. “The intensity of the United States legal profession’s resistance to the RCT is such that, viewed individually, each law RCT appears to be a unicorn, a magical creation with no origin story that appears briefly in a larger setting and then fades away.” They find more than 50 RCTs, but note that “what looking we were able to do generated no evidence that the results of an RCT in the United States legal profession were actually used, in the sense that a program or policy changed because of the study’s results.”…” even when researchers have been able to field RCTs in the United States legal profession, lawyers and judges sometimes undermined them. And the lawyers and judges who did so appeared to have a common motive: certainty as to the “right” answer.”

Weekly links February 12: All-knowing gods and cheating, Kenya lab experiments, Dads on leave, and more…

David McKenzie's picture

Tools of the Trade: The Regression Kink Design

David McKenzie's picture

Regression Discontinuity designs have become a popular addition to the impact evaluation toolkit, and offer a visually appealing way of demonstrating the impact of a program around a cutoff. An extension of this approach which is growing in usage is the regression kink design(RKD). I’ve never estimated one of these, and am not an expert, but thought it might be useful to try to provide an introduction to this approach along with some links that people can then follow-up on if they want to implement it.

Weekly links February 5: the future of the World Bank, education reforms, nutrition evidence, and more…

David McKenzie's picture
  • The latest Journal of Economic Perspectives has two papers on the role of the World Bank: Clemens and Kremer on its role in facilitating international agreements to reduce poverty; and Ravallion on the role as a knowledge bank. Clemens and Kremer have a nice list of policy areas where developing countries have dramatically changed policies following World Bank involvement and conclude that “While it is impossible to quantify the Bank’s policy influence in a precise way, our judgment is that Bank donors are getting a tremendous amount of policy influence with their limited funding. This influence comes both through deals that link Bank finance to policy reform and through the Bank’s soft power. For this reason, allocating more resources to the Bank would be desirable.”
  • The JEP also has a nice summary by Larry Katz of Roland Fryer’s work.
  • The wonkblog on how much evidence there is (or is not) behind nutrition guidelines, and how evidence interacts with public policy demands – and of the difficulties of using RCTs in this context but also the dangers of veering towards nutritional nihilism
  • Finally, if you wonder why your emails don’t get replied to, here is PhD comics

Responses to the policymaker complaint that “randomized experiments take too much time”

David McKenzie's picture

There has obviously been a large increase in the number of rigorous impact evaluations taking place of World Bank projects over the past decade, including increasing use of randomized experiments. But one comment/complaint of a number of operational staff and government policymakers is still that “randomized experiments take too much time”.  In order to avoid repeating myself so often in responding to this, I thought I’d provide some responses on this point here.

Weekly links February 29: migration as development, Heckman on the extreme failure of early experiments, generalizing the family size instrument, and more…

David McKenzie's picture

Weekly links Jan 22: data, online training, not predicting growth, heaps of conferences, and more…

David McKenzie's picture