Syndicate content


Lessons from some of my evaluation failures: Part 1 of ?

David McKenzie's picture

We’ve yet to receive much in the way of submissions to our learning from failure series, so I thought I’d share some of my trials and tribulations, and what I’ve learnt along the way. Some of this comes back to how much you need to sweat the small stuff versus delegate and preserve your time for bigger picture thinking (which I discussed in this post on whether IE is O-ring or knowledge hierarchy production). But this presumes you have a choice on what you do yourself, when often in dealing with governments and multiple layers of bureaucracy, the problem is your potential for micro-management can be less in the first place. Here are a few, and I can share more in other posts.

Weekly links October 21: Deaton on doing research, flexible work schedules, 17,000 minimum wage changes, and more…

David McKenzie's picture
  • Three questions with Angus Deaton – why diversity in researchers is good and directed research can be bad “Everyone of us has a different upbringing. Many people in economics now come from many countries around the world, they have different political views and political backgrounds. There’s a whole different social culture, and so on. I think economics in the United States has changed immeasurably in the last 30 years and been enormously enriched by that diversity with people coming from all over the world. That will only work if people bring with them the stuff they had when they were children or the stuff they did in college, the passions they had early on. Either smash them to pieces in the face of the data and see your professors like me telling them to do something else or turn them into something really valuable. So, don’t lose your unique value contributions. Stick to what is really important to you and try to research that” (h/t Berk).
  • Chris Blattman on the hidden price of risky research, particularly for women.

What’s new in education research? Impact evaluations and measurement – October round-up

David Evans's picture

Here is a curated round-up of recent research on education in low- and middle-income countries, with a few findings from high-income countries that I found relevant. All are from the course of 2016.

If I’m missing recent articles that you’ve found useful, please add them in the comments!

Did Peru’s CCT program halve its stunting rate?

Berk Ozler's picture

On September 30, the Guardian ran several articles (see here, here, and an editorial here) linking the halving of Peru’s stunting rate (from 28 to 14% between mid-2000s and 2015) to its CCT program Juntos. Of course, it is great to hear that the share of stunted children in Peru declined dramatically over a short period. However, as I know that while CCT programs (conditional or not) have been successful in improving various outcomes including child health, the effect sizes are never this dramatic, I was curious to see whether the decline was part of a secular trend in Peru or actually could be attributed primarily to Juntos

Weekly links October 14: fake doctors and dubious health claims, real profits, refugees, reweighting, and more…

David McKenzie's picture
  • In Science this week: what refugees do Europeans want? A “conjoint experiment” with 180,000 Europeans finds they want high-skilled, young, fluent in the local language, who are persecuted and non-Muslim (5 page paper, 121 page appendix!). This involved showing pairs of refugees with randomly assigned characteristics and having them say whether they supported admitting the refugee, and if they could only choose one out of the pair, which one.
  • BBC News covers the recent science paper by Jishnu Das and co-authors on training ‘fake doctors’ in India (or for more study details, see the MIT press release which has a great photo-bomb)

When the Juice Isn’t Worth the Squeeze: NGOs refuse to participate in a beneficiary feedback experiment

Guest post by Dean Karlan and Jacob Appel
Dean has failed again! Dean and Jacob are kicking off our series on learning from failure by contributing a case that wasn’t in the book.
I. Background + Motivation
Recent changes in the aid landscape have allowed donors to support small, nimble organizations that can identify and address local needs. However, many have lamented the difficulties of monitoring the effectiveness of local organizations. At the same time as donors become more involved, the World Bank has called for greater “beneficiary control,” or more direct input from people receiving development services.
While attempts have been made to increase the accountability of non-profits, little research addresses whether doing so actually encourages donors to give more or to continue supporting the same projects. On the contrary it may be that lack of accountability provides donors with a convenient excuse for not giving. It could be that donors give the same amount even with greater accountability. Furthermore, little research indicates whether increased transparency and accountability would provide incentives for organizations to be more effective in providing services. Rigorous research will help determine the impact of increasing accountability, both on the behavior of donors and on the behavior of organizations working in the field.  

Book Review: Failing in the Field – Karlan and Appel on what we can learn from things going wrong

David McKenzie's picture

Dean Karlan and Jacob Appel have a new book out called Failing in the Field: What we can learn when field research goes wrong. It is intended to highlight research failures and what we can learn from them, sharing stories that otherwise might otherwise be told only over a drink at the end of a conference, if at all. It draws on a number of Dean’s own studies, as well as those of several other researchers who have shared stories and lessons. The book is a good short read (I finished it in an hour), and definitely worth the time for anyone involved in collecting field data or running an experiment.

Weekly Links: Capabilities and Skills, More on RDD, Elite capture, Real-time price tracking in South Sudan, and more...

Berk Ozler's picture
  • Heckman turned one of his lectures from last year into an NBER WP titled "Capabilities and Skills." Looks really interesting - here's a quote from the abstract: "We address measurement problems common to both the economics of human development and the capability approach. The economics of human development analyzes the dynamics of preference formation, but is silent about which preferences should be used to evaluate alternative policies. This is both a strength and a limitation of the approach."

  • Advances in Econometrics has a special issue on Regression Discontinuity Design, including many papers by prominent statisticians and econometricians in the field and edited by Cattaneo and Escanciano.

  • Do poor people want more redistributive programs and less public goods? Latest issue of the Journal of the European Economic Association has a paper by Bursztyn that challenges elite capture as the explanation for low levels of investment in public education. Here is the abstract: "A large literature has emphasized elite capture of democratic institutions as the explanation for the low levels of spending on public education in many low-income democracies. This paper provides an alternative to that longstanding hypothesis. Motivated by new cross-country facts and evidence from Brazilian municipalities, we hypothesize that many democratic developing countries might invest less in public education spending because poor decisive voters prefer the government to allocate resources elsewhere. One possible explanation is that low-income voters could instead favor redistributive programs that increase their incomes in the short run, such as cash transfers. To test for this possibility, we design and implement an experimental survey and an incentivized choice experiment in Brazil. The findings from both interventions support our hypothesis."

  • My brilliant former research assistant Utz Pape has a blog post titled: "What did we learn from real-time tracking of market prices in South Sudan?" Read it if you're into innovative use of technology in development.

  • Private Enterprise Development in Low-Income Countries (PEDL), a joint research initiative of the Centre for Economic Policy Research (CEPR) and the Department For International Development (DFID), is offering a competitive research grants scheme for projects related to the behaviour of firms in Low-Income Countries (LICs) that aim to better understand what determines the strength of market forces driving efficiency in these countries. Round 21 of their new Exploratory Research Grants is now open:

A roundup of some new work on labor

Markus Goldstein's picture
Last week I was at the GLM-LIC/IZA research conference and there was a pretty diverse and interesting group of papers – some fairly finished, and some still at the idea stage.   Note that almost every paper here had co-authors, but I’ve left them out unless they also presented – the links will give you more information.  Now, fasten your seat belts as we whizz through 2.5 days of papers in 15 minutes.         

Papers that caught my attention last week

Berk Ozler's picture
A lot of interesting papers came across my desk this past week. Here are a few that I think may be useful to you too (and why):

Practical advice on robust standard errors in (not so small) samples: Imbens and Kolesár have an old working paper just published in REStat that tells you to do three things: