Syndicate content

randomization in practice

Should you oversample compliers if budget is limited and you are concerned take-up is low?

David McKenzie's picture

My colleague Bilal Zia recently released a working paper (joint with Emmanuel Hakizimfura and Douglas Randall) that reports on an experiment conducted with 200 Savings and Credit Cooperative Associations (SACCOs) in Rwanda. The experiment aimed to test two different approaches to decentralizing financial education delivery, and finds improvements are greater when Saccos get to choose which staff should be trained rather than when they are told to send the manager, a loan officer, and a board member.

One point of the paper that I thought might be of broader interest to our readers concerns the issue of what to do when you only have enough budget to survey a sample of a program’s beneficiaries, and you are concerned about getting enough compliers.

Have RCTs taken over development economics?

David McKenzie's picture

Last week the “State of Economics, State of the World” conference was held at the World Bank. I had the pleasure of discussing (along with Martin Ravallion) Esther Duflo’s talk on “The Influence of Randomized Controlled Trials on Development Economics Research and on Development Policy”. The website should have links to the papers and video stream replay up (if not already, then soon).

The first part of Esther’s talk traced out the growth in RCTs in development economics. She pointed out that in 2000 the top-5 journals published 21 articles in development, of which 0 were RCTs, while in 2015 there were 32, of which 10 were RCTs – so pretty much all the growth in development papers in top journals comes from RCTs. She also showed that the more recently BREAD members had received their PhD, the more likely they were to have done at least one RCT.
In my discussion I expanded on these facts to put them in context, and argue against what I see as a couple of strawman arguments: 1) that top journals only publish RCTs, and that RCTs have taken over development research; and 2) that young researchers have a “randomize or bust” attitude and refuse to do anything but RCTs. I thought I’d summarize what I said on both here.

From my inbox: Three enquiries on winsorizing, testing balance, and dealing with low take-up

David McKenzie's picture

I’ve been travelling the past week, and had several people contact me with questions about impact evaluation while away. I figured these might come up again, and so I’d put up the questions and answers here in case they are useful for others.
Question 1: Winsorizing – “do we do this on the whole sample, or do we do it within treatment and control, baseline and follow-up?”
Winsorizing is commonly used to deal with outliers, for example, you might set all data points above the 99th percentile equal to the 99th percentile. It is key here that you don’t use different cut-offs for treatment and control. For example, suppose you have a treatment for businesses that makes 4 percent of the treatment group grow their sales massively. If you winsorize separately at the 95th percentile of the treatment distribution for the treatment group and at the 95th percentile of the control distribution for the control groups, you might end up completely missing the treatment effect. I think it makes sense to do this with separate cutoffs by survey round to allow for seasonal effects and so you aren’t winsorizing more points from one round than another (which could be the case if you used the same global cutoffs for all rounds).

Allocating Treatment and Control with Multiple Applications per Applicant and Ranked Choices

David McKenzie's picture
This came up in the context of work with Ganesh Seshan designing an evaluation for a computer training program for migrants. The training program was to be taught in one 3 hour class per week for several months. Classes were taught Sunday, Tuesday and Thursday evenings from 5-8 pm, and then there were four separate slots on Friday, the first day of the weekend. So in total there were 7 possible sessions people could potentially attend. However, most migrants would prefer to go on the weekend, and many would not be able to attend on particular days of the week.