Consumption or income, valued at prevailing market prices, is the workhorse metric of economic welfare – poverty is almost universally defined in these terms. In low- and middle-income countries these measures of household resource availability are typically assessed through household surveys. Yet the global diversity in survey approaches is vast, with little rigorous evidence concerning which particular approach yields the most accurate resource estimate. (Indeed there may be no one approach that best suits every context – more on this below.)
With this question in mind, Kathleen Beegle, Joachim DeWeerdt, John Gibson, and I conducted a survey measurement experiment in Tanzania that randomized common survey approaches to consumption measurement across a representative sample of households in Tanzania. Previous papers have explored the relative performance of the approaches in terms of mean consumption, inequality, poverty, and the prevalence of hunger (see these papers here, here, and here). Our new working paper seeks to push this data further to understand the nature of the reporting errors that underlie the mean estimates.
- The inaugural issue of Development Engineering is now out (all issues are open access!). I’m delighted that my paper on attempting to use RFID to track small firm sales is in this first issue, along with a paper on how to randomize better in sequential randomized trials, a paper that proposes a “system [which] leverages smartphones, cellular based sensors, and cloud storage and computing to lower the entry barrier to impact evaluation”, a paper on biomass stoves, and one on rural electrification. Note also this from the editor’s introduction “we see major benefits from publishing studies that find weak or no impacts. In global development, there should be no silent failures; there is inherent value in learning from interventions that fail to achieve their intended impacts.”
I’ve been travelling the past week, and had several people contact me with questions about impact evaluation while away. I figured these might come up again, and so I’d put up the questions and answers here in case they are useful for others.
Question 1: Winsorizing – “do we do this on the whole sample, or do we do it within treatment and control, baseline and follow-up?”
Winsorizing is commonly used to deal with outliers, for example, you might set all data points above the 99th percentile equal to the 99th percentile. It is key here that you don’t use different cut-offs for treatment and control. For example, suppose you have a treatment for businesses that makes 4 percent of the treatment group grow their sales massively. If you winsorize separately at the 95th percentile of the treatment distribution for the treatment group and at the 95th percentile of the control distribution for the control groups, you might end up completely missing the treatment effect. I think it makes sense to do this with separate cutoffs by survey round to allow for seasonal effects and so you aren’t winsorizing more points from one round than another (which could be the case if you used the same global cutoffs for all rounds).
In an article in Slate yesterday, co-founders of GiveDirectly announced that they will provide at least 6,000 people in Kenya with a basic income grant (BIG) for a period of 10-15 years, which will cost about $30 million. The proposal is scant in details at the moment, but this article in Vox suggests that dozens of villages will randomly be selected in an already selected region of Kenya for this exercise and everyone within will be given roughly a dollar a day per person for a decade.
In 1997, Peter Singer wrote about a dilemma he’d pose to his students about a drowning child in a pond on their way to class: “would they be willing to save the child at the cost of getting all wet, having to go back home to change, and missing the first period?” After getting the expected answer that they all would, he’d ask about a hypothetical child far away, and ways that the students could save lives elsewhere at “no great cost – and absolutely no danger – to themselves. This would lead to a discussion of a version of effective altruism two decades ago.
- On Quora, Stanford Professor Jon Levin answers questions about economics. Both graduate students and young faculty might be interested in his response to “did you ever feel like economics was not for you even though you enjoy it”: “after I finished my graduate classes,I went through a period where I was trying to find an idea for my job market paper and getting nowhere. I was working every minute but at the end of every day I'd pretty much throw out all my notes. Research can be incredibly frustrating when you are getting nowhere.”
- Friday Links