- The inaugural issue of Development Engineering is now out (all issues are open access!). I’m delighted that my paper on attempting to use RFID to track small firm sales is in this first issue, along with a paper on how to randomize better in sequential randomized trials, a paper that proposes a “system [which] leverages smartphones, cellular based sensors, and cloud storage and computing to lower the entry barrier to impact evaluation”, a paper on biomass stoves, and one on rural electrification. Note also this from the editor’s introduction “we see major benefits from publishing studies that find weak or no impacts. In global development, there should be no silent failures; there is inherent value in learning from interventions that fail to achieve their intended impacts.”
I’ve been travelling the past week, and had several people contact me with questions about impact evaluation while away. I figured these might come up again, and so I’d put up the questions and answers here in case they are useful for others.
Question 1: Winsorizing – “do we do this on the whole sample, or do we do it within treatment and control, baseline and follow-up?”
Winsorizing is commonly used to deal with outliers, for example, you might set all data points above the 99th percentile equal to the 99th percentile. It is key here that you don’t use different cut-offs for treatment and control. For example, suppose you have a treatment for businesses that makes 4 percent of the treatment group grow their sales massively. If you winsorize separately at the 95th percentile of the treatment distribution for the treatment group and at the 95th percentile of the control distribution for the control groups, you might end up completely missing the treatment effect. I think it makes sense to do this with separate cutoffs by survey round to allow for seasonal effects and so you aren’t winsorizing more points from one round than another (which could be the case if you used the same global cutoffs for all rounds).
In an article in Slate yesterday, co-founders of GiveDirectly announced that they will provide at least 6,000 people in Kenya with a basic income grant (BIG) for a period of 10-15 years, which will cost about $30 million. The proposal is scant in details at the moment, but this article in Vox suggests that dozens of villages will randomly be selected in an already selected region of Kenya for this exercise and everyone within will be given roughly a dollar a day per person for a decade.
In 1997, Peter Singer wrote about a dilemma he’d pose to his students about a drowning child in a pond on their way to class: “would they be willing to save the child at the cost of getting all wet, having to go back home to change, and missing the first period?” After getting the expected answer that they all would, he’d ask about a hypothetical child far away, and ways that the students could save lives elsewhere at “no great cost – and absolutely no danger – to themselves. This would lead to a discussion of a version of effective altruism two decades ago.
- On Quora, Stanford Professor Jon Levin answers questions about economics. Both graduate students and young faculty might be interested in his response to “did you ever feel like economics was not for you even though you enjoy it”: “after I finished my graduate classes,I went through a period where I was trying to find an idea for my job market paper and getting nowhere. I was working every minute but at the end of every day I'd pretty much throw out all my notes. Research can be incredibly frustrating when you are getting nowhere.”
- Friday Links
Selective teacher retention policies are both complex and controversial wherever they’re implemented. In Washington, DC, where I live and work, things are no different: pursuit of such policies arguably led to the early departures of a mayor and the chancellor of the public school system (DCPS). A January 2016 paper by Adnot, Dee, Katz, and Wyckoff evaluates the effect of teacher turnover under on student achievement under IMPACT, DCPS’ performance assessment and incentive system for its teachers – introduced in the 2009-10 school year (NBER WP version, gated, can be found here.
It was also a Friday, on April 1, 2011 when we launched Development Impact. We're pleasantly surprised and quite pleased that we have kept it going this long -- there certainly were popular blogs back then that don't exist anymore (remember Aid Watch?); while others have become part of newspaper blogs (Monkey Cage is now part of Washington Post).