Suppose that you’re told that a program reduced the rate of dropping out of school among 15 year-olds by 17% and this reduction was statistically significant. You are also told that the same figure among 12 year-olds is 38%. You would likely take note. Suppose now you’re told that these are the effects of a conditional cash transfer program, where the dropout rate among the control group is 37.7% and 16.8%, respectively for ages 15 and 12, thus the absolute effect sizes are 6.4 percentage points in each case.
When you hear the first sentence, you are likely to miss two things that you might have thought about had you been given the latter facts about the same program effects – simply stated differently. First, in the latter case, you might say: “31.3% of 15 year-olds dropped out of school even though the government offered their families a considerable sum?” Second, you notice the fact that the program averted just sixdropouts for every 100 transfers. Would you be surprised if you were then told that these are the effect sizes of the much-heralded PROGRESA? If you are, feel free to examine Table 1 in this paper (gated, sorry no WP versions around) by Behrman, Sengupta, and Todd (2005).
One of the main issues Mark Rosenzweig takes with Banerjee and Duflo’s Poor Economicsin his recent review (gated) of the book for the Journal of Economic literature is exactly this: that the absolute gains detected in many of the studies covered in the book (and many of the studies we cover in this blog) are small. And when benefits are small, the small inconveniences, seemingly trivial side effects, and self-control problems addressed by many of these interventions can “loom large” for the study subjects.Dr. Rosenzweig makes two important points: first, the benchmark used in Poor Economics or many other studies similar to the ones covered therein is “not the level of living of the non-poor but the typical level of the relevant outcome variable in the poor population” (120). Small absolute gains on small baseline levels produce large percentage gains, but the difference in the lives of the subjects, even in the long run can be quite small: he cites Udry (2011), who calculated a $10-15 gain to a farmer from using no fertilizer to using the recommended dose; his own calculations show that the earnings gains to children from iodine supplements to mothers are 2.6% or those from a school-building program in Indonesia to are less than 0.5%. The point is these programs do make a difference in the lives of poor people, but nowhere near to bridging the gap between poor and non-poor in developing countries, let alone making a dent in global inequality.
Second, Rosenzweig makes a very nice point: it makes sense to report effects in percentages for some things, such as income, where, due to diminishing marginal utility, the same absolute gain indicates higher welfare gains for a poor person than a rich one. But, does it make sense to do the same for schooling? The log-linear Mincerian regression for wages specifies the schooling coefficient to be the same whether you went from Grade 1 to Grade 2 or Form 3 to form 4. So, why report increases in schooling using percentages when they potentially “obscure what is really at stake” (121)?
Many of us fall victim to this – surely sometimes because it sounds better to say a 33% increase than a 3.3 percentage point gain over a baseline mean of 10. Also calculating what these impacts mean can involve mathematical gymnastics that are tenuous in their assumptions. Some of us are better than others in going the extra mile to give the reader a sense of what the effect size means in terms of something else that is tangible with respect to improvements in welfare. I, for one, know that I should be trying harder -- we should be proud of the research we do, not the impact sizes we found in interventions we happen to be studying…Ditto for organizations advocating or implementing these interventions: if impacts are small then we need to figure out whether they can be larger or try something else.
The discussion of the small marginal gains ties in nicely with the debate that has been going on in the past few weeks surrounding the selection of a president for this institution: ‘big development’ vs. ‘small development’. Michael Woolcock wrote the best piece on this that I read, which was then covered by the Economist and the New Yorker blogs: there are the big questions about how to get people permanently out of poverty, which may have to do with markets, infrastructure, trade, institutions and politics (depending on who you’re asking) and there are smaller questions about why people are poor and how their lives can be made better – even in small ways – now. Dr. Woolcock has said that the two visions are not “incompatible, and reasonable people can cast their lot with either one”.Rosenzweig gives a slightly different take, which I prefer. He thinks that these efforts, i.e. “thinking big and focusing on marginally improving the lives of those who do not escape are ultimately complementary agendas… The scale of poverty is too big to only think small.”
If you were inclined to think, however, that Dr. Rosenzweig is down on the “small development” agenda, you would be wrong. Unlike some other critics of the so-called ‘randomistas’, there is an avuncular warmth in his review of this current movement, which he sees for what it is: a young research program that “may or may not have large future payoffs for the poor, but has surely already done some good.” You get the sense that he wants to know more: do we need more replication studies? More studies with multiple treatment arms? How can we maximize the potential of this nascent agenda to answer not only the small questions, but also help address the bigger ones?
Reading this review of Poor Economics from start to finish will make you appreciate what is going on in development economics today and seriously question it at the same time.
Join the Conversation