This post is coauthored with Aletheia Donald
Four years ago, Markus looked at 20 impact evaluations and wrote a post concluding that most of them didn’t have much to say about reducing poverty (where was poverty was defined as expenditure, income, and/or wealth). This summer Shanta Devarajan asked for an update on twitter, so here it is.
We broadened the scope this time, picking 10 impact evaluations each from respected impact evaluation outfits with poverty in their title or in their mission – for a total of 40 impact evaluations. We picked the most recent complete papers on their website in the order they appeared (and none of them come close to 2014, so they’re all new relative to the earlier post).
Let’s start with the paper introductions. Here poverty shows up: 27 of the 40 mention poverty in the introductions. So the motivation does seem to be there.
OK, then, how many of the impact evaluations take place in overwhelmingly poor countries? In 2018, 5 of the 39 evaluations take place in countries with poverty rates above 50% (one country has no recent internationally comparable poverty data). In 2014, this was 3 of 19. So, no better on this metric. It’s a perennial problem – the poorest countries in the world are ones in which it is more difficult to work, particularly since conflict correlates highly with this level of poverty.
Maybe then, these impact evaluations are working with poor populations? Not so clear. Of the 40 impact evaluations we looked at, 9 of them mention the poverty level of the area in which the program takes place and 12 give us poverty statistics for the actual impact evaluation sample and mention them in the text. Accounting for evaluations that do both, we are left with 16 of the 40 where we have some idea of the poverty level of the program participants. Of the impact evaluations in overwhelmingly poor countries, 3 of the 5 tell us that they’re working with the poor.
So, if we add all of this together, in less of half of the evaluations do we know if the results are for poor people or not (keep in mind, that when authors tell us about the poverty status of the beneficiaries, sometimes they are not working with the poor). A potentially interesting tangent: when we took a gander at the affiliation of authors, we found that having a discussion of the poverty level of participants was less common in papers where academics made up a over a third of the authors.
What about impacts on measures of poverty? If we confine ourselves to a money metric notion of poverty (income, expenditures, consumption, assets), only 13 of our 40 include this in the outcomes for which they measure impacts. In 2014, this was 7 of 20. So not great news. There is somewhat better news on the examination of heterogenous effects by poverty levels. Here we find that 8 of the 40 impact evaluations contain this analysis – in 2014, this was only 1.
The sector of intervention most likely to have a discussion of the poverty of the program area or some measure of poverty as an outcome was social protection. Out of the 8 papers that had both, an entire 6 were from social protection. Of course, these interventions, which tilt heavily towards cash transfers (often with accompanying interventions) lend themselves to this. And maybe the push towards direct comparisons (as discussed in last Friday’s post) will help other types of interventions take a harder look at their poverty outcomes, as well as deepen our discussion of which outcomes mean the most for poverty reduction.
This post is the first of two posts on how much impact evaluations tell us about improving life outcomes for the most vulnerable. Post 2 – by David Evans and Fei Yuan – will go up next Monday.
Four years ago, Markus looked at 20 impact evaluations and wrote a post concluding that most of them didn’t have much to say about reducing poverty (where was poverty was defined as expenditure, income, and/or wealth). This summer Shanta Devarajan asked for an update on twitter, so here it is.
We broadened the scope this time, picking 10 impact evaluations each from respected impact evaluation outfits with poverty in their title or in their mission – for a total of 40 impact evaluations. We picked the most recent complete papers on their website in the order they appeared (and none of them come close to 2014, so they’re all new relative to the earlier post).
Let’s start with the paper introductions. Here poverty shows up: 27 of the 40 mention poverty in the introductions. So the motivation does seem to be there.
OK, then, how many of the impact evaluations take place in overwhelmingly poor countries? In 2018, 5 of the 39 evaluations take place in countries with poverty rates above 50% (one country has no recent internationally comparable poverty data). In 2014, this was 3 of 19. So, no better on this metric. It’s a perennial problem – the poorest countries in the world are ones in which it is more difficult to work, particularly since conflict correlates highly with this level of poverty.
Maybe then, these impact evaluations are working with poor populations? Not so clear. Of the 40 impact evaluations we looked at, 9 of them mention the poverty level of the area in which the program takes place and 12 give us poverty statistics for the actual impact evaluation sample and mention them in the text. Accounting for evaluations that do both, we are left with 16 of the 40 where we have some idea of the poverty level of the program participants. Of the impact evaluations in overwhelmingly poor countries, 3 of the 5 tell us that they’re working with the poor.
So, if we add all of this together, in less of half of the evaluations do we know if the results are for poor people or not (keep in mind, that when authors tell us about the poverty status of the beneficiaries, sometimes they are not working with the poor). A potentially interesting tangent: when we took a gander at the affiliation of authors, we found that having a discussion of the poverty level of participants was less common in papers where academics made up a over a third of the authors.
What about impacts on measures of poverty? If we confine ourselves to a money metric notion of poverty (income, expenditures, consumption, assets), only 13 of our 40 include this in the outcomes for which they measure impacts. In 2014, this was 7 of 20. So not great news. There is somewhat better news on the examination of heterogenous effects by poverty levels. Here we find that 8 of the 40 impact evaluations contain this analysis – in 2014, this was only 1.
The sector of intervention most likely to have a discussion of the poverty of the program area or some measure of poverty as an outcome was social protection. Out of the 8 papers that had both, an entire 6 were from social protection. Of course, these interventions, which tilt heavily towards cash transfers (often with accompanying interventions) lend themselves to this. And maybe the push towards direct comparisons (as discussed in last Friday’s post) will help other types of interventions take a harder look at their poverty outcomes, as well as deepen our discussion of which outcomes mean the most for poverty reduction.
This post is the first of two posts on how much impact evaluations tell us about improving life outcomes for the most vulnerable. Post 2 – by David Evans and Fei Yuan – will go up next Monday.
Join the Conversation