I recently was thinking about what impact evaluations in development can tell us about poverty reduction. On one level this is a ridiculous question. Most of the impact evaluations out there are designed to look at interventions to improve people's lives and the work is done in developing countries, so it follows that we are making poor people's lives better, right? That's less obvious.
As I was mulling over this question, I decided to take a look in the literature and see what it might tell me. So I took a sample of 20 recent evaluations - some of them published but many of them working papers. I tried to get coverage of different sectors, but otherwise got fairly close to random sampling what I would think of as good quality impact evaluations. I then skimmed the papers I hadn't read looking for a couple of things. First, did they tell us if the target population for the program was poor? Second, to be narrower on the definition of poverty, do they use income, consumption and/or expenditure as an outcome variable? Third, is there any kind of heterogeneity analysis by wealth or income measures (this shouldn't apply to all, but I was curious if anyone was doing it). Finally, in case none of the other things were telling me anything, is the average person in the country where the intervention takes place poor (using the World Bank's $1.25 a day cutoff)? To answer these questions, I looked through the paper and the tables, and also did keyword searches.
My first conclusion is that a lot of impact evaluations (some of the ones that I have worked on included) don't do a great job of telling you whether the people participating in the program are poor. The obvious exceptions to this are impact evaluations of social protection programs (particularly conditional cash transfers). The ones I looked at were poverty targeted, so this discussion was clearly front and center in the authors' minds.
For the rest of the evaluations (i.e., most of them), it turns out that the words poor, poverty and wealth appear mostly in the introduction and the motivation section and then...nothing. What was really frustrating was that some of the evaluations seem to be working in poor areas or with programs that might be targeting poor populations. But this is really not described in the papers. One paper had something like what I was looking for: it actually looked at the poverty rate for the district they were working in, compared it to the national average, and then talked about other rural areas in Africa to give us a sense of the poverty relevant external validity. But this was a clear outlier.
So if we are concerned about helping governments in developing countries reduce poverty in their countries, one thing that would probably help is to get some sense of whether these results apply to poor people or not poor people. A lot of papers talk about the issues of poor or low-income countries in the introduction, but it's important to remember that in a lot of these countries the majority of the people are no longer poor. Indeed, in 19 of the evaluations I looked at (I exclude the one discussed in the paragraph above) only 3 of them took place in countries with poverty rates at 50 percent or more.
So another way to look at whether these interventions are making a dent in monetary measures of poverty is to look at the outcome measures. Seven of the 20 evaluations contained these as outcome measures. Part of the explanation is that income and expenditure are clearly among the noisier variables, so we would expect them to show up less partially because it's hard to get significant or meaningful results. But there are also some interesting sectoral differences. Only one of the health and education evaluations (n=7) measured income/expenditure. Perhaps not so surprising. But for agriculture (n=4) there was also only 1 evaluation that measured income/expenditure (not even partial measures). And it wasn't universal for entrepreneurship interventions or labor market interventions, although they did better. But it was present in both of the social protection interventions.
If the sample for these evaluations isn't universally poor (or rich) then we might want to understand if the program had larger impacts for the poor or the rich(er). One of the 20 evaluations did this - an education evaluation. Another had used this as a stratification variable, but the stratification failed. As an interesting aside, a couple of the programs talked about differential attrition (or not) by income levels.
So where does this leave us? On one level, even if we only use a money metric measure of poverty as our objective, we care about whether education, health, finance, and other interventions work because we have some evidence that they matter for increasing incomes. But on another level, this exercise left me frustrated because I am not sure if these interventions work for poor people. To fix this, we would need more impact evaluations that actually talk about whether program participants are poor (fairly easy), look at differential effects by poverty level (a bit more work), and seriously think about measuring poverty as an outcome (more work) and reporting it even when the results aren’t significant.
As I was mulling over this question, I decided to take a look in the literature and see what it might tell me. So I took a sample of 20 recent evaluations - some of them published but many of them working papers. I tried to get coverage of different sectors, but otherwise got fairly close to random sampling what I would think of as good quality impact evaluations. I then skimmed the papers I hadn't read looking for a couple of things. First, did they tell us if the target population for the program was poor? Second, to be narrower on the definition of poverty, do they use income, consumption and/or expenditure as an outcome variable? Third, is there any kind of heterogeneity analysis by wealth or income measures (this shouldn't apply to all, but I was curious if anyone was doing it). Finally, in case none of the other things were telling me anything, is the average person in the country where the intervention takes place poor (using the World Bank's $1.25 a day cutoff)? To answer these questions, I looked through the paper and the tables, and also did keyword searches.
My first conclusion is that a lot of impact evaluations (some of the ones that I have worked on included) don't do a great job of telling you whether the people participating in the program are poor. The obvious exceptions to this are impact evaluations of social protection programs (particularly conditional cash transfers). The ones I looked at were poverty targeted, so this discussion was clearly front and center in the authors' minds.
For the rest of the evaluations (i.e., most of them), it turns out that the words poor, poverty and wealth appear mostly in the introduction and the motivation section and then...nothing. What was really frustrating was that some of the evaluations seem to be working in poor areas or with programs that might be targeting poor populations. But this is really not described in the papers. One paper had something like what I was looking for: it actually looked at the poverty rate for the district they were working in, compared it to the national average, and then talked about other rural areas in Africa to give us a sense of the poverty relevant external validity. But this was a clear outlier.
So if we are concerned about helping governments in developing countries reduce poverty in their countries, one thing that would probably help is to get some sense of whether these results apply to poor people or not poor people. A lot of papers talk about the issues of poor or low-income countries in the introduction, but it's important to remember that in a lot of these countries the majority of the people are no longer poor. Indeed, in 19 of the evaluations I looked at (I exclude the one discussed in the paragraph above) only 3 of them took place in countries with poverty rates at 50 percent or more.
So another way to look at whether these interventions are making a dent in monetary measures of poverty is to look at the outcome measures. Seven of the 20 evaluations contained these as outcome measures. Part of the explanation is that income and expenditure are clearly among the noisier variables, so we would expect them to show up less partially because it's hard to get significant or meaningful results. But there are also some interesting sectoral differences. Only one of the health and education evaluations (n=7) measured income/expenditure. Perhaps not so surprising. But for agriculture (n=4) there was also only 1 evaluation that measured income/expenditure (not even partial measures). And it wasn't universal for entrepreneurship interventions or labor market interventions, although they did better. But it was present in both of the social protection interventions.
If the sample for these evaluations isn't universally poor (or rich) then we might want to understand if the program had larger impacts for the poor or the rich(er). One of the 20 evaluations did this - an education evaluation. Another had used this as a stratification variable, but the stratification failed. As an interesting aside, a couple of the programs talked about differential attrition (or not) by income levels.
So where does this leave us? On one level, even if we only use a money metric measure of poverty as our objective, we care about whether education, health, finance, and other interventions work because we have some evidence that they matter for increasing incomes. But on another level, this exercise left me frustrated because I am not sure if these interventions work for poor people. To fix this, we would need more impact evaluations that actually talk about whether program participants are poor (fairly easy), look at differential effects by poverty level (a bit more work), and seriously think about measuring poverty as an outcome (more work) and reporting it even when the results aren’t significant.
Join the Conversation