Do impact evaluations tell us anything about reducing poverty?

|

This page in:

I recently was thinking about what impact evaluations in development can tell us about poverty reduction.  On one level this is a ridiculous question.  Most of the impact evaluations out there are designed to look at interventions to improve people's lives and the work is done in developing countries, so it follows that we are making poor people's lives better, right?   That's less obvious.  
 
As I was mulling over this question, I decided to take a look in the literature and see what it might tell me.  So I took a sample of 20 recent evaluations - some of them published but many of them working papers.   I tried to get coverage of different sectors, but otherwise got fairly close to random sampling what I would think of as good quality impact evaluations.  I then skimmed the papers I hadn't read looking for a couple of things.    First, did they tell us if the target population for the program was poor?   Second, to be narrower on the definition of poverty, do they use income, consumption and/or expenditure as an outcome variable?   Third, is there any kind of heterogeneity analysis by wealth or income measures (this shouldn't apply to all, but I was curious if anyone was doing it).   Finally, in case none of the other things were telling me anything, is the average person in the country where the intervention takes place poor (using the World Bank's $1.25 a day cutoff)?  To answer these questions, I looked through the paper and the tables, and also did keyword searches.   
 
My first conclusion is that a lot of impact evaluations (some of the ones that I have worked on included) don't do a great job of telling you whether the people participating in the program are poor.  The obvious exceptions to this are impact evaluations of social protection programs (particularly conditional cash transfers).   The ones I looked at were poverty targeted, so this discussion was clearly front and center in the authors' minds.  
 
For the rest of the evaluations (i.e., most of them), it turns out that the words poor, poverty and wealth appear mostly in the introduction and the motivation section and then...nothing.     What was really frustrating was that some of the evaluations seem to be working in poor areas or with programs that might be targeting poor populations.    But this is really not described in the papers.   One paper had something like what I was looking for: it actually looked at the poverty rate for the district they were working in, compared it to the national average, and then talked about other rural areas in Africa to give us a sense of the poverty relevant external validity.  But this was a clear outlier.  
 
So if we are concerned about helping governments in developing countries reduce poverty in their countries, one thing that would probably help is to get some sense of whether these results apply to poor people or not poor people.   A lot of papers talk about the issues of poor or low-income countries in the introduction, but it's important to remember that in a lot of these countries the majority of the people are no longer poor.  Indeed, in 19 of the evaluations I looked at (I exclude the one discussed in the paragraph above) only 3 of them took place in countries with poverty rates at 50 percent or more.  
 
So another way to look at whether these interventions are making a dent in monetary measures of poverty is to look at the outcome measures.  Seven of the 20 evaluations contained these as outcome measures.  Part of the explanation is that income and expenditure are clearly among the noisier variables, so we would expect them to show up less partially because it's hard to get significant or meaningful results.   But there are also some interesting sectoral differences.   Only one of the health and education evaluations (n=7) measured income/expenditure.      Perhaps not so surprising.   But for agriculture (n=4) there was also only 1 evaluation that measured income/expenditure (not even partial measures).   And it wasn't universal for entrepreneurship interventions or labor market interventions, although they did better.   But it was present in both of the social protection interventions.  
 
If the sample for these evaluations isn't universally poor (or rich) then we might want to understand if the program had larger impacts for the poor or the rich(er).   One of the 20 evaluations did this - an education evaluation.    Another had used this as a stratification variable, but the stratification failed.   As an interesting aside, a couple of the programs talked about differential attrition (or not) by income levels. 
 
So where does this leave us?   On one level, even if we only use a money metric measure of poverty as our objective, we care about whether education, health, finance, and other interventions work because we have some evidence that they matter for increasing incomes.    But on another level, this exercise left me frustrated because I am not sure if these interventions work for poor people.   To fix this, we would need more impact evaluations that actually talk about whether program participants are poor (fairly easy), look at differential  effects by poverty level (a bit more work), and seriously think about measuring poverty as an outcome (more work) and reporting it even when the results aren’t significant. 
 
Topics

Authors

Markus Goldstein

Lead Economist, Africa Gender Innovation Lab and Chief Economists Office

Aneta
October 18, 2014

To paraphrase Kierkegaard "Poverty is not a problem to be solved, but a reality to be experienced."

Glenn
September 29, 2014

Two words: Welfare Analysis, seldom done (well) if at all in empirical development economics

Monique Buys
October 09, 2014

I am a student and had to do a study on developing countries and their ability to generate FDI in order to develop and eradicate poverty.
Most literature referred to aspects I would call the symptoms of poverty. They would define poverty, chose an area for evaluation based on the prevalence of these symptoms and call it a study on poor and poverty.
I question this approach as it discount the fact that their evaluation might include the non-poor. People that are living a life of poverty purely due to bad spending habits and mismanagement of available income. There is a definite difference between poverty due to no or very small income per capita or household and that due to own bad decisions.

Martina Vojtkova
September 26, 2014

Thanks a lot for highlighting these important issues! In May this year, 3ie commenced work on an evidence gap map of productive safety net programmes. So far, we have identified 216 impact evaluations. Our findings are strikingly similar to your findings. See our recent blog to find out more. The findings are all the more surprising since the included interventions are frequently implemented with the aim to help people escape poverty, so one would expect them to report more relevant information.
(Full link to the blog: http://blogs.3ieimpact.org/unexpected-evidence-on-impact-evaluations-of…)

Markus Goldstein
September 25, 2014

A query I got offline asked about assets. When looking through these evaluations I used the keyword wealth as well and looked in the tables for asset analyses and it doesn't change anything unfortunately.

Iván
September 24, 2014

Very true. Maybe is the famous problem of economics, great analyses were the data are not were the problems are.

Meg
November 13, 2014

Really good point! I see this a lot as well. My sense is that there is a more obvious dilemma: the projects are not targeting the poorest populations in the first place. Most interventions require some good infrastructure, social and economic safeguards, market activity, etc, etc, to be in place (in other words, not the least well-off groups and/or regions) in order to reflect some kind of "results" (whether they are actual outcomes or not) in the short timeframe of the project or program. So they may pay lip-service to poverty reduction, especially in the introduction and justification of the intervention, but they are actually trying enhance or bolster existing systems and activities already in place, only some (or none) of which may be addressing the underlying causes of poverty. The costs of (good) impact assessment and requisite monitoring to track poverty alleviation during the project, let alone afterwards, is also not typically incorporated into the approved budget.

Dominique Habimana
March 03, 2018

I am a PhD student and I am trying to work on non-experimental panel econometric analysis of impact evaluation of social protection program in African country, I am planning to tackle the issues you have mentioned here.