Two years ago, Mike O’Sullivan and I did a post on gender and agriculture. One of the things we pointed out was that there was a pretty dismal lack of evidence on interventions in agriculture (forget gender). So I was pretty excited when the recent Campbell Collaboration systematic review on “the effects of training, innovation and new technology on African smallholder farmers’ economic outcomes and food security” crossed my desk.
Alas, we’re not there yet. And, by a careful reading of this systematic review, not even close. The authors (Ruth Stewart and a bevy of co-authors), have a hard row to hoe: they’re focusing on some key outcomes (income and food security) and they then go on to choose some key interventions (training, innovation, and new technology). Out they go looking for papers: high and low. They found 18,470 citations, but only 19(!) of these are viable studies. And viable isn’t a super-high bar – they’re including not only randomized trials, but also matching and other attempts to construct some kind of control group. Indeed, they conclude that eight of the 19 studies were at risk of serious or worse risk of bias.
So, with this handful, what do they find? First, the good news. Growing orange fortified sweet potatoes (OFSP) is a good thing. There is a handful (i.e. 5) of studies out there showing positive nutritional impacts for members of households (at least women and children) where interventions lead them to grow this crop. Second, they turn to the impact of agricultural input innovations (introducing a new crop variety) on incomes. Here, when they pool the four studies, they find a significant impact on income. But a) only one of these studies has a low risk of bias and b) taken individually, they show not significant impacts in three of the four. The small samples (600 participants when all four are pooled) is likely an issue here.
Moving on to the impacts of practice innovations and their impact on income, Stewart and co. come up with two studies. One of these has serious risk of bias, but the other by Ashraf and co-authors, looks at the impacts of introducing export crops in Kenya (in case you were wondering, Kenya and Uganda account for over half of the studies in the entire systematic review). This study finds nice, big effects. But there is a nasty end to the story – a year later. EU policy changes wiped out the market for these particular farmers and left them with a lot of unsold green beans and baby corn. Stewart and co. also look at the impacts of practice innovations on nutrition. They’ve got one study here, which does find some effects, but it too has a serious risk of bias.
The last area that Stewart and co. cover is the impact of training interventions on farmers’ economic outcomes. Here we have a whopping five studies, three of which are on farmer field schools. They point out that even within farmer field schools, there’s a lot of variation in structure and focus. Bottom line for the pooled analysis: no significant impacts. However, when they throw out the one study with serious risk of bias, they get some positive impacts.
So, in sum: not a lot of evidence, and less high quality evidence. Part of what limits the story here is the geographical focus -- adding India, for instance, would get a significant increase in the number of potential studies. But ultimately it’s limited by the fact that they start with the outcomes of income and nutrition for smallholders. As Stewart and co. explain, they are motivated to answer the (very important) question that donors and NGOs are asking: which intervention should we invest in and why? As noted above, they then limit the interventions to a fairly common/intelligent set of agricultural interventions (i.e. interventions an agriculture ministry or donor would make). This surely makes things tractable (which is key for a systematic review), but it does miss the possibility that the most effective interventions might not be in this space. It might be heresy, but there is some intriguing evidence of agriculture impacts from cash transfers which I blogged about awhile back.
As Stewart and co. note, this tack also takes them away from approaching things from the perspective of what do we know about particular interventions. Here, Stewart and co-authors have another paper, a review of reviews, and it turns out there is quite a profusion of studies – 18 completed systematic reviews (my favorite: residue retention). But as they point out – these don’t have a lot to tell us about income and food security outcomes in Sub-Saharan Africa.
However, if we’re thinking about interventions in agriculture, and we’re thinking about lessons that could apply to countries in Africa, the broader topical approach may still add to our knowledge. Recently, at the World Bank we’ve had two really interesting synthesis presentations from the ATAI initiative. Watching my more seasoned agriculture colleagues in discussion with this evidence was educational, with a lot of operational experience mixing with new evidence. I’ll blog about this work more later, but in the meantime, if you are interested, you can watch videos of Craig McIntosh talking about insurance and risk and Jeremy Magruder talking about information and new technologies.
Alas, we’re not there yet. And, by a careful reading of this systematic review, not even close. The authors (Ruth Stewart and a bevy of co-authors), have a hard row to hoe: they’re focusing on some key outcomes (income and food security) and they then go on to choose some key interventions (training, innovation, and new technology). Out they go looking for papers: high and low. They found 18,470 citations, but only 19(!) of these are viable studies. And viable isn’t a super-high bar – they’re including not only randomized trials, but also matching and other attempts to construct some kind of control group. Indeed, they conclude that eight of the 19 studies were at risk of serious or worse risk of bias.
So, with this handful, what do they find? First, the good news. Growing orange fortified sweet potatoes (OFSP) is a good thing. There is a handful (i.e. 5) of studies out there showing positive nutritional impacts for members of households (at least women and children) where interventions lead them to grow this crop. Second, they turn to the impact of agricultural input innovations (introducing a new crop variety) on incomes. Here, when they pool the four studies, they find a significant impact on income. But a) only one of these studies has a low risk of bias and b) taken individually, they show not significant impacts in three of the four. The small samples (600 participants when all four are pooled) is likely an issue here.
Moving on to the impacts of practice innovations and their impact on income, Stewart and co. come up with two studies. One of these has serious risk of bias, but the other by Ashraf and co-authors, looks at the impacts of introducing export crops in Kenya (in case you were wondering, Kenya and Uganda account for over half of the studies in the entire systematic review). This study finds nice, big effects. But there is a nasty end to the story – a year later. EU policy changes wiped out the market for these particular farmers and left them with a lot of unsold green beans and baby corn. Stewart and co. also look at the impacts of practice innovations on nutrition. They’ve got one study here, which does find some effects, but it too has a serious risk of bias.
The last area that Stewart and co. cover is the impact of training interventions on farmers’ economic outcomes. Here we have a whopping five studies, three of which are on farmer field schools. They point out that even within farmer field schools, there’s a lot of variation in structure and focus. Bottom line for the pooled analysis: no significant impacts. However, when they throw out the one study with serious risk of bias, they get some positive impacts.
So, in sum: not a lot of evidence, and less high quality evidence. Part of what limits the story here is the geographical focus -- adding India, for instance, would get a significant increase in the number of potential studies. But ultimately it’s limited by the fact that they start with the outcomes of income and nutrition for smallholders. As Stewart and co. explain, they are motivated to answer the (very important) question that donors and NGOs are asking: which intervention should we invest in and why? As noted above, they then limit the interventions to a fairly common/intelligent set of agricultural interventions (i.e. interventions an agriculture ministry or donor would make). This surely makes things tractable (which is key for a systematic review), but it does miss the possibility that the most effective interventions might not be in this space. It might be heresy, but there is some intriguing evidence of agriculture impacts from cash transfers which I blogged about awhile back.
As Stewart and co. note, this tack also takes them away from approaching things from the perspective of what do we know about particular interventions. Here, Stewart and co-authors have another paper, a review of reviews, and it turns out there is quite a profusion of studies – 18 completed systematic reviews (my favorite: residue retention). But as they point out – these don’t have a lot to tell us about income and food security outcomes in Sub-Saharan Africa.
However, if we’re thinking about interventions in agriculture, and we’re thinking about lessons that could apply to countries in Africa, the broader topical approach may still add to our knowledge. Recently, at the World Bank we’ve had two really interesting synthesis presentations from the ATAI initiative. Watching my more seasoned agriculture colleagues in discussion with this evidence was educational, with a lot of operational experience mixing with new evidence. I’ll blog about this work more later, but in the meantime, if you are interested, you can watch videos of Craig McIntosh talking about insurance and risk and Jeremy Magruder talking about information and new technologies.
Join the Conversation