Quick thoughts on the deworming brouhaha


This page in:

Our Friday links this week linked to a Givewell summary of a new Cochrane Review of the effectiveness of deworming. In response there was a discussion on the PLOS.org blog, and by Ted Miguel, Michael Kremer and others on the IPA blog here, as well as lots of twitter chatter. Here are a few thoughts on some of the key issues in the discussion:

·         External Validity: According to Givewell’s summary (since the Cochrane Review is gated and not accessible to me at home), one of the critiques in the Cochrane review is that the strongest evidence comes from one location in Kenya with extremely high worm prevalence – I presume this is Busia. In their response, Miguel, Kremer and others cite three more papers to back up their case – but two of them are also from the same area of Kenya – only the Bleakley work on hookworm in the early 20th century helps back up their case that the results aren’t unique to Busia. Givewell raises the further possibility that even the evidence from Kenya comes from a particular time when worm prevalence was especially elevated due to flooding caused by El Nino. Bottom line is that more evidence from other countries seems welcome.

·         Publication Bias: The Cochrane review/Givewell raise the issue of publication bias, noting that a randomized evaluation on 1 million children in India remains unpublished 6 years after completion- however, it appears to focus on mortality, rather than schooling as an outcome, and therefore doesn’t seem to matter so much for the schooling debate. One key point to make here is that I think publication is a particular problem in this area because of the ridiculous restrictions medical journals put on release of results through working papers – if these were economics studies not intended for medical journals, the working papers would at least be available (for example, the Ozier study and Baird et al studies that Miguel et al refer to in their response would not be available as evidence if they were medical studies).

·         Are baseline data needed? One of the critiques that the Cochrane review raises of Miguel and Kremer’s original worms study is that they did not have baseline data available on school attendance. As they note in their response, and as I have heard Berk lament previously, medical journals don’t seem to get that with large samples randomization ensures you have baseline balance – perhaps because they are so used to dealing with small N studies where imbalance can still be important even after randomization? Nevertheless, one caveat to this is that randomization only will ensure there is balance for the full sample – so if there is attrition at follow-up, the lack of a baseline to see how selective this attrition might be can be problematic. Miguel and Kremer have attrition rates of 15 to 17 percent for their exam score outcome, with rates not differing much by treatment status.

In sum, the response by Miguel and Kremer suggests on balance that the Cochrane review misused or misinterpreted some of the evidence from their study. What it doesn’t do is address whether or not the less impressive evidence found in other studies is because those studies are not designed as well as Miguel and Kremer, or because they were in different settings where the impacts differ, or whether the Cochrane review is not comparing like with like. I’ll leave it to someone more familiar with this literature to comment on these points.


David McKenzie

Lead Economist, Development Research Group, World Bank

July 22, 2012

The JPAL crew appears to be the pot calling the proverbial kettle "black" in saying the Cochrane review is misleading.

JPAL has published in several places their considered take on school-based deworming - it's a "best buy" for health AND school attendance.
See here: http://www.povertyactionlab.org/publication/deworming-best-buy-developm…
They base this conclusion on evaluations of one PROJECT - which is not a program, and as such has very limited external validity. That is, implementation of the intervention was largely done by an NGO and delivered through NGO- supported schools. This means the the conditions for implementing the intervention were mostly controlled - so the evaluations are in essence efficacy not effectiveness studies.
Putting this aside, even as an efficacy trial, the findings are not generalizable because of several context factors which clearly enhance the effect of the intervention: unusually high school attendance, and, unusually high worm infection rate.
Because of the long-standing NGO support to the schools, the Project schools had 97-98% attendance, the average for Kenya's schools at the time was 62% (the figures come from the Edu for All database). The average in other countries with high worm infection is LOWER.
And, as the Cochrane review rightly noted, the worm infection rate in the Kenyan study area was also unusually high (92%) during the period of the program. Even if you implemented the exact same intervention in the exact same place with the exact same NGO implementation arrangements in subsequent years you would expect lower impact.

Given such low (infinitesimal?) external validity - the thoughtful policy analyst would want to examine more studies before drawing conclusions or making recommendations. Alas, JPAL's publications don't show such restraint. They are comfortable making recommendations on how to spend aid dollars based on these studies alone. Some might call this...misleading.

For sure there are interpretations I might quibble with here and there in the Cochrane review. Givewell's write up covered most of the quibbles that came to my mind.
Honestly, I feel the JPAL team should be glad that a synthesis has been done. It is certainly a sounder foundation for drawing conclusions about deworming interventions than the Kenya studies alone. Let's see if they will update their "best buy" aid spending recommendations.