Quick thoughts on the deworming brouhaha

Our Friday links this week linked to a Givewell summary of a new Cochrane Review of the effectiveness of deworming. In response there was a discussion on the PLOS.org blog, and by Ted Miguel, Michael Kremer and others on the IPA blog here, as well as lots of twitter chatter. Here are a few thoughts on some of the key issues in the discussion:

·         External Validity: According to Givewell’s summary (since the Cochrane Review is gated and not accessible to me at home), one of the critiques in the Cochrane review is that the strongest evidence comes from one location in Kenya with extremely high worm prevalence – I presume this is Busia. In their response, Miguel, Kremer and others cite three more papers to back up their case – but two of them are also from the same area of Kenya – only the Bleakley work on hookworm in the early 20^th century helps back up their case that the results aren’t unique to Busia. Givewell raises the further possibility that even the evidence from Kenya comes from a particular time when worm prevalence was especially elevated due to flooding caused by El Nino. Bottom line is that more evidence from other countries seems welcome.

·         Publication Bias: The Cochrane review/Givewell raise the issue of publication bias, noting that a randomized evaluation on 1 million children in India remains unpublished 6 years after completion- however, it appears to focus on mortality, rather than schooling as an outcome, and therefore doesn’t seem to matter so much for the schooling debate. One key point to make here is that I think publication is a particular problem in this area because of the ridiculous restrictions medical journals put on release of results through working papers – if these were economics studies not intended for medical journals, the working papers would at least be available (for example, the Ozier study and Baird et al studies that Miguel et al refer to in their response would not be available as evidence if they were medical studies).

·         Are baseline data needed? One of the critiques that the Cochrane review raises of Miguel and Kremer’s original worms study is that they did not have baseline data available on school attendance. As they note in their response, and as I have heard Berk lament previously, medical journals don’t seem to get that with large samples randomization ensures you have baseline balance – perhaps because they are so used to dealing with small N studies where imbalance can still be important even after randomization? Nevertheless, one caveat to this is that randomization only will ensure there is balance for the full sample – so if there is attrition at follow-up, the lack of a baseline to see how selective this attrition might be can be problematic. Miguel and Kremer have attrition rates of 15 to 17 percent for their exam score outcome, with rates not differing much by treatment status.

In sum, the response by Miguel and Kremer suggests on balance that the Cochrane review misused or misinterpreted some of the evidence from their study. What it doesn’t do is address whether or not the less impressive evidence found in other studies is because those studies are not designed as well as Miguel and Kremer, or because they were in different settings where the impacts differ, or whether the Cochrane review is not comparing like with like. I’ll leave it to someone more familiar with this literature to comment on these points.