We are often in a world where we are allowed to randomly assign a treatment to assess its efficacy, but the number of subjects available for the study is small. This could be because the treatment (and its study) is very expensive – often the case in medical experiments – or because the condition we’re trying to treat is rare leaving us with two few subjects or because the units we’re trying to treat are like districts or hospitals, of which there are only so many in the country/region of interest.
Berk Ozler's blog
The demand for pre-analysis plans that are registered at a public site prior available for all consumers to be able to examine has recently increased in social sciences, leading to the establishment of several social science registries. David recently included a link to Ben Olken’s JEP paper on pre-analysis plans in Economics. I recently came across a paper by Humphreys, de la Sierra, and van der Windt (HSW hereon) that proposes a comprehensive nonbinding registration of research. The authors end up agreeing on a number of issues with Ben, but still end up favoring a very detailed pre-analysis plan. As they also report on a mock reporting exercise and I am also in the midst of writing a paper that utilized a pre-analysis plan struggling with some of the difficulties identified in this paper, I thought I’d link to it a quickly summarize it before ending the post with a few of my own thoughts.
Bruce Wydick on the Impact of giving away TOMS Shoes: He gives kudos to TOMS for being open for evaluation and being responsive to findings, but what caught my eye was this observation: "The bad news is that there is no evidence that the shoes exhibit any kind of life-changing impact,..."
David McKenzie pops out from under many empirical questions that come up in my research projects, which has not yet ceased to be surprising every time it happens, despite his prolific production. The last time it happened was a teachable moment for me, so I thought I’d share it in a short post that fits nicely under our “Tools of the Trade” tag.
I thank Annette Brown and Benjamin Wood (B&W from hereon) for their response to my previous post about the 3ie replication window. It not only clarified some of the thinking behind their approach, but arrived at an opportune moment – just as I was preparing a new post on part 2 of the replication (or reanalysis as they call it) of Miguel and Kremer’s 2004 Econometrica paper titled “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities,” by Davey et al. (2014b) and the response (Hicks, Kremer, and Miguel 2014b, HKM from hereon). While I appreciate B&W’s clarifications, I respectfully disagree on two key points, which also happen to illustrate why I think the reanalysis of the original data by Davey et al. (2014b) ends up being flawed.
- 3ie blog on power, replication, and meta-analysis.
- Tim Harford, The Undercover Economist, on the need for corrections for multiple hypothesis testing in the finance literature. He cites the famous jelly beans cause acne cartoon, but I did not find a link to the paper he mentions...
- Michael Hobbes in the TNR on how big ideas are destroying international development. Regular readers won't find much in here, reads like written by someone who familiarized himself with the field over the past year. His recommendation: scale up successful ideas slower and no cookie cutter solutions from Western Kenya to India -- probably not worth 5,000 words in TNR...
A treatment is an instance of treating someone, say, medically. A cure ends a problem. Sometimes, the treatment is a cure. Other times, it just keeps the problem under control without curing it: if you remove the treatment, the problem comes back…
Yesterday, Martin Ravallion wrote a piece titled ‘Taking Ethical Validity Seriously.’ It focused on ethically contestable evaluations and used RCTs as the main (only?) example of such evaluations. It is a good piece: researchers can always benefit from questioning themselves and their work in different ways.