As a PhD student in the late 90s, randomized field trials were not yet common place in empirical development economics. Certain quasi-experimental methods such as regression discontinuity were also fairly exotic. It was the era of the “natural experiment”, when fellow PhD students scoured county newspapers at the university library for research leads. These students were looking for news of policy changes that might plausibly introduce some exogenous variation in the local market environment.
As a fair number of impact evaluations I work on are programs designed by governments or NGOs, I often initially have to have a tricky discussion when it comes time to do the power calculations to design the impact evaluation. The subject of this conversation is the anticipated effect size. This is a key parameter – if it’s too optimistic you run the risk of an impact evaluation with no effect even when the program had worked to some (lesser) degree, if it’s too pessimistic, then you are wasting money and people’s time in your survey.
Millions of dollars are spent each year trying to improve the productivity of firms in Africa (and those in other developing countries), yet we have very little rigorous evidence as to what works. In a new working paper I look at whether it is even possible to learn whether such policies even work, and what can be done to make progress.
Small number of firms + Large heterogeneity = Not much power
In response to an earlier blog post on marketing experiments, we noted that young creative researchers are working with NGOs to try out new innovative ways to alleviate poverty and spur development. A reader wrote with the following question:
I have been thinking about marriage recently. No, not about my own marital status, but marriage among school-age girls and its effects on future outcomes… While many arguments are made to curb teen marriages (and pregnancies), it is not clear whether these events themselves are the cause of poor future outcomes or they are simply correlated with other background characteristics that are prognostic of future outcomes. A brief survey of the literature indeed suggests that the evidence is mixed; especially when it comes to the effects of teen childbearing on future outcomes.
My last post discussed an example of a system intervention (improvements to the pharmaceutical supply chain) and the not uncommon inferential challenge of low power from relatively few units of observation.
More than Good Intentions: How a new economics is helping to solve global poverty is a personalized helicopter tour of many recent randomized controlled trials (RCTs) in developing countries. It is written by Dean Karlan, who has been a researcher in many of these experiments, and Jacob Appel, who worked for Dean in implementing many of these experiments in Ghana.
Update: Lant Pritchett has kindly responded to my invitation and posted his thoughts: "No need for unmet need." Check out the comments section.
A quick look at the burgeoning literature on policy evaluations will reveal a preponderance of evaluations of demand side schemes such as conditional cash transfers. There is an obvious reason for this beyond the promise that such interventions hold: the technology of treatment allows for large sample randomized evaluations, either at the household or community/village level. As long as financing is sufficient to sample an adequate number of study units, study power will not be a concern.