Basing policy decisions on rigorous causal evidence

What key insights have emerged from development economics in the past decade, and how should they impact the work of the World Bank? A new working paper Toward Successful Development Policies: Insights from Research in Development Economics from the Bank’s research department captures 13 of the most significant insights in the world of development economics.

Here’s insight #8 on the difficulty on randomized controlled trials as a tool to evaluate policy interventions. See all previous insights here: Thirteen insights for successful development policies

To inform policy decisions, policymakers need evidence that answers questions about causality such as “how does an SME credit program affect business growth?” This amounts to asking, “how would SMEs that participated in the program have fared in the absence of such a program and how would SMEs that did not participate have fared had they participated?”  Answering such questions is hard.  Simply comparing the same firm’s performance before and after participation in the program will not give a reliable estimate of the program’s impact, since other factors that affect outcomes may have changed since the program was introduced.  Identifying the program’s impact is complicated further if the program is voluntary, and firms that choose to participate are different from those that do not. Recovering the average impact of the program requires a comparison of firms that participated in the program to an otherwise identical group of firms that did not participate. 

The recent Nobel Prize awarded to Abhijit Banerjee, Esther Duflo and Michael Kremer cited their pioneering work in adapting randomized controlled trials (or RCTs) to answer these types of development policy questions. The key idea behind an RCT is that the group that receives the program and the status quo or “control” group are chosen by random lottery.  Since these groups are identical on average, we can compare outcomes in both groups over time to credibly estimate the impact of the program.

Perhaps more than introducing RCTs in development economics, however, the Nobel committee recognized the efforts of these three prominent scholars in building a consensus that policy decisions must be informed by rigorous evidence.  In the words of Michael Kremer, “a new medication goes through the rigor of an RCT even when it only affects a few thousand people; so, it’s crazy that we spend billions of dollars on policies and programs affecting hundreds of millions of people with nothing close to the same level of evidence on effectiveness or lack thereof.”

RCT evaluations of educational policies have shown that adding an extra teacher to a school may not have nearly as much impact on learning as matching instruction to the learning levels of children.^ref1 RCTs have shown that a school deworming program greatly increases school participation and improves long-term outcomes at very low cost.^ref2,ref3 Together, these RCTs show that an extra year of school attendance “costs” much less if policy makers focus on deworming kids instead of providing additional teachers.

While RCTs can establish what works in a particular setting, they often cannot say much about why it works or whether it could work elsewhere.  In response, a new generation of impact evaluations integrate their empirical design with theoretical or “structural” modeling, with three main benefits:

• It allows researchers to quantitatively assess the mechanisms underlying policy impacts. This often helps to understand why interventions did not work as intended or why their effects differed across certain groups of beneficiaries.  For example, an RCT of fertilizer subsidies in Kenya ^ref4 considers a behavioral economics model in which some farmers are patient and others procrastinate, to rationalize low investment in fertilizer in the face of high returns to its use.  A study of microfinance in Thailand^ref5 uses a structural model of entrepreneurship to explain why credit provision can have highly heterogenous impacts on subsequent investment.  A later RCT of a microfinance program in India confirms precisely this pattern of response across beneficiaries.^ref6

• Structural modeling also enables researchers to predict the impact of hypothetical policies that have yet to be implemented.  A study^ref7 of Mexico’s celebrated PROGRESA conditional cash transfer program uses a dynamic decision-making model to show that an alternative subsidy scheme could generate greater schooling impact at similar cost to the existing program.  In the Thailand microfinance study, a counterfactual policy of providing credit explicitly tied to investment is found to substantially outperform the actual policy of unconditional credit provision.
• Finally, a tight link to theory can allow researchers to go beyond cost-benefit analysis and make quantitative welfare statements, identifying winners and losers from a policy (or a range of alternative policies) and the magnitudes of these gains or losses.  In the Kenya study, the authors use the structural model to show that small and time-limited fertilizer subsidies can lead to higher social welfare compared to heavy subsidies or no subsidies at all.  The Thai study shows that large-scale credit provision is not a cost-effective policy on average; although some households value the added liquidity substantially more than it costs to provide, many more value it very little.

The World Bank, as well as many governments and large NGOs, now insist on evaluations – often in the form of RCTs – wherever feasible. Many World Bank researchers regularly use these methods and a specialized unit named DIME was set up to facilitate impact evaluations within World Bank projects.  However, not all development policies or programs can be evaluated with RCTs. For example, policies that affect the whole of the economy or an entire market, like a minimum wage regulation, are not amenable to RCTs. To identify causal effects of market-level policies, researchers rely on non-experimental methods such as difference-in-differences – e.g., comparing employment changes in border-counties across lines of U.S. states with and without minimum wage hikes – and regression-discontinuity – e.g., comparing employment rates among individuals just below the age threshold to qualify for the minimum wage in Denmark to those just above it.^ref8,ref9

In the developing country context, a difference-in-differences strategy was used to study the labor market consequences of a massive school construction program in Indonesia by comparing cohorts that would have had access to the new schools with those that would have graduated by the time of the school construction.^ref10  A regression discontinuity design was used to estimate the impact of India’s huge road-building program targeted to smaller villages by comparing outcomes in villages just below the population cutoff to those just above it.^ref11  

In conclusion, in recent decades development economists have honed several empirical tools that demonstrate both the feasibility and importance of providing rigorous analysis to support evidence-based policy making.