Implementing successful small interventions at a large scale is hard

What key insights have emerged from development economics in the past decade, and how should they impact the work of the World Bank? A new working paper Toward Successful Development Policies: Insights from Research in Development Economics from the Bank’s research department captures 13 of the most significant insights in the world of development economics.

Here’s insight #7 on the difficulty on implementing successful small interventions at a large scale. See all previous insights here: Thirteen insights for successful development policies

The last decade has seen a large increase in the number of policy evaluations, with many pilot programs tested rigorously for their impacts. This is a boon to governments looking to practice evidence-based policy making. However, even when pilots and local development interventions have proven very successful, they have often been difficult to scale up in a cost-effective way to achieve development impact on a large scale. Conversely, several development interventions that have managed to achieve impressive scales at relatively low costs are increasingly under scrutiny for their lack of transformative impacts on the lives of the poor. This raises key questions about why promising programs are so hard to scale, and what can be done to enhance the impacts of large-scale programs.

Taking successful pilots to scale is challenging

A recent meta-analysis of more than 600 research papers covering 20 different types of development interventions found that the larger the study, the smaller the size of the effect, and that programs implemented by governments tend to have smaller effect sizes than academic or NGO-implemented programs.^ref1 Moreover, several recent experiences provide stark examples of successful pilot programs not having the intended impacts when taken to scale:

• A first example comes from efforts to reform education in Kenyan primary schools by changing the way teachers are hired.^ref2 New teachers offered a fixed-term contract by an international NGO significantly raised test scores. However, when the Kenyan government offered identical contracts, this produced zero impact.
• A second example is seen in efforts to help low-income agricultural workers in rural Bangladesh to be less vulnerable to seasonal food insecurity during the agricultural lean season. Researchers tested the impact of offering small grants and interest-free loans to enable these workers to temporarily migrate to cities in search of work.^ref3  This was incredibly successful in increasing internal migration flows, delivering higher income and consumption to program participants. The evidence was so striking that Givewell, a charity evaluator, recommended this program as one of the best values for money approaches of helping the poor, and an NGO scaled up the program dramatically to reach over 150,000 households in 2017. However, evaluations of this scaled-up program found it was unsuccessful in increasing migration rates, and the program was shut down.

Why is scaling up programs so challenging?

There are a variety of reasons why program impacts may become smaller as programs are taken to scale. These can include programs having different effects on different types of people, implementation and political economy issues, and cost challenges.

• Small-scale pilots may concentrate efforts on those who benefit most It is natural for those planning a pilot to target the most promising households or locations first, where they think the program will have most effect. Any program expansion is then likely to involve relaxing eligibility criteria to include those who may not have such large benefits from the program. This is seen clearly in a study of electricity conservation programs carried out in the United States.^ref4 The power company began this program by targeting locations it thought would benefit most from the program, and subsequent expansion to new areas had much smaller effects. This was also one of the reasons behind less impact in the Bangladesh migration example, since scaling up the program meant relaxing eligibility requirements and offering loans to many households for whom lack of credit may not have been the binding constraint.
• Implementation and political economy issues can arise as programs grow.  Small-scale pilots are often closely monitored and implemented with a degree of oversight that becomes challenging to carry out at large scales. This is particularly the case once the government becomes involved in implementation, where a combination of low state capacity, poor bureaucratic management, and capture by vested interests can seriously degrade the quality of the program offered. This was a key feature for the lack of success of government implementation of the Kenyan school reform, as resistance from teacher unions helped prevent the new contracts from working as intended.
• General equilibrium effects can further reduce some impacts. Pilot programs to train a small number of workers may be successful in helping them find new jobs, but when large numbers are trained, they may end up competing with one another for a limited set of jobs and the impact on employment could be much lower.
• Cost issues can make scaling prohibitive. Programs that have large and promising effects when delivered to small numbers of households or firms are often very intensive and expensive, making them challenging to scale up from a fiscal standpoint.

What can be done to enhance the impacts of programs at scale?

It is not simply enough to see which programs are successful in reaching large numbers of people in a cost-effective way. Microfinance is a good example of an institution that has managed to do this but has come under increasing scrutiny for not delivering the transformative development benefits that its proponents promised.^ref5  Instead, efforts to scale up programs should:

• Pay more attention to the quality of the bureaucracy and to improving management within government. The World Bank’s Bureaucracy Lab^ref6  has been set up to facilitate this.
• Experiment with using technological solutions to deliver standardized products at scale. New technologies may enable replicable and scalable ways of maintaining quality while reaching large numbers of people. For example, a World Bank study with small firms in Togo found personal initiative training led to large increases in business profits.^ref7 However, delivering this in person was expensive and relied heavily on quality trainers. An ongoing study in Ecuador is testing the promise of online training to deliver standardized content to over 20,000 students. 
• Continue to evaluate at scale. The above lessons illustrate the importance of testing whether programs work as intended as they scale, and of the number of areas where further experimentation and learning are needed.
• Do not stop trying to improve The basic microfinance model of group loans to women with immediate and regular repayment has not led to large reductions in poverty or growth in small businesses. But recent research has suggested that modifications to this basic model, such as introducing more flexibility in the repayment structure^ref8, can greatly enhance the effectiveness. This highlights the need for continued innovation, even when a program has reached millions of people.