Fast and furious: Crises call for a new kind of research

Inspector Lestrade: “You were told to wait for my orders.”

Sherlock Holmes: “If I had, you'd be cleaning up a corpse”

Sherlock Holmes, 2009 film

The police arriving too late is a familiar trope in fiction. Does research have the same problem?

Many well-intentioned programs provide laptops in schools to improve learning among poor students. Such programs have been implemented in more than 36 countries, distributing millions of laptops. A lot of money has gone into this idea, especially considering that a typical laptop costs around $200, compared to the $53 per child low income countries spend per year on education. It’s only after many years of substantial money spent that research has come out showing the low effectiveness of these types of programs. This research was valuable but came too late.

This is not to say that research that comes after the fact is useless. On the contrary, it can be tremendously useful. It helps enormously with future programs and policies. It also serves broader purposes – deepening our knowledge and understanding what works and what doesn’t. The laptop debacle has taught the world many important lessons for deploying ed-tech in low-income contexts.

Faster research needed for faster impact

But is there a way to do things faster? So that answers come quicker and serve the beneficiaries now instead of later? Our new paper, written with a team of co-authors across five countries, has tried to do this – with both speed and scale. We conducted a five-country randomized trial within just 18 months. We tested the impact of phone-based educational tutorials, providing timely evidence on the effectiveness of this approach during COVID-19 school closures.

Youth Impact, an NGO headquartered in Botswana, built on experience implementing Teaching at The Right Level (TaRL) in schools, a proven approach to teaching foundational literacy and numeracy, to respond quickly to continue to provide education during COVID-19 school closures.

In April 2020, within weeks of schools shutting down in Botswana, they tested a phone-based adaptation of TaRL—one-on-one tutoring from trained facilitators using phones (this approach is now called ConnectEd). Their randomized trial in Botswana showed promising results. By using text messages and phone-based tutoring together, math learning improved by 0.12 standard deviations.

What next? The NGO took a gamble and decided to replicate the approach worldwide with a coalition of partners ranging from governments to NGOs to the multilaterals, such as the World Bank. This was in late 2020 when schools in many low- and middle-income countries remained closed and evidence of learning losses continued to mount. Our other research shows that one month of school closures led to one month of learning losses, on average. Globally, between March 2020 and March 2022 an average school-age child lost about one year (37 weeks) of schooling due to COVID-19-related school closures, causing massive learning losses that the world is still recovering from.

Swift replication across contexts is possible

A unique coalition of NGOs, governments, multilaterals, and researchers came together in five countries to test and adapt ConnectEd in their context. ConnectEd trials were launched in Kenya, Nepal, India, Philippines, and Uganda between December 2020 and October 2021. Some partnerships included government delivery. In Nepal, for example, the World Bank collaborated with the Ministry of Education, Science, and Technology, and two NGOs. This ambitious undertaking yielded some of the most rapid and comprehensive multi-country evidence generated on an education intervention.

The impacts have been impressive and consistent. On average, phone call tutorials improved learning by 0.33 standard deviations—over a third larger than the median effectiveness of typical education interventions. Although children who received phone tutoring learned more in every location, the greatest gains were in the Philippines and Uganda, where school closures were longer than in other countries. In Nepal and the Philippines, where the program was randomized to be delivered by both governments and NGOs, effects were similarly large across both models. 

What did we learn from all this? First, on substance, we learned the mobile phones provide a flexible platform to reach people cheaply and across diverse settings, even during school disruption. Accompanied with high quality pedagogy focused on targeted instruction and foundational skills, mobile phones can enable teachers to tutor students and improve learning. Second, in terms of process, we learned evidence can be generated quickly, across contexts and with governments, to inform real-time challenges. 

It’s time to test sound ideas quickly and replicate them swiftly. Some say better late than never. We say better early than late.