Two new rigorous evaluations of technology use in education

Last week saw a flurry of news reports in response to a single blog post about the well known One Laptop Per Child project. It's dead, proclaimed one news report as a result; it's not dead yet, countered another. Recalling Mark Twain's famous quotation, Wired chimed in to announce that Reports of One Laptop Per Child's death have been greatly exaggerated.

Whatever the status and future of the iconic initiative that has helped bring a few million green and white laptops to students in places like Uruguay, Peru and Rwanda, it is hard to argue that, ten years ago, when the idea was thrown out there, you heard a lot of people asking, ‘Why would you do such a thing?’ Ten years on, however, the idea of providing low cost computing devices like laptops and tablets to students is now (for better and/or for worse, depending on your perspective) part of the mainstream conversation in countries all around the world.

What do we know about the impact and results of initiatives
to provide computing devices to students
in middle and low income countries around the world?

Increasingly, we know a lot about the mechanics of how to (and how not to) roll out such projects. Especially in remote, poor communities, there are some potentially useful 'principles' that have been identified. Reports from Uruguay, for example, continue to emerge documenting and analyzing what has been happening under that country's pioneering Plan Ceibal to provide every student with her own free laptop computer.

Despite such efforts, there remains a dearth of rigorous impact evaluations utilizing randomized control trials (RCTs) that are consider a sort of 'gold standard' by many in the education research community.  A working paper from the the Inter-American Development Bank (DB), Technology and Child Development: Evidence from One Laptop per Child Program in Peru, has been a notable exception to many of the research reports about educational technology initiatives in developing countries that have employed methodologies that haven't been particularly rigorous, and which have often focused on things like changes in attitudes, and perceptions of change, among groups which have been involved in various ways with such projects. There is nothing necessarily wrong with evaluations that make extensive use of things like self-reported data, of course, but the limitations of such efforts should be pretty clear as well. RCTs can be quite expensive, and difficult to do, and, depending on the circumstances at hand, there may well be other research approaches that are more viable and useful. That said, by attempting to set up and assess the results from an RCT, the IDB working paper exploring some of the potential impacts of the OLPC project in Peru has been widely cited by many researchers with serious interests in this area, and has in many ways helped raise the bar for what is expected by many international funders interested in supporting other rigorous research efforts of this sort.

Released to little fanfare back in January, two new working papers from the IDB also draw on the Peruvian experience with educational technology to offer useful contributions to our collective knowledgebase about the impact of educational technology initiatives:

Does Technology in Schools Affect Repetition, Dropout and Enrollment? Evidence from Peru
January 2014: IDB-WP-477
Abstract: Many developing countries are allocating significant resources to expanding technology access in schools. Whether these investments will translate into measurable educational improvements remains an open question because of the limited evidence available. This paper contributes to filling that gap by exploiting a large-scale public program that increased computer and Internet access in secondary public schools in Peru. Rich longitudinal school-level data from 2001 to 2006 are used to implement a differences-in-differences framework. Results indicate no statistically significant effects of increasing technology access in schools on repetition, dropout and initial enrollment. Large sample sizes allow ruling out even modest effects.

The Effects of Shared School Technology Access on Students’ Digital Skills in Peru
January 2014: IDB-WP-476
Abstract: This paper analyzes the effects of increased shared computer access in secondary schools in Peru. Administrative data are used to identify, through propensity-score matching, two groups of schools with similar observable educational inputs but different intensity in computer access. Extensive primary data collected from the 202 matched schools are used to determine whether increased shared computer access at schools affects digital skills and academic achievement. Results suggest that small increases in shared computer access, one more computer per 40 students, can produce large increases in digital skills (0. 3 standard deviations). No effects are found on test scores in Math and Language.

For those not familiar with the writing style and notation conventions of academic works of these sorts, these working papers may make for tough reading at times. In some cases, the papers may raise more questions than they perhaps answer, and the careful language with which conclusions are presented may frustrate policymakers looking for clear, unambigous insight into what the 'impact' of various types of interventions may be. (The reality of work in this area is typically much messier than what is portrayed in the marketing brochures produced by both vendors and governments alike.)  That said, many people who make decisions about large scale investments into the use of ICTs in education would do well to take some time to read through the studies and familiarize themselves with the approaches and language which characterize reports and analysis resulting from the use of research methodologies which utilize randomized control trials.

While reasonable people can (and do!) argue about the correct place of RCTs within the toolkit of options related to evaluation available to researchers and policymakers, and the extent to which such things might be expected to make a difference in the way decisions are made in the 'real world', one expects that such approaches will be increasingly prominent inputs into our collective understanding of how educational technologies are being used in practice in a variety of ways and contexts around the world.

(Those wishing to learn more about such approaches, and how to better understand the research findings that follow, may wish to consider enrolling in, or auditing, the upcoming four-week 'MOOC' on "Evaluating Social Programs" being offered through edX by J-PAL, the group based at MIT which coordinates a global network of researchers who use randomized evaluations to answer critical policy questions in the fight against poverty.)

Two other IDB working papers related to technology use in education which utilize randomization are:

Home Computers and Child Outcomes: Short-Term Impacts from a Randomized Experiment in Peru
Dec 2012: IDB-WP-382
Abstract: This paper presents results from a randomized control trial in which approximately 1,000 OLPC XO laptops were provided for home use to children attending primary schools in Lima, Peru. The intervention increased access and use of home computers, with some substitution away from computer use outside the home. Beneficiaries were more likely to complete domestic chores but less likely to read books. Treatment children scored almost one standard deviation higher in a test of XO proficiency, though there were no effects on objective and self-reported skills for using a Windows-based PC and Internet. There were positive impacts on the Raven's Progressive Matrices test among children who did not have a home computer before the intervention, but no significant effects for the sample as a whole. Finally, there was little evidence for spillovers within schools, although close friends and classmates of laptop recipients did exhibit higher proficiency with the XO computer.

Information Technology and Student Achievement: Evidence from a Randomized Experiment in Ecuador
Jan 2011: IDB-WP-223
Abstract: This paper studies the effects of information and communication technologies (ICT) in the school environment on educational achievement. To quantify these effects, the impact is evaluated of a project run by the municipality of Guayaquil, Ecuador, which provides computer-aided instruction in mathematics and language to students in primary schools. Using an experimental design, it is found that the program had a positive impact on mathematics test scores (about 0.30 of a standard deviation) and a negative but statistically insignificant effect on language test scores. The impact is heterogeneous and is much larger for those students at the top of the achievement distribution.

Taken together, this set of evaluations from the IDB effectively doubles the global set of rigorous impact evaluations related to the use of technology in education in developing countries which featured randomization that we had only a few years ago. (An upcoming blog post will highlight a second, even larger set of recent impact evaluations of this sort from the Rural Education Action Program at Stanford, which focuses on China.) The deliberate language in these papers, which is often characterized by the use of many qualifiers and couched in the argot of researchers, may frustrate those who struggle to see how such academic work can have real world policy implications. For others, the ambiguity of the results advanced and discussed within these papers may call into question investments of this sort. Such frustration and confusion need not always be such a bad thing. Indeed:

"To question computer use in schools is to ask what schools are for, why teachers teach certain content, how they should teach, and how children learn. Unsettling questions as these probe the uneasy silence in public debate of the new technology's use in schools, a silence that helps no one who is truly concerned over the schooling offered to the next generation."

This observation isn't contained in any of the IDB research papers highlighted here -- although it perhaps could have been. In fact, it comes from Larry Cuban's seminal book published in 1986, Teachers and Machines: The Classroom Use of Technology Since 1920.  The quintet of rigorous impact evaluations released by the IDB over the past few years investigating the use and impact of computing technologies in the education sector in low income countries is for me an important marker that many groups are finally getting serious about attempting to learn from the massive recent investments in educational technologies around the world. Whereas such investments once were almost exclusively to be found in the 'rich' countries of the 'global north', large efforts of this sort are now happening almost everywhere -- including (e.g.) Andean communities of rural Peru. Hopefully the research base that is slowly accumulating as a result of papers such as these from the IDB will empower people with evidence that can inform discussions which attempt to break the 'uneasy silence' of which Cuban writes.

---

Some notable IDB publications about education & technology:

• The Effects of Shared School Technology Access on Students’ Digital Skills in Peru (January 2014)
• Does Technology in Schools Affect Repetition, Dropout and Enrollment? Evidence from Peru (January 2014)
• Home Computers and Child Outcomes: Short-Term Impacts from a Randomized Experiment in Peru (December 2012)
• Technology and Child Development: Evidence from the One Laptop per Child Program (February 2012)
• Computers in Schools: Why Governments Should Do Their Homework (May 2011)
   (from Development Connections: Unveiling the Impact of New Information Technologies)
• One-to-One Laptop Programs in Latin America and the Caribbean (April 2011)
• Information Technology and Student Achievement: Evidence from a Randomized Experiment in Ecuador (January 2011)
• The One Laptop Per Child Initiative: A Framework for Latin America and the IDB (October 2006)

(Note: Some of the research efforts in Peru were funded under an IDB activity which has also supported research into the impact of the OLPC project in Honduras. Be on the lookout for an upcoming study from J-PAL about Technology and Education: The Impact of Computers in Honduran Schools.)

---

You may also be interested in the following posts from the World Bank's EduTech blog:

• Next steps for Uruguay's Plan Ceibal
• ICT and rural education in China
• Let them eat laptops?*
• Evaluating One Laptop Per Child (OLPC) in Peru
• Learning from a randomized evaluation of OLPC in Peru
• Evaluating the evaluating of the Millennium Villages Project
• How do you evaluate a plan like Ceibal?
• Why are there so many poor evaluations of ICT use in education?
• The Use and Misuse of Computers in Education: Evidence from a Randomized Experiment in Colombia

 
Note: The image used at the top of this blog post of three students at the Catholic Universty of Peru ("Look, right there, there it is: Impact! (I think ...)") comes from the World Bank Photo Collection on Flickr  and is used according to the terms of its Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0) license.