Learning from a randomized evaluation of OLPC in Peru

The Inter-american Development Bank (IDB) recently released the first set of results from its on-going, multi-year randomized evaluation of the impact of the OLPC project in Peru.
Experimental Assessment of the Program "One Laptop Per Child" in Peru (Spanish version here) is the first rigorous attempt to examine the impact of the largest '1-to-1 computing' initiative in a developing country.  This evaluation, done in concert with the Ministry of Education, looks at the ambitious program to provide computing resources to multi-grade rural elementary schools in some of the poorer communities of Peru.

The contrasts between the OLPC roll-out in Peru and that of the other country in South America to have adopted the program at scale -- Uruguay -- are striking.  A geographically compact, middle income country with good infrastructure known historically for its high literacy rates and much narrower gaps between the rich and poor than are found in most other South American countries, Uruguay provides in many ways an ideal setting in which attempt a quick roll-out of low cost computers to primary school students.  (This is not to say that they aren't still substantial challenges with the project in Uruguay -- of course there are -- just that, comparatively, the challenges in Peru are in many ways of a different sort and magnitude.)   Results from a number of evaluations of various sorts of the Uruguayan experience were shared at a recent event in Montevideo, Ciudadania Digital.  You can read an executive summary of results (in Spanish) here.

While each circumstance and context is of course unique, the types of challenges presented in the Peruvian case (geographical isolated communities, mountainous terrain, poor communities, indigenous languages) present a good test of the how well-intentioned, large-scale educational technology programs targeting populations that face some of the most difficult schooling and learning challenges might (and might not) work in reality.  If you want to know where theory meets the reality (or at least where one theory meets one reality), you could do worse than to look to Peru.

Here are the highlights from the IDB paper:

Even though this program has only recently been implemented, this document presents a few preliminary findings that could be relevant for its future development. On the one hand, we find evidence of better attitudes and expectations among teachers and parents; students that are more critical of school work and of their own performance; and a greater development of technological skills among girls and boys. On the other hand, there seems to be a decrease in the intensity of computer use in the classroom, as time passes and difficulties arise in the implementation of the project. Due to the short interval of time since implementation, no impact was observed in learning. This should be verified in future surveys.

 
The authors of the IDB report concede that it is too early to draw many sweeping conclusions from the progress of the OLPC program in Peru so far.  That said, comments on the report are beginning to appear in many places.  The independent OLPCnews.com web site, which tracks and comments the various OLPC initiatives in great detail (and which features a number of divergent voices, a number of them highly critical, and which has no official connection to the OLPC organizations that coordinate the various OLPC initiatives), weighed in its quick take on the preliminary results.  This followed a fascinating exchange in the comments section of infoDev's EduTech Debate web site between Christoph Derndorfer (who spoke at the World Bank last August) and Oscar Becerra, who directs the OLPC program at the Ministry of Education in Peru.  (You may also be interested in Walter Bender's take on this exchange.)

In addition to its (obvious) potential relevance to the students, teachers, and communities impacted by the OLPC program in Peru, the IDB evaluation is important for another reason as well.  Most evaluations of ICT use in education in developing countries are, at least in my opinion, of regrettably low quality.  To my knowledge, there have been precious few rigorous, large-scale randomized evaluations of an educational technology initiative in a developing country ever attempted at this scale.  (If you are looking for one of the few good examples of a rigorous evaluation of this sort, albeit at a more modest scale, you may wish to check out the work done in Colombia by my World Bank colleague Felipe Barrera, together with Leigh Linden.)

Working together with the Ministry of Education in Peru, the IDB has been investigating the OLPC project from the very start of its implementation there, and this perspective has the potential to provide fascinating insights into some of the critical factors that may lead to 'successes' or 'failures' of various types.  Reasonable people may well disagree about what constitutes 'success' or 'failure' in ICT/education initiatives of this sort, but by engaging with a project and evaluating its roll-out and impact over multiple years, and by publishing its related research methodologies and the data it collects, together with whatever conclusions it draws, the IDB and its partners will potentially offer a valuable tool to enrich our collective understanding of what types of impacts we might reasonably expect from large scale roll-outs of educational technologies in very challenging environments, and how such impacts might be measured.

For those interested in tracking developments with the OLPC project, you may wish to have a look at earlier related posts on the EduTech blog, as well as the following web sites and papers (in addition to those mentioned above):

• The IDB also looked at the early stages of the roll-out of the OLPC project in Haiti.  You may wish to monitor the IDB ICT/education blog for possible sneak peeks at results from the next round of evaluations in Peru (a second round of data collection was meant to be completed in November).
• The official OLPC wiki provides a wealth of information on the project around the world, including an 'Assessment Overview of One Laptop per Child Projects' (link is to a large PDF file).
• The World Bank is also conducting an evaluation of a small OLPC pilot in Sri Lanka.  We expect some results from this work in 2011; we'll share them here once they are available.
   

Please note: The image used at the top of this blog post of Machu Picchu ("sometimes the goals are clear to see -- it's just challenging to get there") comes courtesy of Martin St-Manat via Wikimedia Commons and is used according to the terms of its Creative Commons Attribution-ShareAlike 3.0 Unported license.  It was a finalist in the Fourth Annual Wikimedia Commons Picture of the Year (2009). [Martin St-Amant - Wikipedia - CC-BY-SA-3.0]

Due to a technical glitch, this post disappeared from the EduTech blog for undetermined amount of time.  It was re-posted on 5 January 2011.  We apologize for any confusion or inconvenience this may have caused.