As a result of reading the recent IDB study on the impact of the One Laptop Per Child project in Peru, my World Bank colleague Berk Ozler recently published a great post on the World Bank's Development Impact blog asking "One Laptop Per Child is not improving reading or math. But, are we learning enough from these evaluations?"
Drawing insights from his readings of a few evaluations of technology use (one in Nepal [PDF] and one in Romania) he notes that, at quick glance, some large scale implementations of educational technologies are, for lack of a more technical term, rather a 'mess':
"The reason I call this a mess is because I am not sure (a) how the governments (and the organizations that help them) purchased a whole lot of these laptops to begin with and (b) why their evaluations have not been designed differently – to learn as much as we can from them on the potential of particular technologies in building human capital."
Three members of the team at IDB that led the OLPC Peru evaluation have responded ("One Laptop per Child revisited") in part to question (b) in the portion of Berk's informative and engaging post excerpted above. I thought I'd try to try to help address question (a).
First let me say: I have no firsthand knowledge of the background to the OLPC Peru project specifically, nor of the motivations of various key actors instrumental in helping to decide to implement the program there as it was implemented, beyond what I have read about it online. (There is quite a lot written about this on the web; I won't attempt to summarize the many vibrant commentaries on this subject, but, for those who speak Spanish or who are handy with online translation tools, some time with your favorite search engine should unearth some related facts and a lot of opinions -- which I don't feel well-placed to evaluate in their specifics.) I have never worked in Peru, and have had only informal contact with some of the key people working on the project there. The World Bank, while maintaining a regular dialogue with the Ministry of Education in Peru, was not to my knowledge involved in the OLPC project there in any substantive way. The World Bank itself is helping to evaluate a small OLPC pilot in Sri Lanka; a draft set of findings from that research is currently circulating and hopefully it will be released in the not too distant future.
That said, I *have* been involved in various capacities with *lots* of other large scale initiatives in other countries where lots of computers were purchased for use in schools and/or by students and/or teachers, and so I do feel I can offer some general comments based on this experience, in case it might of interest to anyone.
A number of years ago, infoDev, UNESCO and a few other groups put together a 'toolkit' for use by policymakers that was eventually used as part of ICT/education planning processes in about 30 countries. Watching Wadi Haddad, the principal author of the toolkit, walk senior policymakers through the decisionmaking processes at the heart of the toolkit was always an educational experience for me, especially when he discussed the various pressures that ministers of education faced when trying to figure out to what extent, and how, they should be introducing and utilizing new technologies into the education systems in their countries. A former minister of education himself, he would receive immediate smiles and nods of recognition when he discussed the 'pressures' exerted from various quarters:
vendors: "it's a new world, we have new technological answers to help solve many of your old problems"
business community: "you run your schools like it's still the 19th century, we need workers who know how to use the tools of the 21st century"
parents: "by buying lots of new technologies, you can demonstrate to us -- quickly -- that you care about our children"
academics: "if you want to transform what you are doing today, you need to adopt the new technology-enabled methods and approaches that we, the real experts, are championing"
international competition: "look at the countries with real vision around the world -- if Uruguay and Rwanda and Singapore are doing it, why aren't you?"
fashion (this one should be self-explanatory)
This is not to say that all such pressures are bad, just that they are there. The extent that these essentially political pressures can be leveraged to help bring about useful courses of action is often a function or measure of good leadership. 'Education' is a famously long term enterprise, the results of which, at an aggregate level, may not be felt for a generation or two. The visual and political impact of buying lots of computers is much more immediate, and can (in theory) buy policymakers space to make other, more fundamental changes.
This is not to say that the short term effects are all necessarily positive, of course. In a recent presentation at the World Bank on a pilot initiative in India exploring the use of mobile phones in low income private schools, Matthew Kam shared findings from ongoing research that showed that one good way to spur technology adoption in schools in certain communities is to use ICTs for basic test preparation activities -- something which many parents like a lot, even if the type of pedagogy reinforced by many such activities makes many educators cringe. (See #3 in the recent EduTech blog post on Ten things about computer use in schools that you don't want to hear for more on this topic.)
You can agree that there are lots of political pressures compelling policymakers to invest in educational technologies, note that there are lots of long-standing challenges that may require new approaches to overcome (and that technology use may be integral to such new approaches), and concede there may need to be sacrifices in the short run to help build consensus and momentum that can help in the long run, but this doesn't really answer the question:
Why aren't more decisions to embark on large scale educational technology projects more evidence-based?
Here are the most common answers I hear to this question that can't be simply dismissed as 'politics':
'We can't make evidence-based decisions when we don't have the evidence.' This is especially true, many feel, when it comes to the evidence base related to educational technology investments (a topic frequently explored on the EduTech blog), but, as the IDB team note in their blog post, citing a recent study by Glewwe, Hanushek and others, it still very difficult in many cases to get clear, useful policy guidance about what works in education generally.
'People do make decisions based on the evidence ... only the evidence comes from sources with a financial stake in decision.' Vendors are often blamed in this regard, but it is important to note that NGOs and other groups can be at fault here as well.
'This stuff is so new, the evidence that we do have doesn't really apply.' Whether or not one believes this, it doesn't necessarily follow (at least for me) that there is no value in rigorously testing our assumptions, and documenting and testing what we are doing, so that we can learn as we go somewhere we haven't gone before.
'We don't need to evaluate ... because we just know it works.' This argument, which is seen to be compelling in many circumstances by key decisionmakers and proponents of certain approaches to the use of new technologies in education, is admittedly a hard one to counter with facts. (One response is to say ''OK, we agree with you that it works, but how do we know that it works more/better than if we used the related resources in some other way that might have even greater impact?" Unfortunately, in my experience, folks who don't believe in the value of evaluation in general don't tend to be convinced that there is a great comparative value in evaluation either.)
There are often critical mismatches between the rationales put forward to justify large investments in educational technologies and what is actually implemented and what is actually measured. Despite the scientific rigor that defines good impact evaluations, there is no denying that the act of evaluation itself is often a very political act. I have spoken with many countries who said that they didn't have money for evaluation ... and then, when they did find money for this purpose, the price of access to do research -- e.g. only pursuing certain sorts of questions in certain sorts of places -- was too dear for the researchers to pay. In other cases, researchers make accommodations, calculating that some access is better than no access. In yet other cases, researchers may misunderstand the types of impacts that project proponents really desire. Looking for evidence of the impact of an ambitious program like Uruguay's Plan Ceibal only on student performance in school, for example, ignores that fact that the project there has been to a large extent about 'societal transformation through technology', with the education system the most convenient vector through which to introduce sets of technologies and technology-enabled approaches which are intended to have impacts far beyond the education system.
---
If you don't know what works, there can be an understandable temptation to try to create a picture that more closely resembles things that work. In some of his presentations on the dire state of student learning around the world, Lant Pritchett invokes the zoological concept of isomorphic mimicry: the adoption of the camouflage of organizational forms that are successful elsewhere to hide their actual dysfunction. (Think, for example, of a harmless snake that has the same size and coloring as a very venomous snake -- potential predators might not be able to tell the difference, and so they assume both have the same deadly qualities.) For our illustrative purposes here, this could mean in practice that some leaders believe that, since good schools in advanced countries have lots of computers, it will follow that, if computers are put into poor schools, they will look more like the good schools. The hope is that, in the process, the poor schools will somehow (magically?) become good, or at least better than they previously were. Such inclinations can nicely complement the ''edifice complex'' of certain political leaders who wish to leave a lasting, tangible, physical legacy of their benevolent rule. Where this once meant a gleaming monument soaring towards the heavens, in the 21st century this can mean rows of shiny new computers in shiny new computer classrooms.
---
Near the end of his post, Berk states that
"One important role larger development organizations like the World Bank or IDB can play is in testing big ideas like these across multiple countries or settings. No one with a pulse in 2012 thinks that cheap laptops are not a good thing: we’re just trying to decide whether we should be spending precious funds on subsidizing them for families with young children. Same with Millennium Villages: perhaps the ‘big bang’ approach has merit. But every such idea needs to be assessed properly, allowing us to learn as much as possible from each study. The bigger the idea and the hype, the more important the evidence becomes.
We have come some distance from the days when we used to implement projects and programs with the belief that they would work – without much in the way of thorough evaluations."
I certainly would like to believe that we have indeed ''come some distance'' ... but, at least as relates to the use of ICTs in education, I am not so sure. Given the lack of rigorous evidence to inform related decisions in the United States, a staffer at the U.S. Department of Education once remarked to me that educational technology was in some ways the real faith-based initiative that was being championed in many schools. By repeating this rather clever (if exaggerated) witticism, I don't mean to criticize 'faith' in this context. It is, after all, an important ingredient that can compel action. Combine it with some rigorously gathered and analyzed evidence, however, and you have a potentially potent concoction that could help catalyze real change.
In conclusion, Berk states that
"As researchers and as policymakers, we all have to be more proactive in producing evidence before decisions are made. Until then, studies like the ones covered here will be second-best solutions putting out fires instead of preventing them."
This is certainly true, and well said. In my experience, however, many policymakers become frustrated with 'experts' who are quite adept at putting out and preventing fires, but who offer little practical guidance on alternative courses of action. (I deliberately place the word 'expert' in quotation marks here, as this is a role in which I often find myself cast in such discussions; given my own limitations, this designation is perhaps best understood as convenient shorthand, noting that those assigned this label are often only a little less ignorant about what to do than some others involved in the same discussion.) Given the rather checkered history of many large scale educational technologies initiatives around the world over the past two decades, we do have an increasingly dense knowledgebase on what not to do. If it is true that 'fortune favors the bold', as the ancient Roman saying holds, it is a pity when the bold are advised by people who advocate 'revolutionary' courses of action that lead to sadly predictable dead ends.
Way back in AD 79, noted Roman man of letters Pliny the Younger similarly invoked Fortuna, the Roman goddess of luck, when charting a course that brought his ship closer to Vesuvius. Unfortunately, he and his crew perished soon afterward when that volcano erupted. In retrospect, the truly bold move may have been to turn instead to the open seas and set sail for the unknown. Would he have been more successful as a result? Maybe, maybe not, but at least he would have made a different mistake. To increase the likelihood that his journey might have been a success, however, he undoubtedly could have used some better navigation tools. Providing such tools (and not just hypotheses) to policymakers today as they consider large investments in new technologies to help make current education practices a little more efficient, or cheaper, or more impactful -- let alone to 'transform education', to adopt the rhetoric that so often accompanies announcements about large scale purchases of ICTs in the education sector and which motivates so much of us -- should be a high priority for all of us working in the field. The science fiction author Arthur C. Clarke famously contended that "Any sufficiently advanced technology is indistinguishable from magic." Maybe -- but expecting magic to happen just because you buy a lot of shiny new gadgets often results simply in conjuring up the ghosts of past failures.
*The deliberately provocative title of this blog post is inspired by the rather cheeky title of an article in The Economist on the recent large-scale purchase of tablets for use in education in Thailand ("Let them eat tablets: Trying to stop the rot in Thailand’s schools by giving out tablet computers"), just one of a number of countries currently pursuing large scale educational tablet initiatives. I have been challenged to come up with titles to my blog posts that are pithier and more memorable. With this title here, I certainly mean no offense to the many visionary leaders who have championed various large scale educational technology projects around the world, many of which have been referenced on this blog over the years. That said, and generally speaking, asking some basic and pointed questions of this sort about the motivations behind some of the high profile announcements of big educational technology purchases probably can't hurt.
Note: The image of Marie Antoinette at the top of this blog post from an oil painting by Elisabeth Vigée-Lebrun ("in my hand I have a very precious gift for you") comes from the Google Art project via Wikimedia Commons and is in the public domain.
Join the Conversation