Why are there so many poor evaluations of ICT use in education?

Despite increasing attention to the impact of ICT on teaching and learning in various ways, the ICT/education field continues to be littered with examples of poor evaluation work.  A few of them arrive in my in-box every week.

There are many potential reasons advanced for the general poor quality of much of this work.  One is simple bias -- many evaluations are done and/or financed by groups greatly invested in the success of a particular initiative, and in such cases findings of positive impact are almost foregone conclusions.  Many (too many, some will argue) evaluations are restricted to gauging perceptions of impact, as opposed to actual impact. Some studies are dogged by sloppy science (poor methodologies, questionable data collection techniques), others attempt to extrapolate finds from carefully nurtured, hothouse flower pilot projects in ways that are rather dubious. (The list of potential explanations is long; we'll stop here for now.)

More fundamental to all of this is a belief by many that we still don't know how to evaluate the impact of technology use in schools.  People in this camp often paraphrase, unconsciously or not, the Solow computer paradox ("You can see the computer age everywhere but in the productivity statistics").  It may be, as an infoDev study argues, that "more useful analyses of the impact of ICT can only emerge when the methods used to measure achievement and outcomes are more closely related to the learning activities and processes promoted by the use of ICTs."  Fair enough.  But what advice do we have to share while we wait for this to occur?

Some recent studies of potential interest that you might have missed (all links to direct downloads of PDF files):

• Evaluation of the Jordan Education Initiative (EDC, 2008)
• Too much computer and Internet use is bad for your grades, especially if you are young and poor: Results from the 2001 Brazilian SAEB (Wainer et al, 2008)
• The Use and Misuse of Computers in Education - Evidence from a Randomized Experiment in Colombia (Barrera-Osorio & Linden, World Bank, 2009)
• The Effect of Computer Use on Child Outcomes (Malamud & Pop-Eleches, 2008) (Romania)
• Technology's Edge: The Educational Benefits of Computer-Aided Instruction (Barrow, Markman & Rouse, 2008)

 
Note: The image used at the top of this blog post from Kmarinas86 via Wikimedia Commons is used according to the terms of its GNU Free Documentation License.