How would you design an ICT/education program for impact?


This page in:

where is this road leading us? the path ahead is murky | image attribution at bottomImagine, if you will, that you were an official at an international development organization who has been working with country x for a number of years in helping them think through options and issues related to the use of ICTs in their education sector.  As part of this dialogue, you had regularly preached the virtues of a commitment to rigorous monitoring and impact evaluation.

Country x has, in various ways, been host to numerous initiatives to introduce computers into its schools and, to lesser extents, to train teachers and students on their use, and schools have piloted a variety of digital learning materials and education software applications.  It is now ready, country leaders say, to invest in a rigorous, randomized trial of an educational technology initiative as a prelude to a very ambitious, large-scale roll-out of the use of educational technologies nationwide. It asks:

What programs or specific interventions should we consider?

We are not interested in evaluating a program that does not show tangible, measurable impact -- while there is scientific value in doing so, we feel this has been done already [they cite as one example this World Bank working paper].  We will take as a given that the introduction of computers etc. has a positive impact on student and teacher motivation toward learning, and that students and parents feel that the use of such technologies in schools will have a positive (if rather fuzzy) benefit over time.  We are interested in moving beyond the simple promotion of basic 'ICT literacy' -- we have been doing that for many years already in many places, and feel that we have a good idea of how to do that efficiently.  We would like to target the broadest group of students and teachers possible, and we want to do something where we can build credible control groups so that we can better tease out just what the impact of the technology has or has not been.

If no such program exists, how can we put one together, drawing on experiences from many existing programs from which there is much to learn but, by themselves, have not demonstrated the type of impact we are seeking?

Country x is (not surprisingly) most interested in being able to show a clear link between technology use and gains in standardized test scores, but is quite open to considering (and measuring) other types of impact, if given compelling reasons for doing so. Educational leaders have support from political leaders to be bold, provided they have a solid evidence base to support large funding requests going forward. And: They need to show results in a 'reasonable amount of time'.  (Just how long is 'reasonable' is of course open to interpretation -- experience leads me to assume that, as a practical matter, this usually means 'before the government changes').

One response to such a request would be to attempt to redefine the questions asked.  This is of course an entirely legitimate course of action ... but is simply not possible here.  The decisionmakers in the education sector in country x are firm believers in the potential for ICTs to be used in a variety of ways to help meet a variety of educational goals -- some of them, in fact, have advanced degrees in this topic and/or have led pilot programs of various sorts exploring educational technologies at small scales. There is a vibrant community of voices from the private sector and academia lobbying the country to adopt program A or technology B or approach C -- so much lobbying and marketing, in fact, that policymakers are having trouble finding a space to think away from all of the 'noise'. In some places in country x, computers (and the Internet) are still a novelty in schools, in other places they have been around and used for a decade or so.  The country is not looking for guidance on an 'approach' to making decisions about ICT-related investments in the education sector (it is familiar with holistic, systematic approaches like those advocated in such tools the infoDev-UNESCO ICT in Education Toolkit) -- it is looking for specific guidance on what to do.

What would be a useful response to such inquiries?  How would you design a program for measurable impact in a way that is immediately policy-relevant for decisionmakers contemplating large investments in the use of technology in the education sector, and what would this program look like?

Country x appears to be one of the few developing countries asking such pointed questions today -- let's hope many others will join them soon.

Please note: The image of an unpaved road in rural Georgia used at the top of this blog post ("where is this road leading us? the path ahead is murky") comes from Flickr user mortsan (Morten Oddvik) via Wikimedia Commons and is used according to the terms of its Creative Commons Attribution 2.0 Generic license.


Michael Trucano

Global Lead for Innovation in Education, Sr. Education & Technology Policy Specialist

Join the Conversation

July 20, 2010

At AMREF, we have been implementing a national-scale eLearning programme to upskill nurses using the PPP model since 2005. We adopted the Kirkpatrick model to evaluate this mode of trainingtraining and have done levels 1&2 assessments (user satistaction & performance respectively) and are currently doing the level 3 (practice) assessment.

Using the lessons learnt and good pracatice from our experience we have developed a model that we use to assess the "as is" status of a country with respect to implementing a natioinal-scale eLearning programme taking into consideration that countries context and objectives. From the findings of this model we are then develop an implementation strategy that we use to guide us as we implement a national-scale eLearning programme for that country. We are currently using the model to implement a national scale eLearing programme to upskill enrolled midwives in Uganda. I am very interested in hearing about how other large-scale eLearning programs in the developing world are being evaluated.

Joe Nutt
July 29, 2010

Acerbically pertinent question Mike and one I have a very active professional interest in. Here’s my answer.

The key objective is to decouple the belief that technology can in any meaningful way be linked to educational outcomes. You note yourself that many of the decision makers in country x hold this belief so firmly through their background and training, that they are not open to redefining the questions. So the best way to start that process is to ask, "Why would anyone seek to assume a tight connection between technology and positive educational outcomes?" The answer of course is because they doubt the existence of such a link.

Once you achieve that, then I think you are free to define a much more useful and meaningful range of ways of measuring impact. Since the strongest and most frequently cited argument by funders and government for ICT investment in education relates to employment, then my first choice for country x would be the employability of its students.