Imagine, if you will, that you were an official at an international development organization who has been working with country x for a number of years in helping them think through options and issues related to the use of ICTs in their education sector. As part of this dialogue, you had regularly preached the virtues of a commitment to rigorous monitoring and impact evaluation.
Country x has, in various ways, been host to numerous initiatives to introduce computers into its schools and, to lesser extents, to train teachers and students on their use, and schools have piloted a variety of digital learning materials and education software applications. It is now ready, country leaders say, to invest in a rigorous, randomized trial of an educational technology initiative as a prelude to a very ambitious, large-scale roll-out of the use of educational technologies nationwide. It asks:
What programs or specific interventions should we consider?
We are not interested in evaluating a program that does not show tangible, measurable impact -- while there is scientific value in doing so, we feel this has been done already [they cite as one example this World Bank working paper]. We will take as a given that the introduction of computers etc. has a positive impact on student and teacher motivation toward learning, and that students and parents feel that the use of such technologies in schools will have a positive (if rather fuzzy) benefit over time. We are interested in moving beyond the simple promotion of basic 'ICT literacy' -- we have been doing that for many years already in many places, and feel that we have a good idea of how to do that efficiently. We would like to target the broadest group of students and teachers possible, and we want to do something where we can build credible control groups so that we can better tease out just what the impact of the technology has or has not been.
If no such program exists, how can we put one together, drawing on experiences from many existing programs from which there is much to learn but, by themselves, have not demonstrated the type of impact we are seeking?
Country x is (not surprisingly) most interested in being able to show a clear link between technology use and gains in standardized test scores, but is quite open to considering (and measuring) other types of impact, if given compelling reasons for doing so. Educational leaders have support from political leaders to be bold, provided they have a solid evidence base to support large funding requests going forward. And: They need to show results in a 'reasonable amount of time'. (Just how long is 'reasonable' is of course open to interpretation -- experience leads me to assume that, as a practical matter, this usually means 'before the government changes').
One response to such a request would be to attempt to redefine the questions asked. This is of course an entirely legitimate course of action ... but is simply not possible here. The decisionmakers in the education sector in country x are firm believers in the potential for ICTs to be used in a variety of ways to help meet a variety of educational goals -- some of them, in fact, have advanced degrees in this topic and/or have led pilot programs of various sorts exploring educational technologies at small scales. There is a vibrant community of voices from the private sector and academia lobbying the country to adopt program A or technology B or approach C -- so much lobbying and marketing, in fact, that policymakers are having trouble finding a space to think away from all of the 'noise'. In some places in country x, computers (and the Internet) are still a novelty in schools, in other places they have been around and used for a decade or so. The country is not looking for guidance on an 'approach' to making decisions about ICT-related investments in the education sector (it is familiar with holistic, systematic approaches like those advocated in such tools the infoDev-UNESCO ICT in Education Toolkit) -- it is looking for specific guidance on what to do.
What would be a useful response to such inquiries? How would you design a program for measurable impact in a way that is immediately policy-relevant for decisionmakers contemplating large investments in the use of technology in the education sector, and what would this program look like?
Country x appears to be one of the few developing countries asking such pointed questions today -- let's hope many others will join them soon.
Please note: The image of an unpaved road in rural Georgia used at the top of this blog post ("where is this road leading us? the path ahead is murky") comes from Flickr user mortsan (Morten Oddvik) via Wikimedia Commons and is used according to the terms of its Creative Commons Attribution 2.0 Generic license.
Join the Conversation