How to measure technology use in education

ICTs are increasingly being used in education systems around the world. How do we know what the impact of such use is? How should we monitor and assessment the use of ICTs in education? How can, should and might answers to these questions impact the policy planning process?

Questions such as these are complicated in many countries by a lack of consensus on what can and/or should be measured, and how this measurement can and should take place. Lack of common sets of methodologies and indicators in this regard also hampers cross-national comparison of developments and the impact of related initiatives.

To help address such challenges, many organizations have begun to develop, or propose to develop, common sets of 'ICT in education indicators' to help guide their activities, and those of their developmental partners, in this area.

To promote harmonization of related efforts, representatives from the World Bank, KERIS, the Korean Ministry of Education, Science and Technology, UNESCO Institute of Statistics (UIS), UNESCO-Bangkok, the Inter-american Development Bank, the OECD (CERI), and the European Union - CRELL, joined by experts from universities in South Korea, the Netherlands and Canada, and representatives from the international initiative on the 'Assessment & Teaching of 21st Century Skills' met to:

1. share information about current and upcoming efforts sponsored by key organizations in this topic;
2. provide critical feedback and advice to colleagues leading initiatives in this area;
3. assess potential areas of cross-donor collaboration;
4. identify gaps in existing or proposed initiatives; and
5. propose areas for collaboration and joint activity going forward.

Proceedings from this event are now available on-line.

The UNESCO Institute of Statistics (UIS) has been perhaps the leading organization building consensus in this area to date, through its leadership role on education issues within the Partnership for Measuring ICT in Development.  Informed by work done by other groups, including the pioneering work of UNESCO-Bangkok in the Asia-Pacific region, UIS has proposed, and reached general agreement on, a set of nine initial 'core indicators' which are largely infrastructure-related.  Building off this core set, of indicators, UIS has brought together a group of 25 countries as part of a Working Group for Information and Communication Technology Statistics in Education (WISE) to expand and test a set of around fifty indicators, covering a wider variety of 'conceptual domains' (i.e. topics).  A useful short summary of this work is available here (in PDF), including the full list of extended indicators, together with some useful context and explanation.  This set of indicators is due to be discussed in Montevideo this December.

Global indicators of the sort that UIS is proposing will only get us so far, however.  At a project level, there are widely divergent approaches to monitoring (and ultimately evaluating) the activities and impact of a specific initiative.  Now, 'widely divergent approaches' can of course be a good thing, as individual indicators may be more or less relevant, given the particular objectives of a project.  That said, some agreement on conceptual frameworks to help guide such work could be useful, and last week's worksho featured spirited discussion of proposed frameworks from the Inter-american Development Bank, the European Commission's Joint Research Centre - CRELL, the OECD, and Korea.  More commonalities in approach emerged than differences; joint work on a commonly-endorsed framework should be out by the end of the year.

The workshop also featured presentations on the IDRC-sponsored work to build and maintain a 'Pan-African observatory' on the pedogogical use integration of ICTs and work jointly sponsored by Cisco, Microsoft and Intel to fund academic research on the assessment and teaching of 21st century skills (about which more in subsequent posts).

Also of potential interest:

• Results from the recent CRELL International Research Workshop on "Assessing the effects of ICT in education - indicators, criteria and benchmarks for international comparisons"
• infoDev's Monitoring and Evaluation of ICT in Education Projects: A Handbook for Developing Countries

The image used in this blog posting comes courtesy of the Tango Desktop Project via the Wikimedia Commons and is used according to the terms of its Creative Commons Attribution-ShareAlike 2.5 license.