Why we need more (not fewer) ICT4D pilot projects in education
This page in:
One message that is heard consistently at many ICT4D gatherings is that 'we have too many pilot projects', and that this is especially true for the education sector. 'What we need', or so the sentiment usually goes, 'is to scale up the pilot projects that have been on-going'. Indeed, 'scaling up' seems to be the answer to the funk that many prominent ICT4D organizations currently find themselves in these days, with changes in funding priorities in international donor organizations, foundations and the international private sector provoking many groups to re-examine many of their current practices. Scaling up is then a way to demonstrate (and re-affirm) the relevance of what many organizations have been doing since their inception, and by pursuing no more pilot projects such organizations can better orient themselves to working at scale. Or so the story goes.
I would like to sound a contrary note:
What we need are more ICT4D pilot projects,
not fewer,
especially in the education sector!
When many people talk about 'pilot projects', they are in fact talking about things that just aren't very big. We shouldn't confuse 'small projects' with 'pilot projects'. The purpose of a pilot project is to test something -- a process, a concept, a technology solution, an incentive system, a hypothesis, an implementation arrangement, etc. -- in order to see if it is worth supporting over time and/or at a larger scale. By its very nature, a pilot project is meant to have a component of experimentation. In order to know if something works, how and why it works (or doesn't), it is necessary to systematically (and dispassionately) document the activities and lessons learned from the pilot. Small projects, in contrast, are just things done at a small scale. There is nothing wrong with small projects, of course ('small is beautiful', as E.F. Schumacher remarked). Lots of good can come from them, and lots of NGOs do excellent work in supporting myriad valuable small projects utilizing ICTs in the education sector. But there is an important distinction here that many ICT4D organziations, and their funders, might do well to note.
Many developing countries are now moving implement ICT-related activities in education at scale -- many (regrettably) without being informed by pilot projects at all. Some NGOs are big enough to attempt pilot projects at scale ... but only a few ... and corporate social responsibility initiatives seldom reach scale. Many times, ICT-related small projects in the education sector are supported by groups with a strong (if not exclusive) ICT focus. Scaling up ICT-related initiatives in the education sector is typically done within the education sector itself. Not only is there a clash of cultures often at work here (NGO versus government, or, in CSR activities, private sector versus government), but the levers for change are quite different. For better or for worse, when we talk about doing things at scale in the education sector in most developing countries, we are talking about government action. Based on my experience, it is remarkable, and perhaps a bit depressing, that the lessons learned from so many small ICT4D projects seem to have little or no impact on government planning for such projects at scale in the education sector. The Stockhom Challenge annually recognizes innovative and successful ICT4D projects in a number of sectors, including education. To what extent are lessons from such projects being understood by policymakers planning for ICT-related interventions at scale? Not often enough, many would argue. This is both unfortunate ... and a missed opportunity.
(I do realize that not all of the education projects shortlisted for the Stockhom Challenge are small -- witness initiatives like TOPIC64 in Vietnam -- but a quick scan of all 1272 projects in the education category shows this to be an outlier in terms of its scale.)
If you buy this argument (and I'll be the first to admit that there are some holes* in it), then an important role for some NGOs currently looking to do things at scale might be, instead, to orient themselves on supporting and pursuing true pilot projects, exploring and testing solutions and implementation arrangements in ways that governments are often ill-equipped to do. Things are changing so quickly in the IT and ICT4D industries, and the demands and challenges contronting education systems are increasingly urgent (and daunting), that a focus on continuous learning is vital to help avoid large 'white elephant' ICT-related investment in education. Pilot projects can and should play an important role here.
How can we help to transmit lessons from small projects to the people making decsions at scale? When the World Bank's infoDev program was re-organized a few years ago from a small grant facility to an ICT4D 'knowledge shop', it was expressly to get at this issue (with mixed success). GeSCI is consciously deciding to work upstream in the ICT/education policy and planning process in a few key countries. Many NGOs working on e-content or digital learning materials issues (like the the Open Learning Exchange and IADP, to name just two) are consciously trying to pilot things that are potentially relevant to policymakers planning for ICT-related educational initiatives activities at scale. There is room for many more. There are certain things that you can learn only at scale. But you can also learn things from small projects that are very relevent to discussions to scale up the use of ICTs in education -- if you are serious about treating them as real pilot projects. This involves taking risks (and in some cases even experimenting), and accepting that not all small projects can (or should) be deemed a 'success' at the end of their initial funding period and sustained over time. This may put some NGOs in conflict with their donors, for whom project 'sustainability' is often a key criterion for whether an NGO continues to receive funds for its initiatives. But as we move from an era of many small projects to (expensive) projects at large scale in many developing countries, these are risks that groups have to be willing to take.
---
This blog entry was inspired by provocative discussions this week at the International Institute for Communication and Development (IICD), a Dutch NGO well-known in ICT4D circles for its support of grassroots partner organizations in Africa and Latin America. [Disclaimer: I serve on the IICD International Advisory Board.]
IICD has been sharing lessons from some of the projects (some small pilots, some pilot projects) it has helped to support in a number of sectors, including education. Many of these were collected in a publication with the (perhaps unsurprising) title, ICTs for Education: Impact and lessons learned from IICD-supported activities [pdf]. Additional IICD briefing sheets that might be of interest include:
- Leveraging the National ICT Programme for Education in Bolivia
- ICT for Education in Burkina Faso
- ICT in education in Tanzania
- Enhancing the Zambian Education Sector with ICT
- Lessons learned from the Education Support Network Project (ESNet) in Zambia
[Apologies for the delay in posting -- the author was traveling and made a technical mistake when attempting to publish this weekly entry.]
Mike,
I agree. I also think that evaluation plays a big role in what we learn from pilot projects and how far that knowledge is leveraged.
It seems that along with more pilot projects, well-documented and honest evaluations are much needed. Unfortunately, many of the overly laudatory ICT4D evaluations make objectively analyzing pilots a difficult task! Any steps we can take to leverage useful evaluations of ICT4D pilots would be beneficial.
Mike
EXCELLENT entry... I think there are political, operational, technical, and human-capacity-related barriers to establishing "real" pilot projects... You point out some of them at the end of your article. I tend to think that the World Bank (along with other key actors in the field) has a key role to play in fostering an "evaluation culture" to help overcome those barriers -although it's going to be a difficult, long-term task...
Thank you
Raul
Hi Mike and all,
First, I completely agree that impartially and effectively evaluated pilot projects _should_ be key tools in improving ICT projects in education.
There are many noteworthy challenges wrapped up in this position, however. Two of the chief ones (in my experience) are:
- Failure to link proposed ICT use to learning/skills/knowledge
The sub-problems here are also many, and range from over-reliance on test scores as indicators of learning to _under_-reliance on accurate and well-formulated (pre- and post-intervention, for example) assessments. It's very easy to restrict investigation of pilot projects in schools to "metrics" of implementation (how many labs, how many were functional, how many kids used them how often, etc.) and to perceptions (gauging the motivational impact of computers--which IS valuable, but not necessarily cost-effective).
- Failure to address findings generated by pilot projects
About half of the evaluations that I've been associated with have revealed serious problems in project conception and design -- asking teachers / students to do things that they have no incentive to do, for example, or basing a project on the expectation that IT teachers will support teachers of other subjects. In the majority of these instances, evaluation identifies the problem and characterizes it through several means, BUT those findings are simply too inconvenient or too fundamental to be addressed through changes in project design.
Absent the commitment to increased traction between project design, project implementation and learning in a scaled version, pilot-project evaluations often don't result in much improvement in terms of implementation.
Some thought provoking points Mike. I think that one of the bigger laments about ICT pilots though is not so much that there were small, but that in reality they were planned as experiments, yes, but in isolation and as once-off, supply-side interventions. I agree that we certainly need more pilots - especially given the rapidity with which new innovations appear to be emerging - but an important part of the planning process has to integrate monitoring and evaluation that considers also, the prospects of going to scale. my two pennies worth :)
Mike, Shafika,
Good points,
I agree the role of pilots, the ideas must first be invented and tested in real life. However, another question is how many of these pilots are done in isolation from the current policy environment and reality in the country?
What is the missing link, I think, is how the knowledge gained from pilots can be utilized at policy level and national planning process. This is a real gap. I also would not speak of up-scaling of one single pilot, but using the pilot as a learning experience for planning the national solutions. The national solution can then be, like it is in Finland, that every school follow their own plan (or idea) , or as it is in many developing countries, that there is a national roll-out model.
So, we should ask, why the pilots are done, what is the impact at national level, and what kind of policy environment (and resources) then can make it possible to utilize those piloted ideas in larger scale.
GeSCI (http://www.gesci.org) is in a process to create a research and innovation fund to facilitate this kind of research ( piloting, verifying ideas), evaluation, and looking the policy requirements and impacts of the innovations. GeSCI's main mission is to help policy makers to understand ICT related innovations in education, and therefore this kind of research is very valuable for our staff in the field.
Jyrki
Many thanks for your useful comments -- and questions! -- Jyrki.
For those of you interested in more information that hints at some of GeSCI's upcoming work in this area, have a look on the GeSCI blog at
http://www.gesci.org/Who-really-needs-policies-30072009.html.
I'm coming back to this ancient thread because I'm thinking about scalability and sustainability. Here is my current $00.02:
- Pilot projects should be designed from the get-go to be scalable. The _test_, such as it is, is a test of something that will work across a whole system. CapEx, OpEx, electricity, TPD, maintenance etc. should all be designed to reflect "real-world" and systemwide conditions, including how big the system is.
- Research projects can test the effectiveness of specific approaches (game-based learning, for example) without so much attention to real-world or system-scale constraints.
A practical progression, then, might be 1) research; 2) pilot; 3) scaled replication.