Impact evaluations are key to how we think about development. Pilot programs suggesting statistically significant impacts are hailed as breakthroughs and as candidates for scaling up. Programs without such clear impact tend to be looked down upon and may be terminated. This may not be warranted. A primary function of impact evaluations should be to improve existing programs, especially in fields where evidence of positive impacts remains scarce. The experience of OLE Nepal, which is part of the OLE network and aims to improve learning and teaching through technology, is instructive in this regard.
Last week, Rabi Karmacharya shared his experience at OLE Nepal at a seminar co-sponsored by the World Bank, the Global Partnership for Education and the Rotary Club of Capitol Hill, a supporter of Rotary Club of Kathmandu Mid Town in expanding OLE Nepal’s programs. After a successful career in enginerring and technology in California, he launched OLE Nepal in 2007. Now a social entrepreneur, Rabi wants to use technology to transform the way children learn through engagement, exploration and experimentation.
OLE Nepal has achieved quite a bit over the last eight years: deploying 5,000+ laptops in 100+ schools, training 600+ teachers on integrating ICT in the classroom teaching-learning process, developing 600+ learning modules for use by teachers, and creating a digital library with 6000+ books and other items used in schools and community libraries. Its ultimate objective is to help transform and improve Nepal’s education system with technology, working closely with the Ministry of Education and other partners.
What I found most interesting about Rabi’s presentation is how OLE Nepal – as well as donors that support the NGO such as the World Food Program, responded to an evaluation of its programs in 2009-11. The evaluation used a quasi-experimental design: English and mathematics tests administered to students, with collection of additional data through student, teacher and household surveys.
The results were disappointing, with no effect on student test scores even though both teachers and students reported liking the digital contents and finding them useful. The results contradicted field observations and other evidence from teachers and students on the fact that the program was making a difference.
Several factors may have led to an apparent lack of impact on student learning, according to Rabi. First, the program had been implemented only for a relatively short period of time, and improving student learning takes time. Second, it turned out that not all teachers reported to the one-week intensive training session that was held before launching the program, which may have decreased overall impacts in the schools that benefitted from the program. Third, some teachers may not have used as the available digital resources as much as expected due to the increase in workload that this entailed. Fourth, after the initial training, the support provided to teachers was limited. Fifth, it could be that the digital content, while following the curriculum taught in school, was too difficult for students to grasp.
These and a number of factors (including questions about variations in exam difficulty between the baseline and endline tests in the evaluation, and the possibility that some students spent more time on games available through the digital libraries rather than on study for examinations) may have led to the results.
What is important is that the evaluation did not lead to the demise of the program. Witnessing firsthand the changes in classroom dynamic and student engagement brought about by the program, donors continued to fund the NGO, which has been able to grow with support from the World Food Program, the Embassies of Denmark and Finland, and most recently the Air Asia Foundation.
In response to the evaluation, OLE Nepal introduced a number improvements in its programs:
- It now promotes a shared model between grades for a more effective use of laptops and digital resources by schools and teachers;
- The use of resources by students is better supervised and teachers receive more extensive support after the initial one-week training. In addition, a volunteers program is available for additional support;
- The educational contents have been revised to follow the curriculum even more closely; and
- More emphasis will be put on enhancing the ability of students to read in early grades, since this is a prerequisite for them to be able to learn subsequently and use digital resources effectively.
OLE Nepal’s experience is a great lesson in social entrepreneurship, humility, and resilience from an innovative NGO that strives to help teachers and students harness the power of technology in the classroom.
Follow the World Bank Group education team on Twitter @wbg_education
Join the Conversation