As efforts to distribute low cost computing devices and connectivity to schools pick up steam in developing countries around the world , many ministries of education are systematically thinking about the large scale use of digital educational content for the first time. Given that many countries have already spent, are spending, or soon plan to spend large amounts of money on computer hardware, they are often less willing or able to consider large scale purchases of digital learning materials -- at least until they get a better handle on what works, what doesn't and what they really need. In some cases this phenomenon is consistent with one of the ten 'worst practices' in ICT use in education  which have been previously discussed on the EduTech blog: "Think about educational content only after you have rolled out your hardware". Whether or not considerations of digital learning materials are happening 'too early' or 'too late', it is of course encouraging that they are now happening within many ministries of education.
As arguably the world's highest profile digital educational content offering in the world -- and free at that! -- with materials in scores of languages, it is perhaps not surprising that many ministries of education are proposing to use Khan Academy content in their schools.
The promise and potential for using materials from Khan Academy (and other groups as well) is often pretty clear. Less is known about the actual practice of using digital educational content in schools in middle and low income countries in systematic ways.
Researchers at SRI recently published  the first rigorous study looking at how Khan Academy content is being used in formal educational settings as part of math instruction in the United States. Although Khan Academy originally developed as a supplemental aid in individual tutoring outside of school, it is now, in the words of SRI, "also working closely with schools to explore ways of transforming how instruction can be organized, delivered, and experienced by both students and teachers." Some of the findings from the SRI study, which was supported by the Gates Foundation, may offer important insights into potential implementation models of relevance to middle and low income country contexts -- as well as highlight where things may be a little more complicated than they may first appear. Dr. Robert Murphy , who helped lead the research team at SRI International's well-known Center for Technology in Learning  that produced the Research on the Use of Khan Academy in Schools  report, stopped by the World Bank earlier this week  to share some related key lessons and observations.
For what it’s worth, here are five things I took away from Murphy's presentation and the subsequent Q&A session, as well as my reading of the excellent SRI study (I'll note here that these are some of *my* takeaways; I hope I have not misrepresented anything that is said in the report itself, or by Bob during his excellent talk):
1. What you set out to study may not always be (exactly) what you find in practice
The SRI researchers expected to find one dominant usage model (the 'flipped classroom') and that teachers would be using the Khan Academy videos as a core curriculum resource. SRI found that a wide variety of usage models were in evidence in practice (and suspect that even more may be out there). The Khan Academy is perhaps the leading global example offered in support of ‘flipping the classroom’, the pedagogical movement which seeks to capitalize on better access to computers and connectivity at home so that students watch lectures as their ‘homework’ on their computers instead of listening to a teacher lecture at school, and then work through their problem sets at school with the help of teacher. However, at least in the schools studied by SRI, there were no examples of this actually happening in practice. While most people associate the Khan Academies with videos, in the formal school settings studied it turned out that the instructional videos were not watched all that much -- it was the problem sets that were most widely used by teachers to provide students with productive opportunities to practice new skills introduced through traditional teacher lectures.
2. What you set out to do may not always be that possible (and this isn't always such a bad thing)
Originally this research was conceived of as an evaluation of the impact of Khan Academy. However, the SRI research on Khan Academy evolved over time into more of an implementation than an impact study. In a way typical of many Silicon Valley start-ups, Khan Academy staff were eager to iterate their product quickly, based in part on preliminary feedback they received from a variety of sources, including from users and from the researchers at SRI, and so doing something like a randomized control trial (RCT), considered the 'gold standard' evaluation technique in some quarters, was simply not practical. How do you evaluate something that is constantly evolving, and improving? This was a challenge faced by the SRI research team.
3. Implementation studies, as opposed to evaluation studies, should be considered more often by those wishing to learn from educational technology initiatives
I really like that this SRI study is an implementation, not evaluation, study. Where you introduce something that is so new, 'evaluating' the 'impact' right away isn't always that practical, especially where (as in the case of Khan Academy) what was being used changed a lot during the course of the period being studied (in part as a result of what the researchers were finding). In my experience, technology folks don't operate on the same timeline as researchers hoping to publicizing results from RCTs in peer reviewed journals -- if they learn something, they'll iterate immediately and change their product or service so that things potentially improve *today*. They can't (and won't) wait around for a year or two to see what made an impact and then start to redesign their product. This is a reality that more than a few of the RCTs of educational technology projects that I know are being discussed may eventually confront, especially where the projects being evaluated include products or services that sit in the cloud and which can therefore be modified almost immediately by the vendor/developer where/when they think things can be improved in some way. This is not to advocate against RCTs, or impact evaluation studies, of technology use in education, but rather to say that we need to be careful that they are appropriate for the context and needs of the places that are being studied. Where groups are looking to fund impact evaluation studies of technology use in education (of whatever sort), I often suggest that they may want to consider funding some sort of related implementation study as well. The impact evaluation may not actually tell us anything, but even if it does, it may (depending on how it is set up) may tell us about yesterday's product or service and not offer that much value for decision making going forward. Implementation studies are almost always useful, though, in getting at the reality of use cases and user needs. Doing both of these sorts of things, where feasible, may be something to consider, even if it means scaling back the evaluation part a bit.
4. Even in an age of technology-enabled personalized learning, teachers still remain 'in charge'
Not all students are ‘independent learners’. While much is made of the potential for technology use to enable student-directed, personalized learning, in practice most students will need a lot of assistance in helping to develop the tools and skills to become more independent learners. (As a former teacher myself, this is consistent with my experience in the classroom as well.) While it is assumed that most usage of Khan Academy today is by users independently viewing the content and using the problem sets and analytical tools at home, this was not how things worked when Khan Academy content was used in schools studied by SRI. This is not to say that teachers and students studied by SRI did not find value in the Khan Academy content -- they clearly did. Rather, the usage models observed by SRI suggests that, in some teacher-directed environments (which school classrooms today around the world clearly are, for better or for worse), the use of tools meant to enable 'personalized learning' might in practice be used to *help teachers* better adapt what it is that they teach. In the schools studied, teachers for the most part did not take great advantage of many of the (increasingly robust) analytical tools that are offered by Khan Academy to monitor and assess student performance. This might be because they didn't understand them. It might be that they didn't find them of value. Or it might be that they felt that they had other assessment tools at their disposal (regular quizzes, tests, personal interactions with students) that they were trusted and with which they were more comfortable. It may be that many teachers didn't have their students watch the Khan Academy videos because they thought that they were already doing a decent job with their lectures and demonstrations. What they needed was not a replacement for lectures, but rather a bank of problem sets (and ways to provide students with immediate feedback and support on their work), and so that is how the Khan Academy was used.
5. No digital educational content repositories are 'comprehensive', no experience entirely immersive
For many people scrolling through the vast store of mathematics materials on the Khan Academy site, it might appear at first glance that most 'everything' is covered. Over the past few years, Khan Academy has greatly built out its offerings, filling in content gaps and reorganizing the way some of the materials are presented and sequenced. Critics may argue about the utility and relevance of U.S. Common Core standards (there is no shortage of places on the Internet where you can find discussions on that topic, so I'll politely refer folks who like to engage in such discussions to other sites where they can post and argue about such things), but the standards have been of undeniable use in helping Khan Academy map out areas where additional content needed to be developed. In none of the schools studied by SRI was Khan Academy content the only mathematics instructional materials available, however. One challenge for teachers, and for students, was to figure out where, when and how other materials might be utilized as compared with Khan Academy content (and vice versa). Even in low income schools in the United States, there is typically no shortage of educational content available. It may not always be all that great. It may not all be coherent, consistent or aligned. But a lot of it is available. Even where digital education content (from Khan Academy or elsewhere) is pretty comprehensive in its treatment of a given topic and is designed in a very modular format, students may have specific questions that are better answered outside of the digital environment. To me, one of the great attributes of the Khan Academy videos is that they are so easy to rewind. Didn't understand something the first time? Just watch it again (and again and again, if need be -- you aren't bothering anyone else). That said, SRI found that it may often be easier, and more productive, for students to simply ask a question of their peers or teachers than to try to locate and then rewind their way through a video to find the moment of instruction or demonstration that may address their confusion. That is to say: Even where digital education content offerings are quite comprehensive and useful, they are only one piece of a larger whole. (This conclusion is rather obvious, but from time to time it may perhaps be worth re-stating -- especially in response to groups who propose investing only in a single, comprehensive, all-encompassing digital learning tool as part of efforts to 'transform' teaching and learning.)
The SRI research study into the use of Khan Academy is quite rich in detail and contains a lot of valuable food for thought. I recommend you read it , not only for its insights into Khan Academy (which, given its widespread use and prominence, is perhaps reason alone to read the study), but also because it offers larger insights in how education technology products and services are used (and not used) in actual practice. There is no shortage of hype and nonsense about a lot of the products and 'solutions' being marketed to education systems around the world. There is also a lot of really good and useful stuff going on. Reading and reflecting on this SRI study may help you as you try to tell the difference.
This is the first in a short series of blog posts looking at various insights emerging from the development and use of the Khan Academy as a way of exploring a number of related larger issues.
Note: The image used at the top of this blog post of a man demonstrating chemical properties in front of an audience ("this is fascinating, but wouldn't it be better online?") comes via Wikimedia Commons  and is in the public domain.