Contextualizing lessons from the use of the Khan Academy

Based on my interactions with educational policymakers, and those who advise them, it seems to be a truth (almost) universally acknowledged that there is insufficient research into models of educational technology use, the impact of such use, and related costs. This is not to say that there is *no* useful research into the use of technology in education around the world, of course. Online tools like ERIC and SSRN can help you find some useful studies; the popular press and the blogosphere increasingly reference such work (sometimes even in ways where you can actually track down the referenced studies!); there are of course a lot of academic, industry and professional journals dedicated to the subject; and a healthy amount of 'grey' literature circulates informally (including stuff commissioned by companies that is never formally published). Firms also circulate 'white papers' touting the 'impact' of their products and services, something which I tend to place into its own separate category, given the commercial and marketing imperatives that often animate such work. That said, just because a lot of 'research' is produced doesn't mean that such research is helpful to meet the practical information demands of educational policymakers, planners and educators.

Even if you *are* of the opinion that there is indeed a lot of useful, policy- and practice-relevant research out there related to the use of technologies in education, the fact remains that most of our collective knowledgebase has been constructed as a result of studying and attempting to learn from experiences in 'highly developed' (OECD) countries. While there is always danger when trying to draw generalized lessons from a research study that examines a specific context, it would seem reasonable to me that the difficulties when looking to draw lessons from experiences in Quebec that might be relevant to Kansas or Canberra pale in comparison to those when trying to extend such lessons to policymakers making decisions which will affect students and teachers in places like Quito or Kampala -- let alone rural Cambodia.

Thankfully, there are a number of promising moves afoot which hope to direct more energy and resources to investigate issues and circumstances of relevance to those exploring the use of ICTs in education in middle and low income learning environments and contexts around the world. (As such efforts kick off, and especially as related studies emerge, we would hope to feature them on the EduTech blog.) Until we start to see results from these sorts of efforts, however, the practical reality is that, in most cases, policymakers in middle and low income countries who wish to draw lessons 'from the research' in order to inform their policymaking related to potential educational technology initiatives will continue to try to contextualize results from research in higher income countries in the attempt to divine what lessons (if any) might be relevant to their own circumstances, even in places where contexts for use and typical use cases may be quite different.

---

A previous EduTech blog post, "Evaluating the Khan Academy", explores some of the lessons that have emerged as a result of research by SRI into the use of Khan Academy in a number of schools in Northern California. For those who don't know it: The Khan Academy is a widely known and much celebrated educational website which features thousands of short video tutorials on educational topics, as well as linked sets of over 100,000 practice problems and a 'personal learning dashboard'. Policymakers from a number of countries have approached the World Bank for specific advice and guidance on how they might make use of Khan Academy resources within their schools, and during the course of related conversations (some of which were catalyzed by a talk given by SRI's Robert Murphy at the World Bank earlier this year, which was discussed in a subsequent blog post) we have passed along and discussed the Research on the Use of Khan Academy in Schools published by SRI -- as well as a report that appeared a few months ago from EDC that looked at the use of Khan Academy in Chile. While individual lessons and insights drawn from an analysis of the use of Khan Academy content at sites examined in the SRI and EDC reports may or may not be more broadly generalizable to other contexts, examining the various usage models documented and explored in those studies may help raise *questions* that might be relevant to educators and policymakers who work in other contexts.

Indeed:

What are some useful questions that policymakers in middle and low income countries might ask as they attempt to contextualize insights from the use of the Khan Academy in California and Chile as part of their efforts to investigate and plan for the use of digital learning resources from the Khan Academy (and from other sources as well) within their own education systems?

For what it's worth, and in case presenting them here might be helpful to groups working on such efforts, here are some of the related questions that have surfaced during discussions of the SRI and EDC reports in which I have been involved. This list is not meant to be exhaustive or comprehensive -- and indeed there may well be other, more important, and even more fundamental, questions to ask in many places.

The intention here is not to imply that the Khan Academy is the 'right' or indeed 'best' choice for a given education system to consider (Khan Academy is one of the most prominent of untold numbers of digital learning tools, products and services of potential relevance to education systems around the world – and of course there are lots of other ‘non-digital’ ways in which countries can spend scarce resources dedicated to teaching and learning!), but rather to enrich a number of ongoing discussions about the potential use of Khan Academy (and similar) content in education systems in a number of countries.

Ten (sets of) questions

1. Variety of usage models
One thing that is very clear from the SRI report is that the Khan Academy is being used in many different ways in different schools -- and sometimes even in different ways within the same school. The SRI study documents Khan Academy being used as a tool for additional practice; as remediation for students who are behind and as enrichment for those who are highly motivated or advanced; as well as to monitor student learning. Such uses are of course quite relevant and useful in schools and educational contexts in low and middle income countries as well. This flexibility in usage models is potentially quite valuable -- especially in schools and classrooms where you have highly capable, experienced teachers who can find the right usage model that works well with their students. That said, is this flexibility a good or bad thing in 'low capacity' education systems where teacher capacity and quality is a major concern, and where many educational planners experience challenges as they plan for many of the 'basics' (getting textbooks to schools, ensuring the correct number of desks, making sure that teachers are at work, etc.)?
 

2. Mechanical processes and procedures
One persistent criticism of the Khan Academy from some quarters is that it focuses on the mastery of various mechanical processes and procedures (e.g. factoring numbers, long division) at the expensive of developing an understanding of higher order concepts. Whether or not one agrees with such criticism, many policymakers in middle and low income may (rightly or wrongly) wonder about the relevance of such criticism to their own immediate circumstances. (You can find many related discussions quite easily on the Internet if you're interested in them -- some of them thoughtful, well informed and useful; some rather breathless; some quite ideological; still others vituperative and snarky -- so I'll not attempt to address and consider them here.) For some education officials, it is precisely this emphasis on process and procedure that they find most alluring about the Khan Academy. Other policymakers may concede that a focus on the development of the mastery of various 'mechanical' processes is only one piece of a larger puzzle (and perhaps not even the most important piece), but look at the situations in classrooms in their countries' schools today, where very little learning is taking place and students (and indeed many teachers) struggle with things at very basic level, and wonder: In our circumstances, wouldn't helping our students with the development of some very basic mechanical processes and procedures be a step in the right direction -- and leave us in a much better shape than where we are today?
 

3. Teacher use of student learning data
Khan Academy has developed -- and continues to develop, at great expense -- an impressive set of tools so that teachers and students can monitor student progress through Khan Academy content. Despite the availability of easy-to-use 'dashboards' that can help teachers monitor student progress, however, the SRI study did not seem to find that a lot of teachers were actually using them. No matter how potentially useful they may be, should we expect that teachers in low capacity education systems will use such tools, especially given that access to, and familiarity with, digital technologies may be quite limited? Or: Will teachers in such circumstances perhaps actually use these sorts of monitoring tools *more*, given that they don't have other sources of formative assessment data?

4. Some key facilitating factors: Access to computers, longer class periods
The SRI reports note a number of key factors which seem to have facilitated the use of Khan Academy in the schools studied. Notable among these include the existence of 1-to-1 computing efforts in some schools, which helped facilitate access to the website and content, as well as the fact that some schools were able to utilize longer, extended blocks of time during the schedule of the school day. In most middle and low income countries, efforts to provide broad access to educational computing devices (especially those which are meant to enable so-called '1-to-1 computing', where each student has her own device) are still in their infancy. Reconfiguring the school day to be able to dedicate longer blocks of time dedicated to specific subjects in order to better accommodate the use of related Khan Academy content may simply not be practical in many educational settings in middle and low income countries -- especially those which are notably rigid in their approach to scheduling or where large numbers of students and limited numbers of classrooms made such changes very, very difficult. Absent the widespread availability of computers within schools for student use, and the potential to lengthen class periods so as to be able to make better use of computer resources to access Khan Academy content, what other facilitating factors might education systems wish to explore in order to ensure sufficient access to the Khan Academy?
 

5. Support structures for learning
The SRI study also makes note of a number of other important 'support structures for learning' that exist in the schools in Northern California that were studied. As difficult and challenged as the learning environments in under-resourced schools in poor communities in the United States may be, students in those schools still have access to educational content from a variety of sources (including textbooks), to teachers (who, no matter how well or poorly qualified, due show up to school in sufficient numbers, unlike in many other parts of the world), as well as to peers and parents who may be tapped to help explain and discuss various concepts being studied. The use, and utility, of the Khan Academy in such contexts may be quite different than in environments in low income countries where such support structures for learning may be much weaker, and in some cases may not exist at all. In environments where the Khan Academy is meant to be used in large part because of the absence of such support structures for learning, what steps might educational policymakers and educators consider in order to make the 'successful' use of the Khan Academy more likely? More broadly: What are the important things that need to be in place (human resources, technology, incentive structures, pedagogical practices, physical space, etc.) for tools like the Khan Academy to be introduced most productively, with the greatest impact?

6. The videos, and 'flipping the classroom'
The Khan Academy is held out as a prominent example of 'flipping the classroom', where students listen to lectures outside class (e.g. by viewing Khan Academy videos at home) and then do their 'homework' when in class, aided by their teachers (and their peers). Despite many assumptions that this is how Khan Academy is actually utilized within formal educational settings, the SRI study did not see much of this happening in practice, for a variety of reasons. Indeed, to a large extent teachers appeared to want to continue to teach as they had already done, and largely conceptualized the use of the Khan Academy as a complement to their traditional teaching methods. In other words: the Khan Academy video lectures were not used to replace the lectures of teachers, but were instead positioned as resources for students to consult and review after the traditional lecture was delivered. Is the flipped classroom model relevant in middle and low income countries, given existing pedagogical practices and norms, especially in places where access to connectivity and computers outside of schools is low?
 

7. Planning for content that changes
For better or worse, the content in printed textbooks does not change very often. Indeed, in many countries the same textbooks may be used for many years and are updated very infrequently (they may be in critically short supply as well). One potential advantage of digital learning resources, as made available through efforts like the Khan Academy, is that they can be updated much more frequently than printed materials. That said, in places without reliable connectivity, such advantages may be more theoretical than practical. In environments where connectivity is less of a challenge, teachers and education systems may be challenged to adapt to the emergence of new digital learning content that they have not already been exposed to, have not vetted and which has not been included as part of official teacher professional development and training programs. This is especially true in 'low capacity' education systems where teachers are inexperienced, poorly trained and often struggle with mastery of the content they are supposed to deliver. How can education systems and educators plan for and take advantage of updates to digital learning content at a pace and schedule which is much faster than that which previously marked their use of printed textbooks?

8. Competency-based advancement and standardized tests
One virtue identified by many educational policymakers of tools like the Khan Academy is that they enable competency-based advancement, whereby students continue to study a particular concept or topic until they demonstrate a certain level of related mastery. This makes a lot of sense, especially in subjects where concepts explicitly build upon each other, as is quite often the case in mathematics. If you can't multiply single digit integers this month, you're probably going to have a tough time multiply two and three digit integers when you are asked to do so next month. Why move on to multiplying 55X77 when you haven't demonstrated that you can successfully multiply 5x7? In most classrooms, students move in lock step through the curriculum, with many students frustrated/bored because things are moving too fast, and many other students bored/frustrated because things are moving too slow. That said, in education systems that rely on high stakes end-of-year exams that cover the entirety of a given curriculum -- which is the reality in most middle and low income countries -- many students who utilize tools like the Khan Academy to advance through a given curriculum based on demonstrating various competencies and content mastery before moving on to the next topic may in the end be tested on material that they have never seen before. How can education systems which rely on standardized summative assessments utilize tools which promote competency-based advancement through the curriculum?
 

9. Who's teaching -- and how -- and does it make a difference?
The Matthew Effect in Educational Technology might not only be relevant to discussions about the use of technology tools and devices by students, but by teachers as well. What if the Khan Academy is meant to be used in classrooms with 'poor' teachers, but it is in classrooms overseen by the 'best' teachers that the use of the Khan Academy has the greatest impact? In other words: What do you do if the use of the Khan Academy is explicitly meant to address issues of low teacher quality, but it turns out that teacher quality is actually an important ingredient into the successful use of such tools and resources? The EDC study on the use of the Khan Academy in a few schools in Chile notes that such use "changes the ways and the degree to which students engage with and are engaged by the math content; it also changes the way teachers and students interact with each other." What if educational policymakers and educators -- and parents -- aren't comfortable with such changes?

10. ____
As with other 'lists of ten' that appear occasionally on the EduTech blog, #10 here is left intentionally blank, as an acknowledgement that there are many more useful questions which can be asked (including presumably some that we haven't yet been able to identify and formulate, but which will emerge as people learn more practical lessons as the result of implementing resources like the Khan Academy in learning environments in middle and low income countries). In my experience, posing the sorts of questions presented above typically leads to additional questions, along the way helping to bring to the surface issues that educational policymakers, planners and educators may not have anticipated. Feel free to raise other questions you might have as a result of reading the studies from SRI and EDC in the comments section below, using the email tool at the right side of this web page, or on Twitter (where we can be found @WBedutech; I'm at @trucano). Even if we don't have all the answers to how we will get to where we want to go, asking better questions will make it more likely that we are headed in the right direction.

Note: The image used at the top of this blog post of two middle school boys comparing what they have found ("at first glance things look the same, but upon further reflection there are some potentially important differences to consider") comes from Flickr user VA State Park Staff via Wikimedia Commons. The image was initially uploaded to Flickr and is used according to the terms of its Creative Commons Attribution 2.0 Generic (CC BY 2.0) license.