Contextualizing lessons from the use of the Khan Academy

|

This page in:

at first glance things look the same, but upon further reflection there are some potentially important differences to consider
at first glance things look the same,
but upon further reflection there are some
potentially important differences to consider

Based on my interactions with educational policymakers, and those who advise them, it seems to be a truth (almost) universally acknowledged that there is insufficient research into models of educational technology use, the impact of such use, and related costs. This is not to say that there is *no* useful research into the use of technology in education around the world, of course. Online tools like ERIC and SSRN can help you find some useful studies; the popular press and the blogosphere increasingly reference such work (sometimes even in ways where you can actually track down the referenced studies!); there are of course a lot of academic, industry and professional journals dedicated to the subject; and a healthy amount of 'grey' literature circulates informally (including stuff commissioned by companies that is never formally published). Firms also circulate 'white papers' touting the 'impact' of their products and services, something which I tend to place into its own separate category, given the commercial and marketing imperatives that often animate such work. That said, just because a lot of 'research' is produced doesn't mean that such research is helpful to meet the practical information demands of educational policymakers, planners and educators.

Even if you *are* of the opinion that there is indeed a lot of useful, policy- and practice-relevant research out there related to the use of technologies in education, the fact remains that most of our collective knowledgebase has been constructed as a result of studying and attempting to learn from experiences in 'highly developed' (OECD) countries. While there is always danger when trying to draw generalized lessons from a research study that examines a specific context, it would seem reasonable to me that the difficulties when looking to draw lessons from experiences in Quebec that might be relevant to Kansas or Canberra pale in comparison to those when trying to extend such lessons to policymakers making decisions which will affect students and teachers in places like Quito or Kampala -- let alone rural Cambodia.

Thankfully, there are a number of promising moves afoot which hope to direct more energy and resources to investigate issues and circumstances of relevance to those exploring the use of ICTs in education in middle and low income learning environments and contexts around the world. (As such efforts kick off, and especially as related studies emerge, we would hope to feature them on the EduTech blog.) Until we start to see results from these sorts of efforts, however, the practical reality is that, in most cases, policymakers in middle and low income countries who wish to draw lessons 'from the research' in order to inform their policymaking related to potential educational technology initiatives will continue to try to contextualize results from research in higher income countries in the attempt to divine what lessons (if any) might be relevant to their own circumstances, even in places where contexts for use and typical use cases may be quite different.

---

A previous EduTech blog post, "Evaluating the Khan Academy", explores some of the lessons that have emerged as a result of research by SRI into the use of Khan Academy in a number of schools in Northern California. For those who don't know it: The Khan Academy is a widely known and much celebrated educational website which features thousands of short video tutorials on educational topics, as well as linked sets of over 100,000 practice problems and a 'personal learning dashboard'. Policymakers from a number of countries have approached the World Bank for specific advice and guidance on how they might make use of Khan Academy resources within their schools, and during the course of related conversations (some of which were catalyzed by a talk given by SRI's Robert Murphy at the World Bank earlier this year, which was discussed in a subsequent blog post) we have passed along and discussed the Research on the Use of Khan Academy in Schools published by SRI -- as well as a report that appeared a few months ago from EDC that looked at the use of Khan Academy in Chile. While individual lessons and insights drawn from an analysis of the use of Khan Academy content at sites examined in the SRI and EDC reports may or may not be more broadly generalizable to other contexts, examining the various usage models documented and explored in those studies may help raise *questions* that might be relevant to educators and policymakers who work in other contexts.

Indeed:

What are some useful questions that policymakers in middle and low income countries might ask as they attempt to contextualize insights from the use of the Khan Academy in California and Chile as part of their efforts to investigate and plan for the use of digital learning resources from the Khan Academy (and from other sources as well) within their own education systems?


For what it's worth, and in case presenting them here might be helpful to groups working on such efforts, here are some of the related questions that have surfaced during discussions of the SRI and EDC reports in which I have been involved. This list is not meant to be exhaustive or comprehensive -- and indeed there may well be other, more important, and even more fundamental, questions to ask in many places.

The intention here is not to imply that the Khan Academy is the 'right' or indeed 'best' choice for a given education system to consider (Khan Academy is one of the most prominent of untold numbers of digital learning tools, products and services of potential relevance to education systems around the world – and of course there are lots of other ‘non-digital’ ways in which countries can spend scarce resources dedicated to teaching and learning!), but rather to enrich a number of ongoing discussions about the potential use of Khan Academy (and similar) content in education systems in a number of countries.

Ten (sets of) questions

1. Variety of usage models
One thing that is very clear from the SRI report is that the Khan Academy is being used in many different ways in different schools -- and sometimes even in different ways within the same school. The SRI study documents Khan Academy being used as a tool for additional practice; as remediation for students who are behind and as enrichment for those who are highly motivated or advanced; as well as to monitor student learning. Such uses are of course quite relevant and useful in schools and educational contexts in low and middle income countries as well. This flexibility in usage models is potentially quite valuable -- especially in schools and classrooms where you have highly capable, experienced teachers who can find the right usage model that works well with their students. That said, is this flexibility a good or bad thing in 'low capacity' education systems where teacher capacity and quality is a major concern, and where many educational planners experience challenges as they plan for many of the 'basics' (getting textbooks to schools, ensuring the correct number of desks, making sure that teachers are at work, etc.)?
 

2. Mechanical processes and procedures
One persistent criticism of the Khan Academy from some quarters is that it focuses on the mastery of various mechanical processes and procedures (e.g. factoring numbers, long division) at the expensive of developing an understanding of higher order concepts. Whether or not one agrees with such criticism, many policymakers in middle and low income may (rightly or wrongly) wonder about the relevance of such criticism to their own immediate circumstances. (You can find many related discussions quite easily on the Internet if you're interested in them -- some of them thoughtful, well informed and useful; some rather breathless; some quite ideological; still others vituperative and snarky -- so I'll not attempt to address and consider them here.) For some education officials, it is precisely this emphasis on process and procedure that they find most alluring about the Khan Academy. Other policymakers may concede that a focus on the development of the mastery of various 'mechanical' processes is only one piece of a larger puzzle (and perhaps not even the most important piece), but look at the situations in classrooms in their countries' schools today, where very little learning is taking place and students (and indeed many teachers) struggle with things at very basic level, and wonder: In our circumstances, wouldn't helping our students with the development of some very basic mechanical processes and procedures be a step in the right direction -- and leave us in a much better shape than where we are today?
 

Some low income countries are considering using the Khan Academy not as a tool for students, but rather to help teachers learn the content they are supposed to already know – and teach.


3. Teacher use of student learning data
Khan Academy has developed -- and continues to develop, at great expense -- an impressive set of tools so that teachers and students can monitor student progress through Khan Academy content. Despite the availability of easy-to-use 'dashboards' that can help teachers monitor student progress, however, the SRI study did not seem to find that a lot of teachers were actually using them. No matter how potentially useful they may be, should we expect that teachers in low capacity education systems will use such tools, especially given that access to, and familiarity with, digital technologies may be quite limited? Or: Will teachers in such circumstances perhaps actually use these sorts of monitoring tools *more*, given that they don't have other sources of formative assessment data?


4. Some key facilitating factors: Access to computers, longer class periods
The SRI reports note a number of key factors which seem to have facilitated the use of Khan Academy in the schools studied. Notable among these include the existence of 1-to-1 computing efforts in some schools, which helped facilitate access to the website and content, as well as the fact that some schools were able to utilize longer, extended blocks of time during the schedule of the school day. In most middle and low income countries, efforts to provide broad access to educational computing devices (especially those which are meant to enable so-called '1-to-1 computing', where each student has her own device) are still in their infancy. Reconfiguring the school day to be able to dedicate longer blocks of time dedicated to specific subjects in order to better accommodate the use of related Khan Academy content may simply not be practical in many educational settings in middle and low income countries -- especially those which are notably rigid in their approach to scheduling or where large numbers of students and limited numbers of classrooms made such changes very, very difficult. Absent the widespread availability of computers within schools for student use, and the potential to lengthen class periods so as to be able to make better use of computer resources to access Khan Academy content, what other facilitating factors might education systems wish to explore in order to ensure sufficient access to the Khan Academy?
 

A few years ago some educators created and posted to YouTube a combination parody/critique of a Khan Academy tutorial that looked at multiplying and dividing integers. The response to this parody video, which went vital in certain educational circles in the United States, was very interesting to follow. However one feels about this parody, and the response to it, there is no denying that it helped catalyze some useful and valuable discussions of pedagogical approaches to teaching certain concepts. It is also perhaps interesting to note that, in response to the video and some of the related discussion, Khan Academy actually modified, and improved, the original video. Given enough eyeballs, all bugs are shallow, as they like to say in Silicon Valley. When I used to teach, I don't recall ever getting this sort of feedback on anything I ever taught, or the way I taught it.


5. Support structures for learning
The SRI study also makes note of a number of other important 'support structures for learning' that exist in the schools in Northern California that were studied. As difficult and challenged as the learning environments in under-resourced schools in poor communities in the United States may be, students in those schools still have access to educational content from a variety of sources (including textbooks), to teachers (who, no matter how well or poorly qualified, due show up to school in sufficient numbers, unlike in many other parts of the world), as well as to peers and parents who may be tapped to help explain and discuss various concepts being studied. The use, and utility, of the Khan Academy in such contexts may be quite different than in environments in low income countries where such support structures for learning may be much weaker, and in some cases may not exist at all. In environments where the Khan Academy is meant to be used in large part because of the absence of such support structures for learning, what steps might educational policymakers and educators consider in order to make the 'successful' use of the Khan Academy more likely? More broadly: What are the important things that need to be in place (human resources, technology, incentive structures, pedagogical practices, physical space, etc.) for tools like the Khan Academy to be introduced most productively, with the greatest impact?


6. The videos, and 'flipping the classroom'
The Khan Academy is held out as a prominent example of 'flipping the classroom', where students listen to lectures outside class (e.g. by viewing Khan Academy videos at home) and then do their 'homework' when in class, aided by their teachers (and their peers). Despite many assumptions that this is how Khan Academy is actually utilized within formal educational settings, the SRI study did not see much of this happening in practice, for a variety of reasons. Indeed, to a large extent teachers appeared to want to continue to teach as they had already done, and largely conceptualized the use of the Khan Academy as a complement to their traditional teaching methods. In other words: the Khan Academy video lectures were not used to replace the lectures of teachers, but were instead positioned as resources for students to consult and review after the traditional lecture was delivered. Is the flipped classroom model relevant in middle and low income countries, given existing pedagogical practices and norms, especially in places where access to connectivity and computers outside of schools is low?
 

In some important ways, the Khan Academy videos make transparent what was previously hidden. How did your child's teacher conduct today's lesson about, say, fractions? Unless you were physically in the classroom, it is rather difficult (if not impossible) to know. If you have access to the Internet, however, you can see exactly how this was addressed in the related Khan Academy video. (And: If you missed part of it or didn't understand something, no worries: Just re-wind and watch it again.) While Khan Academy content and pedagogical approaches may come in for legitimate criticism, is this compared to an ideal, or actual practice? We should perhaps be careful where we set the bar, and note that in different contexts, different expectations may be in order, at least in the near term. This is not to argue for mediocrity or mistakes, but rather to note the potential utility in critically analyzing things in comparison to not only what is considered *ideal* practice, but also to what is currently happening in classrooms around the world, especially in many of the ones that reasonable people would consider to be 'low performing'. This doesn't mean that the bar for analysis should be set low, but rather to observe that the bar may be set at different levels in different places right now, and what's immediately important for educational policymakers in many places is that approaches be found to raise the bar in ways that are useful and achievable -- even if they appear modest by the standards of 'highly developed' countries -- as part of a longer-term process of raising it to where it should eventually be placed.

7. Planning for content that changes
For better or worse, the content in printed textbooks does not change very often. Indeed, in many countries the same textbooks may be used for many years and are updated very infrequently (they may be in critically short supply as well). One potential advantage of digital learning resources, as made available through efforts like the Khan Academy, is that they can be updated much more frequently than printed materials. That said, in places without reliable connectivity, such advantages may be more theoretical than practical. In environments where connectivity is less of a challenge, teachers and education systems may be challenged to adapt to the emergence of new digital learning content that they have not already been exposed to, have not vetted and which has not been included as part of official teacher professional development and training programs. This is especially true in 'low capacity' education systems where teachers are inexperienced, poorly trained and often struggle with mastery of the content they are supposed to deliver. How can education systems and educators plan for and take advantage of updates to digital learning content at a pace and schedule which is much faster than that which previously marked their use of printed textbooks?


8. Competency-based advancement and standardized tests
One virtue identified by many educational policymakers of tools like the Khan Academy is that they enable competency-based advancement, whereby students continue to study a particular concept or topic until they demonstrate a certain level of related mastery. This makes a lot of sense, especially in subjects where concepts explicitly build upon each other, as is quite often the case in mathematics. If you can't multiply single digit integers this month, you're probably going to have a tough time multiply two and three digit integers when you are asked to do so next month. Why move on to multiplying 55X77 when you haven't demonstrated that you can successfully multiply 5x7? In most classrooms, students move in lock step through the curriculum, with many students frustrated/bored because things are moving too fast, and many other students bored/frustrated because things are moving too slow. That said, in education systems that rely on high stakes end-of-year exams that cover the entirety of a given curriculum -- which is the reality in most middle and low income countries -- many students who utilize tools like the Khan Academy to advance through a given curriculum based on demonstrating various competencies and content mastery before moving on to the next topic may in the end be tested on material that they have never seen before. How can education systems which rely on standardized summative assessments utilize tools which promote competency-based advancement through the curriculum?
 

I once helped organize a study tour to an Asian country for a number of Western academics, including some well-known experts on the use of educational technologies. Many of these folks were rather appalled at the uses of technologies they observed in a few schools in a country which has been lauded for the strength of its education system. One such expert expressed to me her exasperation that technologies were not being used in ways that were 'transformative', in support of 'modern' pedagogical approaches, etc. Compared with schools in your country you may well be right, I responded, but I am not sure if that comparison is the operative one for many people here. Go to the classroom just down the hall where students and teachers are not using any technology at all and look at the pedagogical practices in evidence there. That may also be a useful point of comparison.


9. Who's teaching -- and how -- and does it make a difference?
The Matthew Effect in Educational Technology might not only be relevant to discussions about the use of technology tools and devices by students, but by teachers as well. What if the Khan Academy is meant to be used in classrooms with 'poor' teachers, but it is in classrooms overseen by the 'best' teachers that the use of the Khan Academy has the greatest impact? In other words: What do you do if the use of the Khan Academy is explicitly meant to address issues of low teacher quality, but it turns out that teacher quality is actually an important ingredient into the successful use of such tools and resources? The EDC study on the use of the Khan Academy in a few schools in Chile notes that such use "changes the ways and the degree to which students engage with and are engaged by the math content; it also changes the way teachers and students interact with each other." What if educational policymakers and educators -- and parents -- aren't comfortable with such changes?


10. ____
As with other 'lists of ten' that appear occasionally on the EduTech blog, #10 here is left intentionally blank, as an acknowledgement that there are many more useful questions which can be asked (including presumably some that we haven't yet been able to identify and formulate, but which will emerge as people learn more practical lessons as the result of implementing resources like the Khan Academy in learning environments in middle and low income countries). In my experience, posing the sorts of questions presented above typically leads to additional questions, along the way helping to bring to the surface issues that educational policymakers, planners and educators may not have anticipated. Feel free to raise other questions you might have as a result of reading the studies from SRI and EDC in the comments section below, using the email tool at the right side of this web page, or on Twitter (where we can be found @WBedutech; I'm at @trucano). Even if we don't have all the answers to how we will get to where we want to go, asking better questions will make it more likely that we are headed in the right direction.

Note: The image used at the top of this blog post of two middle school boys comparing what they have found ("at first glance things look the same, but upon further reflection there are some potentially important differences to consider") comes from Flickr user VA State Park Staff via Wikimedia Commons. The image was initially uploaded to Flickr and is used according to the terms of its Creative Commons Attribution 2.0 Generic (CC BY 2.0) license.

Authors

Michael Trucano

Global Lead for Innovation in Education, Sr. Education & Technology Policy Specialist

Bornwell Mwewa
February 08, 2015

I completely disagree with your opening line on contextualising the research on the value of ICT in education from high income industrialised countries to low income states. The impact of the context in such studies is historically zero.
I will use the case of Zambia and Africa in general. The Zambian education context we are told is markedly different from the British context. This is a falsehood because everything else in the Zambian education is based on the british education system including language and content. Until recently even the matriculation exam for all Zambians was prepared by the Cambridge University. The current standards and requirements for students in the Zambian education system are matched closely to those of the British. The British student with whom the Zambian is expected to compete has access to technology which by the way has progressively been implemented with no researched evidence because none existed. Meanwhile the Zambian student is told she cannot have these technologies because there is no proven STUDY!
Where did Britain get proven study on the efficacy of Computers in the classroom in 1997 when almost all British schools started using computers in the classroom? There was probably a short pilot study and quick roll out. Inumerous pilot studies have been carried out on the value of ICT in Education in Zambia and in Africa but we still hear the story of caution against using computers in Africa schools.
The discussion, I would like to propose, should be on logistical challenges in implementing ICT in Education. We must move ahead
Bornwell Mwewa
CEO
Camara Zambia

Michael Trucano
February 24, 2015

Hi Bornwell,
Thanks a lot for your comment
Just because other places decided to invest lots of money in edtech roll-outs without trying to learn from the mistakes and successes of others (or indeed from their own emerging failures and successes) doesn't make such a thing 'good practice'.
So I guess we'll have to agree to disagree here -- or maybe we are just talking about slightly different things.
In my experience, few decisions about whether or not to invest in ICTs in education systems are really evidence-based. When people cite the evidence, or lack of it, they often do so to justify a decision that they have already made for other reasons.
As you note, many 'developed' countries have invested large amounts of money in rolling out ICTs into their education system despite the fact that there was scant 'evidence' about the potential impact of what they were intending to to.
This is precisely why I spoke in the first sentence above not only about the impact, but about usage models and costing exercises as well. Frankly, we don't have good research about this stuff. Costing studies are almost non-existent -- especially in real-life contexts in developing country environemtns (whether in Africa or elsewhere).  Usage models are poorly documented -- and rarely 'tested' or compared with other potential usage models.
But I think it is also important to note that impacts can be quite different as well. Investing in more computers in education systems where there are lots of computers already -- both in schools, as well as in larger society -- is frankly not the same as introducing ICTs into places where such tools are scarce. In fact, one can construct compelling hypotheses that posit that the potential impact of technology use can be even greater, *in some circumstances*, in places where needs and contexts are different because they are more acute.
This is, however, a hypothesis, because we don't actually have much useful practical data about this stuff to help guide our investments and activities. There are *huge* projects underway in places around the world where there are pretty much no on-going research efforts designed to help guide the directions of the projects based on what is working, and what is not. In my experience, the most impactful efforts are a result of 'iterating your way to success', rather than a result of some grand, perfect plan conceived beforehand. Applied research can help establish critical feedback loops -- and often the most relevant research is one based on your own contexts and user needs, not those of people and environments halfway around the world.
And:
I fully agree that we should move beyond a simple, existential "should we?" to a more practical "how to?" when discussing the use of ICTs in education. As a practical matter, more and more places are simply deciding "we should" and moving forward, but don't have much useful rigorous evidence that they can use to help guide their activities. Perhaps a rigorous discussion of 'how to?" may over time lead folks to question "should we?" as well -- but whatever the case, I hardly think that proceeding blindly forward simply because others have done so is a terribly good idea. And, as countries in Africa move forward with plans to introduce large numbers of ICTs into their education systems, it would, I think, be rather useful to have a research base, upon which they can draw as they make decisions, that is derived not only from what is happening in North America and Northern Europe, but from contexts more like their own as well. Until then, policymakers and implementers will have to continue to attempt to translate and contextualize lessons from other places and hope (fingers corssed!) that their processes of translation and contextualization are on the mark.
-M