So you’ve designed an awesome impact evaluation, you’ve carried out a rich baseline survey, you’ve presented the baseline results to the government of Brigadoon, and now you….wait two years until the follow-up survey? What else can you do with this baseline data? You can do a lot! You can write a report, you can write a brief, you can publish papers, you can test targeting strategies, and you can even [drumroll] affect policy.
External validity is a recurring concern in impact evaluation: How applicable is what I learn in Benin or in Pakistan to some other country? There are a host of important technical issues around external validity, but at some level, policy makers and technocrats in Country A examine the evidence from Country B and think about how likely it is to apply in Country A. But how likely are they to consider the evidence from Country B in the first place?
As efforts to distribute low cost computing devices and connectivity to schools pick up steam in developing countries around the world, many ministries of education are systematically thinking about the large scale use of digital educational content for the first time. Given that many countries have already spent, are spending, or soon plan to spend large amounts of money on computer hardware, they are often less willing or able to consider large scale purchases of digital learning materials -- at least until they get a better handle on what works, what doesn't and what they really need. In some cases this phenomenon is consistent with one of the ten 'worst practices' in ICT use in education which have been previously discussed on the EduTech blog: "Think about educational content only after you have rolled out your hardware". Whether or not considerations of digital learning materials are happening 'too early' or 'too late', it is of course encouraging that they are now happening within many ministries of education.
As arguably the world's highest profile digital educational content offering in the world -- and free at that! -- with materials in scores of languages, it is perhaps not surprising that many ministries of education are proposing to use Khan Academy content in their schools.
The promise and potential for using materials from Khan Academy (and other groups as well) is often pretty clear. Less is known about the actual practice of using digital educational content in schools in middle and low income countries in systematic ways.
In the course of my job at the World Bank helping ministries of education in middle and low income countries think about, analyze, plan for, implement and evaluate issues, ideas and projects at the intersection of the worlds of technology and education, I spend a fair amount of time considering issues related to the digital publishing of educational materials. The World Bank has over the years funded the purchase of lots of textbooks around the world and we maintain active dialogues with scores of education ministries, helping to provide related advice and technical assistance.
In many countries, especially poor ones where disposable income is very low and where there is not an established culture of leisure reading, the educational publishing industry is sometimes largely indistinguishable from the publishing industry as a whole, and government purchases of textbooks (and/or government directives about which textbooks families or schools should purchase, where such things are not centrally procured) have huge, often determining influence on the direction of the publishing industry in general. In order to better understand how all of this impacts educational publishing, I talk regularly with lots of 'traditional' educational publishers, big and small, both international and local. I also talk a lot with technology companies who do things that look a lot like educational publishing to me, or who provide the tools and services to enable and support related activities.
Last year I participated in two fascinating events a few weeks apart, the EdTech Industry Summit in San Francisco and a symposium convened by the International Publishers Association (IPA) at the London Book Fair. (I was lucky enough to be at the IPA symposium this year as well.) At these meetings, the agendas and items discussed were largely the same, but were often approached from quite different perspectives. For the sake of argument here -- and I admit I am greatly oversimplifying things by making this characterization -- EdTech Summit participants were mainly 'tech companies', while the London Book Fair event was mainly attended by 'traditional publishers'. (I concede that such distinctions are increasingly difficult, and less useful, to make as time goes on; in my opinion all publishers are technology companies these days, whether they self-identify as such or not and/or whether outsiders see them that way.)
At both events, a data point that was quoted quite often was that '1% of national education budgets around the world are devoted to the purchasing of textbooks and other learning materials'. While I have never been able to find this assertion supported by hard data, I have heard it expressed so many times over the years by people who work in or around the educational publishing industry that I have taken it as almost 'conventional wisdom'. Whether this figure is actually .5% or 2% or 4% (or whatever), what has struck me when in conversation with many vendors is that many traditional publishers have, in the face of the steady rise in many countries around the world of large scale purchases of laptops and tablets for students and teachers, worried that technology purchases are eating into traditional budgets for the purchases of textbooks.
As one traditional publisher put it to me at the London event in 2013, "we need to figure out how to protect this 1% so that it is not tapped to buy iPads". Contrast this with a statement made to me by an enthusiastic founder of an edtech start-up in San Francisco, who said that the goal of firms like his was to "eventually capture 20% of education budgets" by transforming the way education is delivered as a result of the use of new technologies. Whether or not such figures are accurate, they for me exemplify a difference in perspective and ambition that is consistent with many stereotypical characterizations of brash young tech entrepreneurs in their hoodies (and/or khakis) versus the tweedy old-school booksellers whose business model that I have been told on many occasions -- especially by those not in that business -- was one for "dinosaurs".
(While conceding that the business models for selling books will have to change rather radically going forward, a concession to which no educational publisher I know would object in the age of e-readers and hypertext, of apps and APIs, I am fairly confident that extinction rates for edtech startups will remain much greater than that of book publishers for the foreseeable future, although in the end I wouldn't be too surprised if the most successful 'tech' firms doing business in this space end up buying up a lot of the 'publishers' -- some of whom will themselves be buying and merging with 'tech' firms along the way.)
If we accept the premise that educational publishing industry in the 'developed' countries of Europe and North America are being, and will continue to be, increasingly radically disrupted -- a contention with which I expect few people would disagree -- what might this mean for business models for educational publishing in less developed, 'poor' countries? Will the related business models from OECD contexts simply, and eventually, be transplanted to middle and low income countries? Or: Might some new business models for digital educational publishing emerge from less developed countries, based on specific local contexts and consumer demands in an increasingly digital -- and mobile -- age?
In response to a recent EduTech blog post on “the 'ideal’ educational technology devices for developing countries”, I received numerous responses that effectively said: “We already know what this ideal device is: the mobile phone”. While the use of mobile phones in education is a regular topic explored on this blog, and the mobile phone is a device that I regularly recommend that ministries of education consider when planning for technology use in schools more than they currently do (in my experience few education authorities do consider utilizing phones as tools for learning in any real way), I would not go so far as to say that it is the ‘ideal’ device for use in educational settings in developing countries. Context is always king.
It may be true that, in many cases, the ‘best device is the one you already have, know how to use and can afford’. In some contexts, mobile phones conform to this definition quite well (although many school systems around the world do continue to ban or severely limit their use on school property). Depending on the context and usage scenario, others do too, including the two that I used to compose the first draft of this blog post: a ballpoint pen and a notepad (the old fashioned kind with actual paper, not the one that comes bundled with Microsoft Windows).
Because I often prominently highlight the potential of mobile phones to be used in educational contexts in developing countries in the course of my work at the World Bank, I am often asked for specific examples of this use. Here’s a rather interesting one that you may not have heard much about:
This list certainly isn't comprehensive. As with all posts on the EduTech blog, the standard disclaimers should apply (e.g. these are the views of the author and do not necessarily represent official views of the World Bank, etc.). It is perhaps worth noting that these sorts of suggestions are typically made and discussed within a specific context: A country has decided, for better or for worse, that it will consider significant new investments in digital teaching and learning materials. With this decision already made, policymakers are looking for some additional perspectives and inputs to help guide their thinking as they move forward.
In other words: These sorts of recommendations typically are not meant to inform higher level discussions about fundamental strategic priorities in the education sector (although, where they may help trigger reconsideration of some broader decisions made at higher levels, that may not always be such a bad thing). They are not meant to help, for example, policymakers assess whether or not to spend money on digital textbooks versus buying related hardware, let alone whether or not investments in digital learning resources should be made instead of spending money on things like school feeding programs, improvements in instruction at teacher training colleges, or hiring more teachers. Rather, they are more along the lines of:
Here is some potential food for thought.
With that context and those caveats in place, here are ten general recommendations that education officials contemplating the use of digital teaching and learning materials at scale across a country’s education system may wish to consider during their related planning processes:
Across Africa, a variety of devices are increasingly being used to disseminate and display teaching and learning materials in electronic and digital formats. As costs for such devices continue to fall, and as the devices themselves become more widely available and used across communities, the small pilot, and largely NGO-led, projects that have characterized most efforts to introduce educational technologies in schools across Africa will inevitably be complemented, and in many cases superseded, by large-scale national initiatives of the sorts now taking place in Rwanda and Kenya, where hundreds of thousands of devices are being, or will soon be, distributed to schools.
Few would argue that the use of such devices do not offer great promise and potential to improve the access to and quality of education by providing access to more educational content than is currently available inside and outside of schools. Internet connectivity can provide access to millions of educational materials available on the Internet; low cost, handheld e-reading devices can hold more than a thousand books. Depending on the availability of connectivity, or local resourcefulness in transferring materials to devices manually, digital content used in schools can be updated more regularly than is possible with printed materials. Depending on the device utilized, this content can be presented as ‘rich media’, with audio, video and animations helping content be displayed in ways that are engaging and interactive. It is possible to track electronically how such content is used, and, depending on the technologies employed, to present content to teachers and learners in personalized ways. In some cases , this content can be delivered at lower costs than those incurred when providing traditional printed materials.
Given the increased availability and diffusion of consumer computing technologies across much of the continent in less than a decade, it is perhaps not surprising that a number of widespread misconceptions about the promise and potential of using digital technologies and devices across Africa to increase access to learning materials appear to have taken hold. On one level, this is consistent with the ‘hype cycle’ model of technology diffusion in which, according to Gartner, a technology breakthrough is soon followed by a period of time of “inflated expectations” about what sort of changes might be possible as a result.
A few countries across Africa are considering rather ambitious initiatives to roll out and utilize digital textbooks, a general catch-all term or metaphor which I understand in many circumstances to be ‘teaching and learning resources and materials presented in electronic and digital formats’.
How much will such initiatives cost?
Reflexively, some ministries of education (and donors!) may think this is a pretty straightforward question to answer. After all, they have been buying textbooks in printed formats for a long time, they have a good handle on what such materials traditional cost, and so they may naturally presume that they can think about the costs of ‘digital textbooks’ in pretty similar ways.
Many people are surprised to discover that calculating costs associated with the introduction and use of digital teaching and learning materials is often a non-trivial endeavor. At a basic level, how much an education system spends will depend on what it intends to do, its current capacity to support such use – and of course what it can afford. As they investigate matters more deeply (and sit through many presentations from publishers and other vendors, sometimes wowed at what is now possible and available while at the same time rather confused about what is now possible and available), education officials seeking to acquire digital teaching and learning materials for use at scale across an education system may find costing exercises to be, in reality, rather challenging and (surprisingly) complex when compared to their ‘standard’ textbook procurement practices.
In the decade since the term 'open education resources' was formally identified and adopted by UNESCO, related "teaching, learning or research materials that are in the public domain or released with an intellectual property license that allows for free use, adaptation, and distribution" have been slowly but surely creeping into mainstream use in many education systems around the world. North America has recently seen prominent announcements about projects to provide free, online open textbooks in British Columbia and California, following similar sorts of headlines out of Poland earlier in the year. In June, the so-called 'Paris Declaration' [pdf] was released as part of a prominent international effort both to "increase government understanding of the significance of open education resources and to encourage more governments to support the principle that the products of publicly funded work should carry such licenses." In conversations with education ministries in many low and middle income countries over the past year, I have seen a marked increase in the interest in exploring the relevance of the 'OER movement' to national efforts to procure and develop digital learning resources. Traditional educational publishers have been monitoring such efforts closely, identifying both potential threats to existing business models, and in some cases, ingredients for potentially new business models as well.
How might we be able to track related initiatives around the world?
The World Bank is currently working with a few countries that are planning for the procurement of lots of digital learning materials. In some cases, these are billed as 'e-textbooks', replacing in part existing paper-based materials; in other cases, these are meant to complement existing curricular materials. In pretty much all cases, this is happening as a result of past, on-going or upcoming large scale procurements of lots of ICT equipment. Once you have your schools connected and lots of devices (PCs, laptops, tablets) in the hands of teachers and students, it can be rather useful to have educational content that runs on whatever gadgets you have introduced into to help aid and support teaching and learning. In this regard, we have been pleased to note fewer countries pursuing one of the prominent worst practices in ICT use in education that we identified a few years ago: "Think about educational content only after you have rolled out your hardware."
At least initially, many education authorities in middle- and low-income countries seem to approach the large-scale procurement of digital learning materials in much the same way that they viewed purchases of textbooks in the past. On its face, this is quite natural: If you have tried and tested systems in place to buy textbooks, why not use them to buy 'e-textbooks' as well? (We'll leave aside for a moment questions about whether such systems to procure textbooks actually worked well -- that's another discussion!) As with many things that have to do with technology in some way, things become a little more complicated the more experience you have wrestling with them.