National online testing as an indicator for edtech readiness?


This page in:

there are so many ingredients ... what if I could only choose one?
there are so many ingredients ...
what if I could only choose one?

A recent post on the EduTech blog explored the potential usefulness of an 'edtech readiness index', a new tool to help inform conversations and decisions currently occurring in many ministries of education around the world about how to best (most impactfully, most equitably, most strategically) roll-out and utilize new technologies across an education system.

When creating an index of most any sort, there is a natural tension regarding how many indicators or types of data to include. Chose too few, and you're likely to exclude a lot of important stuff. The more you include, however, the more complicated things get. This is especially true if there are few or no existing sources of the data you would like to utilize, and so you need to build related data collection mechanisms from scratch. (When it comes to edtech-related topics, there is very little rigorously collected, globally comparable data available in most countries. That's the stark reality.) 

If I were forced to choose one indicator that would suggest to me that an education system was equipped to utilize educational technologies at scale to (as the slogan often proclaims) 'transform teaching and learning', it would be whether or not the country implements national online testing of some sort.

Let me be clear: I am not arguing that implementing national online testing should be a goal for an education system, nor that it should be a priority, nor that it is even useful. Whether or not a tool or approach 'works' in a particular environment will depend on a lot of factors, a good number of them specific to a given operating context. I leave it to others to enter into and comment on related debates, which can quickly become ideological and polarized, in other forums. (Those interested and willing to do so have no shortage of opportunities and outlets elsewhere on the Internet to do so, whether they are looking to have their own perspectives validated, pick a fight with someone with whom they disagree, or engage in conversation that falls somewhere in between.)

Rather, I am observing that, in order for national online testing to occur, a lot of things need to be in place, and working, and that such things can provide the foundation upon which other large scale edtech projects can be built.

When I speak with big companies that sell edtech-related products and services who are exploring moving into new markets, they often tell me that they closely monitor which countries are making the move to online testing. I am not talking about the (small handful of international) companies in the 'assessments business' -- of course they track such stuff. Rather, I am talking about most everyone else. I have heard countless tech firms remark that, 'If a country is implementing online testing at a national scale, then we know they are serious about edtech. Once this particular genie is out of the bottle, it can be difficult to stuff it back in.' (Usually such a pronouncement is accompanied by a rather large smile.)

In order to institute online testing, a country needs to buy lots of hardware. It needs to improve connectivity. It needs a lot more than just this, however. Lots of countries already buy lots of hardware, spend lots on connectivity, and create and purchase digital learning resources of various types. Alongside such things, lots of money is often spent on complementary training and professional development (of teachers, of technical staff, of administrators -- and programs need to be put in place to ensure that students have a basic level of 'digital skills'). When this happens at a national scale, this often represents big business, and can potentially mean big opportunities for many businesses.

This isn't all that happens, though, for national online testing to take place:

  • Enabling policies, regulations and guidelines (and sometimes even laws) need to be formulated and/or updated -- and implemented.
  • National examination bodies usually need to be reformed, and the skills of their staff upgraded, so that they can successfully oversee, manage and advise on the details of implementing national online testing. Local academic and research communities need to be engaged to help with this effort.
  • Lots of different stakeholder groups need to be consulted and brought into the process. Cases need to be made, for example, to parents, to teachers, to civil service employees, as well as to other stakeholder and interest groups.
  • Within government itself, policymakers and technical staff need to be able to make informed decisions and to manage related budgeting and procurement processes (a scenario often complicated by the fact that they may be negotiating with large international testing companies that know a lot more about the topic than they do).
  • New approaches to data security and privacy often need to be formulated and implemented. (Testing them before they are put in place is usually a good idea as well.)
  • There are huge logistical challenges in pulling off national online testing -- in scheduling, in use of school facilities and infrastructure, in ensuring adequate connectivity and electricity, in providing local technical support, in preventing cheating, in identifying and correcting problems quickly, and in coordinating all of the multitude of moving parts in a way so that, at a given moment in time, everything is in sync.
  • (etc. etc. etc.)

Even if national online tests are not 'high stakes' for students, where the results determine whether a student passes on to the next level of schooling or teachers or administrators are given bonuses or fired (worthy topics for investigation and debate that I'll leave aside for the purposes of this essay), such an effort tends to be high stakes politically for government. And, at least in the short term -- disruptive for most everyone.

Let's be honest (and this is the dirty little secret of many educational technology initiatives in many countries, even quite high-profile ones):

In many schools in most countries, even where education systems have spent *lots* of money on tablets, connectivity, digital textbooks, and/or training, ensuring that everything 'works' is often not a terribly high priority. Yes, it would be great if students actually used the devices that were purchased for them. It'd be great if teachers put to use in the classroom some of what they learning in their technology training classes. It'd be nice if teachers and students actually accessed and used the national education portal put up at great expense by the education ministry or the digital content or applications that a local educational authority made available. But if none of this happens, life will go on, much as it had previously, for better or for worse. No doubt many people and groups will complain (about the expense, about wasted opportunities), and their criticisms will likely be quite valid, but such complaints and criticisms may be unlikely to incentivise practical action in a way that leads to real change. This is an unfortunate but observable reality around much of the world.

But if national online testing rolls out and it doesn't work -- that's another story! Students (and teachers, and school administrators) get demoralized. Parents and community groups mobilize. Vendors may be in breach of provisions in their contracts. Education and government officials face pressure in the media, and politically. As such, there are often big incentives in place to ensure that, when it comes to national online testing, things work.

And when they don't work -- and in practice they often don't work all that well the first time this sort of thing is attempted (and often not the second or fifth time either) -- there are usually compelling incentives in place for groups to work together to fix what's wrong, at least eventually.

When things (finally) do work, the preconditions are in place for an education system to do other things (perhaps more useful and impactful things) with educational technologies at scale.

Doing something at scale with technology in which *all* students and *all* schools participate is, as they like to say in Silicon Valley, non-trivial. National testing initiatives must, by their very nature, include *all students*. Unlike national edtech efforts that aim to connect all schools to the Internet but only end up connecting 95% or so (with the excluded 5% typically including schools in some of the poorest or most geographically isolated communities), national tests need to happen everywhere, and anything less than 100% coverage is a failure. This means that whatever approach or 'solution' is conceived and implemented needs to work for everyone -- not just for the kids in the nice schools in the big cities where edtech initiatives usually are first attempted and considered to be 'successful'.

That's why, when I'm asked about what's the one indicator I look for that might suggest whether a country is prepared to utilize educational technologies at scale across its entire education system to impact teaching and learning, I usually look to see if national online testing is occurring.

If it is not: Well, it's certainly possibly that an effort to introduce digital textbooks for use by every learner will be successful, or that a project to help schools offer more 'personalized learning' experiences to all of their students will meet its objectives. By no means does the existence of national online testing in a country ensure that all of the prerequisites to do something else with technology in education at scale are in place. But I expect that a lot of them might be.

When conceiving and embarking on something that's quite ambitious, the devil is often to be found in the details. When it comes to pulling off online testing at a national scale, there are lots of details to attend to, big and small. Get them right, and many of the ingredients and tools will be in place to cook up something else. That's no guarantee that the eventual result will be healthy, or applauded, or even edible or digestible, of course. No one said this stuff is easy. But pull this sort of thing off, and it's much more likely that you'll be able to succeed at whatever you would like to appear next on your plate. 

(At minimum, you'll at least have a better sense of what you might be getting yourself into!)

You may also be interested in the following posts on the EduTech blog

Note: The image used at the top of this blog post ("there are so many ingredients ... what if I could only choose one?") comes via Pixabay member fxxu and is used according to the related Pixabay license.


Michael Trucano

Global Lead for Innovation in Education, Sr. Education & Technology Policy Specialist

Ed Gaible
August 26, 2019

Hi again.
I'm very interested in the idea. And I'm much less tied in to national EdTech milieux in the donor, government and private sectors than you are. And, finally (kind of) I do understand that you are not advocating national online testing. The idea's use as an indicator, however, seems to call into question the concept of the readiness of complex systems and the use of indicators in relation to these. To wit: "To determine if a country is ready to deploy a bunch of EdTech to support systemic change, find countries that have already deployed a lot of EdTech." Granted, such countries might be ready on some level to change everything they are doing in education _except_ deploying a lot of EdTech. But it also seems that national online testing might be precisely an indicator of decisionmakers' commitments to the status quo, or even to increasing the efficiency to the status quo.

It's vital, IMHO, to recognize that the Global Education Industry (notably the big EdTech players) are looking at proper places to invest in education systems, not in systemic change of those systems; those interests are likely best supported by infrastructure, policy, hardware management, stakeholder involvement and--to cut to the chase--a rasher of debt or other obligation possibly related to procurement and deployment of hardware. But this last consideration is way above my paygrade. Clearly, they will look for 'stability,' which by definition undercuts any potential commitment to change (as well as entailing, at least potentially again, partnership with under-ethical governments.)

In general, the readiness index is best constructed as multivalent--comprised of many indicators--precisely because national actors will take different paths to education improvement. Singapore, for example, developed its world-leading education system largely before EdTech was an option; Finland, similarly in a leading position, has downplayed the use of technology heretofore, although that's clearly changing in response to many factors; the Republic of Korea, which rocks educationally, has as I understand it, achieved great success with the support of online content (and testing) and national programs that support use by all kids; Slovenia, with its world-leading commitment to openness, is walking yet a different path, one that presumes that the government will walk the walk fast enough and far enough to achieve stability while delivering enough improvement to retain the good wishes of the people it governs.

I know we agree that the goal is way other than the deployment of a lot of hardware, whether it's used to support online testing or education improvement. I discourage the use of that goal (which _is_ held as such by some) as an indicator.

August 26, 2019

Hi Ed, Thanks for your comments. There's a lot to unpack here! My post was meant to be deliberately provocative in a few ways (some of which you've hinted at), which is why I put it up. I get lots of great feedback that helps clarify my thinking, and teaches me things along the way, when I 'think aloud in public' like this.

The question of whether or not a country is 'ready' to support change in its approach to education is a much larger, more fundamental question, whether through the use of edtech or not (although of course technology is often considered to be integral to such efforts in many places, even if sometimes only rhetorically). It is of course highly plausible that some (many? most?) countries seek to (and do) utilize new technologies *not* to transform or reform their current system, as is so often the stated rationale, but rather to further entrench existing practices and players. I expect we both agree on this. (It is a truth often observed that 'progressive' approaches to pedagogy championed by many education reformers, including many very good teachers, stand in stark contrast to the inherent conservatism of the education system in which they operate.)

My intention in noting that big edtech players often use this as an indicator for where they may see business opportunities was meant to share this observation with groups for whom this is a surprise. (On numerous occasions I have mentioned this to donor groups and governments and it almost always came as a bit of a surprise to them. To me, this surprise was a useful and quite crude indicator of the understanding within some governments and donors of the motivations and perspectives of 'the private sector', with whom many of them were looking to 'partner'.) Perhaps the post would have been stronger had I removed this comment. (I must confess that my blogging skills are a bit rusty.)

In a subsequent blog post, I will brainstorm about some potential components of an 'edtech readiness index', and I will point to the blog post here to provide background context to my consideration of online testing in that post, to save myself (and the reader) from inserting what may be a lengthy and rather tedious aside into that post.

(And just to be clear, and to repeat a point I made in the post, and which you acknowledge in your comment: I am not advocating for national online testing. In fact, I am not commenting on that topic *at all* here. That's a separate discussion -- and one better led by folks much more expert and experienced in the topic than I am.)<br><br>

When it comes to 'edtech readiness' at a national scale: Whether it is to wonder aloud about the potential existence of a single indicator (which is, to be clear, preposterous on its face), or to speculate about what a collection of indicators might be (perhaps combined in some way into a related 'index'), the point is to help provoke related conversation and discussion. Doing so might help convince groups that expanding on and moving forward with related initiatives is a good idea -- or indeed a bad one. I've got no particular dog in such a fight, to quote a rather distasteful folk saying. But hopefully an open share of related views can help inform processes by which such decisions are currently being considered and made. To the extent that this post (and others) are useful in that regard, I'll consider them a success.

Alfred Okello
January 31, 2020

Thank you Michael. I can't agree with you anymore! National online testing is arguably the ultimate, as it speaks a lot about the extent to which everything, I mean literally everything to do with edtech is right, and in place. It is the confirmatory test for the success of technology integration in education across the board. Perhaps, as a measure, its only unfairness (or disadvantage) is that even if everything else is right, ranging from policies, institutional resources, leadership, hardware to software, and so on are in place, but if connectivity isn't, then the national online test can't be done, and the score is zero, thus a fiasco! In other words, everything must be right; it's some kind of "All or nothing " Law.......where it is either everything or nothing!
However, while I agree, leaving it as .....national online testing renders it (perhaps) too broad for accurate and consistent interpretation. I believe it can be disaggregated to measurable dimensions, like it's in the blog of "edtech readiness index" with skills, content, infrastucure, policy & institutions and many others....

Thank you.