“Test and punish”?
There’s a debate raging in American schools today: how (and how much) should children be tested?
The No Child Left Behind (NCLB) Act created a system where all children in all schools from grades 3 to 8 must be tested each year. Critics refer to this accountability architecture as “test and punish,” with stakes such as school funding (or closings!), bonuses for teachers, or grade promotion for students all riding on performance. There is evidence that NCLB improved learning outcomes, but improvements came at a high cost: In addition to teaching to the test, this approach can lead to a number of perverse incentives, like keeping weaker students at home on test day, narrowing the curriculum, or downright cheating. Worse, some have said they can serve to mask and contribute to the structural race and class inequalities in the United States.
“Test and punish”?
Kenyan schools are not doing well. About a 120 of them were set alight in arson attacks last year alone which were largely blamed on fears arising from a government crackdown on cheating in national exams. Amid national schooling reforms, many pupils and parents continue to be unhappy about the changes. Where do the teachers figure within this period of heavy reform?
Both the best and worst performers in East Africa are in Kenya
Although school enrolment has gone up steadily, over a million children are still out of school. In terms of learning outcomes, Kenya performs relatively better than its neighbours, but results from internationally recognised competency test, Uwezo, shows that learning levels are poor, and have stagnated over time. For instance, in the 2014 Uwezo assessment, 39% of children aged 7-13 years passed a test that required them to demonstrate competence of Standard 2 level numeracy and literacy. This was not significantly different from the performance in previous years: 40% in 2011, 37% in 2012 and 41% in 2013. Looking at student learning levels, both the best and worst performing districts in East Africa are in Kenya. The extremities in quality within Kenyan education are huge. For instance, according to the same Uwezo data, “a child in the Central region is over seven times more likely to have attained a Standard 2 level of literacy and numeracy than a child in the North Eastern region”.
Fixing the education system in Kenya is an onerous task. The Government of Kenya has time and time again, reiterated its commitment to improving the state of education, and has outlined its vision in the National Education Sector Plan 2013- 2018. Alongside, a host of national and international development agencies in Kenya have over the years, financed numerous programmes, targeting various components of the education sector. These efforts have yielded a wealth of evidence. One should consider such evidence, while attempting to answer the question – how can we improve the quality of schooling in Kenya?
International evidence shows that investing in high-quality early childhood programs can have large economic returns, especially for children from socially disadvantaged groups. In response, developing countries are looking to increase public investments in the early years, especially in early education programs. As they do so, one of the challenges policymakers face is deciding what to fund. After all, there are a wide range of opportunities for early childhood education that already exist in local settings such as playgroups and kindergartens. As a result, different children can often have very different early childhood education experiences on their way to primary school.
The inefficiency and inequity caused by age differences in testing is not news. On the contrary, it is a well-documented fact. The proposed solution to this problem is to age-adjust test scores. But the truth is, we are nowhere near to implementing such a solution.
For all of the recent explosion in data related to learning -- as a result of standardized tests, etc. -- remarkably little is known at scale about what exactly happens in classrooms around the world, and outside of them, when it comes to learning, and what the impact of this has.
This isn't to say that we know nothing, of course:
The World Bank (to cite an example from within my own institution) has been using standardized classroom observation techniques to help document what is happening in many classrooms around the world (see, for example, reports based on modified Stallings Method classroom observations across Latin America which seek to identify how much time is actually spent on instruction during school hours; in many cases, the resulting data generated are rather appalling).
Common sense holds various tenets dear when it comes to education, and to learning; many educators profess to know intuitively what works, based on their individual (and hard won) experience, even in the absence of rigorously gathered, statistically significant 'hard' data; the impact of various socioeconomic factors is increasingly acknowledged (even if many policymakers remain impervious to them); and cognitive neuroscience is providing many interesting insights.
But in many important ways, education policymaking and processes of teaching and learning are constrained by the fact that we don't have sufficient, useful, actionable data about what is actually happening with learners at a large scale across an education system -- and what impact this might have. Without data, as Andreas Schleicher likes to say, you are just another person with an opinion. (Of course, with data you might be a person with an ill-considered or poorly argued opinion, but that’s another issue.)
|side observation: Echoing many teachers (but, in contrast to teaching professionals, usually with little or no formal teaching experience themselves), I find that many parents and politicians also profess to know intuitively ‘what works’ when it comes to teaching. When it comes to education, most everyone is an ‘expert’, because, well, after all, everyone was at one time a student. While not seeking to denigrate the ‘wisdom of the crowd’, or downplay the value of common sense, I do find it interesting that many leaders profess to have ready prescriptions at hand for what ‘ails education’ in ways that differ markedly from the ways in which they approach making decisions when it comes to healthcare policy, for example, or finance – even though they themselves have also been patients and make spending decisions in their daily lives.|
One of the great attractions of educational technologies for many people is their potential to help open up and peer inside this so-called black box. For example:
- When teachers talk in front of a class, there are only imperfect records of what transpired (teacher and student notes, memories of participants, what's left on the blackboard -- until that's erased). When lectures are recorded, on the other hand, there is a data trail that can be examined and potentially mined for related insights.
- When students are asked to read in their paper textbook, there is no record of whether the book was actually opened, let along whether or not to the correct page, how long a page was viewed, etc. Not so when using e-readers or reading on the web.
- Facts, figures and questions scribbled on the blackboard disappear once the class bell rings; when this information is entered into, say, Blackboard TM (or any other digital learning management system, for that matter), they can potentially live on forever.
|A few years ago I worked on a large project where a government was planning to introduce lots of new technologies into classrooms across its education system. Policymakers were not primarily seeking to do this in order to ‘transform teaching and learning’ (although of course the project was marketed this way), but rather so that they could better understand what was actually happening in classrooms. If students were scoring poorly on their national end-of-year assessments, policymakers were wondering: Is this because the quality of instruction was insufficient? Because the learning materials used were inadequate? Or might it be because the teachers never got to that part of the syllabus, and so students were being assessed on things they hadn’t been taught? If technology use was mandated, at least they might get some sense about what material was being covered in schools – and what wasn’t. Or so the thinking went ....|
Yes, such digital trails are admittedly incomplete, and can obscure as much as they illuminate, especially if the limitations of such data are poorly understood and data are investigated and analyzed incompletely, poorly, or with bias (or malicious intent). They also carry with them all sorts of very important and thorny considerations related to privacy, security, intellectual property and many other issues.
That said, used well, the addition of additional data points holds out the tantalizing promise of potentially new and/or deeper insights than has been currently possible within 'analogue' classrooms.
But there is another 'black box of education' worth considering.
In many countries, there have been serious and expansive efforts underway to compel governments make available more ‘open data’ about what is happening in their societies, and to utilize more ‘open educational resources’ for learning – including in schools. Many international donor and aid agencies support related efforts in key ways. The World Bank is a big promoter of many of these so-called ‘open data’ initiatives, for example. UNESCO has long been a big proponent of ‘open education resources’ (OERs). To some degree, pretty much all international donor agencies are involved in such activities in some way.
There is no doubt that increased ‘openness’ of various sorts can help make many processes and decisions in the education sector more transparent, as well as have other benefits (by allowing the re-use and ‘re-mixing’ of OERs, teachers and students can themselves help create new teaching and learning materials; civil society groups and private firms can utilize open data to help build new products and services; etc.).
What happens when governments promote the use of open education data and open education resources but, at the same time, refuse to make openly available the algorithms (formulas) that are utilized to draw insights from, and make key decisions based on, these open data and resources?
- Are we in danger of opening up one black box, only to place another, more inscrutable back box inside of it?
Testing: For better and/or for worse, many education systems are built around it (although some perhaps not as much as others). In numerous countries, a long-standing joke asks whether 'MOE' stands for 'Ministry of Education', or 'Ministry of Examination'? (This joke is not meant to be funny, of course.)
'Testing' is a source of and trigger for controversies of all different sorts, in different places around the world. The word 'standardized' is considered negative by many people when it is used to modify 'testing', but it is perhaps worth noting that a lack of 'standardized' tests can have important implications for equity. Within the U.S., the Obama administration recently declared that students should spend no more than 2% of classroom instruction time taking tests. However one feels about the wisdom of setting such hard targets (one could argue, for instance, that it's not 'testing' per se that's the problem, to the extent that it is indeed a problem, but rather 'bad testing') and the various types of time accounting shenanigans that might predictably emerge so that the letter but not the spirit of such declarations are met (a school could be creative about what it feels constitutes a 'test' or 'instruction time', for example), there is no denying the centrality of testing to approaches to education in schools around the world.
'Testing' means different things to different people. There are important distinctions between assessments that are formative (i.e. low stakes means to provide feedback to teachers and students on how much students are learning, as a way to identify strengths and weaknesses and act accordingly) and those that are summative (e.g. high stakes final exams).
It's also potentially worth noting that tests can be utilized not only as means of assessment, but explicitly as tools to help learning we well (an approach sometimes called 'studying by testing'; here's an interesting related paper: When Does Testing Enhance Retention? A Distribution-Based Interpretation of Retrieval as a Memory Modifier [pdf]).
The point here is not to get into a debate about testing, as illuminating and energetic (or frustrating and political) as such a debate might be. Rather, it is to shine a light on some related things happening at the frontier of activities and experiences in this area that are comparatively little known in most of the world but which may be increasingly relevant to many education systems in the coming years.
The nature of tests and testing is changing, enabled in large part by new technologies. (Side note: One way to predict where there are going to be large upcoming public sector procurement activities to provide computing equipment and connectivity to schools is to identify places where big reforms around standardized testing are underway.) While there continues to be growing interest (and hype, and discussion, and confusion) surrounding the potential for technology to enable more 'personalized learning', less remarked on in many quarters is the potential rise in more personalized testing.
The science fiction author William Gibson has famously observed that, The future is already here, it's just not evenly distributed. When it comes to educational technology use around the world, there are lots of interesting 'innovations at the edges' that are happening far away from the spots where one might reflexively look (like Seoul, Silicon Valley or Shanghai, to cite just a few examples that begin with the letter 'S') to learn practical lessons about what might be coming next, and how this may come to pass.
When it comes to testing, one such place is ... Georgia. This is not the Georgia in the southern United States (capital = Atlanta, where people play football while wearing helmets), but rather the small, mountainous country that borders the Black Sea which emerged as a result of the breakup of the Soviet Union (capital = Tbilisi, where people play football with their feet).
Georgia is the first country in the world to utilize computer adaptive testing for all of its school leaving examinations.
What does this mean,
what does it look like in practice,
what is there to learn from this experience,
and why should we care?