Collecting data in education can be a tricky business. After spending considerable resources to design a representative study, enlist and train data collectors, and organize the logistics of data collection, we want to ensure that we capture as true a picture of the situation on the ground as possible. This can be particularly challenging when we attempt to measure complex concepts, such as child development, learning outcomes, or the quality of an educational environment.
Data can be biased by many factors. For example, the very act of observation by itself can influence behavior. How can we expect a teacher to behave “normally” when outsiders sit in her or his classroom taking detailed notes about everything they do? Social desirability bias, where subjects seek to represent themselves in the most positive light, is another common challenge. Asking a teacher, “Do you hit children in your classroom?” may elicit an intense denial, even if the teacher still has a cane in one hand and the ear of a misbehaving child in another.
Assessments make a lot of people nervous, and I’m not just talking about the students who have to take them. As a psychometrician (assessment expert) and World Bank staffer, I’ve worked on assessment projects in more than 30 countries around the world over the past 10 years. Time and again, I’ve found great interest in student assessment as a tool for monitoring and supporting student learning coupled with great unease over how exactly to go about ‘doing’ an assessment.
Over the past several decades, developing countries have made remarkable progress in achieving quantitative education targets. Since the turn of the millennium, almost 50 million children around the world have gained access to basic education – and most are reaching completion. But as recent PISA data shows, this is not typically the case for qualitative improvements in education. A persistent learning gap remains for an estimated 250 million children who are unable to read and do math, even after spending three or more years in the classroom.
The headlines started to stream as soon as the PISA results were in: “Asian countries top OECD's latest PISA survey.” “Poor academic standards.” “Students score below international averages.” It depends on the country, of course. A time to celebrate for some, a time to lament for others.
I recently came across a report card from my secondary school days in Ireland. It was an interesting read. My progress in areas as diverse as mathematics, singing, Irish language, and physical education was reported on in the form of marks, grades, and narrative feedback. Some teachers provided little information on my learning. Others went into detail. I was impressed by the number of areas in which my progress had been assessed (less so by my lack of singing ability, which, evidently, had been spotted early on!).
Flash forward to 2013, and there is a conversation raging in the development community about how to measure and report on learning globally. A huge concern is the fact that too often children leave school without acquiring the basic knowledge and skills they need to lead productive lives. To make matters worse, there is a global data gap on learning that is impeding efforts to better understand this crisis and how to achieve learning for all.
Most educational interventions are widely considered successful if they increase test-scores -- which indicate cognitive ability. Presumably, this is because higher test-scores in school imply gains such as higher wages later on.
However, non-cognitive outcomes also matter---a lot.
All eyes are focused on South Africa this year: it both hosts the World Cup and celebrates its 20th anniversary since the end of apartheid when Nelson Mandela walked those historic steps to freedom. In post-aparteid South Africa, education promised to hold part of the answer towards creating a fairer society. Development through education – would lead to freedom. The burning question remains - has this been achieved?
In a 2007 World Bank publication, Shafika Isaacs summarized the desired changes South Africa hoped to undertake:
A lot of my work is on test-scores. Here is a (very partial) list of what we know about test-scores, why they may be important (and why they may not). In future blogs we hope to take up each of these topics in greater detail. We cite the papers we know below.
Test-scores, Enrollment and Growth: Macro Picture
- Enrollment and growth are not related according to Pritchett’s classic paper on “Where has all the education gone”. The missing ingredient may well be test-scores. There is a positive association between test-scores and growth as Hanushek and Woessmann argue.
- Further work by Hanushek and Woessmann suggests that these effects may be causal. Particularly interesting is the fact that home-country cognitive skills affects the earnings of immigrants to the United States.