Learning for All: Beyond an Average Score

Before COVID-19, the world was already in a learning crisis. At historical rates of progress, the Sustainable Development Goals (SDG) target of ensuring that all girls and boys complete free, equitable, and quality primary and secondary education, leading to relevant and effective learning outcomes by 2030, is out of reach. More than half of children in low- and middle-income countries are either out-of-school or fail to learn to read with comprehension by age ten. At the secondary level, the situation is even direr.

Learning losses due to the pandemic will likely be large. We are only starting to grasp how learning inequalities  may play out. Looking at historical data, can we find patterns that can help us better plan our actions in the near future? As countries build back educational systems that are more resilient, equitable, and inclusive – capable of delivering learning for all – how likely is that learning inequality might undermine this process?

Education SDGs: beyond average scores

Let us consider lower-secondary students' learning outcomes, as assessed by the Programme for International Student Assessment (PISA). PISA provides comparable data of 15-year-old pupils' performance in mathematics, science, and reading. Much attention is spent discussing country ranks and improvements of the average PISA test scores. But the average gives an incomplete picture in an unequal world. And we know that multiple sources of inequality generate variations in learning at school and at home across gender, ethnicity, socio-economic status, and disabilities.

Through the SDG process, countries have agreed to monitor learning for all through the share of students achieving a minimum proficiency level. For instance, in reading, lower secondary students should recognize the main idea in a text, understand relationships, or construe meaning within a limited part of the text when the information is not prominent and make low level inferences. Failing to master these basic competencies will preclude them from participating effectively and productively in life as continuing students, workers and citizens.  To go beyond the test score of the average student reflects a strong commitment to focus our attention on improving learning for the kids that fail to reach these minimum standards of skills/competences.

These two performance measures of education systems – average scores and the share of students achieving minimum proficiency convey important information. They are akin to observing the changes in average income per capita and in the share of the population living in poverty, when assessing a country's growth and welfare performance over time. The average income per capita and the average test score are summary measures of all individuals. The share of the population living in poverty and the share of students below the minimum proficiency level refer to the percentage of students failing to acquire a minimum set of competences and are deprived of minimum standards of living and of acquired competences, respectively. In education, just like in the economy, those measures do not necessarily move in the same direction.

Unbundling progress in the share of students achieving minimum proficiency

Economists have long investigated the relationship between poverty, growth, and inequality, and understand that those are jointly determined. Changes in poverty rates can be unpacked into the contributions of changes in the mean (average income per capita) and income distribution (inequality). A similar decomposition can be insightful in the education context. An improvement in the average test score with growing inequality will be qualitatively different from a process in which the average test score increases by the same amount, but inequality remains constant.

We explore data from countries that participated in multiple PISA rounds. Each bar in Figure 1 represents the decomposition of the annualized change in students' share below the SDG threshold in reading between two rounds of PISA for a given country. The countries in red performed worse in recent years. That is, the share of students who cannot read proficiently increased by at least 0.1 percentage at an annualized rate. The countries in green have seen improvement: more students achieving minimum proficiency in reading. Lastly, for the countries in yellow, the proficiency share remains reasonably similar. We decompose the change in the share of students below minimum proficiency (black line) into the contribution from the average score (dark bars) and the distribution of scores (light bars).

Figure 1- Decomposition of changes in the share of students below minimum proficiency (BMP)

The reader can use our interactive visualization to extend this analysis to mathematics and science and subgroups of students by gender, location, and quintile of socioeconomic status. We also include any two years of PISA participation – not just the longest interval of time in which the subject was the major focus of assessment. An alternative decomposition methodology is also used, but more of that in another blog. Results remain qualitatively similar – as you can see by interacting with the filters in Figure 1. Another way to visualize these results is by averaging all countries that improved or worsened. This is done in Figure 2, which summarizes the decomposition for reading in over 300 observations (each corresponding to a country and pair of years of participation).

Figure 2 Absolute and relative  contributions to  changes in the share of students below minimum proficiency (BMP)

Some stylized facts emerge:

➤ A rising tide [almost always] lifts all boats

• All countries which improved their proficiency level also have increased average test scores. In Peru, the share of students who cannot read the basics fell from 80% to 54%, or an annualized rate of change of approximately -1.4p.p. per year.  During this period, average scores rose from 327 to 401 points between 2000 and 2018, while inequality in learning worsened - meaning if the inequality in test scores would have remained the same as in 2000, the improvement on the share of students below the MPL would have been 3p.p. higher, or the equivalent of two additional years of average improvement.
• Only in a few countries, the share of students achieving proficiency worsened despite small improvements in average test scores.  In Georgia, the mean score rose by 6 points between 2009 and 2018; nevertheless, the students' share below the minimum proficiency threshold increased by 2.4 percentage points. So it was students with higher test scores that improved the most.

➤ When average test scores fall, the inequality in learning always got worse; and there was no clear pattern when proficiency level improved

• In all countries where the share of students below minimum proficiency increased, the inequality in learning worsens the situation. In South Korea, the average scores in 2018 were almost the same as in 2000, but the share of youth who cannot read at minimum proficiency increased from 6% to 15% as inequality in learning increased.
• The proficiency level can improve even when inequality in learning worsens. Between 2006 and 2018, the share of students below minimum proficiency in Colombia increased by almost 6 percentage points, although average test scores surged 27 points, reflecting larger improvements in learning for children with relatively higher learning achievement.

Visualizing country examples can clarify the decomposition. Figure 3 shows overlaid distributions of reading scores in a country for two PISA rounds - the first in blue and the latest in red. In countries like Peru, the two snapshots in time have similar shapes but different average test scores, given the impression that the curve slid, the rising tide that lifted all boats. In South Korea, the mean remained the same, but the distribution's shape (inequality in learning) changed significantly between the years.

Figure 3- Distribution of reading scores in two rounds of PISA for selected countries

Note: The animated GIF cycles through countries that participated in multiple rounds of PISA. For each country, the distribution of reading scores is displayed in the first (blue) and last (red) year of participation. The shaded areas denote students' share below the minimum proficiency (BMP), represented by the vertical line (407.47 points). The mean and BMP are displayed for both years in the legend box, and the footnote reports the decomposition of the BMP change into the contribution of the mean score and the distribution of test scores (the bar values in Figure 1).

Design policies for scale with a focus on learning for all

The SDGs gave us a consensus on minimum proficiency levels of learning. We must build on that to document and understand the drivers of change. Learning for all requires overcoming inequalities across and within countries.

Across countries, we must be obsessed with achieving results at scale. Small-scale successful interventions that benefit a small group of children will not suffice to address the magnitude of the learning crisis. Interventions and policies need to be designed for scale. This does not neccesarily mean that we need to overhaul the entire education system in the short term to achieve improvements in learning. More focused policies at scale, as described in the “Ending Learning Poverty Report” and exemplified by the state of Ceará in Brazil, can generate significant short-term improvements, while at the same time countries take on the more difficult longer term task of building robust education systems.

In order to ensure that learning occurs for all, we must measure and track changes in learning for kids failing to reach minimum proficiency levels. Assessing the performance of educational systems based on average scores or achievement gaps between groups is not enough. We should be intentional and systematic on how we document and understand the learning inequality within systems. Not all educational systems propagate inequalities in the same way.

We expect COVID-19 to negatively affect the learning distribution, as it has happened before in all episodes of decline in learning proficiency. Periods of reduction of learning proficiency have never been distributionally neutral. Going forward, we must mitigate and remediate the learning losses of the most vulnerable. We have the opportunity and responsibility to reimagine and build back educational systems that are more resilient and able to better protect the learning from the most vulnerable, especially at moments of crisis.