Learning trajectories through primary school: Lessons from a low-income setting

Many children in low-income countries cannot read a full sentence or perform basic mathematical operations, such as addition, after spending 5-6 years in primary school.  The drivers of low learning remain unclear because of the lack of the high-quality, longitudinal test score data needed to understand how children learn. Our forthcoming paper uses one of the first such datasets to uncover new facts about how children learn in a low-income setting.

Our paper uses the Learning and Educational Achievement in Pakistani Schools (LEAPS) dataset, assembled by Tahir Andrabi, Jishnu Das and Asim Ijaz Khwaja. The LEAPS dataset follows more than 12,000 children as they progress from 3rd to 6th grade and are among the first data from a low-income setting to allow us to examine children’s test scores in a consistent manner across four years. The data are from Punjab province, Pakistan (the 12th largest schooling system in the world). The fact that Pakistan scores similarly to other low-income countries in internationally benchmarked tests suggests that the patterns in these data may be equally applicable elsewhere.

Our first finding, not surprising, is that children do learn in every grade. For instance, 58% of children could correctly multiple “4 x 5” in grade 3, and this fraction then increases to 60%, 73% and 79% in each subsequent year. We see similar patterns across every question and subject, and an aggregate measure of learning (which can be placed on the same scale across tests using item response theory) shows that test scores increase by 1.2sd between grades 3 and 6. This relative rate of learning is similar to what we find in the Young Lives countries of Vietnam, Peru, India (Andhra Pradesh) and Ethiopia and also to the U.S. state of Florida, all of which have gains of approximately 1 standard deviation over a similar 4-year period. This implies that in all these school systems the top 30% of children in grade 3 score (roughly) the same as the bottom 30% of children in grade 6. It does not imply that they learn the same amount since the tests and starting learning levels are very different.

One question is whether this learning is due to schooling as opposed to natural gains as children age. To distinguish learning due to aging versus schooling, we tracked and tested children who had dropped out between grades 5 and 6. While dropouts reported slightly lower test scores in previous grades than children who continued in school, learning gains between grades 3 and 5 were identical for dropouts and non-dropouts. However, once children dropped out, their learning stalled, while for those who remained in school it continued along the same trend (Figure 1). So, learning gains observed through the years are due to schooling (not aging) and a narrative suggesting that dropouts would not benefit from schooling anyway needs to be revisited. Our results suggest that if test score gains have constant returns for adult outcomes, dropouts and non-dropouts would experience similar returns from remaining in school.

We next consider whether low learning rates are driven by less-prepared children (with ex-ante lower test scores) falling further and further behind. We might expect these children to experience more mismatch between their academic preparation and the curriculum, leading to lower levels of learning. Surprisingly, as Figure 2 shows, this is not the case. Children whose test scores were in the bottom 20% in grade 3 learned significantly more between grades 3 and 6 than children ranked in the top 20%. Correcting for mean reversion in test scores does not change the qualitative pattern. In fact, we see a marked convergence in test score levels, rather than divergence, over time. Schooling reduces inequality in learning.

Results that are so much at odds with the perceived wisdom must be carefully scrutinized. Does attrition affect our results? What about measurement error? Is test score scaling an issue? We address each of these questions in the paper and show that the core results are robust to how we handle these challenges. We also show that our results are fully consistent with the previous work of Muralidharan, Singh and Ganimian (2019), who have argued that children who fall behind subsequently learn less in school. We find that the difference in our findings is due to the nature of the comparison. When we compare test score gains across the distribution in both datasets, we find similar results. Children who start out behind learn more, although this learning is not sufficient to allow them to catch up to the curriculum.

[Technical Aside: Muralidharan, Singh, and Ganimian are interested in whether children with low test scores learn less than what they are predicted to learn, given the overall dynamics of test score gains in their data. We instead ask whether children with low test scores learn less in absolute terms. The answer to the latter question, in both their data and ours, is ‘no.’]

To understand how these facts can be consistent with low test scores, we then measure the patterns of gains and losses on a question-by-question basis. We find that test scores do not increase monotonically year by year. In fact, 20% of children see declines in test scores on a yearly basis, and 10% reported lower test scores in grade 5 compared to grade 3. We propose a new term for the patterns in the data: ‘fragile learning.’ Children learn in one year but are about as likely to forget as to consolidate their learning. In fact, the proportion of ‘fragile learners,’ or those who learn and then forget (and then sometimes learn again), is worryingly high.

This kind of fragile learning could be driven by random guessing on multiple choice questions, but interestingly, we find very similar patterns when we look at questions such as “23 + 14” that did not offer options but instead asked children to fill in the right answer. Many children who correctly answered ‘37’ in one year failed to do so in the next (these tests were administered by our own highly trained team with tests recovered each year after the exam, so we can rule out cheating as a potential explanation). The key message is that performance in school has as much to do with forgetting as it does with learning.

Our plea therefore is not to label all schooling as next to useless based on low levels of learning. The basic facts are that children learn, schools tend to equalize test scores and a conflagration of factors lead to dropouts, not just how much children are learning. Further emphasizing the importance of schools vs. household characteristics, our paper shows that household characteristics, such as parental wealth and education, only account for 6% of the variation in year-to-year learning gains. Schools really matter, and if we can keep children in school longer—even with the status quo — children will be better off in terms of what they know.

As more and better longitudinal data in low and middle-income becomes available, new evidence on learning trajectories will emerge, and we will be able to observe if these findings are also applicable to other contexts. Our hunch is that the consolidation of knowledge, which may vary across contexts, will emerge as an important factor; unfortunately, no data currently allows us to disentangle learning from forgetting. Our own attempts to understand this are heuristic rather than statistically grounded as the combination of measurement error and poorly measured rates of persistence makes this a very difficult exercise. One interesting hypothesis is that recent pedagogic breakthroughs such as Teaching at the Right Level may be very effective precisely because they tailor material to children’s current learning levels and therefore consolidate their knowledge before moving on. If so, studies designed to better measure forgetting may greatly contribute to our knowledge and ultimately better ways of helping children learn.