The strange case of missing textbook impacts

|

This page in:

Public programs are designed on assumptions - nice, tidy, convenient assumptions. Then they hit the real world and very little goes as planned. The culprit, some philosophically inclined would argue, is human behavior. After all, human beings are impossible to predict. They can react in ways entirely unexpected and fairly baffling …

… until you dig deeper.

We found this happening in one of the most commonplace and straightforward of public programs: textbook provision to schools.  In 2008, the Government of Sierra Leone implemented a program for delivering textbooks to primary schools – one core set of textbooks for each student. There were not enough books for all primary schools and the government was interested in establishing the relationship between textbook provision and learning outcomes, so implementation was done using a randomized trial. The school sample was randomly divided into a treatment group (books) and a control group (no books).

Straightforward, right?

The first touch of mild-ish puzzlement came when we found no impacts on student performance. This was disappointing but not entirely unexpected. At least one other study – in Kenya – found similar results (at least for most students).

What was stranger – at least to the naïve researchers – was what had happened to the program textbooks. In many treatment schools, program textbooks had not been distributed to students; instead they were being stored on school premises. They hadn’t disappeared; we found them! On follow-up visits, we saw storage rooms filled with textbooks, while in classrooms students were still sharing - 3 or 4 students to 1 textbook (or worse). What’s more, students in treatment schools were not being allowed to take textbooks home.

What? What were head teachers thinking keeping textbooks from students who clearly needed them? What could prompt head teachers to (uselessly?) let these textbooks gather dust in some storage room?  We explored several theories and carefully analyzed the correlates of book storage. We asked teachers if they thought the books were useful; they did. We asked students if they already had books at home; they didn’t. It turns out that head teachers who had high uncertainty about government transfers of textbooks were significantly more likely to store them: We measured this uncertainty by whether or not the head teacher knew how many textbooks were allocated to the school the previous year by the government.

Once we took this underlying ‘uncertainty’ into account, the storing behavior started to make sense. Think of consumption theory: if there is uncertainty about future transfers then current transfers are seen as a one-time (or transitory) shock. If this is the case then the impact of current transfers on current consumption (defined as intensive use of textbooks by students) will be limited. This is because when there is uncertainty, head teachers have incentives to store part of the current transfers in order to smooth consumption over time, particularly for inputs like textbooks that depreciate quickly when used. Remember the Permanent Income Hypothesis? This is a permanent public input hypothesis.

Taking ‘uncertainty’ into explicit consideration makes the seemingly irrational, rational. Clearly, head teachers in this context are acting as forward looking agents who – in the classic spirit of homo-economicus – are deciding how to respond to the public program based on their resources, constraints, and expectations.

And these results illustrate what this decision-making looks like when prevailing expectations around government largess and reliability are bleak. And why wouldn’t they be bleak? Our survey data shows that in 2009, 17 percent of the head teachers and 36 percent of classroom teachers reported not receiving their full pay in the last year. In Zambia, the unpredictability of government transfers to line ministries was well documented a decade ago. Anecdotal evidence tells us this is widespread.

This smoothing behavior isn’t unique to textbooks. In a project providing grants to schools in the Gambia, schools “were directed to use the grant towards some aspect of the school development that relates directly to teaching and learning.” Yet a quarter of schools reported their biggest expenditure was on infrastructure, durable inputs rather than quickly depreciated learning materials.

Is this simply ill-informed decision-making OR in the face of uncertain future transfers, are school committees trying to convert current transfers into goods that can be consumed not just in the present but also in the future?

When program goals are subverted by beneficiaries (or intermediate agents, as in this case), it is common to blame low capacity, myopic decision-making, lack of information, human error, corruption, or increasingly (and more charitably) the poverty-induced burden on mental bandwidth. But what we find here is rational behavior.

Our modest argument is that public provision of inputs that depreciate quickly when used, like textbooks, will be fully consumed only if agents have expectations of replenishment. Hence, for public programs that are designed as recurring transfers, there is a need to establish a reputation of consistent delivery and to reliably communicate the timing of the next transfer to agents. Otherwise, an environment plagued with uncertainty can easily wreak havoc on the nice, tidy, and convenient assumptions underlying program design.
Topics

Authors

Shwetlena Sabarwal

Senior Economist, Education Global Practice, World Bank

David Evans

Senior Fellow, Center for Global Development

Join the Conversation

Alexis
September 21, 2014

Don't you think the randomization process has increased the uncertainty? In this case the external validity of your results might be low.

David Evans
September 24, 2014

This is an interesting point. The books included in the evaluation were additional to regular allotments, but it's true that a school may have received more than expected, which could have added to uncertainty. That said, the question we use to proxy for uncertainty is whether they knew how many they should expect, not whether they were actually correct. Many head teachers really had no idea how many to expect, which would have been true regardless of the intervention. It seems like this is a space of extremely high uncertainty regarding inputs, and this is consistent across many low-income settings. If the behavior is highly sensitive to the magnitude of the uncertainty, than I agree that external validity to [Sierra Leone in absence of intervention] would be decreased.
 

emmanuel
September 15, 2014

Interesting finding, as well as the way it has been associated with theoretical explanation from a different field ,,, it seems we have to revisit all empirics which concluded observed behaviors as irrational!

Anonymous
September 16, 2014

Did you ask the headteachers why they kept them in storage?

Shwetlena
September 21, 2014

Thank you - this is a great point. At the time the endline survey was designed and administered, we were unaware of the extent of the storage phenomenon. Therefore, unfortunately, we did not include any direct questions on it. If only we had known what we were going to find. we could have done a much better job of asking the right questions! As they say, hindsight is 20-20.

Tim Horsler
October 01, 2014

You've put your finger(s) on the spot. However, I can tell you that after about 40 years involvement in publishing and distributing books in Africa (starting off in Nigeria in 1969 and involving inter alia frequent visits to Sieera Leone starting in 1974), these problems are neither new nor unknown. What remains depressingly constant is the inability of Ministers of Education and their officials to understand that the planned and continuing funding of textbooks on a regular rolling basis is vital to avoid the usual cycle of (a) a glut of large quantities of books following usually) the receipt of donor aid or World Bank loans, to be immediately followed by (b) famine for some years until another project/loan/donor comes along. I am also far from convinced that all the various aid agencies and banks involved in education projects get the point, or take a sufficiently long term view to play their part in helping governments to avoid the problems you describe.
Such peaks/gluts and troughs/famines also play havoc with the whole business of book authorship, publishing, production and distribution, to the detriment of the development of the book sector in geneal.
Tim Horsler
(formerly of Longman Group, now an educational publishing consultant and Associate Consultant for Education for Change Ltd (www.efc.co.uk)

Tim Horsler
October 01, 2014

Here is a PS to my previous note: with virtually no exceptions, no Ministries of Education have the skills or capacity to handle cost-efficient and timely book supply. Distribution is costly, and even more so in countries with poor transport and storage facilities which are significanly affected by rainy seasons. Where aid money is involved, the entire procurement cycle for existing books can easily take 18 months from approval of the procurement budget and plan to the delivery of books to the furthest school. If new boooks are involved - and NB new curriculums means new books - you can add in another 18 months of writing, testing, rewriting and designing and illustrating time to this cycle.
In addition, most MoEs do not understand that successful distribution requires high levels of planning capability, nor that publishing requires both extended periods of time plus skill before quality books can be produced.
Finally - and once again - I remain unconvinced that most donors and lenders understand these matters. A noble, recent exception is in South Sudan, where DFIF grasped the nettle of funding the nationwide book supply programme to be carried out by a commercial distribution company following full ICB procedures. The winning company did a great job. The distribution cost was three times the cost of the books. BUT the books did get delivered...
The next questions, of course, are 1. Are they being used? 2. What effect are they having on educational attainments? 3. Have the teachers been trained in their use? 4. I could go on...