A Data Guide to Sir Michael Barber’s “The Good News from Pakistan”

Shanta’s blog reported on Sir Michael Barber’s approach to implementing service delivery or “Deliverology”. Sir Michael was back at the World Bank on June 6th to present “The Good News from Pakistan”, where he outlined the impressive changes in Punjab, Pakistan as a result of his leadership in delivering deliverology. As a discussant, with Dhushyanth Raju’s inputs (Dhushyanth is a Senior Economist in the World Bank's South Asia education team), I examined and triangulated the existing data. Despite my original excitement about the method and the results after reading the report, I am reluctantly forced to conclude that at the moment the data do not support the report’s claims (see my presentation). That’s not to say there’s no good news from Pakistan on education. Just the opposite in fact: the good news is the large increase in enrollment and learning that predated Sir Michael’s deliverology intervention. 

What does “The Good News from Pakistan” say?

Sir Michael has led an ambitious education reform in the province of Punjab, Pakistan since 2010. In his own words:

“This essay tells the story of the Punjab education reform called the Punjab Schools Reform Roadmap. Potentially it is a story of redemption for Pakistan. I am publishing it now, ahead of Pakistan’s elections, because it sets an agenda which any party or government could adopt and pursue after an election. Potentially, too, it is a story about how aid programs can be so successful that they won’t be needed any more.” (Page 11, “The Good News from Pakistan”)

His striking results on whether deliverology is working in Punjab, Pakistan are summarized on Page 13 of the report:

“As of January 2013, on a conservative estimate, there are approaching one and a half million extra children enrolled in school. In addition, student attendance daily is now over 90 per cent, 81,000 new teachers have been hired on merit and more than 35,000 more teachers are present at school every day than two years ago. Over 90 per cent of schools now have basic facilities in place as opposed to less than 70 per cent two years ago. Importantly, across all the indicators there has been a narrowing of the gender gap, although there is more to do, especially in the south of Punjab.”

How was this done? Sir Michael outlined in his presentation how his tireless and ceaseless effort combined with ambitious target setting and the Punjab Chief Minister’s backing brought about this change. For instance, Sir Michael and his team made impressive progress getting textbooks to children:

“The fact that under the Roadmap we had prepared and distributed lesson plans within a few months showed it (providing textbooks) could be done, so we turned up the heat on the textbook board. In the summer of 2011, I chaired a meeting of all the key stakeholders to set the ball rolling. Ali Abid Hussein from my team became the driving force. Some months (and three Chairmen of the Punjab Textbook Board) later, we were able to show the Chief Minister the new textbooks at a stocktake meeting” (Page 47, “The Good News from Pakistan”)

What do I know about this?

I was asked to discuss Sir Michael’s presentation because this is a sector and province that I know a bit about. Since 2003, as part of the Learning and Educational Achievements in Punjab Schools, or LEAPS project, together with Asim Ijaz Khwaja at Harvard University and Tahir Andrabi at Pomona College, I have followed 112 villages in 3 districts of the province, testing 12-24,000 children, surveying teachers, schools and conducting a full-scale longitudinal household survey among 1,800 households. We have also been fortunate to collaborate with a stellar team working on education in South Asia at the World Bank during this time, a team with a detailed grasp of programs and education data in Pakistan. 

So, what can we say?

Because the program is a province-wide program, there is a basic problem of constructing counterfactuals to assess the impact of deliverology. But this is something that, as researchers, we are used to dealing with, and even in country-wide or province-wide programs, we can find innovative ways of arriving at comparable control groups.  Examples include Esther Duflo’s work on wages and education in Indonesia; Karthik Muralidharan and Nishith Prakash’s work on the impact of reducing travel costs for girls on school enrollment and our own work in Pakistan on the impact of constructing secondary schools on private school location decisions or the impact of maternal education on their children’s schooling outcomes. 
With recourse to the appropriate data, we usually manage to find micro-variations in administration, timing or geography to deduce control and treatment groups.

More serious is the lack of any consistent data source on enrollment, school inputs and learning between 2000 and 2013. For instance, till 2010, the primary source of enrollment data was the Government’s own Pakistan Social and Living Standards Measurement or PSLM survey, but the rounds after 2010 have not been released. Instead, the report relies on a household-level dataset collected by the company Nielsen in 2011, 2012 and 2013, which has not been publicly released and externally scrutinized with comparisons to other data. Data on inputs comes from the government’s own monitoring database and data on learning from the Annual State of Education Report, Pakistan for 2011 and 2012. In each of these cases, the LEAPS data help triangulate, but they cover 3 (out of 36) districts and are currently from 2003-2010. Therefore, the best we can do is triangulate and see if a consistent story emerges. 

Here is the broad summary; the attached presentation has further detail.

Enrollment: There was a massive jump in enrollment between 2003 and 2007/8 with net primary enrollments shooting up from 54% to 70%. We find precisely the same numbers in the LEAPS data and in the government’s household survey, the PSLM. With increasing political turmoil and an economy in real trouble, enrollment increases abruptly stopped in 2008, and in 2010, the numbers remained at 70%. In 2011, the data source changed and the Nielsen household survey shows an overnight jump to 85%, a mystery currently lacking any explanation. From 2011 to 2012, enrollment went up from 85% to 86% and from 2012 to 2013 went to 86.8%. This is the increase that leads Barber to say that “After January 2013, there are approaching one and a half million extra children aged 5-16 enrolled in school.” Even this small increase, which contrasts sharply with the 16 percentage point jump under the World Bank-supported Punjab Education Reform Program between 2003 and 2008, runs into problems when I try to triangulate with other data sources. Specifically, the Nielsen report also claims that the share of enrollment in public schools increased over the same time, so that the numbers in public schools must have also risen. In fact, in the Annual Census of Schools, which is the Government’s data on public school enrollment for all the input-based monitoring numbers in the report, there is no such increase—there were 10,640,000-odd children in 2009 in public schools and exactly the same number in 2012/13.

{NB: After the discussion, I became aware of the release of the 2011-2012 PSLM report from the Government of Pakistan. The PSLM report shows that in 2011/2012, the net enrollment rate in Punjab for children aged 6-10 increased from 70% in 2010/2011 to 74%, with much of that increase coming from an increase in private school enrollments. This is much more in line with previous increases and is broadly consistent with constant enrollment numbers in public schools. It is 11 percentage points lower than the Nielsen-reported enrollments. Clearly, the Nielsen survey got it wrong. But without publicly available data and sampling files from both Nielsen and PSLM, it is impossible to tell what went wrong at this point.}

Inputs: The story on inputs is clearer. From the Annual Survey of Education Report, Pakistan and the Government’s monitoring database, there have been impressive improvements in school inputs. But are these a deviation from historic trends? Not necessarily. In the LEAPS villages, we documented an equally impressive increase in school infrastructure between 2003 and 2010, with the number of permanent classrooms increasing from 3.89 (per public school) to 5.67, the number of boys’ toilets increasing from 0.71 to 1.25 and the number of girls’ toilets from 0.41 to 1.23. And as in the report, in LEAPS we have also seen declines in teacher absence between 2003 and 2010. The input trend under deliverology looks much like the trend before it. 

Learning: The LEAPS data, which use a rigorous test that is perfectly comparably across years (technically, we rotate some common questions in every year to achieve test comparability), provide consistent data on test-scores for Grade 4 children in 2004, 2006 and 2011. The provisional data (subject to revision) suggest that between 2004 and 2011, Mathematics test-scores in public schools declined by 0.15 standard-deviations, Urdu test-scores increased by 0.07 standard-deviations and English test-scores improved dramatically, by 0.43 standard-deviations. There have been no LEAPS surveys since 2010; we are now trying to go back to the field in 2015. The only other data source is an NGO-run Annual Survey of Education Report, Pakistan, which uses a volunteer army with home-based testing to implement a short test in Urdu, Mathematics and English. This test shows impressive gains in Punjab in just one year. And these gains are very large—taken at face-value they would suggest a half standard-deviation increase in test-scores in just one year. This is an improvement we have never seen before. However, it turns out that the gains in the neighboring province of Khyber Pakhtunwa (where deliverology had not yet started) are even larger, suggesting that the changes reflect differences in something else—perhaps the nature of the sample, the training or the test-instruments.

What can we conclude?

At this point then we simply don’t have data to support the deliverologists’ strong claims. There seem to have been real changes in monitoring and management practices in the Punjab Government, and much more centralized management appears to have been enforced. The evidence is consistent with continued improvements in school inputs, and potentially small changes in enrollment. On learning, we can’t say much at this point. 

Does deliverology deliver?

The lack of credible public data and the lack of third-party evaluations of the program make it difficult to go to bat for the deliverologists at this time. The critical steps needed to understand the program better are independent and public sources of data (including the government’s own monitoring data) and evaluations of what the program is doing in a manner that fully accounts for management costs and increased budgetary allocations. There are broader points about whether deliverology is a fundamentally new approach, or a rediscovery of the economics of incentives and mechanism design. There is also an important discussion on the merits of a centralized system in a population with enormous heterogeneity, but the first step is to credibly understand what deliverology actually delivers.