Syndicate content

A Data Guide to Sir Michael Barber’s “The Good News from Pakistan”

Jishnu Das's picture

Shanta’s blog reported on Sir Michael Barber’s approach to implementing service delivery or “Deliverology”. Sir Michael was back at the World Bank on June 6th to present “The Good News from Pakistan”, where he outlined the impressive changes in Punjab, Pakistan as a result of his leadership in delivering deliverology. As a discussant, with Dhushyanth Raju’s inputs (Dhushyanth is a Senior Economist in the World Bank's South Asia education team), I examined and triangulated the existing data. Despite my original excitement about the method and the results after reading the report, I am reluctantly forced to conclude that at the moment the data do not support the report’s claims (see my presentation). That’s not to say there’s no good news from Pakistan on education. Just the opposite in fact: the good news is the large increase in enrollment and learning that predated Sir Michael’s deliverology intervention. 


What does “The Good News from Pakistan” say?

Sir Michael has led an ambitious education reform in the province of Punjab, Pakistan since 2010. In his own words:

This essay tells the story of the Punjab education reform called the Punjab Schools Reform Roadmap. Potentially it is a story of redemption for Pakistan. I am publishing it now, ahead of Pakistan’s elections, because it sets an agenda which any party or government could adopt and pursue after an election. Potentially, too, it is a story about how aid programs can be so successful that they won’t be needed any more.” (Page 11, “The Good News from Pakistan”)

His striking results on whether deliverology is working in Punjab, Pakistan are summarized on Page 13 of the report:

As of January 2013, on a conservative estimate, there are approaching one and a half million extra children enrolled in school. In addition, student attendance daily is now over 90 per cent, 81,000 new teachers have been hired on merit and more than 35,000 more teachers are present at school every day than two years ago. Over 90 per cent of schools now have basic facilities in place as opposed to less than 70 per cent two years ago. Importantly, across all the indicators there has been a narrowing of the gender gap, although there is more to do, especially in the south of Punjab.

How was this done? Sir Michael outlined in his presentation how his tireless and ceaseless effort combined with ambitious target setting and the Punjab Chief Minister’s backing brought about this change. For instance, Sir Michael and his team made impressive progress getting textbooks to children:

The fact that under the Roadmap we had prepared and distributed lesson plans within a few months showed it (providing textbooks) could be done, so we turned up the heat on the textbook board. In the summer of 2011, I chaired a meeting of all the key stakeholders to set the ball rolling. Ali Abid Hussein from my team became the driving force. Some months (and three Chairmen of the Punjab Textbook Board) later, we were able to show the Chief Minister the new textbooks at a stocktake meeting” (Page 47, “The Good News from Pakistan”)

What do I know about this?

I was asked to discuss Sir Michael’s presentation because this is a sector and province that I know a bit about. Since 2003, as part of the Learning and Educational Achievements in Punjab Schools, or LEAPS project, together with Asim Ijaz Khwaja at Harvard University and Tahir Andrabi at Pomona College, I have followed 112 villages in 3 districts of the province, testing 12-24,000 children, surveying teachers, schools and conducting a full-scale longitudinal household survey among 1,800 households. We have also been fortunate to collaborate with a stellar team working on education in South Asia at the World Bank during this time, a team with a detailed grasp of programs and education data in Pakistan. 

So, what can we say?

Because the program is a province-wide program, there is a basic problem of constructing counterfactuals to assess the impact of deliverology. But this is something that, as researchers, we are used to dealing with, and even in country-wide or province-wide programs, we can find innovative ways of arriving at comparable control groups.  Examples include Esther Duflo’s work on wages and education in Indonesia; Karthik Muralidharan and Nishith Prakash’s work on the impact of reducing travel costs for girls on school enrollment and our own work in Pakistan on the impact of constructing secondary schools on private school location decisions or the impact of maternal education on their children’s schooling outcomes
With recourse to the appropriate data, we usually manage to find micro-variations in administration, timing or geography to deduce control and treatment groups.

More serious is the lack of any consistent data source on enrollment, school inputs and learning between 2000 and 2013. For instance, till 2010, the primary source of enrollment data was the Government’s own Pakistan Social and Living Standards Measurement or PSLM survey, but the rounds after 2010 have not been released. Instead, the report relies on a household-level dataset collected by the company Nielsen in 2011, 2012 and 2013, which has not been publicly released and externally scrutinized with comparisons to other data. Data on inputs comes from the government’s own monitoring database and data on learning from the Annual State of Education Report, Pakistan for 2011 and 2012. In each of these cases, the LEAPS data help triangulate, but they cover 3 (out of 36) districts and are currently from 2003-2010. Therefore, the best we can do is triangulate and see if a consistent story emerges. 

Here is the broad summary; the attached presentation has further detail.

Enrollment: There was a massive jump in enrollment between 2003 and 2007/8 with net primary enrollments shooting up from 54% to 70%. We find precisely the same numbers in the LEAPS data and in the government’s household survey, the PSLM. With increasing political turmoil and an economy in real trouble, enrollment increases abruptly stopped in 2008, and in 2010, the numbers remained at 70%. In 2011, the data source changed and the Nielsen household survey shows an overnight jump to 85%, a mystery currently lacking any explanation. From 2011 to 2012, enrollment went up from 85% to 86% and from 2012 to 2013 went to 86.8%. This is the increase that leads Barber to say that “After January 2013, there are approaching one and a half million extra children aged 5-16 enrolled in school.” Even this small increase, which contrasts sharply with the 16 percentage point jump under the World Bank-supported Punjab Education Reform Program between 2003 and 2008, runs into problems when I try to triangulate with other data sources. Specifically, the Nielsen report also claims that the share of enrollment in public schools increased over the same time, so that the numbers in public schools must have also risen. In fact, in the Annual Census of Schools, which is the Government’s data on public school enrollment for all the input-based monitoring numbers in the report, there is no such increase—there were 10,640,000-odd children in 2009 in public schools and exactly the same number in 2012/13.

{NB: After the discussion, I became aware of the release of the 2011-2012 PSLM report from the Government of Pakistan. The PSLM report shows that in 2011/2012, the net enrollment rate in Punjab for children aged 6-10 increased from 70% in 2010/2011 to 74%, with much of that increase coming from an increase in private school enrollments. This is much more in line with previous increases and is broadly consistent with constant enrollment numbers in public schools. It is 11 percentage points lower than the Nielsen-reported enrollments. Clearly, the Nielsen survey got it wrong. But without publicly available data and sampling files from both Nielsen and PSLM, it is impossible to tell what went wrong at this point.}

Inputs: The story on inputs is clearer. From the Annual Survey of Education Report, Pakistan and the Government’s monitoring database, there have been impressive improvements in school inputs. But are these a deviation from historic trends? Not necessarily. In the LEAPS villages, we documented an equally impressive increase in school infrastructure between 2003 and 2010, with the number of permanent classrooms increasing from 3.89 (per public school) to 5.67, the number of boys’ toilets increasing from 0.71 to 1.25 and the number of girls’ toilets from 0.41 to 1.23. And as in the report, in LEAPS we have also seen declines in teacher absence between 2003 and 2010. The input trend under deliverology looks much like the trend before it. 

Learning: The LEAPS data, which use a rigorous test that is perfectly comparably across years (technically, we rotate some common questions in every year to achieve test comparability), provide consistent data on test-scores for Grade 4 children in 2004, 2006 and 2011. The provisional data (subject to revision) suggest that between 2004 and 2011, Mathematics test-scores in public schools declined by 0.15 standard-deviations, Urdu test-scores increased by 0.07 standard-deviations and English test-scores improved dramatically, by 0.43 standard-deviations. There have been no LEAPS surveys since 2010; we are now trying to go back to the field in 2015. The only other data source is an NGO-run Annual Survey of Education Report, Pakistan, which uses a volunteer army with home-based testing to implement a short test in Urdu, Mathematics and English. This test shows impressive gains in Punjab in just one year. And these gains are very large—taken at face-value they would suggest a half standard-deviation increase in test-scores in just one year. This is an improvement we have never seen before. However, it turns out that the gains in the neighboring province of Khyber Pakhtunwa (where deliverology had not yet started) are even larger, suggesting that the changes reflect differences in something else—perhaps the nature of the sample, the training or the test-instruments.

What can we conclude?

At this point then we simply don’t have data to support the deliverologists’ strong claims. There seem to have been real changes in monitoring and management practices in the Punjab Government, and much more centralized management appears to have been enforced. The evidence is consistent with continued improvements in school inputs, and potentially small changes in enrollment. On learning, we can’t say much at this point. 

Does deliverology deliver?

The lack of credible public data and the lack of third-party evaluations of the program make it difficult to go to bat for the deliverologists at this time. The critical steps needed to understand the program better are independent and public sources of data (including the government’s own monitoring data) and evaluations of what the program is doing in a manner that fully accounts for management costs and increased budgetary allocations. There are broader points about whether deliverology is a fundamentally new approach, or a rediscovery of the economics of incentives and mechanism design. There is also an important discussion on the merits of a centralized system in a population with enormous heterogeneity, but the first step is to credibly understand what deliverology actually delivers. 
 

Science of Delivery in Education

Comments

Submitted by Muhammad Luqman on

The non-availability of credible data on economic conditions and more precisely on social sector, has been obstructing the policy making process in Pakistan. We don't need any model but political will on the part of the government to execute plans for development.

Submitted by Ian Morris on

It is to be expected that there are leads and lags in education enrolments. One big shift was under the social action plan where basic education expenditure doubled in Punjab over 5 years. Expenditure on social sectors (basic education, primary health, population welfare and rural water supplies and sanitation increased from 12% to 24% of total provincial expenditures. It was also under this program that schools began to be built and located according to agreed criteria, and teachers were hired on merit (monitored by the auditor general with Donor funds under the SAP.

Education Reform in Punjab:

Pakistan has the second highest out of school population in the world. Learning levels are low. Successive governments have not allocated sufficient resources in the right places. Successive governments have struggled to take many of the tough political decisions required to tackle a range of barriers to reform. Even in Punjab where indicators are more positive than other parts of Pakistan, there are significant variations between districts, with children, especially girls, in southern districts much more likely to suffer educational deprivation than elsewhere in the province.

Yet there are also some hopeful signs. Across Pakistan, and in Punjab in particular, the low cost private sector is burgeoning, stimulated by demand for better quality education. Education activists are finding innovative ways to reach children in rural areas. And the Government of Punjab has demonstrated a long term commitment to reform, working consistently with DFID, the World Bank and CIDA to strengthen institutions and address issues such as the way teachers are recruited. For all these reasons DFID has committed up to £350 million between 2013 and 2018 to support the transformation of education in Punjab.

The Chief Minister’s Education Reform Roadmap, launched in 2011, is an important element of DFID’s work alongside direct financial aid to government in partnership with the World Bank, and substantial investment in the low cost private sector with an emphasis on reaching the poorest and most marginalised children. The Roadmap is based on monthly data for core indicators such as teacher presence and student attendance, collected through the Programme Management & Implementation Unit, an institution built with the World Bank, DFID and other donor support over the past decade.

The strength of the Roadmap lies in securing political leadership at the most senior levels for delivering free, quality education to all children in Punjab, in line with the 2010 18th Amendment of the Constitution of Pakistan. The Chief Minister has met with us and our partners every two months for the last two years to review progress and set direction, and education has moved up the political agenda in the province. Early signs following this year’s election are that this commitment will not only continue, but that the new government is prepared to take the tough political decisions required to transform Punjab’s schools.

While top down commitment to identifying and tackling binding constraints to reform is crucial, it is essential that this approach is complemented not only by technical assistance and support to implement reform, but also by developing bottom up demand for change. This is why the next phase of the Roadmap work will include new indicators on learning outcomes with the aim not only of tracking improvements in the quality of education offered in Punjab’s schools, but also of equipping parents with information to make informed decisions about their children’s education.

In its first two years, the Roadmap has focused strongly on improving student participation and enrolment. The Programme Management & Implementation Unit has produced monthly data by district for the Roadmap that indicates that participation has increased from a baseline of 82.8% to 92.1% in December 2012. On enrolment, newly released Pakistan Social & Living Standards Measurement data indicates that while net enrolment at primary level remained static between 2007 and 2011, there was a 4% point increase between 2010 and 2011. It is difficult to make firm statements about attribution for the increase, although more enrolments in the private sector are likely to be a major factor, as well as the government’s own reform programme, including the Roadmap.

PSLM is conducted annually and currently the government is not able to release data at pace. For this reason, DFID has commissioned a bi-annual household survey to track enrolment on a more regular basis and produce information more quickly. It is important to note that this survey does not measure net enrolment but does provide regular snapshots on how many more children are attending school and pre-school. It therefore reports how many children are in school, regardless of grade (around 86% of 6-10 year olds) as opposed to the net enrolment rate (NER), which excludes children in preschool and is around 74%. Estimates indicate that gross enrolment in Punjab has increased by between 1,000,000 and 1,500,000 children since the survey began in December 2011.

While this increase, for which attribution is also complex, does not translate directly to progress towards meeting the Millennium Development Goal which is measured using NER, it does indicate that many more children are enrolled in school than before. The challenge now for the government will be to help those children enrolling late to catch up with their peers, and to ensure that preschool, where all are required to enrol when starting school for the first time, provides an excellent foundation for future learning.

At the same time, the Roadmap has helped secure real progress in some important areas. For example, teacher absenteeism in Punjab has declined from a baseline of 19.3% to less than 9% in December 2012. Unauthorised leave has been squeezed to less than 1%, while the difficulties some districts are facing in ensuring teachers are in the classroom every day has opened up a policy discussion with government on the complex and political issue of teacher pay and incentives. Progress in providing toilets, drinking water, electricity and boundary walls to schools accelerated greatly, with the proportion of these facilities available and functioning increasing from 69% to 88% during the first year of the Roadmap. The Roadmap has also made significant progress in improving some of the inputs to learning, particularly textbooks and teacher guides, though as yet there has been no measured impact on learning itself.

DFID is deeply committed to the transformation of education in Punjab. Our range of interventions including the Roadmap is designed to progress this. Our close collaboration with the World Bank is crucial to achieving our ambitions for the province’s children, teachers and schools. And we are keen to see World Bank colleagues working with us more closely to make effective use of the political opportunities that the Roadmap offers to deliver better education for current and future students across Punjab.

Moazzam Malik and Sir Michael Barber

Moazzam Malik is Director, West Asia and Stabilisation Division at the UK's Department for International Development. Sir Michael Barber is DFID’s Special Representative for Education

Add new comment