Policymakers need to focus on the quality and not just the quantity of services delivered

What key insights have emerged from development economics in the past decade, and how should they impact the work of the World Bank? A new working paper Toward Successful Development Policies: Insights from Research in Development Economics from the Bank’s research department captures 13 of the most significant insights in the world of development economics.

Here’s insight #2 on the importance of focusing interventions on improving the quality – and not just the quantity – of services provided in health and education. See all previous insights here: Thirteen insights for successful development policies

Education and health policymakers often focus on indicators of the quantity of services provided: Are children enrolled in school? Are women delivering their babies in a health facility? Are newborn babies receiving postnatal care? This assumes that children in school will automatically learn, and that health outcomes such as maternal and child mortality will automatically improve as service coverage increases.

This assumption often is wrong. Even in countries on track to hit enrollment targets, children often have low levels of mastery of reading, writing and mathematics. Indonesia and Mexico, for example, had both almost reached the universal primary completion MDG target by the mid-2000s, but at the time 68 percent of Indonesian youth and 50 percent of Mexican youth lacked even minimally adequate competence in mathematics. ^ref1 In health, evidence shows a similar tenuous link between service coverage and outcomes. For example, having women deliver their babies in a health facility (an MDG and SDG target) has been found not to lead to lower maternal or neonatal mortality rates. ^ref2

In both education and health, poor quality of service delivery is the key reason why service coverage does not necessarily translate into better outcomes. We know this because research – much of it done at the World Bank – has zeroed in on the quality of service delivery.

Early research focused on a very basic aspect of quality: are teachers and health workers actually teaching and delivering health services? Random unannounced visits to schools and health centers in six developing countries uncovered high absenteeism rates – 19% in the case of teachers and as much as 35% in the case of health workers. ^ref3 Averaging across seven African countries, more recent research found that 23% of teachers were absent from school, and 44% were absent from the classroom. ^ref4 Absenteeism – coupled with teachers spending time in class but not teaching – resulted in, on average, just 2¾ hours of teaching time per day instead of the scheduled 5½ hours.

Later research looked at another aspect of the quality of education and health services – provider knowledge. The study of seven African countries also tested primary school teachers’ knowledge of their subject by asking them to grade mock student tests in language and mathematics. Teachers did well enough on spelling and simple grammar, with two-thirds grading at least 80% of the questions correctly.

But they did much worse in other areas, averaging 44% on vocabulary and comprehension, and just 25% on composition. In mathematics, the results also were worse the more complex the task. On average, 91% of teachers could add double digits, but only 68% could multiply double digits. Less than one-third could understand a Venn diagram, and only 11% could interpret data on a graph.

Results of studies testing health provider knowledge are similarly sobering. The approach is similar to that used with teachers – health providers are asked to “treat” a hypothetical patient. These “vignette” exercises have been done for a variety of medical conditions including child pneumonia, diabetes and tuberculosis. A study of nine African countries painted a rather bleak picture. On the tuberculosis vignette, 83% of health providers got the correct diagnosis, but only 54% prescribed the correct treatment. On the pneumonia vignette, 70% prescribed the correct treatment, but on the diabetes vignette, only 52% did.

Having providers at their workplace and having them know their subject are necessary conditions for high- quality service delivery, but recent research shows that even this is not sufficient. This research shows that there is often a gap – sometimes a large one – between what service providers know in terms of their mastery of their subject and how well they perform on the job.

In the case of education, teacher performance has been assessed by looking at teachers’ pedagogical knowledge (e.g., comprehension of factual text and formulating aims and learning outcomes), their ability to assess students (e.g., formulating questions to check understanding), and their success in applying their knowledge in the classroom (e.g., introducing and summarizing the topic of a lesson). On these criteria, primary school teachers across the same seven African countries performed poorly. They scored on average just 23% on an exercise gauging their ability to formulate aims and learning outcomes and just 23% on an exercise gauging their ability to formulate questions to check students’ understanding. And when observed in their class, only 41% of teachers introduced and summarized the topic of the lesson.

In health, too, there have been studies employing the direct observation approach – observing real-life interactions between patients and health workers. But these have their drawbacks. Providers may perform differently when observed. And while in education interactions between teacher and students are limited to just a few age groups and just a few subjects, a health provider in a clinic could be dealing with any number of medical conditions. An alternative approach is to have actors pretend to be a real patient. The provider consents beforehand to participating in the study, knowing that at some stage during the study period an actor may show up pretending to be a patient, but not when. To get around the fact actors can’t be subjected to invasive exams, in some studies the actor pretends to be the parent of a sick child who is at home, with, for example, diarrhea or symptoms of pneumonia. ^ref5 Providers mostly did better in the vignettes than they did when confronted with an actor, suggesting a “know-do gap”: in the case of diarrhea, 74% got the right diagnosis in the vignette but only 3% did in the real-life interaction with the actor; 3% recommended the correct treatment in the vignette but none recommended the correct treatment in the real-life interaction.

What to do to improve service delivery quality? One rather sobering finding from one health study ^ref6 is that while providers in better-equipped facilities asked more questions and did more tests, they were not more likely to recommend the correct treatment. Another health study ^ref7 found that providers with dual public-private practices performed better in their private practice than in their public practice, suggesting financial incentives may play a role – a theme explored in a separate blog post in this series.

The above assumes that poor service quality is indeed the reason why getting children into school does not automatically ensure learning and why getting patients into health facilities does not automatically ensure better health. Studies support this hypothesis. One study ^ref8 found that going from having a teacher in the bottom quality decile to one in the top is equivalent to a full additional business-as-usual year of learning for students. Another study ^ref9 compared mortality in high-income countries and mortality in low- and middle-income countries (LMICs), and concluded that of the 8.6 million excess deaths in LMICs that were amenable to medical care, the bulk (5.0 million) were due to receipt of poor-quality care rather than non-receipt of care.