What’s new in education research? Impact evaluations and measurement – January 2018 round-up

Here is a selected round-up of recent research on education in low- and middle-income countries, with a few findings from high-income countries that I found relevant. This is mostly but not entirely from the “economics of education” literature. If I’m missing recent articles that you’ve found useful, please add them in the comments!

What is education good for?

• Education saves lives, but only some of them! “Education is strongly associated with better health and longer lives.” But is that mere correlation, or is a causal link? It depends! This review finds no impact on obesity, inconsistent impact on smoking, and “an effect of education on mortality exists in some contexts but not in others, and seems to depend on (i) gender; (ii) the labor market returns to education; (iii) the quality of education; and (iv) whether education affects the quality of individuals’ peers” (Galama, Lleras-Muney, and van Kippersluis).

Quality of education!

•  “The largest globally comparable panel database of education quality. The database includes 163 countries and regions over 1965–2015. The globally comparable achievement outcomes were constructed by linking standardized, psychometrically-robust international and regional achievement tests” (Altinok, Angrist, and Patrinos). [Blog post]
• Girls are taking down boys on school enrollment in rich (and some middle-income) countries. “The dramatic expansion of enrollment in education worldwide over the past decades was accompanied by a … striking phenomenon. As individuals acquired more schooling over time, females not only caught up with males, but surpassed them in educational attainment, a drastic shift from the historical standard. This phenomenon, sometimes referred as the gender gap reversal in education, is widespread globally.” The authors test a hypothesis to explain this phenomenon (Bossavie and Kanninen). 
• “We draw upon evidence from the Service Delivery Indicators program—an ongoing Africa-wide program with the aim of collecting informative and standardized measures of what primary teachers know, what they do, and what they have to work with” (Bold et al.; more detail working paper) [I wrote about this paper here.]
• Many African countries participate in one of two regional assessments. Integrating those two tests into a common ranking shows “considerable variance when comparing three commonly-used equating methods, suggesting precise country rankings are unreliable.” And yet: “Learning levels in this sample of African countries are consistently…significantly lower than predicted by African per capita GDP levels; and…converging slowly, if at all, to the rest of the world during the 2000s” (Sandefur).

But wait, maybe we know less than we think we know about the quality of education!

• International tests, which tend to be low-stakes, rely on intrinsic motivation of students. In other words, how much do they care about doing well in a test that doesn’t matter to them personally? In Singapore, they seem to care a lot: Adding extrinsic incentives didn’t change outcomes. In the USA, they don’t care so much: Adding extrinsic incentives improved performance substantially. “We estimate that increasing student effort on the test itself would improve U.S. mathematics performance by 22 − 24 points, equivalent to moving the U.S. from 36th to 19th in the 2012 international mathematics rankings” (Gneezy et al.). I wonder how much intrinsic test motivation varies between Burundi and Senegal?
• In Tanzania, “estimates of the returns to education vary by questionnaire design…varying from higher returns of 5 percentage points among the most well educated men to 16 percentage points among the least well educated women” (Serneels, Beegle, and Dillon).

Closing the gender gaps at home, not just at school!

• “Using PISA test scores from 11,527 second-generation immigrants coming from 35 different countries of ancestry and living in 9 host countries, we find that the positive effects of country-of-ancestry gender social norms on girls’ math test scores relative to those of boys expand to other subjects (namely reading and science). … Gender norms do not seem to particularly influence math-related stereotypes, but instead, preferences for math (Rodríguez-Planas and Nollenberger).
• What drives gender gaps in math achievement? In Ecuador, there are “steep socioeconomic gradients and a substantial boy-girl gap in math test scores. However, among children of mothers with university education, there is no difference in the math achievement of girls and boys, which suggests that maternal education specifically, and home environments generally, are important” (Carneiro, Cruz-Aguayo, and Schady).
• “High school math and science teacher gender affects student interest and self-efficacy in STEM” in the US. But why? “Such effects become insignificant once teacher behaviors and attitudes are taken into account” (Sansone).

You know what else you can do at home? Improve parent literacy

• In India, “Households were assigned to receive either adult literacy (language and math) classes for mothers, training for mothers on how to enhance their children's learning at home, or a combination of the two programs. All three interventions had significant but modest impacts on children’s math scores. The interventions also increased mothers' test scores in both language and math” (Banerji, Berry, and Shotland). [Blog post]

How do we get from pilot experiments to scale?

• “The promise of randomized controlled trials is that evidence gathered through the evaluation of a specific program helps us—possibly after several rounds of fine-tuning and multiple replications in different contexts—to inform policy.” This paper lays out 6 key challenges to scale-up and then “describes the journey from the original concept to the design and evaluation of scalable policy. We do so by evaluating a series of strategies that aim to integrate the nongovernment organization Pratham's "Teaching at the Right Level" methodology into elementary schools in India. The methodology consists of reorganizing instruction based on children's actual learning levels, rather than on a prescribed syllabus, and has previously been shown to be very effective when properly implemented” (Banerjee et al.).

Education technology!

• Can tablets help instructional coaches to do their jobs? In Kenya, results of a survey around the Tusome tablets program show “high levels of tablet program utilization, increased accountability, and improvements in learning outcomes” (Piper et al.).
• When textbooks were replaced with laptops (with textbook content on them) in Honduras, there was no impact on student learning. Laptops had higher fixed costs but lower marginal costs, so cost-effectiveness is in favor of laptops only if enough textbooks are replaced (Bando et al.).  

Early learning!

• What do we learn from early learning programs in Ethiopia, Kenya, Liberia, Malawi, Tanzania, and Uganda? “In several countries with completed impact evaluations, there are significant and important learning gains of between 0.2 and 2.57 SD in effect size; in one case the percentage of students reaching grade-level reading proficiency increased from 12% to 47%” (Gove et al.).
• Reviewing 18 early grade reading interventions, “the study finds that early grade reading interventions are consistently effective, although not infallible. The large majority had highly significant impacts on at least one reading subtask. However, only for a few interventions were effect sizes large enough to equate to more than a year’s worth of schooling or create fluent readers on average” (Graham and Kelly).

Early child education!

• Introducing preschools in Mozambique had important impacts: “Children who attended preschool experienced gains in cognitive development, communication, fine motor skills, and socio-emotional skills, scoring 0.33 standard deviations higher on a child development screening test. Preschoolers were 21 percentage points more likely to be enrolled in primary school” (Martinez, Naudeau, & Pereira)

Teachers!

• Why are there such large pupil-teacher ratio disparities? In Malawi, “most teachers [are]  concentrated near commercial centers and in rural schools with better amenities. … Political economy network mapping reveals that teachers leverage informal networks and political patronage to resist placement in remote schools, while administrative officials are unable to stand up to these formal and informal pressures, in part because of a lack of reliable databases and objective criteria for the allocation of teachers” (Asim et al.). More data for decisionmakers, please! [Blog post] This reminds me of Pugatch and Schroeder’s work on the impact of hardship allowances for teachers in the Gambia.
• In-service teacher training in Georgia (the country, not the U.S. state) increased student learning (Ome, Menendez, and Le).
• But in South Africa, coaching had a much bigger impact than training and was more cost effective, despite costing more (Cilliers et al.).
• In Argentina, a “diagnostic feedback” intervention in which “standardized tests were administered in math and reading comprehension at baseline and two follow-ups and the results were made available to the schools through user-friendly reports” increased math scores by 0.28-0.34 SDs and reading scores by 0.36-0.38 SDs. Adding “capacity-building” workshops did not help (de Hoyos, Ganimian, & Holland). [Blog post]

School resources!

• When schools in Jakarta (Indonesia) had a performance component – based on student test scores – added to their grant, inequality in primary school outcomes narrowed, but both by improvements in the lowest performers and lower outcomes among the highest performers. (I’m not sure if they can rule out mean reversion.) But junior secondary schools showed an unambiguous increase (Al-Samarrai et al.).

Cash transfers!

• “We meta-analyze for impact and cost-effectiveness 94 studies from 47 conditional cash transfer programs in low- and middle-income countries worldwide. … Educational effect size estimates from long-standing national CCT programs are statistically indistinguishable from those of pilot 1- or 2-year CCT programs. We also do not find evidence consistent with stronger effects for transfers that target mothers. … For some educational outcomes, we find that effect sizes are greater when other schooling conditions, such as grade promotion or test scores, are imposed on beneficiaries, beyond the typical requirements of enrollment and minimum attendance” (García and Saavedra; open access working paper)
• In the Philippines, a cash transfer that didn’t cover the full cost of schooling led to more schooling but also to more child work: Kids worked more to cover the rest of their schooling costs (De Hoop et al.).
• “After a years’ worth of transfers, we find the Malawi SCTP [Social Cash Transfer Program] both improves enrollment rates and decreases dropouts. The main intervening pathway between the program and schooling is education expenditures, suggesting that the cash improves the demand for education by reducing financial constraints” (Kilburn et al.).

Private schools and vouchers!

• Why do parents send their children to “low-cost private schools”? In Kenya, “parents who chose LCPS for their children were more driven by quality concerns than were public school parents. … Despite being termed ‘low cost’, the fees charged by schools primarily serving the poor were often a heavy burden on families” (Zuilkowski et al.).
• Increasing the value of vouchers for low-income students in Chile has the result that “student test scores increased markedly and income-based gaps in those scores declined by one-third in the five years after the passage.” But why? “The combination of increased support of schools and accountability was the critical mechanism through which the implementation of SEP increased student scores, especially in schools serving high concentrations of low-income students” (Murnane et al.).
• Yet in the USA, new evidence from the state of Louisiana suggests that a school voucher program actually reduced student performance in math, reading, science, and social studies. “These effects may be due in part to selection of low-quality private schools into the program” (Abdulkadiroglu, Pathak, and Walters).

School management!

• Do school superintendents matter? In Israel, “we exploit a quasi-random matching of superintendent and schools, and estimate that superintendent value added has positive and significant effects. … One standard deviation improvement in superintendent value added increases test scores by about 0.04 of a standard deviation in the test score distribution.” That’s not particularly big, but it “increases sharply for superintendents in the highest-quartile of the value added distribution, and is larger for female superintendents” (Lavy and Boiko).

Don’t forget about access to schooling!

• A program in Haiti that “provided public financing to nonpublic schools conditional on not charging tuition,” resulting in “more students enrolled, more staff, and slightly higher student-teacher ratios. The program also reduces grade repetition and the share of overage students” (Adelman, Holland, and Heidelk; ungated working paper)

You can previous education-focused research round-ups below:

• How can education systems be better? A round-up of the 2017 RISE conference
• The Latest Quantitative Research on Education in South Africa (and What It Tells Us about the Rest of the World)
• What’s new in education research? Impact evaluations and measurement – March round-up (2017)
• What’s new in education research? Impact evaluations and measurement – October round-up (2016)
• What’s new in education research? Impact evaluations and measurement – July round-up (2016)
• What does research on education systems look like? A run-down of the RISE 2016 conference