Gonzalo was part of a panel with David McKenzie at a recent meeting of the Impact Evaluation Network (IEN). One of the questions during this discussion was whether there were good examples of cases where impact evaluations had found null or negative results, and policymakers had actually changed policy as a result. We thought others would be interested in hearing his examples from Mexico.
It feels like a cold water shower when impact evaluations (IEs) do not show positive impacts. Those studies are neither sexy for academic publication nor for public policy use. But the fact that some IEs show no impact of certain programs or projects, it’s an important piece of information!
I would like to suggest here that if a country has an institutional and relatively credible Monitoring and Evaluation System (M&E), the chances of using IEs with no impact increase.
Mexico started evaluating single programs in 1997, but in 2002 the country built an M&E system for social programs. This system was created by law and two important features are a) evaluations are compulsory and b) all evaluations should be public.
The M&E system does not make compulsory the use of the evaluations’ findings, but it is true that government officials have some degree of social pressure because all evaluations are public, both evaluations showing good and bad results.
Here are three examples, where IEs didn’t show good results, but where policy makers used this information. In two cases the social programs were cancelled. In the last example, the program was improved due to the evaluation findings.
Estrategia Microrregiones
The Mexican government has developed several approaches to mitigate regional inequalities within the country prioritizing territories with the highest deprivation conditions. One of those strategies was Estrategia Microregiones (EM) launched in 2001 to integrate public policy to support actions on infrastructure and services in order to achieve a minimum wellbeing level. The EM was implemented in 250 regions across 476 municipalities with high marginalization levels, measured by the Marginalization Index in 2000, distributed in 17 states. The strategy sought to assist 5.5 million people, 3.4 million considered indigenous people.
In order to measure the effectiveness of the EM, an impact evaluation was carried out in 2007; the purpose was to estimate the impact on the absolute marginalization Index of the treated communities. In general, there was no impact on the absolute marginalization index, and at the component level, no impact was identified on 5 of the 7 estimated components of the Marginalization Index, including illiteracy, individuals without primary education, households without drainage or exclusive sanitary service, households without electricity and overcrowding. For households without piped water and with ground floor, although the results suggested a positive impact of the EM, those were not clear or conclusive.
After this evidence, in 2007, the strategy was replaced by another Strategy (Estrategia 100x100) implemented during the period 2006-2012. This intervention also used a territorial approach but now focused on the 125 municipalities with the lowest Human Development Index.
In this case, it was relatively easy to stop the Microrregiones strategy because the one who made this decision was not the President who designed it, but the new (elected) President. The evaluation findings were presented to the new elected President’s team and they decided to have a new strategy to reduce poverty in rural poor areas.
Programa Primer Empleo
The Programa Primer Empleo (PPE) was a federal program implemented by the Mexican Social Security Institute (IMSS) in 2007 to subsidize private demand for formal employment for new entry workers by a worker subsidy -employer quotas that employers pay when they register a new employee to the Institute.
The idea was that subsidizing non-wage costs could encourage the demand of private firms for formal jobs for new entry workers, which traditionally cannot access formal employment.
CONEVAL, the evaluation institution, decided to have both an impact and a design evaluation for this important program. A Design Evaluation of the PPE in 2007 found that a minimum proportion of the population defined to attend had effectively enrolled in the program and there was not a clear definition of target population. This resulted in questions about whether it was appropriate to maintain the intervention given another efforts already taking place in the Minister of Labor and Social Provision.
Given these results, CONEVAL decided not to carry out the impact evaluation for this program anymore; as a cheaper evaluation tool had showed that the program had important problems.
In 2008 the target population was redefined. However, the budget of the program in the following years was reduced and eventually in 2010 no budget was assigned to the program, and eventually it disappeared.
In this case, it was the same government who designed the program which was the one who stopped it. The evidence both from the design evaluation and from administrative records, published every year and presented to the Under-minister of Finance, was clear that the program needed important changes, which implied big changes. It’s also true that the financial crisis of 2008-2009 helped to make this decision. Public finances were tight and those resources were needed somewhere else.
Programa Escuelas de Calidad
The Programa Escuelas de Calidad (PEC) is an initiative of the Federal Government whose implementation began in 2001 in order to improve educational quality through establishing a school-centered management model. The intervention consisted of a grant to the schools applying, and encouraging them to diversify their funding sources through donations from local governments and the school community, including parents.
According to an impact evaluation on PEC carried out by Shapiro and Skoufias in 2006, effects were found in the decrease of the failure rate (0.31%) and dropout rates (0.24%), however, these results are not maintained for indigenous schools. Another study by Murnane, Willett and Cardenas, also in 2006, raises the hypothesis that states have different capacities for the development of PEC; classified in three groups based on their Human Development Index (HDI), the results showed that states with a medium and high HDI had a reduction in dropout at the school level of up to 0.33 percentage points after one year in the program, but the impact on states with a low HDI was not significantly different from zero. In addition, a study conducted in 2016 by CONEVAL using public information and administrative data shows no impact of PEC on the ENLACE test over the 2008-2013 period for schools classified as highly and very highly marginalized, while in the full sample of schools there was a positive effect.
The evidence has encouraged efforts to improve the targeting by adjusting the selection criteria to support the schools with the highest deprivation level. In 2015, this program was included as a component of a broader program integrating several types of support to improve school management, basic infrastructure and digital technologies access.
There were two reasons why the government made efforts to have a better targeting for this program precisely in 2015. The first one was that multidimensional poverty (which included access to basic schooling) increased between 2010 and 2014, according to CONEVAL figures. This result made the government to design a coordinated strategy in all social ministries to reduce poverty in all dimensions. One of these steps was to choose budgets to improve potentially good programs, such as PEC, but also to reduce the budget of programs with no evidence enough to have an impact on poverty.
The second reason was that in 2013 and 2014 CONEVAL produced a “Social Programs Registry” saying that the federal government where handling almost 270 social programs in an un-coordinated way and we should have less and more effective programs. The PEC program was transformed and now it is part of a bigger one. We eventually need to evaluate if this change was a good idea!
It’s clear that evaluations should be used more often by policy makers. But it’s also true that evaluations are only one element of the decision-making process in public policy. We’ll never witness that everything said by evaluators will be addressed by governments, but at least we avoid alternative facts whenever evaluators and politicians meet over rigorous IEs.
Gonzalo Hernández Licona is Executive Secretary of the National Council for the Evaluation of Social Development Policy (CONEVAL) in Mexico.
Join the Conversation