I’ve read several research proposals in the past few months, as well engaged in discussions, that touch on the same question: how to use the spatial variation in a program’s intensity to evaluate its causal impact. Since these proposals and conversations all mentioned the same fairly recent paper by Markus Frolich and Michael Lechner, I eagerly sat down to read it. In this paper the authors use the regional treatment intensity of Swiss labor market programs to estimate the causal effect of state provisioned training for unemployed workers. This is a great paper and I learned interesting extensions to standard IE methods. Unfortunately their approach, though, is not generalizable to many of the settings that I read about in the proposals. Let me explain.
Frolich and Lechner want to determine the effectiveness of Swiss labor training programs for the subsequent employment and earnings of the unemployed Swiss workers who enroll in the program. The authors take advantage of widespread regional variation in program participation as the source of causal identification. Now in general it is quite common for participation rates to vary across regions or other relevant geo-spatial units – for example poorer areas may have higher rates of participation in anti-poverty programs.
Can we correlate the spatial variation in participation with the spatial variation in the outcomes we care about in order the assess program effectiveness? The ready answer is no because the variation in participation doesn’t satisfy the requirements of a valid instrument: (a) the variation may not be due to exogenous factors so, as a consequence, (b) the characteristics of the local area partly determine the outcomes that the program hopes to influence and hence the variation in program participation doesn’t satisfy the necessary “exclusion restriction” – the spatial variation in program participation is related to the outcomes through many possible channels aside from the effects of the program.
However, Frolich and Lechner note two unique characteristics of their setting:
To operationalize the analysis the authors need to define local labor markets and then search for such markets that are bisected by Cantonal boundaries. So they look for clusters of regional employment offices (REOs, the entities tasked with supervising unemployed workers) that are spread over at least two Cantons. They further narrow their search according to other factors that would suggest a common local labor market, such as whether the two Cantons share a common language and whether the REOs are within no more than 30 minutes of each other by car. This process identified 18 local labor markets around Switzerland with different training quotas on either side of the Cantonal border.
This approach is essentially the recognition of a series of natural experiments created by the rules of the Swiss federal government, akin to the famous analysis of minimum wage effects on teen employment by Card and Krueger that compared bordering areas of New Jersey and Pennsylvania. Only here in the Swiss case there are 18 experiments, not one, so Frolich and Lechner average across these experiments in order to estimate an average treatment effect.
They find that the receipt of training significantly increases the likelihood of employment. Over the 12-month period following training, a program recipient works two months more than a non-recipient – in the world of job training programs this is a rather large effect. There may also be an increase in earnings (per month worked) though this effect is not precisely estimated.
A couple issues of note.
This technical paper is a great example of how to leverage spatial variation in treatment intensities to estimate program impacts. But note the fundamental requirements of (a) exogeniety of treatment intensity with respect to the local area and/or (b) a valid exclusion restriction. Many of the proposals I reviewed failed one or both of these requirements. But without these features there is no way to sensibly use the regional differences in program intensity.
Frolich and Lechner want to determine the effectiveness of Swiss labor training programs for the subsequent employment and earnings of the unemployed Swiss workers who enroll in the program. The authors take advantage of widespread regional variation in program participation as the source of causal identification. Now in general it is quite common for participation rates to vary across regions or other relevant geo-spatial units – for example poorer areas may have higher rates of participation in anti-poverty programs.
Can we correlate the spatial variation in participation with the spatial variation in the outcomes we care about in order the assess program effectiveness? The ready answer is no because the variation in participation doesn’t satisfy the requirements of a valid instrument: (a) the variation may not be due to exogenous factors so, as a consequence, (b) the characteristics of the local area partly determine the outcomes that the program hopes to influence and hence the variation in program participation doesn’t satisfy the necessary “exclusion restriction” – the spatial variation in program participation is related to the outcomes through many possible channels aside from the effects of the program.
However, Frolich and Lechner note two unique characteristics of their setting:
1. The Swiss federal government in 1997 mandated district specific quotas for the number of unemployed workers that should be trained in calendar year 1998. The quotas were determined in part by the population of the district, known as the Canton, and in part by the Canton’s unemployment rate. The employment offices in Cantons with a high quota relative to the number of local unemployed assign and/or encourage a greater proportion of the unemployed to attend training programs. And indeed the spatial variation in program intensity induced by the quota system is substantial – it ranges from 12 to 28% of the unemployed within the Canton. Now, as stated just above, this variation per se is not a valid instrument since it is in part determined by previous unemployment rates that in turn are surely related to employment prospects in 1998. However the authors note a second key characteristic …
2. … the Cantonal borders often bisect local labor markets. In many localities in Switzerland (as well as elsewhere) the relevant labor market for workers crosses several administrative boundaries. It is this fact that generates the counterfactual because workers on both sides of the border face the same employment prospects even though they face different training program intensities (as a result of different quotas). If we accept the common labor market argument than the regional variation in participation can be used as a valid instrument.
2. … the Cantonal borders often bisect local labor markets. In many localities in Switzerland (as well as elsewhere) the relevant labor market for workers crosses several administrative boundaries. It is this fact that generates the counterfactual because workers on both sides of the border face the same employment prospects even though they face different training program intensities (as a result of different quotas). If we accept the common labor market argument than the regional variation in participation can be used as a valid instrument.
To operationalize the analysis the authors need to define local labor markets and then search for such markets that are bisected by Cantonal boundaries. So they look for clusters of regional employment offices (REOs, the entities tasked with supervising unemployed workers) that are spread over at least two Cantons. They further narrow their search according to other factors that would suggest a common local labor market, such as whether the two Cantons share a common language and whether the REOs are within no more than 30 minutes of each other by car. This process identified 18 local labor markets around Switzerland with different training quotas on either side of the Cantonal border.
This approach is essentially the recognition of a series of natural experiments created by the rules of the Swiss federal government, akin to the famous analysis of minimum wage effects on teen employment by Card and Krueger that compared bordering areas of New Jersey and Pennsylvania. Only here in the Swiss case there are 18 experiments, not one, so Frolich and Lechner average across these experiments in order to estimate an average treatment effect.
They find that the receipt of training significantly increases the likelihood of employment. Over the 12-month period following training, a program recipient works two months more than a non-recipient – in the world of job training programs this is a rather large effect. There may also be an increase in earnings (per month worked) though this effect is not precisely estimated.
A couple issues of note.
First, what is explicitly identified is the local average treatment effect estimated off of the compliers – those workers induced into participation as a result of the exogenous variation introduced by the federal quotas. There are other groups of unemployed workers – the always-participants – those who would’ve taken up training no matter what – and the never-participants – those who would never take up training in either scenario. The estimated treatment effect only applies to the first group mentioned – the subpopulation responsive to the differences in training probability as a result of differing quotas.
Second, the authors not only report the IV estimate but the Conditional IV estimated (CIV) (a method first presented in this 2007 paper by Frolich) that, by conditioning on worker observables, is better able to balance the characteristics of the workers used to estimate the treatment effect. The CIV must make additional assumptions beyond the standard IV case, such as the complete absence of defiers (workers who choose not to participate in areas with high quotas but who would participate in training in low quota areas), which appear reasonable in this context. The impact estimates are slightly greater in magnitude for the CIV than the IV, suggesting that the characteristics of unemployed on either side of the Canton border are slightly different on average and this difference should be accounted for where possible.
Second, the authors not only report the IV estimate but the Conditional IV estimated (CIV) (a method first presented in this 2007 paper by Frolich) that, by conditioning on worker observables, is better able to balance the characteristics of the workers used to estimate the treatment effect. The CIV must make additional assumptions beyond the standard IV case, such as the complete absence of defiers (workers who choose not to participate in areas with high quotas but who would participate in training in low quota areas), which appear reasonable in this context. The impact estimates are slightly greater in magnitude for the CIV than the IV, suggesting that the characteristics of unemployed on either side of the Canton border are slightly different on average and this difference should be accounted for where possible.
This technical paper is a great example of how to leverage spatial variation in treatment intensities to estimate program impacts. But note the fundamental requirements of (a) exogeniety of treatment intensity with respect to the local area and/or (b) a valid exclusion restriction. Many of the proposals I reviewed failed one or both of these requirements. But without these features there is no way to sensibly use the regional differences in program intensity.
Join the Conversation