Most experiments in development economics involve giving the treatment group something they want (e.g. cash, health care, schooling for their kids) or at least offering something they might want and can choose whether or not to take up (e.g. business training, financial education). Indeed among the most common justifications for randomization is that there is not enough of the treatment for everyone who wants it, leading to oversubscription or randomized phase-in designs.
However, when it comes to issues like taxation and regulatory enforcement, the typical government intervention is something that nobody privately wants (e.g. a tax audit, a health and safety inspection, a traffic fine) but is thought to be socially or publicly desirable (we want others to pay their taxes, prepare our food safely, not drive like lunatics). This raises both ethical and logistical challenges for trying to evaluate these types of interventions in an experimental setting. I don’t know of any papers which have dealt with this issue, so thought I would share some experiences from a recent experiment (working paper, 2-page summary) trying to formalize informal firms in Belo Horizonte, Brazil.
The Intervention
Together with Miriam Bruhn, I worked with the State Government of Minas Gerais to test several different policy approaches designed to get larger informal firms (monthly profits averaging US$1000) to register for tax purposes. Two of the treatments were the standard “give them stuff” types of treatment discussed above, giving the firm owners information and, in some cases, free registration and the use of an accountant. But in addition to these “carrots”, the government wanted to test the effectiveness of better enforcement, by sending municipal inspectors to some firms to check their registration status, and if they weren’t formal, tell them to formalize, and in theory fine them and ultimately shut them down if they didn’t register. You can imagine that if we followed a standard approach of asking firms if they wanted to participate in this study, with a much higher chance of getting inspected if they agree, we wouldn’t have many participants!
The Ethical and Logistical Issues
The first challenge we faced is that we lacked a sample frame of informal firms – after all, these were firms which hadn’t registered their businesses. The standard approach would be to then do a screening survey, asking firms if they were informal, and if so, conducting a baseline survey of them before allocating them to treatment. This was the approach I used with Suresh de Mel and Chris Woodruff in an experiment on formalizing firms in Sri Lanka, where firms only got carrots/incentives to register, but no sticks. But in the Brazilian case, there were two issues with doing this:
First, ethically researchers cannot force firms to answer a survey, and if firms were not told that the purpose of the survey would be to determine whether an inspector was likely to be assigned to them they would most likely refuse to answer, while if they weren’t told this information, this would be deceiving them and unethical. Second, governments on the other hand can legally require firms (and individuals) to provide information, and so the government could ask firms their legal status and require a baseline survey if they weren’t formal. But then if firms linked this survey to the inspection, they would be likely very hard to get to do a follow-up survey, and even if the government forced them, one might doubt the reliability of the answers given.
Our solution
Our solution was to use a listing survey, where enumerators walked through randomly selected census blocks, and noted the details of each business that were observable from outside the business (name of the business, business type, approximate size of the premises, approximate number of employees), without ever speaking to the business owner. Then this information was matched against government records of registered firms to remove firms who were definitely formal, leaving a sample of possibly informal firms, who could then be assigned to the different treatments or the control group.
The government then sent municipal inspectors to the randomly selected firms, and 9 months later a follow-up survey was taken, which followed standard procedures of firm owners being able to decide whether or not to take part in the survey. Since the survey was removed in time from the intervention, and wasn’t preceded by a baseline survey, it was not linked in the firm owner’s minds to getting the inspection happen, and we saw no difference in response rates between inspector treatment firms and control firms. We also had administrative data on registrations for all firms, which could be used to measure the registration response of firms to this treatment.
We find that an additional inspection results in a 21-27 percent increase in municipal registration, and that the costs of the added inspections are more than covered by the tax liabilities of the firms over the first couple of years of being formal. So from a public finance point of view, this is a treatment that is socially beneficial, even though it is not one that any private firm would volunteer for.
Alternative approaches
The only related experimental work I am aware of uses administrative data and the fact that for fairness reasons, governments often use a (risk-adjusted) random inspection process. In a paper in Science last year, David Levine and co-authors look at the impact of workplace safety inspections in California on worker injury rates and injury costs. They use the fact that OSHA randomly selected which workplaces in high-injury industries they inspected, and then match on observables to construct a control group of very similar facilities that were eligible for randomized inspections but not selected. Injury data were then available from the workers’ compensation system, and they find that the inspections reduce injury rates and injury costs, but don’t affect employment or sales.
Am I missing examples of privately bad, but socially good treatments from other experiments? If so, let me know, since it would be useful to learn from the experiences of how these get implemented.
However, when it comes to issues like taxation and regulatory enforcement, the typical government intervention is something that nobody privately wants (e.g. a tax audit, a health and safety inspection, a traffic fine) but is thought to be socially or publicly desirable (we want others to pay their taxes, prepare our food safely, not drive like lunatics). This raises both ethical and logistical challenges for trying to evaluate these types of interventions in an experimental setting. I don’t know of any papers which have dealt with this issue, so thought I would share some experiences from a recent experiment (working paper, 2-page summary) trying to formalize informal firms in Belo Horizonte, Brazil.
The Intervention
Together with Miriam Bruhn, I worked with the State Government of Minas Gerais to test several different policy approaches designed to get larger informal firms (monthly profits averaging US$1000) to register for tax purposes. Two of the treatments were the standard “give them stuff” types of treatment discussed above, giving the firm owners information and, in some cases, free registration and the use of an accountant. But in addition to these “carrots”, the government wanted to test the effectiveness of better enforcement, by sending municipal inspectors to some firms to check their registration status, and if they weren’t formal, tell them to formalize, and in theory fine them and ultimately shut them down if they didn’t register. You can imagine that if we followed a standard approach of asking firms if they wanted to participate in this study, with a much higher chance of getting inspected if they agree, we wouldn’t have many participants!
The Ethical and Logistical Issues
The first challenge we faced is that we lacked a sample frame of informal firms – after all, these were firms which hadn’t registered their businesses. The standard approach would be to then do a screening survey, asking firms if they were informal, and if so, conducting a baseline survey of them before allocating them to treatment. This was the approach I used with Suresh de Mel and Chris Woodruff in an experiment on formalizing firms in Sri Lanka, where firms only got carrots/incentives to register, but no sticks. But in the Brazilian case, there were two issues with doing this:
First, ethically researchers cannot force firms to answer a survey, and if firms were not told that the purpose of the survey would be to determine whether an inspector was likely to be assigned to them they would most likely refuse to answer, while if they weren’t told this information, this would be deceiving them and unethical. Second, governments on the other hand can legally require firms (and individuals) to provide information, and so the government could ask firms their legal status and require a baseline survey if they weren’t formal. But then if firms linked this survey to the inspection, they would be likely very hard to get to do a follow-up survey, and even if the government forced them, one might doubt the reliability of the answers given.
Our solution
Our solution was to use a listing survey, where enumerators walked through randomly selected census blocks, and noted the details of each business that were observable from outside the business (name of the business, business type, approximate size of the premises, approximate number of employees), without ever speaking to the business owner. Then this information was matched against government records of registered firms to remove firms who were definitely formal, leaving a sample of possibly informal firms, who could then be assigned to the different treatments or the control group.
The government then sent municipal inspectors to the randomly selected firms, and 9 months later a follow-up survey was taken, which followed standard procedures of firm owners being able to decide whether or not to take part in the survey. Since the survey was removed in time from the intervention, and wasn’t preceded by a baseline survey, it was not linked in the firm owner’s minds to getting the inspection happen, and we saw no difference in response rates between inspector treatment firms and control firms. We also had administrative data on registrations for all firms, which could be used to measure the registration response of firms to this treatment.
We find that an additional inspection results in a 21-27 percent increase in municipal registration, and that the costs of the added inspections are more than covered by the tax liabilities of the firms over the first couple of years of being formal. So from a public finance point of view, this is a treatment that is socially beneficial, even though it is not one that any private firm would volunteer for.
Alternative approaches
The only related experimental work I am aware of uses administrative data and the fact that for fairness reasons, governments often use a (risk-adjusted) random inspection process. In a paper in Science last year, David Levine and co-authors look at the impact of workplace safety inspections in California on worker injury rates and injury costs. They use the fact that OSHA randomly selected which workplaces in high-injury industries they inspected, and then match on observables to construct a control group of very similar facilities that were eligible for randomized inspections but not selected. Injury data were then available from the workers’ compensation system, and they find that the inspections reduce injury rates and injury costs, but don’t affect employment or sales.
Am I missing examples of privately bad, but socially good treatments from other experiments? If so, let me know, since it would be useful to learn from the experiences of how these get implemented.
Join the Conversation