Doing Experiments with Socially Good but Privately Bad Treatments

|

This page in:

Most experiments in development economics involve giving the treatment group something they want (e.g. cash, health care, schooling for their kids) or at least offering something they might want and can choose whether or not to take up (e.g. business training, financial education). Indeed among the most common justifications for randomization is that there is not enough of the treatment for everyone who wants it, leading to oversubscription or randomized phase-in designs.
However, when it comes to issues like taxation and regulatory enforcement, the typical government intervention is something that nobody privately wants (e.g. a tax audit, a health and safety inspection, a traffic fine) but is thought to be socially or publicly desirable (we want others to pay their taxes, prepare our food safely, not drive like lunatics). This raises both ethical and logistical challenges for trying to evaluate these types of interventions in an experimental setting. I don’t know of any papers which have dealt with this issue, so thought I would share some experiences from a recent experiment (working paper, 2-page summary) trying to formalize informal firms in Belo Horizonte, Brazil.
The Intervention
Together with Miriam Bruhn, I worked with the State Government of Minas Gerais to test several different policy approaches designed to get larger informal firms (monthly profits averaging US$1000) to register for tax purposes. Two of the treatments were the standard “give them stuff” types of treatment discussed above, giving the firm owners information and, in some cases, free registration and the use of an accountant. But in addition to these “carrots”, the government wanted to test the effectiveness of better enforcement, by sending municipal inspectors to some firms to check their registration status, and if they weren’t formal, tell them to formalize, and in theory fine them and ultimately shut them down if they didn’t register. You can imagine that if we followed a standard approach of asking firms if they wanted to participate in this study, with a much higher chance of getting inspected if they agree, we wouldn’t have many participants!
The Ethical and Logistical Issues
The first challenge we faced is that we lacked a sample frame of informal firms – after all, these were firms which hadn’t registered their businesses. The standard approach would be to then do a screening survey, asking firms if they were informal, and if so, conducting a baseline survey of them before allocating them to treatment. This was the approach I used with Suresh de Mel and Chris Woodruff in an experiment on formalizing firms in Sri Lanka, where firms only got carrots/incentives to register, but no sticks. But in the Brazilian case, there were two issues with doing this:
First, ethically researchers cannot force  firms to answer a survey, and if firms were not told that the purpose of the survey would be to determine whether an inspector was likely to be assigned to them they would most likely refuse to answer, while if they weren’t told this information, this would be deceiving them and unethical. Second, governments on the other hand can legally require firms (and individuals) to provide information, and so the government could ask firms their legal status and require a baseline survey if they weren’t formal. But then if firms linked this survey to the inspection, they would be likely very hard to get to do a follow-up survey, and even if the government forced them, one might doubt the reliability of the answers given.
Our solution
Our solution was to use a listing survey, where enumerators walked through randomly selected census blocks, and noted the details of each business that were observable from outside the business (name of the business, business type, approximate size of the premises, approximate number of employees), without ever speaking to the business owner. Then this information was matched against government records of registered firms to remove firms who were definitely formal, leaving a sample of possibly informal firms, who could then be assigned to the different treatments or the control group.
The government then sent municipal inspectors to the randomly selected firms, and 9 months later a follow-up survey was taken, which followed standard procedures of firm owners being able to decide whether or not to take part in the survey. Since the survey was removed in time from the intervention, and wasn’t preceded by a baseline survey, it was not linked in the firm owner’s minds to getting the inspection happen, and we saw no difference in response rates between inspector treatment firms and control firms. We also had administrative data on registrations for all firms, which could be used to measure the registration response of firms to this treatment.
We find that an additional inspection results in a 21-27 percent increase in municipal registration, and that the costs of the added inspections are more than covered by the tax liabilities of the firms over the first couple of years of being formal. So from a public finance point of view, this is a treatment that is socially beneficial, even though it is not one that any private firm would volunteer for.
Alternative approaches
The only related experimental work I am aware of uses administrative data and the fact that for fairness reasons, governments often use a (risk-adjusted) random inspection process. In a paper in Science last year, David Levine and co-authors look at the impact of workplace safety inspections in California on worker injury rates and injury costs. They use the fact that OSHA randomly selected which workplaces in high-injury industries they inspected, and then match on observables to construct a control group of very similar facilities that were eligible for randomized inspections but not selected. Injury data were then available from the workers’ compensation system, and they find that the inspections reduce injury rates and injury costs, but don’t affect employment or sales.
Am I missing examples of privately bad, but socially good treatments from other experiments? If so, let me know, since it would be useful to learn from the experiences of how these get implemented.
 
Countries

Authors

David McKenzie

Lead Economist, Development Research Group, World Bank

Jack
May 29, 2013

The barriers that small firms face in joining the formal economy are well-documented. In countries where local governments provide little in return, it seems like the ethical issues in this sort of study run deeper. While the possibility that an intervention may hurt study participants is always present in an RCT, it seems especially pertinent when informed consent is not possible and the firms in question would rather not be part of the treatment group. How do you handle these sorts of ethical issues? How did the IRB approval process work?

David McKenzie
May 30, 2013

Hi Jack,
In this case because it was the government deciding whether or not to implement the intervention, the intervention did not have to go through an IRB process - just through their usual government channels for decision-making. The challenges are even more severe when researchers are the ones implementing the intervention for the reasons you mention,
David

Anonymous
May 30, 2013

I'm trying nudges to patients in the waiting room to reduce antibiotics use in order to delay antibiotic resistance. This would be a borderzone, as in the long run prudent use of antibiotics is good for everyone - including the individual patient - but in the short run the patient quite likely prefers as speedy a recovery as possible. I hope to have a working paper version out soon, to be able to share it. Do let me know if this sounds interesting to you.

David McKenzie
May 30, 2013

Yes, please let us know when the working paper is out. It would also be interesting to hear about the IRB process for this work if it is not discussed in the paper.
David

Anonymous
September 04, 2018

I was wondering if you've perhaps published your study anywhere? I'm very interested in the ethical aspects of this study

Anonymous
May 30, 2013

Ok, will do!
EPN, the Swedish equivalent to IRB, decided this project did not even need approval, strange as it might seem. They are mainly worried about collection of personal and/or sensitive information, and I keep everyone anonymous.

David McKenzie
September 04, 2018

The paper was published in the World Bank Economic Review in 2016https://academic.oup.com/wber/article/30/1/24/2461702

Anonymous
May 23, 2019

Thanks. It'll be an interesting read. Do you happen to know anything about the Swedish study on reduction of antibiotics use mentioned in one of the comments above?
Or perhaps are you aware of any other studies/experiments that faced a similar dilemma as identified in your blogpost (i.e. subjects of the study expected to be worse off against the prospects of benefits to the wider society)?
Thanks

David McKenzie
May 23, 2019

I think this is the study referred to "Nudging to prudence? The effect of reminders on antibiotics prescriptions", JEBO 2017:https://www.sciencedirect.com/science/article/pii/S016726811730001X?via…
 

Anonymous
May 23, 2019

Many thanks for the quick reply!
Great blogpost and also the discussion in the comments