Syndicate content

(Mis)Leading from Behind (the Scene)

Berk Ozler's picture

To deceive or not to deceive? David’s last post about Beaman and Magruder’s experiment, in which there is a small, and seemingly harmless deception, got me thinking about this continually uncomfortable issue. David claims that this is now an increasingly popular strategy, which gives me some worry about the future of experiments in economics.

There must be a reason why we keep coming back to this subject: it makes us uncomfortable. Jed wrote about the conditions under which it is ethical to lie to the subjects in an experiment. I wrote about informed consent and mystery clients here. In a way, though, mystery clients are significantly different than deception: they are like clinical trials with placebo drugs or sham surgeries, except with people rather than a medical procedure. There is no deception: every subject is “read in” about the probability of receiving the treatment.

Jamison, Karlan, and Schechter (2008) discuss the pros and cons of deception in lab experiments and find that deceiving subjects and letting them know about it causes extensive and intensive margin effects: deceived individuals are less likely to return to participate in a subsequent experiment, and those that return behave differently. So, it may perhaps be both ethical and optimal to deceive at an individual project level, but in a world where these experiments are more common AND it is widely known that the experimenters cannot be trusted with their word, we are likely to get results that are decreasingly valid externally. If there are no rules, we cannot function. Partly due to these types of considerations, Jamison et al. favor a proscription to deceive in economic experiments.

I feel that this is even more important in field experiments. Sometimes, our subjects are not individuals, but institutions: we evaluate the behavior of health clinics, communities, NGOs, etc. Deceiving institutions or individuals that are part of institutions with long-term memory would be potentially very detrimental to development efforts. For example, it does not seem acceptable, say, to tell subjects that the upcoming elections will be monitored by international or local monitors and then not monitor the elections at all. This seems quite harmful if subjects were led to believe this is happening with 100% probability and then it does not. This, as opposed to telling everyone that some polling sites will be randomly monitored while others not and then following through on this promised design: the latter is no different than the “mystery client” case.

My suggestion is the following. In economics, we should keep the rule that we should NOT deceive. However, there may be some research questions for which the researchers think some amount of deception is necessary. In such cases, reviewers and IRBs should push the researchers to carefully think about whether the deception is truly necessary or whether there is an alternative design that is an acceptable compromise for the research question at hand and can avoid deception. For example, in the Beaman and Magruder case, if the recruited people were also paid for performance, then side payments would be less of a concern. In that case, the authors could only identify the ability component over and above the performance component, but that does not seem so bad – perhaps as externally valid if not more so. Only after the researchers convince some reviewers and IRBs that the proposed research design with deception is the only way to get at this question AND that the benefits from this knowledge is greater than the cost to individuals (as well as future researchers due to potential negative spillovers), then and only then we should allow deception.

P.S. I am off to New Zealand tonight to visit family and to enjoy the Rugby World Cup, so this will be my last post for about a month (look for me in the stands on September 24 during the All Blacks-Les Bleus match!). However, we have lined up some good guest bloggers in the upcoming weeks, in addition to the regular posts from David, Jed, and Markus.

Comments

Submitted by Transplant on
Deception is not a good rule in any field.To deceive in economics means hiding figures,facts and digits.Do deception really help in economy?There are two school of thooughts,one consider it fair and other school of thought is strictly against it.

Submitted by julian on
Berk - thanks for bringing up this topic (and mentioning our paper). I'm actually working on a 'sequel' now, not experimental but rather a review piece. We will definitely discuss field as well as lab research, since the issues are interesting and slightly different just as you point out. One of the complications is that deception is not so easy to define, and (as some of your examples show) in particular isn't simply the same as lying. I would argue that it lies on a continuum, and like you that the burden should be on the researcher to defend why it's necessary. However, my guess is that at least for 'lesser' deception, there will be quite a few cases that pass this test, so the outcome is likely to involve more deception than now.

Submitted by Berk on
Hi Julian, I agree about the difficulty of defining deception: I struggled the most with this while writing the post and if I had more time to work on it before leaving for vacation, I would have probably touched on this. This is why a hard proscription seems impractical. I agree that there'll be an interior solution, but I hope it'll be nearer the correct corner. Berk.

Submitted by Helen Abadzi on
Our mission is to get results for the poor, and we are not doing well. Little is truly known about what people with zero education will decide to do under various financial conditions, so despite all the microfinancing, beneficiaries don't really rise out of poverty. Many important behavioral discoveries were found through deception that today would be viewed as unethical. An example is Milner's famous finding that authority figures will lead perfectly normal people to deliver fatal electric shocks to invisible fellowmen. This would be an actionable finding in the right hands, that simply could not have been produced otherwise. I think the dissatisfaction expressed in the article is that many experiments involving deception have trivial goals. Resarchers have disserations to write and research to produce, so the benefitis are too small for the risks. Rather than decide up front about methodology, it's worth deciding up front which big questions need to be answered. Then you can see if deception is the only way to get the results.