Reporting requirements for ethical considerations in economics RCTs

|

This page in:

Two weeks ago, there was a lively discussion on Twitter about this paper by Bursztyn et al., which is about the persistence of political engagement among students in Hong Kong who were given monetary incentives to attend the annual July 1 pro-democracy protests in 2017. David also covered the online discussion here in our Friday links. The paper was a first draft, submitted to the NBER Summer Institute (fooling me into thinking that it was an NBER WP), but quickly found its way into online discussions.

You can read the online criticism and support for the study in the links provided, but the gist of the former was the presence of risks to the students and their families – of imprisonment, of relational repression, of future harassment via the use of facial recognition software, phone tracking, etc. It is certainly not hard to Google these protests and find news of arrests of protesters and protest organizers - in the year in question (2017) and in other years. Using local knowledge on the exact number of arrests of ordinary protesters on the actual day of the protests, rather than of organizers and movement leaders, and on expected attendance, it is possible to put some probabilities on risk of arrest, being held without charges, or emotional or bodily harm. This would allow the researchers to assess whether the risk to study participants is minimal or not. Future harm to students (or their families living in mainland China or colleagues of PIs of this study from other institutions traveling to China) is harder to assess and depends on a bunch of assumptions, including whether the regime is likely to backslide into harsher authoritarianism in the near future, how long a memory it has, etc. I can see reasonable researchers in economics or political science with decent knowledge of the study context disagreeing about the size of these risks.

But, actually, for the first part of this blog, none of this is the actual point I want to make. What I think is regrettable is that the entire brouhaha over this paper could have been avoided if the paper simply had a short section, titled “Ethical considerations, risk assessments, and IRB protocols.” When I wrote to the authors of the paper to suggest that such a section might be useful, they sent me an ethics statement that I am able to share with you here. Furthermore, in further email exchanges, they informed me that they will incorporate the content of this statement into a new ethics section and post an updated working paper. My guess is that had such a section been a part of the draft paper from the beginning, even the harshest critics would have struck a different tone. It is likely that some people would still argue that the risk to the subjects is more than minimal, but the situation is very different if all the cards are transparently on the table and we can argue about our assumptions and expected probabilities of risks, and learn the questions (hopefully serious enough) IRBs asked, the PIs’ answers to those questions, exact study protocols, and perhaps even about our risk aversion levels.

So, lesson #1: If you are conducting an RCT, please include a sub-section on ethical considerations:

  • Devote one sentence to routine data collection, informed consent issues;
  • Provide IRB approval numbers, but please also provide a link to a document that contains your detailed study protocols (possibly your IRB submission) or include this in an appendix; and
  • Discuss in detail more thorny ethical issues that have been covered in the study protocols, provide an assessment of risks, compare them to reasonable benchmarks, give a sense of equipoise and the importance of the study to provide benefits by generating new knowledge…

If you have a really benign intervention (and data collection methods) with negligible risks, this section can be really short. But, it feels better to have a section like this be the default going forward, rather than the current status quo of authors assuming that everything is OK (because they went through an IRB) or that the audience should trust them that they did the right things…

In Part #2, I will take up bigger picture questions that came up during the discussion of this paper, such as:

  • Is it OK for researchers to conduct an intervention if it is legal for others to do so?
  • Is paying people money to do X coercive? If so, when? How much is too much?
  • Researchers “studying” vs. “doing” development…

Stay tuned…

Authors

Berk Ozler

Lead Economist, Development Research Group, World Bank

Join the Conversation

Reader
July 22, 2019

I appreciate this post, but there's some irony in this statement: "What I think is regrettable is that the entire brouhaha over this paper could have been avoided if the paper simply had a short section, titled “Ethical considerations, risk assessments, and IRB protocols.”" The entire brouhaha could have been avoided if the internet didn't pile on to a group of thoughtful, experienced researchers who had, as it turns out, taken proper steps to design and evaluate their study. I think it's particularly problematic that you implicitly shift the blame to the researchers for not having anticipated the frenzy in this first draft of their paper, when you were one of the most prominent voices fanning the flames. Another way to frame this post would have been "It turns out I was wrong -- the experiment studying political engagement of students in Hong Kong was thoughtfully and responsibly designed, and I jumped to conclusions."

July 22, 2019

I disagree: it's not my job to discern the intentions of the researchers and the details of their work. Just like the transparency movement in reporting analysis and methods, we have the same thing here. You don't get to say: "trust me: attrition is fine." You have to show us. Or, "believe me I pre-specified this analysis." You have to show me your time-stamped registration.

And, my post does not agree or disagree with the authors' calculations: it simple says we needed to know them to help assess the risks. You'll have to wait until part II for some of that. I hope reasonable people can disagree about the wisdom of nudging 18-year-old adolescents into risks like those described below for learning about mechanisms for persistence in engagement in protests...

https://www.nytimes.com/2019/07/22/world/asia/hong-kong-protest-mob-att…

Sincerely,

Berk (not anonymous).

Dina
July 22, 2019

IMHO there is responsability on both sides for these kinds of issues. Writers of papers should try to be as clear as possible (but all of us have probably tried and failed at this many times, so some generosity by readers is probably in order). And readers commenting on Twitter should aim to fully understand and ask follow-up questions before rushing to judgement.

These discussions about ethics in research are important and can benefit from nuanced exchanges. And there were some constructive threads. But clearly not all of what happened on Twitter here was conducive to that. Definitely saw some strong condemnations and personal attacks from some people having self-declaredely only skimmed the intro.

In my view, we move the needle forward on ethical considerations (and other issues) in the research community by engaging in vigorous dialogue, which combines both a great commitment for improvement and a basic presumption of good intent and the possibility of learning together.

July 22, 2019

Hi Dina,

Thanks. I agree with most of what you said. Indeed, we did have a nuanced exchange in Twitter ourselves...However, two points of minor disagreement:

1. This case/paper is a bit different than most. When you read even just the abstract, the first question that pops up in many readers' heads is "Hold on, what did they do? Is that (providing financial incentives to university students to attend anti-government protests) OK?" To have the paper be mainly silent about that very question, which was not that hard to anticipate, was a mistake.

The paper was tweeted initially as an example of an amazing study (not by the authors and not their fault). But, If the study makes the rounds on Twitter and people have questions, where else should they go? Not everyone is invited to NBER SI or knows one of the authors, etc. So, they voiced their worries on the same platform. If the authors are happy with the enhanced recognition that they get from their paper being circulated on Twitter, then they ought to be able to deal with questions. The authors were encouraged to chime in: by staying silent, they only helped raise the temperature (at least one of the authors is on Twitter)...

2. A basic presumption of good intent is all fine and well. With all the protocols in the world, without some trust, the system will crumble. But, that cannot be the basis of how we evaluate scientific work: we have to ask how the authors dealt with potentially very thorny questions. We can't just say "I am sure they were thorough." This does not mean we assume the worst of the authors: quite the opposite. Please see this thread from me, in which I am explicitly giving the authors the benefit of the doubt while simply searching for answers: https://twitter.com/BerkOzler12/status/1148922736850472960. That led to my email exchanges with the authors, which led to the ethics statement linked in my post...

The authors had the paper out circulating. These days, it could be NYT picking up such a draft paper, or a blog, or, most likely, Twitter. If you're not prepared for criticism, then either don't put out the paper in circulation or don't leave anything in it that can so obviously be subject to criticism/questioning. In this case, the threads I saw (by Greitens or Dube) and took part in, seemed to be measured and debating the many real issues surrounding the study. I am not the biggest fan of #EconTwitter, but in this case, I don't think it was at fault. All platforms have asinine comments, including blogs and newspapers, but I don't think it's wise to condemn all the criticism because a few that might have taken it too far or too personal or done it ignorantly (like without at least skimming the paper).

Thanks for commenting. Sincerely,

Berk.

Reader
July 23, 2019

I do think it is your job to discern the details of a paper before publicly casting aspirations on it on Twitter. You questioned whether the authors had collected data before they had IRB approval and suggested the appointment of an AEA ombudsperson to look into possible violations of research ethics. Your exact words, quoting a tweet about the paper, were "Maybe we need to appoint another #AEAombudsperson to look into possible violations of research ethics...". I certainly didn't read those as neutral comments. If you just had questions, why not email the authors and wait a reasonable amount of time for a response? When you did email the authors, they provided a reply that seems to have answered most of your concerns.

Frankly, your tweet about my comment doesn't come off as in good faith, either. You take my characterization of the authors as my main point, when in fact my point was that the authors had taken appropriate steps to assess the risks associated with their experiment -- which you seem to agree with, since you think the description of the steps could have avoided the entire brouhaha. I don't argue that it's the authors' personal traits that legitimize the study; I argue that they had been thoughtful in their consideration of risks and experienced in the local context, benefits of the doubt you did not afford them.

You portray yourself as the hero here, defending research ethics. On Twitter, you facetiously suggest that I should thank you for getting a statement from the authors. However, they *had* followed accepted and appropriate procedures before they conducted their experiment. The paper they posted included the information that research had been approved by four IRB committees and provided the protocol numbers for each. Your proposal to include an ethics section in every paper describing an RCT goes beyond what you have done in your own published research. It hardly seems fair to hold the authors to a practice you made up and haven't followed yourself, even if there is merit to the idea itself. That is really my point. You have some good ideas to contribute to this conversation, but you helped generate the initial and overblown controversy, and instead of acknowledging your role in it, you blame the authors for the fire you helped fuel, and then take credit for putting it out.

Finally, in response to your reply to another comment, below, I don't think it's reasonable further pin responsibility for the uproar on the authors' failure to respond on Twitter ("The authors were encouraged to chime in: by staying silent, they only helped raise the temperature (at least one of the authors is on Twitter)..."). Twitter may be one of your preferred mediums of communication but it isn't everyone's. They didn't start the discussion on Twitter and they weren't obligated to participate, especially given the tone that the conversation took early on.

July 23, 2019

I did discern its details - from what was publicly available: the paper and the registration. The registration had a IRB date that was after baseline data collection (feel free to check). The paper reported ages of 20 with a SD of 1.5, so the possibility of minors in the sample, with no discussion in the paper. I read the paper before commenting - I was not satisfied and still am not.

People can't just email the authors whenever they have a question. I got a response because I know Leo. Otherwise, you would not have that statement.

You're welcome...

July 23, 2019

You seem to have a lot of faith in the system, evidenced by you assertion that since the authors "had" followed accepted practices, we should take it for granted that everything must be fine and we should not question anything, but dutifully wait until we're privileged enough to see them at a seminar, catch them at a conference, or something. Your repeated use of the past participle (they had done this..., that had though that...) acknowledges that none of that was evident from the paper that was circulating. You have to be pretty gullible to think that all Econ RCTs with an IRB approval have crossed their "t"s and dotted their "i"s. It's the wild west out there at the moment and we will be better served if there is some accountability and regulation - something that can deliver a bit more than university IRBs simply trying to avoid legal liability.

With this intervention, especially if you thought so carefully about the risk-benefit calculations, you would have to know that these questions would come up as soon as the paper started circulating. The authors may have gotten caught unprepared with their NBER SI submission being posted online. I have some sympathy but I don't feel bad for them. You put out something that raises questions, those questions will get asked - if not one medium then another. The authors admit that the paper can (and will) be improved. The authors will be fine. They got no worse than they would from a referee. If that referee was me, they would know my name and I would ask more or less the same questions.

You seem to have a yearning for not rocking the boat; deferring to accepted practices; trusting and waiting dutifully for researchers to reveal pertinent information when they feel like it. That is not how it works. It never worked that way - but it was just hidden before (anonymous referees and senior men at seminars skewered people in smaller circles). Change is messy, but people will adjust. A little more care a forethought before circulating a paper won't hurt anyone. These people are presumably enjoying the benefits of all these platforms discussing their work. But, there are two sides to the coin. You know what they say about the heat in the kitchen...

Concerned economist
July 24, 2019

I agree completely with "Reader". Your post is either extraordinarily disingenuous or simply ignorant.

1. If you are concerned about ethics, the correct response is to email the authors to voice your concerns and give them reasonable time to respond (a week seems adequate) BEFORE stoking the flames on Twitter with comments such as ""Maybe we need to appoint another #AEAombudsperson to look into possible violations of research ethics". To say that you are giving the authors the "benefit of the doubt" and then a day or two later to post a Tweet implying the authors have acted unethically is laughable.
2. The draft was made available privately as part of NBER SI, and it was not circulated beyond SI. If you were at SI, as I was, you would have heard David discuss ethical concerns at length. I am sure that the final paper will include much of this discussion, just as the authors' previous papers in this setting did.
3. The authors have no responsibility to respond on Twitter, as you appear to suggest. Not everyone lives and dies by the 140 character and 24-hour-response rule. Twitter is a great platform for many types of discussions, but complicated social science research is often not one of them.
4. Free and open discussion of research ethics is welcome and important. If your aim were to contribute to the conversation, you would have posted a Tweet after hearing from the authors linking to the author's public statement and/or and assuring your followers that the proper ethical considerations were met. At the very least, you would amend or follow up on your aforementioned Tweet suggesting that the authors had acted unethically. The fact that you did not suggests that you are more interested in stoking controversy on Twitter than making meaningful progress on what is obviously an important issue.
5. I do agree with you that the experimental community as a whole should move to being more transparent about research ethics, just as we are with robustness checks. In fact, I think your suggestion to include an ethics section in published papers is a good one. But just as one wouldn't accuse a colleague of p-hacking because he/she didn't have the specific robustness check one was looking for in a first draft, one should also not impugn researchers' ethics over a (private, not-circulated) draft simply because it doesn't contain a paragraph or two, the content of which would be easily accessible either by attending the associated presentation or emailing the authors.

July 24, 2019

You seem to live in a different world, where papers are private and not circulated, anyone can email the authors and get a response, and attend NBER SI. Maybe you should walk outside for a minute. The paper was circulating, that's how I read it. The cover page does not have any disclaimers of preliminary, do not circulate, etc. If you don't want it circulating, tell the organizers to not circulate. Everyone seems to enjoy the attention and accolades they get from their paper being circulated on Twitter, but don't want to deal with any criticism. There is not final paper, working paper anymore: your first draft will get much more discussed than the published version. Just be thorough from the beginning - especially on addressing an issue that was so easy to anticipate ex ante...

I am glad you agree with my post re: transparency in research ethics. But, you seem to think that paying 18 year-olds to go to dangerous protests is akin to p-hacking. It's not. I care about both, but I am more likely to condemn you for not providing details about one than the other.

I have nothing to amend or apologize for. I did exactly what I thought was right and would do it again. One thing I will no longer do is to respond to anonymous comments. You refuse to have any accountability or take responsibility for your opinions. You can say anything you want without any of the associated reputation costs. I don't have a lot of respect for that. Enjoy NBER SI and your ivory tower...

B.

Another concerned colleague
July 24, 2019

Disclosure: I know and admire Dina's work and follow her Twitter page (which is how I ended up here).

Berk makes some valid points about the importance of the profession collectively moving towards an equilibrium in which we embrace greater transparency in ethical processes. Yet, I am inclined to agree with Dina and the other two posters on this blog. It is deeply irresponsible to publicly insinuate that the authors have acted unethically before emailing them and waiting for their response. To say that others do not have the option of emailing the authors because they will not respond -- what is Berk's evidence for this statement? What makes him think that he "got a response only because [he knows] Leo"? As an experimentalist himself, I would hope he would have a better grasp of the importance of a control group... And to demand that authors conform to a standard of including an ethical disclosure in the paper when he himself does not do so in his public work (second disclosure: I admit I was not familiar with Berk's work beforehand, but skimmed a few of his papers to check) is simply hypocritical. I will repeat that I like the suggestion of mandating an ethical section in final work, and I do think the profession should adopt the practice - but that does not at all take away from the hypocrisy of the statement.

And finally, to criticize colleagues for posting anonymously when he seems to enjoy publicly questioning colleagues' ethics and motives on Twitter - I don't even need to point out the obvious there.

Pierre
July 24, 2019

Well said.

July 24, 2019

Here is an omnibus response to all the reactions this post got:

1. More than a year ago, when I challenged GiveDirectly's interpretation of new evidence (http://blogs.worldbank.org/impactevaluations/givedirectly-three-year-im…), people thought that I was attacking the authors Haushofer and Shapiro. I was not. We had civil exchanges, they wrote a post in response to my post, I reciprocated, end of story. It is no different today: there was some initial Twitter exchange (not with the authors in either case), followed by some email exchanges, and a public statement from the authors. I had limited but polite and pleasant exchanges with the authors, even though we disagreed on some issues, and, that is all the evidence I have - as far as the authors go.

2. I don't enjoy asking difficult questions about ethical considerations or, in the former case, pointing to flaws in arguments not supported by the evidence. Actually, I don't enjoy it at all. But, it is part of my job. Cash transfers are an important policy question, so are thorny ethical issues in RCTs: I find it important to inform larger audiences with alternative viewpoints, arguments, and facts - when today's social media and clique-ey and small academic circles can often generate echo chambers. It does not make me very popular, but I have learned to be comfortable with this role.

3. Now, let's look at the current case at hand: There is a paper circulating, the abstract of which indicates that researchers provided financial incentives to students to attend anti-government protests in Hong Kong. It came out (as a conference paper) at a time when government crackdown on protesters is all over the news. If the first question that pops into your head is not "How did this pass IRB?" or "Is that really OK?", I find that really strange. And, that was indeed the question many people asked on Twitter and private conversations, etc.

4. Well, there is a simple solution to that - just open the paper and look at the authors' discussion that sheds light on the answer. The problem is that the paper did not have any. So, people, including me, asked on Twitter, debated various related issues, etc. Unsatisfied with silence from the authors, whose right to not engage on Twitter I respect but do not find particularly wise, I wrote to them.

(An anonymous commentator above is right that I do not have any evidence that I would not get a response from the authors had I not met one of the authors previously. But, it is equally a privileged point of view that anyone that has a question can simply email the authors. In addition to being extremely inefficient, I think that it is reasonable to suspect that many people would have their emails ignored, missed, or deleted. That is not a knock on the authors - it is a fact of life...)

5. Anyway, after a quick intervention from Chris Blattman (as a result of Bursztyn copying him on his quick response), I heard back from the authors with a response and their statement. I wrote the blog above and posted it.

6. The result? Lots of good: First, we now have a better sense of the missing information. Second, the paper when it comes out will have significantly improved. I don't have RCT evidence of this, but I suspect that the issue came up at NBER SI and got debated a few seconds longer because of the discussion on Twitter and the blog post. It's hard to gauge the counterfactual, but I have seen questions on ethics be waved off or shouted down on more than one occasion at econ seminars. Third, there seems to be general agreement with my suggestion that papers, at least those that raise obvious concerns, should have a section discussing ethical considerations and include an appendix with IRB submissions and study protocols. If we follow through, we might change something for the better in the profession. These are all positives, I think: more information, better paper, authors receiving important feedback, and a potential improvement for professional standards.

(By the way, I find the suggestion that it is hypocritical to question others for not including such a section on ethics, because I have not done so myself in my past papers a bit harsh - although I do get the sentiment. Clearly it is impossible for me do so for an idea/practice that I only thought of in light of this paper. Ping me if I don't do it in future papers. There is a reason why this paper is the one that brought up this discussion...)

7. So, for the commentators here (and any silent ones out there) to be really upset, it has to be the case that they think the authors were harmed and that risk of harm was more than minimal. The irony here, for those willing to see it, is inescapable: one could argue that critics on Twitter, including me, might have exposed the authors to a small amount of risk of reputational damage, but the public good that came out of it is so valuable that it was worth doing it. You don't agree: after all, it is exactly the argument that the authors are making as to why they were justified to pay a day's worth of wages to young people to expose them to a minimal risk violence - to answer interesting questions about persistence of engagement.

(Furthermore, the authors themselves got to issue a public statement on a blog that is widely read within our small, niche community; they presented and got a chance to explain their side to conference attendees; and, as far as I know, are not overly upset with the discussion. There was not a lot to be upset about: people asked questions and the authors chose to ignore what they might thought was a rogue Twitter mob. But, if so, they must not have thought of me as one, because if they did they would have ignored me as well.)

8. Two more issues to sort out. First, people seem to be upset about this tweet that I posted: https://twitter.com/BerkOzler12/status/1149691036618223616?s=20 . If we want to parse it exactly and literally, I was suggesting that it might not be a bad idea, in the context of interventions that raise serious questions, to have an ombudsperson for research ethics. I was thinking of something like the NYTimes' The Ethicist (Kwame Appiah), to whom we could ask questions that we could not sort out, as these issues are far from black and white. Literally, I did not say that the authors behaved unethically (this is unassailable), rather I did wonder out loud, in the context of another tweet that worried that the experiment in question seemed ethically problematic, whether it would not be a good idea to have an ombudsperson to whom we could ask questions about ethical quandaries - just like we can do now for gender- and race-based discrimination, harassment, etc.

But, it is also important to note that the tweet in question was posted right after this tweet, which was critical of a tweet by one of the authors: https://twitter.com/MartinRavallion/status/1149680009855979520?s=20. In the subtweet, one of the authors has a picture of a street crossing (perhaps in HK, I cannot be sure) which is covered in cameras too many to count, with the caption "Duck your head when you pass by street crossings like this one ;)". It seems like an allusion to the dangers of surveillance...

9. Which brings me to my final point re: Dina Pomeranz' comments, to whom I thank for posting non-anonymously and trying to keep the discourse (or everyone) civil. She also emailed me separately, which I further appreciate. My educated guess from her Twitter feed is that she did not enjoy the tweet criticizing Dr. Yang's tweet (and, by implication, his RCT). This is what I believe she is referring to when she said that people implied that the author was "callous." Personally, I am not going to go that far. The tweet was well before the discussion of the paper on Twitter and also in response to another tweet that seems to be private (I can't see it). Likely being personally invested in the protest movement himself, I doubt that he could callously joke about other young people that his RCT had a nudging hand in exposing to such surveillance.

10. But, it is not a good look. It is awkward to say the least. Here are a bunch of people on Twitter criticizing you of your judgment re: risks to subjects and you have a tweet like that, which seems to confirm those worries. Worse, it tells me that he is on Twitter and is actively ignoring the tweets about the paper that is now circulating. Again, I respect his right, but question the wisdom. This was not a mob: people were asking legitimate questions and there were a lot of good discussions. One could interpret the reluctance to engage as arrogance: as he thinks he does not owe anyone an explanation; as he is above all this jibber jabber...

11. As I told the authors, they should take the criticism on Twitter not as an assault, but as an opportunity to set the record straight. They did that and I thank them for it. I think that they may have been mildly annoyed for the past couple of weeks, but it's hard to say that we would be better off if none of this happened. In fact, I believe the opposite...

12. I am not going to get into the name calling by anonymous readers (or Twitter followers) above. As this tweet put it well, none of them emailed me personally asking to clarify my position and waited a week for a response before publicly impugning my actions and intentions: https://twitter.com/ChrisProttas/status/1154096549913382913?s=20

Dina
July 24, 2019

Many thanks for clarifying your thoughts. I look forward to our planned phone conversation, where we'll have time to discuss with more time and more directly. I'm personally very grateful for that, since I'm feeling a bit sad about how some of our exchanges ended up going.

Just responding here on the one specific point you made, suggesting my comments were about a particular tweet in the thread you mentioned. They weren't. There were quite a few other tweets and online discussions in and outside of econ about this paper.

In any case, I appreciate this opportunity for all of us to learn together how to discuss on- and offline about such important issues regarding our research community, and how to continue to strengthen our research processes and practices ongoingly.

Be well and looking forward to connecting soon.

Another concerned colleague
July 24, 2019

This is, on the whole, a thoughtful post. I do think you have mischaracterized much in point (8) and (12) is a bit laughable -- in the vein of a schoolyard bully who gets offended when his peers call him out. Other than those points, I agree with most of what you had said and I'm glad you put your earlier remarks in context. If only you had taken a deep breath and struck this this tone from the beginning of this controversy, I think the debate would have been much more productive. But, better late than never.

Jörg Peters (concerned economist, too)
July 25, 2019

Berk is right. It is not okay to nudge young people to protest against an authoritarian regime. This is my opinion, a normative verdict, obviously, but at the very least it is okay to raise it. Also on Twitter. If you disagree, just respond, put forward your arguments and if it's only your differing normative judgement, fine. In any case, writing anonymous comments should be the peculiar right of people living in authoritarian regimes - here it is just plainly spineless.