Doing surveys ethically: Practical tips

Despite increases in data transparency, there is still very little transparency on ethical survey implementation and management . Surveys bring many ethical challenges, from obtaining truly informed consent to safely conducting interviews with vulnerable populations and respecting respondents’ privacy. As part of DIME’s Manage Successful Impact Evaluation Surveys course earlier this year, we hosted a session of lightning talks on ethical challenges in data collection (full recording here). Here, we share highlights of both existing ethical guidance and ethical challenges raised during the session the research community should explore in more detail . 

What we already know - Ethical guidance for surveys 

How to safely interview minors?  

Parental or guardian permission is necessary when it comes to interviewing minors per IRB regulations. However, the minor must get the final say in their participation and have the right to decline for any reason. Further, emancipated minors should be allowed to provide their own consent. If they have, for example, already consented to marriage, they should be allowed to answer questions about it, even if they are below the age of majority (which by itself can vary based on factors like location and context).  

Despite these recommendations, challenges persist in this regard. Parents may be "tempted" by financial incentives to enroll their children in studies that are unsafe or even damaging. To prevent it, incentives should be age-appropriate, not enticing to influence participation (e.g., a pencil case or a book, rather than candy or toys) and given directly to the minor  

How to safely collect data on sensitive topics like gender-based violence? 

Questions on sensitive topics can trigger traumatic memories or emotions in respondents. Enumerators should be trained to be able to discern if the respondent is in distress and direct them to relevant counseling resources where needed. A professional counselor may accompany the enumerators if counseling resources are not otherwise accessible to respondents.   

On the other hand, research teams should avoid underreporting on sensitive topics. That is, respondents report fewer instances of undesired behaviors compared to what they have experienced. Many argue that it is unethical not to use valuable data on sensitive topics to inform relevant policies or programs.  

How to safely collect data on taboo topics such as sexually transmitted infections?  

Recently, biometrics have emerged as a more reliable alternative to self-reported data, which may be subject to desirability bias or recall error. For example, researchers routinely gather biomarkers for STIs as an alternative to asking about condom-use. However, there are serious ethical concerns when collecting such biomarkers, as health information is extremely private, and disclosure may bring unwanted consequences such as a potential for psychological damage, for instance, from learning that one has an STI. De-identification of the data at the earliest feasible stage is therefore essential. Further, when handling data on STIs, the research team may provide respondents with resources on how to receive treatment for those who end up testing positive.  

What we don’t yet know – Current ethical challenges  

How to make consent truly informed? 

While they are a standard requirement for any survey, informed consent statements are often long and complex. Do respondents really understand what they are agreeing to? Do they feel that it is indeed their choice to participate? A study in rural Pakistan found that respondents often agreed to the consent statement without understanding its components - only 36% of people knew they had a right to not respond. Engaging respondents in dialogue improves understanding, while showing an animated video decreases understanding.  

A deeper analysis may be warranted to ensure that not only are participants truly informed, but also informed consent remains reasonable and concise. The research team may explore– whether the positive impact of dialogue generalize beyond rural Pakistan? Could a different type of video be more effective? What other modes could help? Could adding more elements to the process encourage participation? 

How much should respondents be compensated?  

There is no consensus in the development research community about whether and how to compensate respondents. Typically, respondents are:(1) reimbursed (paid for transportation or airtime); (2) compensated (paid what the local labor market would pay for the time spent on the survey, even if the respondent doesn’t have a job); (3) offered incentives, which can aid recruitment for all, or a subset of, respondents. It is potentially unethical not to compensate respondents for their time at all, particularly when significant amount of time is required. Yet, it is also potentially unethical to compensate respondents too much, by making an offer the sampled individual cannot refuse. How should researchers determine the right level of compensation? What is right? And what is good?  

Where to go from here - call to action 

Despite increased discourse around ethical guidelines, like this discussion on Twitter, there is scope for more transparency and clarity in establishing ethical standards for data collection. One such area is the proposal by Asiedu et.al. to implement what they call a “structured ethics appendix” approach in primary data collection. This involves a set of questions that provide more holistic information to readers and include issues that do not fall within the purview of IRBs, but are important, nonetheless.  

Another area to explore further is compensation and its effects. For instance, what are the tradeoffs for participants? To what extent do the socioeconomic conditions of participants affect their response to compensation? Are people willing to participate at all in studies involving topics that they find objectionable?  

As we continue to explore these issues in greater depth at DIME in our own work going forward, we invite engagement from the broader research community on the ideal way to approach these concerns, as well as other possible areas where the global research community should be doing more to safeguard the interests of respondents. 

The expert panel for the lightning talks comprised the following: 

• Alexandra Avdeenko, Research Director, Center for Evaluation and Development (C4ED)  

• Anja Sautmann, Research Economist, World Bank’s Development Research Group (DECRG) 

• Heather Lanthorn, Program Director,  Mercury Project 

• Rachael Pierotti, Research Specialist, World Bank’s Africa Gender Innovation Lab (GIL) 

• Victor Orozco, Senior Economist, World Bank’s Development Impact Evaluation (DIME)