Using remote surveying methods to collect data during challenging times

Collecting research survey data in hard-to-reach settings can be difficult. But when layered with the unique challenges of COVID-19, finding the right methods to reach intended participants becomes even more complex . During COVID-19, it was important to monitor the pandemic’s impact on households’ economic and social wellbeing. Such research efforts required collecting basic household-specific data—including individual characteristics and beneficiary status of social programs— and sometimes sensitive information such as violence in the household and mental distress. But collecting data under mobility restrictions and health concerns for both enumerators and respondents was a daunting task.  

Our team faced these challenges while working on a project in El Salvador that required surveying caregivers on their mental health and parenting skills. We wanted to understand if caregivers would be better able to manage stress within the family if provided with information on positive parenting and behavioral skills. But how could we conduct surveys— with limited budget and time—when these families were under a strict lockdown due to COVID-19? 

In our new paper, we examine the effectiveness of collecting data on sensitive topics and in hard-to-reach settings through two remote survey methods: phone-based surveys and self-completion surveys over WhatsApp . Our research found that phone surveys can be a viable remote survey option, outperforming, at least in this context, automated, self-response surveys delivered via mobile phones.   

Before the pandemic, our team had discussed what methods were most appropriate for collecting sensitive household information. Specifically, we discussed the use of list experiments and vignettes, as well as training the enumerators on best practices for conducting face-to-face interviews for maximizing privacy and minimizing measurement error. But when the pandemic hit, our plans for conducting one-hour face-to-face interviews were no longer possible.  Our only other option was remote surveying. Remote surveying to measure the well-being and economic conditions is not new and is often used by researchers to access hard-to-reach populations such as migrants, refugees, and households affected by natural disasters.  

We faced two main challenges: First, we did not have a full understanding of the data quality and sampling implications of carrying out phone or online self-completion surveys. We also didn’t have extensive guidelines and scientific knowledge on remote surveying of households in hard-to-reach groups and even less information on how to proceed when the research topic related to sensitive issues such as mental health and family violence. Second, because the situation under COVID was so uncertain, we wanted to make sure attrition was low to ensure sample sizes were adequate to statistically identify the causal effects of the intervention.  

Given these limitations, we decided to use two remote survey method options: self-completion through WhatsApp message and a phone-based survey. Although we knew that a phone-survey (with an enumerator making calls) would cost more, we were unsure about how much, if any, we could gain in terms of response effectiveness. We were also inclined to rely on the potential advantages of digital methods— people can respond when they are available instead of when the phone is ringing; and they can answer more thoughtfully when writing instead of verbally improvising a response to the enumerator. But is it true in practice? If so, how do the two methods compare to each other?  

To evaluate these two alternatives, we selected a sub-sample of 600 participants from the main study. Next, we randomly placed the individuals in two groups: the first would be contacted by phone and the second would be contacted by a WhatsApp message. Since we wanted to isolate the causal effect of the mode of survey, we kept everything else in the experiment constant, including respondent incentives, quantity of reminders, quantity of attempts to reach a respondent, survey structure, and survey length. Given that the objective was to understand how we would survey respondents at the endline, our main goals were to see how effective these methods were in terms of (i) maximizing survey response rates and (ii) maximizing the quality of survey data. 

We drew four main lessons on the effectiveness of remote surveying: 

First, phone surveys increased the response rate by 42 percentage points relative to those who were contacted by WhatsApp. Overall, the results suggest that when accounting for the probability of survey completion, rather than doubling the implementation costs, phone-based surveys are 25 percent less costly than self-completing surveys due to the higher response rate .  

Second, survey methods can affect the reasons for non-response. We found that for each person who did not respond due to connectivity issues in phone-based surveys, 1.6 people refused to complete the survey. This ratio was 2.2 in self-completion surveys . This finding is important for populations with connectivity issues such as low bandwidth and internet speed. It also shows that using surveys with self-completion questionnaires can lead to high respondent fatigue resulting in fewer completion rates.  

Third, response rates increase substantially using a phone-based survey compared to a text option like WhatsApp when targeting certain groups such as women or older people. Compared to men, women are 15 percentage points more likely to answer a phone-based survey.  The same is true for older populations due to their reluctance to complete a text-based survey. For example, compared to respondents 40 years or younger, the response rate of those 40 years or older was 41 percentage points higher using a phone versus WhatsApp—a finding consistent with the apparent reluctance of younger generations to talk on the phone.  

Finally, when comparing the consistency of self-reporting issues like stress and perceived physical abuse depending on the survey method, we find no differences in the quality of reporting of these sensitive outcomes . This finding is important not only because we were collecting data on sensitive issues, but also because it suggests that the differences in the outcomes of the study are due to the way data was collected and not the fundamental differences across groups, such as age and gender.