What's the right way to pick the respondent for a household survey?
This page in:
I have just come back from the pilot for a survey on perceptions of inequality in Lao Cai, near the northern border of Vietnam. Many tourists visit via the overnight train from Hanoi to trek through the green hills filled with terraced rice paddies and see something of the culture of the region’s ethnic minority groups. Despite all the tourist money, the region remains one of Vietnam’s poorest.
The story of persistent ethnic minority poverty in Vietnam is one I will save for a future post. Here I want to write about an issue that came up during our pilot: how to select respondent(s) within the household.
There are many guides to household survey design (here’s one) with extensive discussion of sampling design, but they typically give minimal attention to choosing the people who actually answer the questions. An otherwise quite useful book The Power of Survey Design which I have on my bookshelf, says nothing on the topic.
Who provides the information matters, and it matters more in some cases than others. People in the household might all respond the same way when reporting on simple measures of household conditions, e.g. the material used for the roof, but for much of what is captured in surveys different people in the household will have different responses.
Bardasi, Beegle, Dillon, and Serneels (2011) [ungated version] examined this issue by experimentally varying whether labor data was collected via self-reporting (from the worker) vs. via proxy reports from another randomly selected member of the household. They found the following:
(The paper also reviews the literature on self-reports vs. proxy reports, which comes almost entirely from developed countries.)
The employment difference for men between the two types of report came from the fact that proxy reporters were much more likely to categorize men as being out of work rather than working in agriculture. The authors point out that it’s not clear whether the proxy reports are necessarily closer to the “truth” than the self-reports. One could imagine that some men who are objectively unemployed men might prefer to describe themselves as working.
The Bardasi et al. study shows that who reports can matter even for seemingly objective information. For a survey that captures subjective data the choice of respondent is more crucial, because there is no ultimate “true” answer other than what the particular respondent says.
Here are roughly the set of different practices I believe are used in LSMS-type household surveys (entirely or almost entirely objective information):
Approach H3 is usually the recommended approach, but I believe what is actually done in practice is more often H2. Talking to multiple members of the household will often require visiting several times, and the enumerator’s incentives are almost always to just get the information as quickly as possible. Even if it is possible to get enumerators to stick to a protocol that requires them to work harder, multiple visits can substantially push up the cost of a survey.
For perception surveys, the procedures include the following:
P4 is clearly the way to get a representative sample of the adult population. This is the procedure at least on paper for the Afrobarometer perception surveys, with the additional stipulation that “Each interviewer alternates in each household between interviewing a man and interviewing a woman to ensure gender balance in the sample.”
For the High Frequency South Sudan Survey pilot, which collects a mix of objective and subjective data, we tried using the Afrobarometer approach. As with interviewing multiple members of the household, this will often mean visiting the household more than once, and in practice we found it difficult to prevent the enumerator from picking someone who is home during first visit. As a result, the sample is disproportionately women—in many households only women are home during the day when most visits take place.
For the Vietnam survey pilot, we realized that the selection of the respondent is absolutely critical. One focus of this survey is the frame of reference people have when they think of inequality. We expected that rural households would have narrower frames, and indeed that appears to be the case beyond our expectations: in the ethnic minority communities we visited, some residents had never traveled beyond their villages, and many cannot speak Vietnamese fluently. Many had a hard time even describing wealthier and poor people. Frames of reference also varied greatly with individual experience, which was correlated with age and gender. Older respondents, particularly women, appeared less likely to have been much beyond their villages, while many young men and some women had travelled at least locally for work. Interviewing only older household heads, or only those present at the time of first visit, would generate a sample of people with less experience with the broader world and thus narrower frames of reference. In a previous pilot, youth in isolated villages had far broader frames of reference than their parents and grandparents.
We will try for the Vietnam survey to get the enumerators to use the “P4” approach, but with expectation that compliance by enumerators will not be perfect. We may end up reweighting the survey respondents in the end (using age, gender, and education) to match the profile of the target population. An ex-post weighting scheme cannot, however, correct for selection on unobservables, which may be a factor in this case.
Even conditional on observables, household members who are home when enumerators show up will probably have less experience with the broader world than those are away.
Our struggle with this issue reflects a broader problem in conducting household surveys (and implementing development programs in general): what actually happens on the ground may be far from what the pristine survey manual says. In many surveys, these deviations-from-protocol are hidden from view. But it’s much better to be up front and correct for them as best as possible rather than sweep them under the rug. What experiences have you had where the survey didn’t go quite as you had planned? (Feel free to answer anonymously!)
The story of persistent ethnic minority poverty in Vietnam is one I will save for a future post. Here I want to write about an issue that came up during our pilot: how to select respondent(s) within the household.
There are many guides to household survey design (here’s one) with extensive discussion of sampling design, but they typically give minimal attention to choosing the people who actually answer the questions. An otherwise quite useful book The Power of Survey Design which I have on my bookshelf, says nothing on the topic.
Who provides the information matters, and it matters more in some cases than others. People in the household might all respond the same way when reporting on simple measures of household conditions, e.g. the material used for the roof, but for much of what is captured in surveys different people in the household will have different responses.
Bardasi, Beegle, Dillon, and Serneels (2011) [ungated version] examined this issue by experimentally varying whether labor data was collected via self-reporting (from the worker) vs. via proxy reports from another randomly selected member of the household. They found the following:
Response by proxy leads to substantially lower male employment rates due to lower reporting of participation in agriculture – while for women there is no effect. The discrepancies between proxy and self-reporting are reduced when the spouse is selected as the proxy and when the proxy is an individual with some education….
These findings suggest that there is no substantial benefit of self-reporting for women (a similar result was found for children, as discussed in Dillon et al., 2010), but there is for men, whose employment rates and distribution across sector of activity are affected by proxy reporting.
These findings suggest that there is no substantial benefit of self-reporting for women (a similar result was found for children, as discussed in Dillon et al., 2010), but there is for men, whose employment rates and distribution across sector of activity are affected by proxy reporting.
(The paper also reviews the literature on self-reports vs. proxy reports, which comes almost entirely from developed countries.)
The employment difference for men between the two types of report came from the fact that proxy reporters were much more likely to categorize men as being out of work rather than working in agriculture. The authors point out that it’s not clear whether the proxy reports are necessarily closer to the “truth” than the self-reports. One could imagine that some men who are objectively unemployed men might prefer to describe themselves as working.
The Bardasi et al. study shows that who reports can matter even for seemingly objective information. For a survey that captures subjective data the choice of respondent is more crucial, because there is no ultimate “true” answer other than what the particular respondent says.
Here are roughly the set of different practices I believe are used in LSMS-type household surveys (entirely or almost entirely objective information):
H1) The household head is interviewed.
H2) Whomever is available when the enumerator shows up is interviewed.
H3) Multiple members of the household are interviewed, with the most knowledgeable respondents providing different pieces of information wherever possible.
H2) Whomever is available when the enumerator shows up is interviewed.
H3) Multiple members of the household are interviewed, with the most knowledgeable respondents providing different pieces of information wherever possible.
Approach H3 is usually the recommended approach, but I believe what is actually done in practice is more often H2. Talking to multiple members of the household will often require visiting several times, and the enumerator’s incentives are almost always to just get the information as quickly as possible. Even if it is possible to get enumerators to stick to a protocol that requires them to work harder, multiple visits can substantially push up the cost of a survey.
For perception surveys, the procedures include the following:
P1) The household head is interviewed
P2) Whomever is available when the enumerator shows up is interviewed
P3) A random adult among those present at the time of visit is interviewed
P4) A random adult among all those on the household roster is interviewed.
P2) Whomever is available when the enumerator shows up is interviewed
P3) A random adult among those present at the time of visit is interviewed
P4) A random adult among all those on the household roster is interviewed.
P4 is clearly the way to get a representative sample of the adult population. This is the procedure at least on paper for the Afrobarometer perception surveys, with the additional stipulation that “Each interviewer alternates in each household between interviewing a man and interviewing a woman to ensure gender balance in the sample.”
For the High Frequency South Sudan Survey pilot, which collects a mix of objective and subjective data, we tried using the Afrobarometer approach. As with interviewing multiple members of the household, this will often mean visiting the household more than once, and in practice we found it difficult to prevent the enumerator from picking someone who is home during first visit. As a result, the sample is disproportionately women—in many households only women are home during the day when most visits take place.
For the Vietnam survey pilot, we realized that the selection of the respondent is absolutely critical. One focus of this survey is the frame of reference people have when they think of inequality. We expected that rural households would have narrower frames, and indeed that appears to be the case beyond our expectations: in the ethnic minority communities we visited, some residents had never traveled beyond their villages, and many cannot speak Vietnamese fluently. Many had a hard time even describing wealthier and poor people. Frames of reference also varied greatly with individual experience, which was correlated with age and gender. Older respondents, particularly women, appeared less likely to have been much beyond their villages, while many young men and some women had travelled at least locally for work. Interviewing only older household heads, or only those present at the time of first visit, would generate a sample of people with less experience with the broader world and thus narrower frames of reference. In a previous pilot, youth in isolated villages had far broader frames of reference than their parents and grandparents.
We will try for the Vietnam survey to get the enumerators to use the “P4” approach, but with expectation that compliance by enumerators will not be perfect. We may end up reweighting the survey respondents in the end (using age, gender, and education) to match the profile of the target population. An ex-post weighting scheme cannot, however, correct for selection on unobservables, which may be a factor in this case.
Even conditional on observables, household members who are home when enumerators show up will probably have less experience with the broader world than those are away.
Our struggle with this issue reflects a broader problem in conducting household surveys (and implementing development programs in general): what actually happens on the ground may be far from what the pristine survey manual says. In many surveys, these deviations-from-protocol are hidden from view. But it’s much better to be up front and correct for them as best as possible rather than sweep them under the rug. What experiences have you had where the survey didn’t go quite as you had planned? (Feel free to answer anonymously!)
Great post.
I'm surprised there's so little discussion of this important issue out there - so kudos for highlighting it. Also good (and reassuring) to know that others are grappling with this. Having just implemented the first round of a panel survey (asking HH-level livelihoods questions and individual-level perceptions questions), and now embarking on designing a separate survey looking at tax, livelihoods and governance in Sierra Leone and Nepal, we're dealing with very similar issues around who it is we should be interviewing (and indeed if it is just one individual per HH we should be interviewing). As one quick example, can we be sure that by only interviewing the HH head (is this even a meaningful concept in every context...?) we are generating accurate information on the full range of payments and taxes a HH is making...? Are we safe to work on the assumption that the HH head alone has the answers to our questions, and that therefore our data will be much more reliable (or at least more complete) than if we were to ask someone else within the household? That often seems to be the assumption built into sampling designs, but I think you're definitely right to question it. In any case, it's something we're trying to deal with at the moment...so any thoughts welcome!
On a related note, some very initial analysis of our previous survey work appears to support the claim above that 'who reports can matter even for seemingly objective information'. We've found that both gender and age of respondent appears, in some cases, to have significant effects on the kinds of answers given to apparently objective questions (e.g. HH asset ownership, receipt of social protection transfers, distance to school, experience of shocks). Does this mean that there is one person per HH with the 'right' information and that we failed to ask them? I'm not convinced it does - I think it instead raises a far broader set of questions around the nature of truth in surveys; that we can't simply assume there is a 'an answer' and that all we need to do as researchers is to 'find it'. Food for thought, that's for sure...
Rich
Rich,
Thanks for your comment. I think what I would emphasize is the importance of consistency and transparency. Particularly for a panel survey of perceptions, it's important to make sure that it's the same person (if at all possible) answering the questin over time. In many household surveys, its often not clear in the data who is answering the questions, and it's easy to imagine that there could be variation in responses across households created merely by differences in the choice of respondents. At the very least, we should always try to be clear about how respondents are chosen, and have the respondents clearly indicated in the data. Sorry I don't have any clearer guidance to give! Please share the results of your work when you have them written up.
Gabriel
For a study on differences in responses to 'household level' questions on food security in Bangladesh you may want to have a look at:
Coates et al. (2010) “He said, she said”: who should speak for households about experiences of food insecurity in Bangladesh? Food Security, 1, 81-95. DOI 10.1007/s12571-010-0052-9.
The paper reports that men and women in the same household respond differently to individual questions about household food insecurity (average degree of discordance in this study was 15%), and that the mean masks differences across the questions ranging from barely 1% to upwards of 50%. The fact that the degree of discordance varied so dramatically from one question to the next suggests that there is something about the questions themselves that affects how men and women will respond.
Patrick:
Thanks for the interesting reference. This doesn't seem too surprising for the food security questions, which are somewhere in between an objective and subjective measure. I've always wondered how to intrepret these (and I have some ongoing work related to this question in South Sudan.)
One interesting point about this: for this paper, they asked the food security questions of BOTH people in the household. In the Bardasi et al. paper, in each household, the labor information was asked of either the person in question OR a proxy respondent. I imagine this was done to simulate how things would be done in a real survey. Asking two different people in the household the same set of questions could influence how each individual responds. Apparently, they didn't worry about this in the Coates study. This raises the question of whether the answers were influenced by the knowledge that the questions were being asked of someone else in the household. But it also allows them to analyze the mismatch in answers by household, something the Bardasi et al study couldn't do.
A tendency - as I see it -(also from assessing poverty research in Lao Cai and other sites in Vietnam) is that researchers both for qualitative and quantitative surveys tend to interview "the member of the family who can speak the Kinh (Vietnamese) language". This is particularly the case in ethnic minority communities where men tend to speak Kinh better than women (since men generally have more relations with people outside the community and often have gone to school longer).
When neither of the adults speak Vietnamese, the child who goes to school in Vietnamese is asked to speak on behalf of both parents and the household.
This is clearly not appropriate. There is an inherent gender bias in that type of approach. And children should not be asked to respond on behalf of their parents on issues as sensitive as poverty (not to talk about gender violence!).
We are raising this gender bias (and potentially putting children at risk of self-stigma and intimidation) in relation to our ongoing review, so I am glad to find similar thinking here on this blog. / Charlotte
Can we select two respondents from each house hold and how we will present them as a single respondent or house hold. For example if we have to survey with 75 households and we select two respondents from each household than our house hold will be 75 with a 150 respondents or it will be considered 150 household?