I have just come back from the pilot for a survey on perceptions of inequality in Lao Cai, near the northern border of Vietnam. Many tourists visit via the overnight train from Hanoi to trek through the green hills filled with terraced rice paddies and see something of the culture of the region’s ethnic minority groups. Despite all the tourist money, the region remains one of Vietnam’s poorest.
The story of persistent ethnic minority poverty in Vietnam is one I will save for a future post. Here I want to write about an issue that came up during our pilot: how to select respondent(s) within the household.
There are many guides to household survey design (here’s one) with extensive discussion of sampling design, but they typically give minimal attention to choosing the people who actually answer the questions. An otherwise quite useful book The Power of Survey Design which I have on my bookshelf, says nothing on the topic.
Who provides the information matters, and it matters more in some cases than others. People in the household might all respond the same way when reporting on simple measures of household conditions, e.g. the material used for the roof, but for much of what is captured in surveys different people in the household will have different responses.
Bardasi, Beegle, Dillon, and Serneels (2011) [ungated version] examined this issue by experimentally varying whether labor data was collected via self-reporting (from the worker) vs. via proxy reports from another randomly selected member of the household. They found the following:
(The paper also reviews the literature on self-reports vs. proxy reports, which comes almost entirely from developed countries.)
The employment difference for men between the two types of report came from the fact that proxy reporters were much more likely to categorize men as being out of work rather than working in agriculture. The authors point out that it’s not clear whether the proxy reports are necessarily closer to the “truth” than the self-reports. One could imagine that some men who are objectively unemployed men might prefer to describe themselves as working.
The Bardasi et al. study shows that who reports can matter even for seemingly objective information. For a survey that captures subjective data the choice of respondent is more crucial, because there is no ultimate “true” answer other than what the particular respondent says.
Here are roughly the set of different practices I believe are used in LSMS-type household surveys (entirely or almost entirely objective information):
Approach H3 is usually the recommended approach, but I believe what is actually done in practice is more often H2. Talking to multiple members of the household will often require visiting several times, and the enumerator’s incentives are almost always to just get the information as quickly as possible. Even if it is possible to get enumerators to stick to a protocol that requires them to work harder, multiple visits can substantially push up the cost of a survey.
For perception surveys, the procedures include the following:
P4 is clearly the way to get a representative sample of the adult population. This is the procedure at least on paper for the Afrobarometer perception surveys, with the additional stipulation that “Each interviewer alternates in each household between interviewing a man and interviewing a woman to ensure gender balance in the sample.”
For the High Frequency South Sudan Survey pilot, which collects a mix of objective and subjective data, we tried using the Afrobarometer approach. As with interviewing multiple members of the household, this will often mean visiting the household more than once, and in practice we found it difficult to prevent the enumerator from picking someone who is home during first visit. As a result, the sample is disproportionately women—in many households only women are home during the day when most visits take place.
For the Vietnam survey pilot, we realized that the selection of the respondent is absolutely critical. One focus of this survey is the frame of reference people have when they think of inequality. We expected that rural households would have narrower frames, and indeed that appears to be the case beyond our expectations: in the ethnic minority communities we visited, some residents had never traveled beyond their villages, and many cannot speak Vietnamese fluently. Many had a hard time even describing wealthier and poor people. Frames of reference also varied greatly with individual experience, which was correlated with age and gender. Older respondents, particularly women, appeared less likely to have been much beyond their villages, while many young men and some women had travelled at least locally for work. Interviewing only older household heads, or only those present at the time of first visit, would generate a sample of people with less experience with the broader world and thus narrower frames of reference. In a previous pilot, youth in isolated villages had far broader frames of reference than their parents and grandparents.
We will try for the Vietnam survey to get the enumerators to use the “P4” approach, but with expectation that compliance by enumerators will not be perfect. We may end up reweighting the survey respondents in the end (using age, gender, and education) to match the profile of the target population. An ex-post weighting scheme cannot, however, correct for selection on unobservables, which may be a factor in this case.
Even conditional on observables, household members who are home when enumerators show up will probably have less experience with the broader world than those are away.
Our struggle with this issue reflects a broader problem in conducting household surveys (and implementing development programs in general): what actually happens on the ground may be far from what the pristine survey manual says. In many surveys, these deviations-from-protocol are hidden from view. But it’s much better to be up front and correct for them as best as possible rather than sweep them under the rug. What experiences have you had where the survey didn’t go quite as you had planned? (Feel free to answer anonymously!)
The story of persistent ethnic minority poverty in Vietnam is one I will save for a future post. Here I want to write about an issue that came up during our pilot: how to select respondent(s) within the household.
There are many guides to household survey design (here’s one) with extensive discussion of sampling design, but they typically give minimal attention to choosing the people who actually answer the questions. An otherwise quite useful book The Power of Survey Design which I have on my bookshelf, says nothing on the topic.
Who provides the information matters, and it matters more in some cases than others. People in the household might all respond the same way when reporting on simple measures of household conditions, e.g. the material used for the roof, but for much of what is captured in surveys different people in the household will have different responses.
Bardasi, Beegle, Dillon, and Serneels (2011) [ungated version] examined this issue by experimentally varying whether labor data was collected via self-reporting (from the worker) vs. via proxy reports from another randomly selected member of the household. They found the following:
Response by proxy leads to substantially lower male employment rates due to lower reporting of participation in agriculture – while for women there is no effect. The discrepancies between proxy and self-reporting are reduced when the spouse is selected as the proxy and when the proxy is an individual with some education….
These findings suggest that there is no substantial benefit of self-reporting for women (a similar result was found for children, as discussed in Dillon et al., 2010), but there is for men, whose employment rates and distribution across sector of activity are affected by proxy reporting.
These findings suggest that there is no substantial benefit of self-reporting for women (a similar result was found for children, as discussed in Dillon et al., 2010), but there is for men, whose employment rates and distribution across sector of activity are affected by proxy reporting.
(The paper also reviews the literature on self-reports vs. proxy reports, which comes almost entirely from developed countries.)
The employment difference for men between the two types of report came from the fact that proxy reporters were much more likely to categorize men as being out of work rather than working in agriculture. The authors point out that it’s not clear whether the proxy reports are necessarily closer to the “truth” than the self-reports. One could imagine that some men who are objectively unemployed men might prefer to describe themselves as working.
The Bardasi et al. study shows that who reports can matter even for seemingly objective information. For a survey that captures subjective data the choice of respondent is more crucial, because there is no ultimate “true” answer other than what the particular respondent says.
Here are roughly the set of different practices I believe are used in LSMS-type household surveys (entirely or almost entirely objective information):
H1) The household head is interviewed.
H2) Whomever is available when the enumerator shows up is interviewed.
H3) Multiple members of the household are interviewed, with the most knowledgeable respondents providing different pieces of information wherever possible.
H2) Whomever is available when the enumerator shows up is interviewed.
H3) Multiple members of the household are interviewed, with the most knowledgeable respondents providing different pieces of information wherever possible.
Approach H3 is usually the recommended approach, but I believe what is actually done in practice is more often H2. Talking to multiple members of the household will often require visiting several times, and the enumerator’s incentives are almost always to just get the information as quickly as possible. Even if it is possible to get enumerators to stick to a protocol that requires them to work harder, multiple visits can substantially push up the cost of a survey.
For perception surveys, the procedures include the following:
P1) The household head is interviewed
P2) Whomever is available when the enumerator shows up is interviewed
P3) A random adult among those present at the time of visit is interviewed
P4) A random adult among all those on the household roster is interviewed.
P2) Whomever is available when the enumerator shows up is interviewed
P3) A random adult among those present at the time of visit is interviewed
P4) A random adult among all those on the household roster is interviewed.
P4 is clearly the way to get a representative sample of the adult population. This is the procedure at least on paper for the Afrobarometer perception surveys, with the additional stipulation that “Each interviewer alternates in each household between interviewing a man and interviewing a woman to ensure gender balance in the sample.”
For the High Frequency South Sudan Survey pilot, which collects a mix of objective and subjective data, we tried using the Afrobarometer approach. As with interviewing multiple members of the household, this will often mean visiting the household more than once, and in practice we found it difficult to prevent the enumerator from picking someone who is home during first visit. As a result, the sample is disproportionately women—in many households only women are home during the day when most visits take place.
For the Vietnam survey pilot, we realized that the selection of the respondent is absolutely critical. One focus of this survey is the frame of reference people have when they think of inequality. We expected that rural households would have narrower frames, and indeed that appears to be the case beyond our expectations: in the ethnic minority communities we visited, some residents had never traveled beyond their villages, and many cannot speak Vietnamese fluently. Many had a hard time even describing wealthier and poor people. Frames of reference also varied greatly with individual experience, which was correlated with age and gender. Older respondents, particularly women, appeared less likely to have been much beyond their villages, while many young men and some women had travelled at least locally for work. Interviewing only older household heads, or only those present at the time of first visit, would generate a sample of people with less experience with the broader world and thus narrower frames of reference. In a previous pilot, youth in isolated villages had far broader frames of reference than their parents and grandparents.
We will try for the Vietnam survey to get the enumerators to use the “P4” approach, but with expectation that compliance by enumerators will not be perfect. We may end up reweighting the survey respondents in the end (using age, gender, and education) to match the profile of the target population. An ex-post weighting scheme cannot, however, correct for selection on unobservables, which may be a factor in this case.
Even conditional on observables, household members who are home when enumerators show up will probably have less experience with the broader world than those are away.
Our struggle with this issue reflects a broader problem in conducting household surveys (and implementing development programs in general): what actually happens on the ground may be far from what the pristine survey manual says. In many surveys, these deviations-from-protocol are hidden from view. But it’s much better to be up front and correct for them as best as possible rather than sweep them under the rug. What experiences have you had where the survey didn’t go quite as you had planned? (Feel free to answer anonymously!)
Join the Conversation