Often in IE (and in social research more generally) the researcher wishes to know respondent views or information regarded as highly sensitive and hence difficult to directly elicit through survey. There are numerous examples of this sensitive information – sexual history especially as it relates to risky or taboo practices, violence in the home, and political or religious views. In a previous blog I wrote about how new survey technologies may help elicit truthful responses. But non-truthful response remains a common challenge. Fortunately direct elicitation is not the only survey method available to researchers.
One alternative survey method – the list method – takes advantage of the fact that social research is largely concerned with drawing inferences on a population rather than assessing any one individual view. This method also takes advantage of the ability to randomize the questions asked to survey respondents in order to estimate the average incidence of a sensitive view or behavior in the studied population.
Also known as the item count technique, the list method presents respondents with a list of items and asks for the total number of items with which they agree. The population incidence related to the sensitive question is identified through the differences in the summary total across respondents given the “control” version of the questionnaire, which does not include the sensitive item, and the “treatment” version, which does.
For example, Blair, Imai, and Lyall (BIL), working in the highly challenging research environment of conflict regions in Afghanistan, seek to measure population support for the international coalition of forces. Among control respondents, they present the following list:
Karzai Government, National Solidarity Program, Local Farmers
and ask only for the total number of groups or individuals that he (or she) broadly supports (this would range from 0 to 3 here). The same instructions are given to the treatment group of respondents, but instead they are read the following list:
Karzai Government, National Solidarity Program, Local Farmers, Foreign Forces
By comparing the responses between treatment and control, the authors estimate a 16% support rate among the target population for the foreign forces.
In order to yield a valid estimate, this type of analysis requires that the addition of a sensitive item to the list does not alter the aggregate response for the control items. Often this seems like an innocuous assumption but one that is very hard to verify. Another possible drawback with this technique: it is not entirely free of bias since respondents recognize they automatically reveal their preference for the sensitive issue if they agree with either all of the items or none. Hence respondents may avoid choosing extremes. These “floor” and “ceiling” effects are unavoidable. Nevertheless the list method is often seen as an improvement in accuracy over previous randomized response efforts that attempt to get at sensitive private information.
Fortunately there is another indirect method, the endorsement method, which can sometimes be used to cross-validate the responses given by the list method. The endorsement method asks respondents to rate their support for an issue or policy, but randomizes across respondents whether the issue is described as supported by a sensitive social group or actor. Again, in the case of BIL, control respondents were asked something like:
A recent proposal calls for the sweeping reform of the Afghan prison system, including the upgrading of facilities… How do you feel about this proposal?
In contrast, the treatment group was read the following:
A recent proposal by the ISAF (Foreign Forces) calls for the sweeping reform of the Afghan prison system, including the upgrading of facilities… How do you feel about this proposal?
Typically in the endorsement method multiple questions regarding different proposals are read so that measurement does not rely on the view of a single proposed policy. BIL actually lists four policies that were widely discussed in the media at the time of the survey and estimates an overall support rate for the ISAF of 17.5%.
The fact that the two methods yield almost identical support rates is reassuring, especially since we might expect indirect methods to be subject to a higher degree of measurement error (in part because responses are highly sensitive to implementation details). That is, these indirect methods may indeed have less bias then direct methods, but may be more imprecise. BIL proposes to improve the precision of indirect elicitation by combining both methods in a joint analysis.
To enact this joint analysis we must assume that there is an unobserved, i.e. latent, level of support that drives responses to both types of question. (The close estimates of overall support from the two methods give us some confidence that this is a safe assumption.) BIL then adopts a specified parametric form to yield a maximum likelihood estimator that in turn gives an overall approval rating and can explore how approval varies with covariates of interest.
In the joint estimate the overall support rate is 19% (and more precisely estimated than from either solo method). The estimated support also co-varies in what seems to be sensible and expected ways – for example households that report victimization by Taliban view ISAF more favorably whereas households that report victimization from ISAF view them less favorably. The details are in the paper, as well some interesting discussion on social desirability bias among different sub-groups.
While a “magic survey truth machine” remains far in the realm of science fiction, I’d like to think the inventive methods discussed here take us a little further down the road of progress.
Join the Conversation