Last year, I was in Nairobi, Kenya, along with some of my colleagues from the World Development Report (WDR) 2015, Mind, Society, and Behavior. We were there to set up the data collection efforts for a four-country study. One of the goals of this study was to replicate results from lab experiments that suggested poverty is a context that shapes economic decision-making amongst households.
One of our replication questions was a vignette proposed by Sendhil Mullainathan and Eldar Shafir in their book Scarcity.[1] This vignette measures whether the experience of poverty eliminates some behavioral inconsistencies exhibited by the more affluent. Shafir and Mullainathan’s findings show that commuters in Princeton, NJ were more likely to say that they would travel to another store for a $50 discount when purchasing a $100 product than when purchasing a $1,000 product. Less affluent individuals at a soup kitchen in Trenton, NJ, however, did not display this kind of inconsistency. For them, the marginal utility of money remained constant, regardless of the cost of the hypothetical product.
With the aim of replicating this question in four developing countries, we took the vignette to field, starting in Nairobi. After spending two days field-testing this hypothetical scenario, I was surprised that most of the respondents, particularly those from low-income households, said that they would not travel for the discount. Why was everyone hesitant to travel, regardless of the discount amount? Why are they seemingly exhibiting the preferences of the more affluent in the US?
Somewhat surprised by this discrepancy, I asked multiple respondents’ the reasons behind their choice. One person replied quite plainly, “There is no guarantee that the product will still be there once I go across town. It’s very likely that the product is gone by the time I get there.” Of course! By assuming the availability of the product, we had let our own implicit biases, based on our mental models, influence the design of the question. Since the original question was conducted in the United States, a developed country, implicit in the question was the assumption that availability is generally not a problem. However, for the respondents from less affluent communities, this assumption was not explicit. The principle of general scarcity applied, and the availability of an item could not be taken for granted, especially if it required travelling across town. Hence, by failing to put the word ‘guarantee’ in the vignette, we weren’t measuring their willingness to travel for the discount (and indirectly their marginal utility of money) but instead their hesitance that the product would be available across town.
This raises broader questions about challenges of collecting high quality data, especially when the objective is to measure the behaviors and attitudes of poor households. Based on our understandings of the behavioral science literature, few techniques that we used in the field to ensure better quality data were:
1) Designing shorter surveys: As the WDR 2015 explains, we all have limited cognitive capacity. The constant, day-to-day hard choices associated with poverty can tax an individual’s mental bandwidth and can lead to limited attention. Therefore by designing shorter surveys, one can ensure (still not guaranteed!) greater attention by the respondents and higher quality of responses.
2) Introducing personal distance: This can help overcome courtesy bias and elicit more accurate behavioral responses, especially when the topics are sensitive and stigmatized. One method of introducing distance is by asking questions in form of vignettes or hypothetical situations about themselves or a friend/neighbor.
And most importantly,
3) Field Test and Pilot: Context matters. And it varies across countries, income groups, and communities. Field-testing and piloting is crucial to make sure our own biases have not influenced the format and the structure of the questions asked. Without piloting, our intuition about those living in poverty will go untested and as the WDR 2015 informs us, even the development professionals have many biases!
One of our replication questions was a vignette proposed by Sendhil Mullainathan and Eldar Shafir in their book Scarcity.[1] This vignette measures whether the experience of poverty eliminates some behavioral inconsistencies exhibited by the more affluent. Shafir and Mullainathan’s findings show that commuters in Princeton, NJ were more likely to say that they would travel to another store for a $50 discount when purchasing a $100 product than when purchasing a $1,000 product. Less affluent individuals at a soup kitchen in Trenton, NJ, however, did not display this kind of inconsistency. For them, the marginal utility of money remained constant, regardless of the cost of the hypothetical product.
With the aim of replicating this question in four developing countries, we took the vignette to field, starting in Nairobi. After spending two days field-testing this hypothetical scenario, I was surprised that most of the respondents, particularly those from low-income households, said that they would not travel for the discount. Why was everyone hesitant to travel, regardless of the discount amount? Why are they seemingly exhibiting the preferences of the more affluent in the US?
Somewhat surprised by this discrepancy, I asked multiple respondents’ the reasons behind their choice. One person replied quite plainly, “There is no guarantee that the product will still be there once I go across town. It’s very likely that the product is gone by the time I get there.” Of course! By assuming the availability of the product, we had let our own implicit biases, based on our mental models, influence the design of the question. Since the original question was conducted in the United States, a developed country, implicit in the question was the assumption that availability is generally not a problem. However, for the respondents from less affluent communities, this assumption was not explicit. The principle of general scarcity applied, and the availability of an item could not be taken for granted, especially if it required travelling across town. Hence, by failing to put the word ‘guarantee’ in the vignette, we weren’t measuring their willingness to travel for the discount (and indirectly their marginal utility of money) but instead their hesitance that the product would be available across town.
This raises broader questions about challenges of collecting high quality data, especially when the objective is to measure the behaviors and attitudes of poor households. Based on our understandings of the behavioral science literature, few techniques that we used in the field to ensure better quality data were:
1) Designing shorter surveys: As the WDR 2015 explains, we all have limited cognitive capacity. The constant, day-to-day hard choices associated with poverty can tax an individual’s mental bandwidth and can lead to limited attention. Therefore by designing shorter surveys, one can ensure (still not guaranteed!) greater attention by the respondents and higher quality of responses.
2) Introducing personal distance: This can help overcome courtesy bias and elicit more accurate behavioral responses, especially when the topics are sensitive and stigmatized. One method of introducing distance is by asking questions in form of vignettes or hypothetical situations about themselves or a friend/neighbor.
And most importantly,
3) Field Test and Pilot: Context matters. And it varies across countries, income groups, and communities. Field-testing and piloting is crucial to make sure our own biases have not influenced the format and the structure of the questions asked. Without piloting, our intuition about those living in poverty will go untested and as the WDR 2015 informs us, even the development professionals have many biases!
[1] Mullainathan, Sendhil, and Eldar Shafir. Scarcity: Why Having Too Little Means so Much.
Join the Conversation