Ten years ago when I was a graduate student piloting questionnaires in rural Indonesia, I sat with a translator and an elderly farmer in his front yard. Mid-way through the interview I asked this farmer the first of several standard questions related to general well-being and life satisfaction: “Thinking about your own life and personal circumstances, how satisfied are you with your life as a whole?” The farmer stared at us with a look of bewilderment on his face. So we asked a second time in a slow sympathetic tone. The farmer appeared to understand our words but it was clear he was greatly troubled by the notion of satisfaction. He responded: “How could I answer this? There is no other life I have lived so how do I know if I am satisfied?” Flummoxed by his statements, we still pressed for a response. He started shaking his head as he realized my team expected an answer to this mystery. “I suppose I am fairly satisfied” he shrugged. Next question.
This anecdote stayed with me over the years, even as I have read numerous papers (and written some myself) that analyze the correlates and determinants of perceptual data such as the life satisfaction question above. We call data of this type subjective data, but how reliable is it? For example, when I look at various patient surveys of client satisfaction with health services, often the reported satisfaction is nearly universal – perhaps 90% of all patients report some degree of satisfaction. Is really almost everyone satisfied with their doctor visit? And if so, what does such a finding imply for the quality of care provided, which is presumably what researchers care about the most?
Economists have long been suspicious of subjective data. In the psychological survey literature a great deal of work has explored the meaning attributed to the responses to subjective question. Bertrand and Mullainathan summarize a good bit of this literature and highlight certain problems including the possibility that subjective views and attitudes may not exist in a coherent form – often different responses are given to the same questions even across a relatively short period of time. Another problem, as witnessed in the anecdote above, is that people may be reluctant to admit they don’t have a view or attitude and so report what they think the interviewer wants to hear. In these cases, the general analytic challenge is that the subjective response does not accurately measure what the researcher hopes to measure and, even worse, this mis-measurement may be correlated with the very characteristics that the researcher uses to explain the subjective response. As Bertrand and Mullainathan conclude “subjective variables cannot be reasonably used as dependent variables, given that the measurement error likely correlates in a very causal way with the explanatory variables.”
But economists have long used subjective data in their analyses, hoping to uncover important determinants and correlates of subjective response. And often they succeed in this endeavor. I’ve blogged before about some of these successes in relation to life satisfaction and to job satisfaction. The volume of work is immense. Here are some interesting studies I’ve come across just in the past few days:
- Beliefs concerning the role of the state in ensuring individual economic security appear to be strongly related to past exposure to political systems, at least when comparing individuals raised in East vs. West Germany before unification, as Alesina and Schundeln have shown.
- The self-employed consistently report higher job satisfaction than workers employed in organizations, even after controlling for income and hours worked, as found by Benz and Frey. People appear to not only value the “outcomes” of a job – the earnings and time used – but also the process leading to these outcomes. In particular, workers particularly value the freedom to follow their own initiatives. These effects persist after controlling for individual fixed effects so this association between satisfaction and self-employment is not merely the selection of satisfied individuals into self-employed occupations.
- Ryff and colleagues find that psychological well-being is significantly linked to numerous biological markers that reflect various dimensions of affect and physical health. For example, a sense of “purpose in life” is significantly associated with higher levels of HDL cholesterol (i.e. the good kind) and lower levels of cortisol – a hormonal marker of stress.
So there are interpretive problems with subjective data, but also associations that appear to convey a good bit of meaning. Obviously the discipline will continue to engage in research of subjective data, as I believe it should. But I also believe a little hermeneutics – the study of the practice of interpretation – will go a long way in enriching the analysis of subjective data. With the example of life satisfaction at the start of this blog, I wanted to understand better how the farmer interprets the term “satisfaction” (and the term “life” for that matter). What is the referent when a respondent says he or she is satisfied? And does this referent vary across the population – if rich and poor individuals have a different referent in mind when they assess satisfaction, can we meaningfully contrast satisfaction determinants between the rich and poor?
Field testing can, at a minimum , involve asking these subjective survey questions and carefully observing the responses – i.e. determining whether the respondent “appears” to comprehend the question. Although in this exercise we won’t know the precise interpretation the respondent holds. Perhaps a formalized field hermeneutics can take this process one-step further. Following Williams, Coyle, and Healy, who investigate the meaning of satisfaction with health services in a small sample of British patients, a “hermeneutics of satisfaction study” would first administer the subjective questionnaire to test subjects. After that, qualitative interviews might start with unstructured discussion around notions of adequacy, satisfaction, etc. in a life context. The investigator can then read the responses to the subjective questionnaire and discuss the respondent’s interpretation of each individual question and response.
Imagine that large studies of subjective data would, through this exercise, be able to explicitly define the meaning of the term as held by population(s) of study. That would be pretty satisfying.
Join the Conversation