Coauthored with Quy-Toan Do
In response to my blog post last week, one of my colleagues stopped me in the hall and pointed out that I missed the point. So in response, I invited him to join this week for a discussion. Our discussion follows:
Toan: A survey without an underlying research question is like salt without pepper. What you need to do is talk about what questions the survey is designed to answer.
Markus: Absolutely right. The point last week was a point on how the context that respondents attribute to some kinds of questions can you get you highly variable answers. If you know why you are doing the survey, you can hone in on the context you are after, as well as getting a broader set of relevant questions.
Toan: Is this going to be an interesting blog? Seriously, what do you mean by context? Take the example of the divorced women that you gave. What you really are after is the theory – and that will tell you which states of the world you care about. I don’t know if it is a lack of theory amongst the empiricists or a lack of empirical knowledge amongst the theorists, but that is missing.
Markus: Absolutely – when you ask questions, and particularly when you ask trickier questions about things like property rights in divorce you want some theoretical model in the front (not the back) of your mind. So an example in this case might be that you care about the within household allocation of resources and individual’s outside options matter and hence you want to know how strong the woman’s rights over land are. And, as you point out, this will limit the states of the world you want to ask about (and inform how you ask about them).
Toan: Sure, and so the first thing you need to do is understand the institution of divorce and what divorce means in this country. Obviously, “divorce” means different things in different contexts. So you need to understand that and then marry that with your economic model. And only then can you properly design your survey questionnaire.
Markus: You betcha. And this is where piloting (with that model in mind) through tools like the cognitive interviews that I discussed last week are going to help you to refine both your knowledge of the way things work in that particular country and what kind of economic model might fit them. But of course, there are also cases where you just might want to know how much domestic violence there is in country X, so you can see whether or not this is an issue.
Toan: I don’t deny that there are surveys that you can use to get the scale and the scope of a problem such as intimate partner violence. But when you want to understand behavior and develop policy recommendations, you are going to need to have an economic model in mind – and this will narrow down and refine those broad questions.
Markus: And this will lead to a much longer set of questions (and more work on the setup side), so you want to make sure this is a dimension of the impact that you really care about. It also means we might want to be a little more careful about those results you get from running the program impact on one of those throw-away questions you included for the heck of it.
Toan: OK. The other thing to keep in mind is that there is an issue of comparability as well. The deeper you get, the more tied your questionnaire will be to the context (and your model) and the less you will have cross-country comparability. So that has to be a consideration as well – some surveys are meant to be diagnostic (like the DHS) and others are after the impacts of a specific intervention.
And I want to conclude by saying that what you meant by context is actually theory. So you are a theorist without knowing it, Monsieur Jourdain!
Markus: But what you mean by theory is what I mean by being really thoughtful about context (ie a general model that is shaped clearly by what is actually going on on the ground). So you’re an empiricist without knowing it, my dear Watson!
Join the Conversation