Last week I attended a workshop on Subjective Expectations at the New York Fed. There were 24 new papers on using subjective probabilities and subjective expectations in both developed and developing country settings. I thought I’d summarize some of the things I learned or that I thought most of interest to me or potentially our readers:
Subjective Expectations don’t provide a substitute for impact evaluation
I presented a new paper I have that is based on the large business plan competition I conducted an impact evaluation of in Nigeria. Three years after applying for the program, I elicited expectations from the treatment group (competition winners) of what their businesses would be like had they not won, and from the control group of what their businesses would have been like had they won. The key question of interest is whether these individuals can form accurate counterfactuals. If they could, this would allow us a way to measure impacts of programs without control groups (just ask the treated for counterfactuals), and to derive individual-level treatment effects. Unfortunately the results show neither the treatment nor control group can form accurate counterfactuals. Both overestimate how important the program was for businesses: the treatment group thinks they would be doing worse off if they had lost than the control group actually is doing, while the control group thinks they would be doing much better than the treatment group is actually doing. In a dynamic environment, where businesses are changing rapidly, it doesn’t seem that subjective expectations can offer a substitute for impact evaluation counterfactuals.
Designing Survey Questions on Subjective Expectations: Rounding and Tail Events
A common feature of elicited subjective expectations are that there is often clumping at 0, 50, and 100, and then most other answers are a multiple of 10 or 5; except in the tails where people will sometimes give answers like 1%, or 98%. There were several papers and discussions around how to use, interpret, and refine such data:
- Chuck Manski presented work using 5 rounds of the Health and Retirement Survey. He noted there are a couple of reasons for rounding- they may be used as a way of communicating (I tell you 50% rather than 48.7%), or to convey partial knowledge (I think the answer is in the range (45,55). He provided two ways of trying to distinguish between these approaches. The first, which is possible in surveys with lots of expectations questions, is to look at responses for a person across a range of expectations questions to infer how much they round (e.g. if I see you answer 45% for one question, and 60% for another, I might infer you round to no more than the nearest 5). Using an algorithm he can then infer intervals consistent with the responses across questions, and then suggests using partial identification methods to work with these intervals. The other approach is to just ask directly a follow-up question -“ when you said X percent, did you mean this as an exact number or were you rounding or approximating? What number or range of numbers did you have in mind when you said X?” – which comes at the cost of more questionnaire space.
- Wandi Bruine de Bruin pointed me to this paper of hers which provides an example of getting more precise answers to subjective probabilities of small events. Her setting is eliciting expectations of the chance of getting swine flu (H1N1). They use a two-step procedure – people first answer on a standard 0 to 100 visual scale, but then those who answer 0 or 1 are then asked a follow-up question which asks whether they think the chance is really 0, between 0 and 1/100,000, between 1/100,000 and 1/10,000, between 1/10,000 and 1/1,000, between 1/1,000 and 1/100, and 1%. This two-step procedure dramatically reduced the use of 0s.
- Matt Rabin discussed a number of biases in the way people form beliefs and subjective judgements. A couple of these have relevance for the way people answer questions about tail events. 1) If you isolate out a small probability event and ask about it, people are going to exaggerate the odds of it. 2) People (including most of us in the room) don’t understand well how probabilities work in the tails. He gave an example of tossing a coin 1000 times, and asked what was more likely, getting exactly 910 heads, or getting between 911 and 1000 heads? It turns out getting 910 heads is five times as likely as between 911 and 1000, but most people would say the latter is more likely. People do better if you use different range bins.
Doing fieldwork in your own backyard while in grad school
Adam Kapor presented some preliminary findings from work done while he was a grad student at Yale, on understanding school choice in the New Haven public schools. The New Haven public school choice system is not strategy-proof, and parents have to give preferences over elementary and high schools. They worked with the local school system, and surveyed 210 parents with pre-K or 8th grade kids, and elicited subjective beliefs about admission chances, knowledge about school choices, and other information needed to form a model of school preferences – they find they while people are unbiased on average, there are a bunch of people who appear to be wrong in their strategies. I’m always pleased to see more survey work taking place in the non-development literature, and given the many interesting problems and opportunities in many communities where grad schools are housed, it is somewhat surprising we don’t see more such work.
Some areas for contributions/future work by development economists
Not surprising for a conference hosted by the NY Fed, many of the studies were about expectations of inflation, housing prices, and retirement. The NY Fed’s Survey of Consumer Expectations is a monthly panel survey that is done online and keeps respondents in for up to 12 months, and has a whole array of expectations questions that were used for both building structural models and for some information experiments to see how expectations are revised. Here are some thoughts for where there is scope for development economists to contribute to the research frontier on subjective expectations:
- For the macroeconomists: building something like the survey of consumer expectations in developing countries – this regular, timely, information on inflation, labor market, and household finances could be invaluable in countries where the macroenvironment is much more unstable than that in the U.S. Ricardo Perez-Truglia presented a paper which compared responses in surveys in Argentina to those in the U.S. – finding in high-inflation Argentina there appears to be much less “rational inattention” to inflation than in the U.S.
- Learning how expectations are formed and how they are updated: there were a couple of papers that did an online survey which asked expectations, then provided people with some information (e.g. house prices in your area rose X% last year, or inflation last year was Y%), and then re-ask expectations to see how people update their beliefs. I think there is scope for some really interesting work in development on expectations for two types of actions: i) how do people form beliefs about important one-time actions (which school to attend? should I move to another city or country? Should I get married or have a first child?) – here there is little scope to iteratively update expectations in response to outcomes, and so the key is what determines expectations in the first place; and ii) how do people form beliefs about repeated actions, where there can be a lot of learning and updating (e.g. Hanan Jacoby presented work on groundwater markets in India, where farmers form expectations every year of how much groundwater will there be at the end of the season).
- How important are these behavioral biases outside of the lab setting: Matt Rabin noted that much of the evidence for things like the “law of small numbers” and “non-belief in the law of large numbers” (e.g. people predict unbiased coin will come up 50% of the time too much in small samples, and too little in large samples) come from lab settings, and it has been argued that people may make these mistakes less in real settings with high stakes. There may be some interesting real field settings where testing these ideas could be compelling.
Join the Conversation