Angela Duckworth’s new book Grit: The Power of Passion and Perseverance has been launched with great fanfare, reaching number two on the NY Times Nonfiction bestseller list. She recently gave a very polished and smooth book launch talk to a packed audience at the World Bank, and is working with World Bank colleagues on improving grit in classrooms in Macedonia. Billed as giving “the secret to outstanding achievement” I was interested in reading the book as both a researcher and a parent. I thought I’d continue my book reviews series with some thoughts on the book.
- On selecting what variables to gather data for in your impact evaluation: Carneiro et al. have a new paper out – “Optimal Data Collection for Randomized Control Trials” – which argues that if you have a household survey or census in advance, you can use an algorithm to select the right covariates, potentially reducing data collection costs or improving precision substantially.
This post was co-authored by Sacha Dray, Felipe Dunsch, and Marcus Holmlund.
Impact evaluation needs data, and often research teams collect this from scratch. Raw data fresh from the field is a bit like dirty laundry: it needs cleaning. Some stains are unavoidable – we all spill wine/sauce/coffee on ourselves from time to time, which is mildly frustrating but easily discarded as a fact of life, a random occurrence. But as these occurrences become regular we might begin to ask ourselves whether something is systematically wrong.
- In the Richard T. Ely lecture, John Campbell discusses the challenge of consumer financial regulation – he distinguishes 5 dimensions of financial ignorance many households exhibit: 1) ignorance of even the most basic financial concepts (financial illiteracy); 2) ignorance of contract terms (such as not knowing about the fees build into credit cards or when mortgage interest rates can change); 3) ignorance of financial history – relying too much on own experiences and the recent past; 4) ignorance of self- a lot of financially illiterate people are over-confident about their abilities; and 5) ignorance of incentives, strategy and equilibrium – failure to take account of incentives faced by other parties to transactions. Given these problems, and the limits of financial education and disclosure requirements to fix them, he discusses what financial regulation is needed: “consumer financial regulation must confront the trade-off between the benefits of intervention to behavioral agents, and the costs to rational agents….the task for economists is to confront this trade-off explicitly”
- development impact links
Last week I attended a workshop on Subjective Expectations at the New York Fed. There were 24 new papers on using subjective probabilities and subjective expectations in both developed and developing country settings. I thought I’d summarize some of the things I learned or that I thought most of interest to me or potentially our readers:
Subjective Expectations don’t provide a substitute for impact evaluation
I presented a new paper I have that is based on the large business plan competition I conducted an impact evaluation of in Nigeria. Three years after applying for the program, I elicited expectations from the treatment group (competition winners) of what their businesses would be like had they not won, and from the control group of what their businesses would have been like had they won. The key question of interest is whether these individuals can form accurate counterfactuals. If they could, this would allow us a way to measure impacts of programs without control groups (just ask the treated for counterfactuals), and to derive individual-level treatment effects. Unfortunately the results show neither the treatment nor control group can form accurate counterfactuals. Both overestimate how important the program was for businesses: the treatment group thinks they would be doing worse off if they had lost than the control group actually is doing, while the control group thinks they would be doing much better than the treatment group is actually doing. In a dynamic environment, where businesses are changing rapidly, it doesn’t seem that subjective expectations can offer a substitute for impact evaluation counterfactuals.
- This week in macro measurement: “‘Laws are like sausages, it is better not to see them being made’ said Otto von Bismarck. Turns out you can probably add GDP to that list.” Duncan Green gives a useful summary of The Economist’s extensive critique of GDP, how it is becoming decreasingly useful over time, and how it could be better.
- I liked the recent Planet Money podcast #698 (a long way home) – there is an interesting discussion of why a lottery is held for access to a housing assistance program in Connecticut, and how they ended up with a lottery rather than other systems of allocating resources – and a great quote about the mishmash of anti-poverty programs in the U.S. which, paraphrasing, is basically “it is not like Congress ever sat down and said what is the best use of the money we set aside to fight poverty” but rather how many different programs have come up over time, all with their own rules and constituencies.
- The latest Journal of Economic Perspectives has a symposium on inequality beyond income (US focused) and a paper on the billion prices project that I linked to a blog post on last week
- Should policy seek to promote small firms or large ones in Africa? Frances Teal on the CSAE blog: “Policy rhetoric focuses on the problems faced by small firms. Data from Ghana over the period for which we have it suggests that it is large firms that face the problems. Unloved possibly because they are not seen as beautiful they are vital for the output of the sector. Policy, not for the first time in Africa, seems to be focused on completely the wrong problem.”
- The Los Angeles Review of Books has a longish discussion on placebo effects in reviewing an anthology on placebos, and how they really don’t work as much of the time in medicine as many people think, and how the term might be over-used in social sciences.
- New in the working paper series: Is living in African cities expensive? Using data from the “2011 round of the International Comparison Program. Readjusting the calculated price levels from national to urban levels, the analysis indicates that African cities are relatively more expensive, despite having lower income levels. The price levels of goods and services consumed by households are up to 31percent higher in Sub-Saharan Africa than in other low- and middle-income countries, relative to their income levels. Food and non-alcoholic beverages are especially expensive, with price levels around 35 percent higher than in other countries.”
Researchers put a lot of effort into developing survey questionnaires designed to measure key outcomes of interest for their impact evaluations. But every now and then, despite efforts piloting and fine-tuning surveys, some of the questions end up “not working”. The result is data that are so noisy and/or missing for so many observations that you may not want to use them in the final analysis. Just as pre-analysis plans have a role in specifying in advance what variables you will use to test which hypotheses, perhaps we also want to specify some rules in advance for when we won’t use the data we’ve collected. This post is a first attempt at doing so.