In ancient Greek times, important decisions were never made without consulting the high priestess at the Oracle of Delphi. She would deliver wisdom from the gods, although this advice was sometimes vague or confusing, and was often misinterpreted by mortals. Today I bring word that the high priestess and priests (Athey, Abadie, Imbens and Wooldridge) have delivered new wisdom from the god of econometrics on the important decision of when should you cluster standard errors. This is definitely one of life’s most important questions, as any keen player of seminar bingo can surely attest. In case their paper is all greek to you (half of it literally is), I will attempt to summarize their recommendations, so that your standard errors may be heavenly.
David McKenzie's blog
- Another reason to justify random selection – Michael Schulson in Aeon “there are plenty of situations when random chance really is your best option. And those situations might be far more prevalent in our modern lives than we generally admit.” An interesting discussion drawing on anthropology of how different cultures have introduced randomness into decision-making, with the advantage being that it stops you using bad reasons for making decisions. “we might want to come to terms with the reality of our situation, which is that our lives are dominated by uncertainty, biases, subjective judgments and the vagaries of chance”
- Maitreesh Ghatak reviews Jean Dreze’s new book “Sense and Solidarity - Jholawala Economics for Everyone”. See also this twitter thread by Abhijeet Singh on whether Dreze is underappreciated in development economics.
- On the future development blog, Jishnu Das discusses recent experiments on public-private provision of education in Liberia and Pakistan, takes on Bridge Academies, and highlights the importance of good measurement: “in Liberia, Romero et al. tracked students to ensure that schools could not “game” the evaluation by sending weaker children home: “We took great care to avoid differential attrition: Enumerators conducting student assessments participated in extra training on tracking and its importance, and dedicated generous time to tracking. Students were tracked to their homes and tested there when not available at school. Finding children who have left a school is like finding a needle in a haystack. In a country where only 42 percent have access to a cell phone, it’s heroism.”
- On Straight Talk on Evidence, James Heckman and co-authors get taken to task for torturing data to overstate findings in a 2014 Science article on the long-term effects of the Abecedarian ECD program. Specific criticisms on sample size (and its reporting) and multiple comparisons. Response and a rejoinder follow the post...
Randomization inference has been increasingly recommended as a way of analyzing data from randomized experiments, especially in samples with a small number of observations, with clustered randomization, or with high leverage (see for example Alwyn Young’s paper, and the books by Imbens and Rubin, and Gerber and Green). However, one of the barriers to widespread usage in development economics has been that, to date, no simple commands for implementing this in Stata have been available, requiring authors to program from scratch.
This has now changed with a new command ritest written by Simon Hess, a PhD student who I met just over a week ago at Goethe University in Frankfurt. This command is extremely simple to use, so I thought I would introduce it and share some tips after playing around with it a little. The Stata journal article is also now out.
How do I get this command?
Simply type findit ritest in Stata.
[edit: that will get the version from the Stata journal. However, to get the most recent version with a couple of bug fixes noted below, type
net describe ritest, from(https://raw.githubusercontent.com/simonheb/ritest/master/)
- Excellent VoxDev piece by Donaldson and Atkin on how high intra-country trade costs are in Ethiopia and Nigeria (and how they go about measuring this).
- Uri Simonsohn on why using a quadratic to test for a U-shaped relationship is a very bad idea and what to do instead.
- “Glasses askew and gray hair tousled, Scott Rozelle jumps into a corral filled with rubber balls and starts mixing it up with several toddlers”. So begins a feature in Science on Scott’s experiment in progress on parenting and early childhood education in China…including the challenges of keeping a control group in this setting “Rozelle says that when he sees kids in the randomly selected control villages “I often want to take them in my arms and move them to the treatment villages””.
- Let’s start with your approach to teaching development economics at the graduate level. The class when you taught David in 1999 was heavy on the agricultural household model and understanding micro development through different types of market failures. Most classes would involve in-depth discussion of one or at most two papers, with a student assigned most weeks to lead this discussion. There was a lot of discussion of the empirical methods in different papers, but no replication tasks and the only empirical work was as part of a term paper. How has your approach to teaching development changed (or not) since this time?
Try as I might, I have made little progress on changing my basic approach to teaching. The papers and topics have changed, but the essence of my graduate teaching remains the in-depth discussion of a paper or two each class. I’ve tried to expand the use of problem sets, and had a number of years of replication assignments. The first was hindered by my own inadequate energy (it’s hard making up decent questions!). I found that replication exercises required too much time and effort in data cleaning by students relative to their learning gain. Students were spending too much time cleaning, merging and recreating variables and too little time thinking about the ideas in the paper. I’ll reassess assigning replication this year, because there may now be enough well-documented replication datasets and programs available. With these as a starting point, it would be possible to get quickly into substantive issues in the context of a replication.
In the latest JEL, Parker and Todd survey the literature on Progresa/Oportunidades: some bits of interest to me included:
- CCTs have now been used in 60+ countries;
- over 100 papers have been published using the Progresa/Oportunidades data, with at least 787 hypotheses tested – multiple testing corrections don’t change the conclusions that the program had health and education effects, but do cast doubt on papers claiming impacts on gender issues and demographic outcomes;
- FN 16 which notes that at the individual level, there are significant differences in 32% of the 187 characteristics on which baseline balance is tested, with the authors arguing that this is because the large sample size leads to a tendency to reject the null at conventional levels – a point that seems inconsistent with use of the same significant levels for measuring treatment effects;
- Two decades later, we still don’t know whether Progresa led to more learning, just more years in school;
- One of the few negative impacts is an increase in deforestation in communities which received the CCT
- Dave Evans asks whether it matters which co-author submits a paper, and summarizes responses from several editors; he also gives a short summary of a panel on how to effectively communicate results to policymakers.
- On the IPA blog, Rachel Glennerster and Claire Walsh argue that it’s time to rethink how we measure women’s household decision-making power in impact evaluations - congrats also to Rachel for being named DFID’s new chief economist.
- At VoxDev, Alaka Holla blogs about how IT training in Nigeria may have changed aspirations for women, and Natalie Bau and Jishnu Das on the market for teachers in Pakistan
- On the AfricaCan blog, Gautam Bastian and Sreelakshmi Papineni report on a test of whether quarterly or monthly cash payments work better for women in Nigeria – finding similar impacts for both.
- Tips from journal editors for young economists, summarizing a panel session at the European Economics Association.
I recently received an email from a researcher who was interested in trying to re-interview participants in one of my experiments to test several theories about whether that intervention had impacts on political participation and other political outcomes. I get these requests infrequently, but this is by no means the first. Another example in the last year was someone who had done in-depth qualitative interviews on participants in a different experiment of mine, and then wanted to be able to link their responses on my surveys to their responses on his. I imagine I am not alone in getting such requests, and I don’t think there is a one-size-fits-all response to when this can be possible, so thought I would set out some thoughts about the issues here, and see if others can also share their thoughts/experiences.
Confidentiality and Informed Consent: typically when participants are invited to respond to a survey or participate in a study they are told i) that the purpose of the survey is X ,and will perhaps involve a baseline survey and several follow-ups; and ii) all responses they provide will be kept confidential and used for research purposes only. These factors make it hard to then hand over identifying information about respondents to another researcher.
However, I think this can be addressed via the following system:
- While the rest of us took August off blogging, Dave Evans blogged about how information can improve service delivery on Let’s Talk Development.
- There was a lot of discussion about gender and economics. Rebecca Thornton helpfully has put together a list of gender and economics links.
- Marc Bellemare has good advice on how to cite intelligently.
- As more and more papers rely on large admin datasets, there are questions about who gets to use this data and under what conditions. The 74 million has an interesting discussion about this in the context of school lottery data from Louisiana.
- On the data blog – a new LSMS guidebook for using non-standard units like local tins or bunches in measuring food and agricultural quantities.