Syndicate content

Trouble with pre-analysis plans? Try these three weird tricks.

Owen Ozier's picture
Pre-analysis plans increase the chances that published results are true by restricting researchers’ ability to data-mine.  Unfortunately, writing a pre-analysis plan isn’t easy, nor is it without costs, as discussed in recent work by Olken and Coffman and Niederle. Two recent working papers - “Split-Sample Strategies for Avoiding False Discoveries,” by Michael L.

What does a game-theoretic model with belief-dependent preferences teach us about how to randomize?

David McKenzie's picture

The June 2017 issue of the Economic Journal has a paper entitled “Assignment procedure biases in randomized policy experiments” (ungated version). The abstract summarizes the claim of the paper:
“We analyse theoretically encouragement and resentful demoralisation in RCTs and show that these might be rooted in the same behavioural trait –people’s propensity to act reciprocally. When people are motivated by reciprocity, the choice of assignment procedure influences the RCTs’ findings. We show that even credible and explicit randomisation procedures do not guarantee an unbiased prediction of the impact of policy interventions; however, they minimise any bias relative to other less transparent assignment procedures.”

Of particular interest to our readers might be the conclusion “Finally, we have shown that the assignment procedure bias is minimised by public randomisation. If possible, public lotteries should be used to allocated subjects into the two groups”

Given this recommendation, I thought it worth discussing how they get to this conclusion, and whether I agree that public randomization will minimize such bias.

Weekly links July 7: Making Jakarta Traffic Worse, Patient Kids and Hungry Judges, Competing for Brides by Pushing up Home Prices, and More…

David McKenzie's picture
  • In this week’s Science, Rema Hanna, Gabriel Kreindler, and Ben Olken look what happened when Jakarta abruptly ended HOV rules – showing how traffic got worse for everyone. Nice example of using Google traffic data – MIT news has a summary and discussion of how the research took place : “The key thing we did is to start collecting traffic data immediately,” Hanna explains. “Within 48 hours of the policy announcement, we were regularly having our computers check Google Maps every 10 minutes to check current traffic speeds on several roads in Jakarta. ... By starting so quickly we were able to capture real-time traffic conditions while the HOV policy was still in effect. We then compared the changes in traffic before and after the policy change.”All told, the impact of changing the HOV policy was highly significant. After the HOV policy was abandoned, the average speed of Jakarta’s rush hour traffic declined from about 17 to 12 miles per hour in the mornings, and from about 13 to 7 miles per hour in the evenings”
  • From NPR’s Goats and Soda: 4-year kids of Cameroonian subsistence farmers take the marshmallow test, as do German kids – who do you think did best?

Teacher Coaching: What We Know

David Evans's picture
“Teacher coaching has emerged as a promising alternative to traditional models of professional development.” In Kraft, Blazar, and Hogan’s newly updated review “The Effect of Teacher Coaching on Instruction and Achievement: A Meta-Analysis of the Causal Evidence,” they highlight that reviews of the literature on teacher professional development (i.e., training teachers who are already on the job) highlight a few promising characteristics of effect

Weekly links June 30: 7th grade development economics, the beginning at the end approach, stuff that happened a long time ago still impacts today, and more…

David McKenzie's picture
  • How to teach development economics in 20 minutes to 7th graders – Dave Evans explains his method.
  • The “beginning at the end” approach to experimentation – written from the point of view of business start-ups, but could easily apply to policy experiment work too “The typical approach to research is to start with a problem. In business, this often leads to identifying a lot of vague unknowns—a “broad area of ignorance” as Andreasen calls it—and leaves a loosely defined goal of simply reducing ignorance…“Beginning at the end” means that you determine what decision you’ll make when you know the results of your research, first, and let that dictate what data you need to collect and what your results need to look like in order to make that decision.”

Can temporary subsidies and agricultural extension build sustainable adoption?

Markus Goldstein's picture
A fair number of governments in developing countries support agricultural subsidy programs.   One of the arguments for these subsidies is that there is some kind of market failure (information is often cited) that the subsidy is meant to overcome.    So, that means when the subsidy is removed (which is the politically hard part), we should see adoption sustained.    There isn’t much clear evidence on this, but two recent papers provide some insight.
  

Should we require balance t-tests of baseline observables in randomized experiments?

David McKenzie's picture

I received an email recently from a major funder of impact evaluations who wanted my advice on the following question regarding testing baseline balance in randomized experiments:

Should we continue to ask our grantees to do t-tests and f-tests to assess the differences in the variables in the balance tables during the baseline?  

Weekly links June 23: VoxDev launches, uncountable Nigerians, a challenge to prospect theory, and more…

David McKenzie's picture

Odds are you’re measuring son preference incorrectly

Seema Jayachandran's picture
When investigating son-biased fertility preferences, the Demographic and Health Surveys (DHS) offer the go-to survey questions:
  • If you could go back to the time you did not have any children and could choose exactly the number of children to have in your whole life, how many would that be?
  • How many of these children would you like to be boys, how many would you like to be girls, and for how many would it not matter if it’s a boy or a girl?

Pages