Syndicate content

multiple testing

Trouble with pre-analysis plans? Try these three weird tricks.

Owen Ozier's picture
Pre-analysis plans increase the chances that published results are true by restricting researchers’ ability to data-mine.  Unfortunately, writing a pre-analysis plan isn’t easy, nor is it without costs, as discussed in recent work by Olken and Coffman and Niederle. Two recent working papers - “Split-Sample Strategies for Avoiding False Discoveries,” by Michael L.

A pre-analysis plan checklist

David McKenzie's picture

A pre-analysis plan is a step-by-step plan setting out how a researcher will analyze data which is written in advance of them seeing this data (and ideally before collecting it in cases where the researcher is collecting the data). They are recently starting to become popular in the context of randomized experiments, with Casey et al. and Finkelstein et al.’s recent papers in the QJE both using them.

Tools of the Trade: A quick adjustment for multiple hypothesis testing

David McKenzie's picture

As our impact evaluations broaden to consider more and more possible outcomes of economic interventions (an extreme example being the 334 unique outcome variables considered by Casey et al. in their CDD evaluation) and increasingly investigate the channels of impact through subgroup heterogeneity analysis, the issue of multiple hypothesis testing is gaining increasing prominence.

The new big randomized trial that you should know about – randomized Medicaid

David McKenzie's picture

Given the massive debate in the U.S. about government health insurance, the just released results of a new experiment are justly making headlines. In 2004, the state of Oregon, due to budgetary shortfalls, closed its public health insurance program for low-income people. In early 2008, the state decided it had enough budget to fund 10,000 new spots. Given that it expected demand for these new slots to far exceed supply, the state Government opened up a sign-up window, getting 90,000 people to sign-up for a waitlist, and then used random lottery draws to select people from the waitlist.