A Curated List of Our Postings on Technical Topics – Your One-Stop Shop for Methodology

|

This page in:

This is a curated list of our technical postings, to serve as a one-stop shop for your technical reading. I’ve focused here on our posts on methodological issues in impact evaluation – we also have a whole lot of posts on how to conduct surveys and measure certain concepts curated here. In lieu of our regular links this week, it is updated up to January 14, 2020

 

General

IEanalytics: introducing the Development Impact Evaluation wiki

 

Random Assignment

Allocating treatment and control with multiple applications per applicant and ranked choices

Is optimization just re-randomization redux? Thoughts on the "don't randomize, optimize" papers

Be an optimista, not a randomista, when you have small samples

Tips for randomization in the wild: adding a waitlist

How to randomize in the field

Stratified randomization and the FIFA world cup

Doing stratified randomization with uneven numbers in the Strata

How to randomize using many baseline variables

Public randomization ceremonies

Designing experiments to measure spillover effects

Mechanism experiments and opening up the black box

Sample weights and RCT design

Response to the policymaker complaint that randomized experiments take too much time

Have RCTs taken over development economics?

Definitions in RCTs with interference

What does a game-theoretic model with belief-dependent preferences teach us about how to randomize?

Incorporating participant welfare and ethics into RCTs

Are we over-investing in baselines?

Most good you can do, but for whom? (on spillovers)

Reporting requirements for ethical considerations in economics RCTs

Research with adolescents: issues surrounding consent

 

Pre-analysis plans and reporting

Pre-registration of studies to avoid fishing and allow transparent discovery

A joint test of orthogonality when testing for baseline balance

A pre-analysis plan check-list

The New Trial Registries

What isn’t reported in impact evaluations but maybe should be

Randomization checks: testing for joint orthogonality

An addendum to pre-analysis plans: pre-specifying when not to use data

A pre-analysis plan is the only way to take your p-values seriously

Trouble with pre-analysis plans? try these 3 weird tricks

Should we require balance t-tests on observables with randomized experiments?

Registered reports: piloting a pre-results review process at the JDE

Declaring and diagnosing research designs

Power to the Plan

How good are pre-analysis plans in practice? and lessons for writing/reviewing your next one

Registering studies when all you want is a little more credibility

 

Propensity Score Matching

Guido Imbens on clustering standard errors with matching

Testing different matching estimators as applied to job training programs

The covariate balanced propensity score

 

Difference-in-Differences

The often unspoken assumptions behind diff-in-diff

How big data helped us estimate the impact of an intervention with 0.8% take-up

What Are We Estimating When We Estimate Difference-in-Differences?

 

Regression Discontinuity

Curves in all the wrong places: Gelman and Imbens on why not to use higher-order polynomials in RD

Regression discontinuity with an implicit index

Tools of the trade: the regression kink design

 

Spatial Discontinuity

Using spatial variation in program performance to identify impacts

Spatial Jumps

 

Event Study Designs

Econometrics Sandbox: Event Study Designs & Co.

 

Synthetic Controls

Evaluating an Argentine tourism policy using synthetic controls: tan linda que enamora?

The synthetic control method, as applied to regulatory reforms

Do I need to “recruit” a control group for my trial?

 

Instrumental Variables

Rethinking identification under the Bartik/Shift-Share instruments

Judge leniency designs: Now not just for Crime Studies

 

Machine learning

How can machine learning and artificial intelligence be used in development interventions and analysis?

The ethics of machine learning

 

Other Evaluation Methods

Impact as narrative

Small n impact evaluation methods

Can we trust shoestring evaluations?

Using case studies to explore and explain complex interventions

 

Analysis

Another reason to prefer Ancova: dealing with measurement changes between baseline and follow-up

Endogenous stratification: the surprisingly easy way to bias your heterogeneous treatment effects and what to do instead

Why is difference-in-difference estimation still so popular in experimental analysis?

Regression adjustment in randomized experiments (part one, part two)

When to use survey weights in analysis

Adjustments for multiple hypothesis testing

Bounding approaches to deal with attrition

Linear probability models versus probits

Dealing with multiple lotteries

Estimating standard errors with small clusters (part one, part two)

Decomposition methods

Estimation of treatment effects with incomplete compliance

What does Alwyn Young's paper mean for analysis of experiments?

Winsorizing, testing balance, and dealing with incomplete take-up

You ran a field experiment, should you then run a regression?

Sometimes, increasingly, estimating the ITT is not enough in experiments

Finally an easy way to do randomization inference in Stata

When should you cluster standard errors? New wisdom from the econometrics oracle

Your go to regression specification is biased: here is the simple fix

What should you do when your random assignment gets compromised?

Econometrics Sandbox: Attrition strikes back!

Be careful with inference from 2x2 experiments and other cross-cutting designs

Experimental design and Ancova when everyone has the same baseline outcome value

Randomly Drawn Equators? (randomization inference to account for spatial correlation)

 

Power Calculations and Improving Power

Should I work with only a subsample of my control group if I have take-up problems?

Power calculations: what software should I use?

Does the intra-cluster correlation matter for power calculations if I am going to cluster my standard errors?

Power calculations for propensity score matching

Power calculations 101: dealing with incomplete take-up

Collecting more rounds of data to boost power

Improving power in small samples

Did you do your power calculations using standard deviations? Do them again.

Power calculations for regression discontinuity (part 1, part 2, part 3)

Power calculation software for randomized saturation experiments

Statistical power and the funnel of attribution

Should you over-sample compliers if budget is limited and you are concerned about take-up?

Take-up and the Inverse-Square Rule for Power Calculations Revisited: When does power not fall quite so drastically with take-up, and when does lower take-up increase power?

Why ex-post power using estimated effect sizes is bad, but an ex-post MDE is not

 

On External Validity

Getting beyond the mirage of external validity

All those external validity issues with impacts? They apply to costs too

External validity as seen from other quantitative social sciences and the gaps in our practices

Towards a more systematic approach to external validity: understanding site selection bias

Weighting for external validity

Will that successful intervention over there get results here?

Learn to live without external validity

Why the external validity of regression estimates can be less than you think

Why similarity is the wrong concept for external validity

A rant on the external validity double standard

Towards policy irrelevance: on the experimental arms race

What's wrong in how we do impact evaluation?

What's in a title? Signaling external validity through titles in development economics

A framework for taking evidence from one setting to another

Informing policy with research that is more than the sum of the parts

External Validity Musings

 

Jargony Terms in Impact Evaluations

A proposed taxonomy of behavioral responses to evaluation

Quantifying the Hawthorne effect

The Hawthorne Effect

The John Henry Effect

Placebo effects

Clinical Equipoise

Social Desirability Bias/Experimenter Demand Effects and more on Experimenter Demand Effects

 

Stata Tricks

Generating regression and summary statistics tables in Stata

Graphing impacts with Standard Error Bars

Calculating the intra-cluster correlation

Generating regression and summary statistics tables in Stata: A checklist and code

Stata code for correlated random coefficient models

Finally an easy way to do randomization inference in Stata

IEanalytics: introducing ietoolkit

Five small things I've learned recently

Tools of the trade: Using iemargins to graph impacts with standard error bars (IE Analytics update)

(love Stata but think R may be more your thing? Check out DIME's R training course for advanced Stata users!)

 

Replication

Worm wars: the anthology

Worm wars: a review of the reanalysis of the Miguel and Kremer deworming study

Response to Brown and Wood's response 

Brown and Woods response on "how scientific are scientific replications"

how scientific are scientific replications?

The infinite loop of failure of replication in economics

More replication in economics

A cynic's take on papers with novel methods to improve transparency

What development economists talk about when they talk about reproducibility

 

Systematic reviews and meta-analysis

How systematic is that systematic review? The case of learning outcomes

How standard is a standard deviation? A cautionary note on using SDs to compare across impact evaluations

Should we give up on SDs for measuring effect size?

What do 600 papers on 20 types of interventions tell us about what types of interventions generalize?

If you want your study included in a systematic review, this is what you should report

 

Book Reviews for Books on Impact Evaluation

Review of Banerjee and Duflo's Poor Economics (and authors' reply)

Review of Manzi's Uncontrolled

Review of Imbens and Rubin's Causal Inference

Review of Glennerster and Takavarasha's Running Randomized Experiments

Review of Gerber and Green's Field Experiments

Review of Karlan and Appel's Failing in the Field

Review of Ogden's Experiments in Development from Every Angle

Review of Leigh's Randomistas: how radical researchers changed the world

Review of Gugerty and Karlan's The Goldilocks Challenge: Right-fit evidence for the social sector

 

Getting Published 

11 tips for making a short presentation based on your research

State of Development Journals 2017

10 journals for publishing a short economics paper

How to publish statistically insignificant results in development

The State of Development Journals 2018

Writing a papers and proceedings paper

Have descriptive development papers been crowded out by impact evaluations?

Make your research known - 10 tools for increasing consumption of your research

Pre-results Review at the Journal of Development Economics: Lessons learned so far

The State of Development Journals 2019: Quality, Acceptance Rates, Review Times, and Open Science

Join the Conversation