# A Curated List of Our Postings on Technical Topics – Your One-Stop Shop for Methodology

## This page in:

*In lieu of our regular links this week, it is u*

*pdated up to October 25, 2018*

**General**

IEanalytics: introducing the Development Impact Evaluation wiki

**Random Assignment**

Allocating treatment and control with multiple applications per applicant and ranked choices

Is optimization just re-randomization redux? Thoughts on the "don't randomize, optimize" papers

Be an optimista, not a randomista, when you have small samples

Tips for randomization in the wild: adding a waitlist

How to randomize in the field

Stratified randomization and the FIFA world cup

Doing stratified randomization with uneven numbers in the Strata

How to randomize using many baseline variables

Public randomization ceremonies

Designing experiments to measure spillover effects

Mechanism experiments and opening up the black box

Sample weights and RCT design

Response to the policymaker complaint that randomized experiments take too much time

Have RCTs taken over development economics?

Definitions in RCTs with interference

What does a game-theoretic model with belief-dependent preferences teach us about how to randomize?

Incorporating participant welfare and ethics into RCTs

Are we over-investing in baselines?

Most good you can do, but for whom? (on spillovers)

**Pre-analysis plans and reporting**

Pre-registration of studies to avoid fishing and allow transparent discovery

A joint test of orthogonality when testing for baseline balance

A pre-analysis plan check-list

The New Trial Registries

What isn’t reported in impact evaluations but maybe should be

Randomization checks: testing for joint orthogonality

An addendum to pre-analysis plans: pre-specifying when not to use data

A pre-analysis plan is the only way to take your p-values seriously

Trouble with pre-analysis plans? try these 3 weird tricks

Should we require balance t-tests on observables with randomized experiments?

Registered reports: piloting a pre-results review process at the JDE

Declaring and diagnosing research designs

**Propensity Score Matching**

Guido Imbens on clustering standard errors with matching

Testing different matching estimators as applied to job training programs

The covariate balanced propensity score

**Difference-in-Differences**

The often unspoken assumptions behind diff-in-diff

How big data helped us estimate the impact of an intervention with 0.8% take-up

**Regression Discontinuity**

Curves in all the wrong places: Gelman and Imbens on why not to use higher-order polynomials in RD

Regression discontinuity with an implicit index

Tools of the trade: the regression kink design

**Synthetic Controls**

Evaluating an Argentine tourism policy using synthetic controls: tan linda que enamora?

The synthetic control method, as applied to regulatory reforms

**Instrumental Variables**

Rethinking identification under the Bartik/Shift-Share instruments

Judge leniency designs: Now not just for Crime Studies

**Machine learning**

How can machine learning and artificial intelligence be used in development interventions and analysis?

**Other Evaluation Methods**

Impact as narrative

Using spatial variation in program performance to identify impacts

Small n impact evaluation methods

Can we trust shoestring evaluations?

Using case studies to explore and explain complex interventions

**Analysis**

Another reason to prefer Ancova: dealing with measurement changes between baseline and follow-up

Endogenous stratification: the surprisingly easy way to bias your heterogeneous treatment effects and what to do instead

Why is difference-in-difference estimation still so popular in experimental analysis?

Regression adjustment in randomized experiments (part one, part two)

When to use survey weights in analysis

Adjustments for multiple hypothesis testing

Bounding approaches to deal with attrition

Linear probability models versus probits

Dealing with multiple lotteries

Estimating standard errors with small clusters (part one, part two)

Decomposition methods

Estimation of treatment effects with incomplete compliance

What does Alwyn Young's paper mean for analysis of experiments?

Winsorizing, testing balance, and dealing with incomplete take-up

You ran a field experiment, should you then run a regression?

Sometimes, increasingly, estimating the ITT is not enough in experiments

Finally an easy way to do randomization inference in Stata

When should you cluster standard errors? New wisdom from the econometrics oracle

Your go to regression specification is biased: here is the simple fix

What should you do when your random assignment gets compromised?

**Power Calculations and Improving Power**

Should I work with only a subsample of my control group if I have take-up problems?

Power calculations: what software should I use?

Does the intra-cluster correlation matter for power calculations if I am going to cluster my standard errors?

Power calculations for propensity score matching

Power calculations 101: dealing with incomplete take-up

Collecting more rounds of data to boost power

Improving power in small samples

Did you do your power calculations using standard deviations? Do them again.

Power calculations for regression discontinuity (part 1, part 2, part 3)

Power calculation software for randomized saturation experiments

Statistical power and the funnel of attribution

Should you over-sample compliers if budget is limited and you are concerned about take-up?

**On External Validity**

Getting beyond the mirage of external validity

All those external validity issues with impacts? They apply to costs too

External validity as seen from other quantitative social sciences and the gaps in our practices

Towards a more systematic approach to external validity: understanding site selection bias

Weighting for external validity

Will that successful intervention over there get results here?

Learn to live without external validity

Why the external validity of regression estimates can be less than you think

Why similarity is the wrong concept for external validity

A rant on the external validity double standard

Towards policy irrelevance: on the experimental arms race

What's wrong in how we do impact evaluation?

What's in a title? Signaling external validity through titles in development economics

A framework for taking evidence from one setting to another

Informing policy with research that is more than the sum of the parts

**Jargony Terms in Impact Evaluations**

A proposed taxonomy of behavioral responses to evaluation

Quantifying the Hawthorne effect

The Hawthorne Effect

The John Henry Effect

Placebo effects

Clinical Equipoise

Social Desirability Bias/Experimenter Demand Effects and more on Experimenter Demand Effects

**Stata Tricks**

Generating regression and summary statistics tables in Stata

Graphing impacts with Standard Error Bars

Calculating the intra-cluster correlation

Generating regression and summary statistics tables in Stata: A checklist and code

Stata code for correlated random coefficient models

Finally an easy way to do randomization inference in Stata

IEanalytics: introducing ietoolkit

Five small things I've learned recently

**Replication**

Worm wars: the anthology

Worm wars: a review of the reanalysis of the Miguel and Kremer deworming study

Response to Brown and Wood's response

Brown and Woods response on "how scientific are scientific replications"

how scientific are scientific replications?

The infinite loop of failure of replication in economics

More replication in economics

A cynic's take on papers with novel methods to improve transparency

**Systematic reviews and meta-analysis**

how systematic is that systematic review? The case of learning outcomes

How standard is a standard deviation? A cautionary note on using SDs to compare across impact evaluations

should we give up on SDs for measuring effect size?

What do 600 papers on 20 types of interventions tell us about what types of interventions generalize?

If you want your study included in a systematic review, this is what you should report

**Book Reviews for Books on Impact Evaluation**

Review of Banerjee and Duflo's Poor Economics (and authors' reply)

Review of Manzi's Uncontrolled

Review of Imbens and Rubin's Causal Inference

Review of Glennerster and Takavarasha's Running Randomized Experiments

Review of Gerber and Green's Field Experiments

Review of Karlan and Appel's Failing in the Field

Review of Ogden's Experiments in Development from Every Angle

Review of Leigh's Randomistas: how radical researchers changed the world

Review of Gugerty and Karlan's The Goldilocks Challenge: Right-fit evidence for the social sector

**Getting Published**

11 tips for making a short presentation based on your research

State of Development Journals 2017

10 journals for publishing a short economics paper

How to publish statistically insignificant results in development

The State of Development Journals 2018

Writing a papers and proceedings paper

Have descriptive development papers been crowded out by impact evaluations?

Make your research known - 10 tools for increasing consumption of your research

## Join the Conversation

I wish that other bloggers would do something like this. It is always nice to have a mini "database" of articles about a certain topic. You never know when something like that will come in handy.

How is this "curated" exactly? It's just a freaking list. Stop saying things are "curated." You can't "curate" a list of things you already published. I'd contend you can't "curate" a list of anything at all. What an obnoxious trend it is becoming to use the word "curated" for "a bunch of stuff a person happens to like." Ugh!

Curate, verb meaning to select, organize, and look after the items in a collection. This precisely describes the process used to go through almost 5 years of old blog posts, select the ones related to technical topics, and organize them in a way that is hopefully useful for people so that they form a coherent collection.

David - I defer absolutely to the high level of sophistication of your impact benefits analysis (!) - i.e. the calculation of the numerator of your rate of return. However I could not get a good understanding of the cost - i.e. the denominator. I.e. what is the (social) cost of the impact benefits.

If we don't take the cost into account then we of course we might be saying simply that if you give an entrepreneur $50,000 his business is likely to be in better shape in three years time than one that does not get the $50,000 under most scenarios - which would not be surprising.

Please could you explain What is the denominator?

1. just the value of the business grants?

2. 1. plus BPC management costs?

3. 2. plus transaction costs to winning (and losing) entrepreneurs?

4. 3. plus matching investment from the entrepreneur?

5. 4. plus contributed investment funds (funded by outside loans, equity)?

And if it is 1. or 2. how much of the overall return benefits are attributable only to the grants? Could we assume a crowding-in effect which attributes the complementary investment also to some extent to the grants ?? (I see that you have found no crowding in effect overall but it seems likely to have happened in some cases).

Getting back to the numerator how have you accounted for benefit externalities that might show up in linkages and associated technological spillover effects, which are arguably key to the development of industry, banking and business services etc? (That is, indirect effects that cannot be accounted for through performance of the treatment or control group enterprises). OR how do you allow for possible contamination of the control group by technological spillovers generated by enterprises that receive awards?? (Apart from the fact hat Nigeria ia a big country and businesses may not meet face to face).

Id really appreciate understanding this since it is key to the validity of the rate of return analysis and by extension the value of business plans.

This should really be posted as a comment to my post on the Nigeria business plan competition, not to the general technical topic links. But I couldn't figure out how to move where you had put the comment, so will answer here.

Short answer is that:

- the measures of impact take into account transaction costs to the winners, and any crowding out or crowding in of other funding, including matching funding provided by them and/or equity/loan finance that is crowded out. The paper shows a little bit of crowd-out of this other finance, and no evidence of crowding in.

- I am not able to measure the spillovers to the control group, and have to rely on it being a small number of winners spread throughout the country.

- the measure of the cost of the program includes both the cost of the grants and the cost of operating the program (approx $2 million in costs to hand out $58 million in grants).

Great list - knew many of these posts but not others, great to have them all together in one place. Now David, a follow-up request. I knew your small N larg T paper and its ANCOVA recommendation, but I hadn't seen the posts by Lin on regression adjustment:

http://blogs.worldbank.org/impactevaluations/regression-adjustment-in-r…

A linear combination of both would be great, since "your" ANCOVA is a regrssion adjustment on a special case, averaged pre-treatment outcomes, and those may vary substantially with the outcomes and the treatment effects...

Thanks!

Fully Agree with Homira. Thanks a ton David for showing the way. Hope other interesting topics will also have one such 'one-stop-shop'.

Truly outstanding knowledge management.

I’d welcome the Feb 2018 edition of this list!