Steven Glazerman (not verified)

January 17, 2023

This is so great to see! Thank you for writing this clear, informative post on applying Bayesian inference to improve reporting of policy experiments. The only part I'd take issue with is the example of when not to use Bayesian methods for impact evaluation, "lab or online experiment." We found Bayesian hierarchical modeling perfect for an online experiment precisely because it had so many treatment arms (72 in our case!), something you can't often do with field trials.
Here is the paper:
And a backgrounder on the power analysis, how we could use pooling to test so many treatment factors:

We varied 5 factors each with 2 or 3 levels, so 3x3x2x2x2 = 72, but they followed a hierarchical structure.
Intervention we tested was variations on web design to test how low-income parents use info to choose schools for their kids.