This is so great to see! Thank you for writing this clear, informative post on applying Bayesian inference to improve reporting of policy experiments. The only part I'd take issue with is the example of when not to use Bayesian methods for impact evaluation, "lab or online experiment." We found Bayesian hierarchical modeling perfect for an online experiment precisely because it had so many treatment arms (72 in our case!), something you can't often do with field trials.
Here is the paper: https://www.tandfonline.com/doi/abs/10.1080/19345747.2020.1716905
And a backgrounder on the power analysis, how we could use pooling to test so many treatment factors: https://journals.sagepub.com/doi/abs/10.1177/0193841X18818903
We varied 5 factors each with 2 or 3 levels, so 3x3x2x2x2 = 72, but they followed a hierarchical structure.
Intervention we tested was variations on web design to test how low-income parents use info to choose schools for their kids.
This is so great to see! Thank you for writing this clear, informative post on applying Bayesian inference to improve reporting of policy experiments. The only part I'd take issue with is the example of when not to use Bayesian methods for impact evaluation, "lab or online experiment." We found Bayesian hierarchical modeling perfect for an online experiment precisely because it had so many treatment arms (72 in our case!), something you can't often do with field trials.
Here is the paper: https://www.tandfonline.com/doi/abs/10.1080/19345747.2020.1716905
And a backgrounder on the power analysis, how we could use pooling to test so many treatment factors: https://journals.sagepub.com/doi/abs/10.1177/0193841X18818903
We varied 5 factors each with 2 or 3 levels, so 3x3x2x2x2 = 72, but they followed a hierarchical structure.
Intervention we tested was variations on web design to test how low-income parents use info to choose schools for their kids.