Shockingly, I was probably too flippant on twitter when I said you were "too easy on Cartwright & Hardie's book here," David. But I'll try to justify that. Though first I should say that I was excited about this book, bought it and read it immediately, and really enjoyed the early chapters in particular. And the book comes with strong endorsement from Dani Rodrik and Angus Deaton (!), so why should anyone care what I think anyway? Nevertheless, here's where it left me unsatisfied:
1. Data needed. This is a book arguing that hasty empirical generalizations lead us astray, particularly generalizations from RCTs in formulating social policy. I happen to agree, up to a point. But the book offers fairly little empirical evidence that this claim is true, and -- given the subtitle is "A Practical Guide to Doing It Better" -- almost zero guidance on where we should draw the line in practice. There are, to be fair, several real world examples and illustrations scattered throughout, but these feel more like cherry-picked cases to illustrate a negative point, rather than a serious evidence about how evidence-based policy is actually done or could be done better.
Instead, the book offers a long, sophisticated, occasionally insightful elaboration on the theme of "just because it worked here (or today) doesn't mean it'll work there (or tomorrow) -- and what is "it" anyway?" Reading C & H feels sort of like arguing billiards tactics with David Hume.
2. External validity is overrated. Yes, the Moving to Opportunity evaluation cost millions. But its potential implications for the population under study (residents of low-income housing in Baltimore, Boston, Chicago, Los Angeles, and New York City) were also huge -- without considering any extrapolation to other contexts. I don't think we should concede the value of good evaluation hinges on generalizability. I expected a book on "Evidence-Based Policy" to give more weight to the use of relevant evidence, and spend less time decrying the use of irrelevant evidence.
3. Lastly, just to stir up trouble, I guess I could also argue you were too easy on the book because, well, it's pretty antithetical to a lot of the stuff on this blog (which I read regularly and enjoy a lot). For my own personal enrichment, I was hoping you'd put up more of a fight!