Syndicate content

Add new comment

Bill and Sara, thanks both of you for these great comments. I do appreciate that Adam et al. wasn’t billed as a systematic review, and like Bill I have a lot of sympathy with Sara’s point that not all reviews should fit into the systematic review straightjacket. That said, like Bill, I wonder whether there aren’t some key principles of the systematic review approach that ought to be a part of any review. True Adam et al. weren’t seeking to review evidence on impacts that the studies unearthed but rather ask whether the studies were trying to get at a broad set of impacts. But does that justify looking at just a couple of databases? It tells us whether a specific set of journals are publishing holistic studies. But unless we can be sure that the journals indexed by other databases are similar, doesn’t it raise some doubts about the generalizability of the article’s “key messages”? Like Bill, I’m also not convinced that assessing holism eliminates the need to judge the validity of a study’s methods. Would we be interested in a bunch of holistic studies using discredited methods? This raises a broader question about the objective of Adam et al. Isn’t there scope to assemble a comprehensive picture of a health system from partial studies? Might we not prefer to put together an overall assessment of health system X from a bunch of partial but well executed studies of health system X than rely on a bunch of holistic but poorly executed studies of the same health system? Would we really want to discourage someone from doing an opportunistic retrospective evaluation of a reform simply because the data they had stumbled on didn’t allow them to shed light on all outcomes of interest, or didn’t allow them to tell a fully fleshed-out story about the process? Just a thought!