Syndicate content

Add new comment

Submitted by Bill Savedoff on
I appreciated Adam's blog because I do think that the growing number of systematic reviews for research in applied policy (not just health) needs some attention to standards and quality. I also think Sara is right in saying that this particular article is not the most appropriate one for Adam to use in highlighting this point. I say this because the paper's focus, as I understand it, was not to draw policy conclusions from the evaluations. Rather it was a critique of the literature for failing to address health system questions in ways relevant to policy. The paper is very carefully qualified from beginning to end about what it is, and isn't, trying to do. For all that, it still looks like a systematic review. It is written in the format of a systematic review and it draws conclusions by referring to included studies which are supposed to be representative in some way. The methods section presents information that is normally the basis for judging whether the included studies are comprehensive and representative. I suspect that including searches of RePEc and Google Scholar would have strengthened the conclusion that very little "holistic" analysis is being published. Yet that is exactly what is necessary for the validity of the paper - evidence from the entire body of literature, not just from Medline and Embase. I'm more troubled by the paper's dismissal of the need to assess study design and methods. I noted this line: "The objective is not to appraise the quality of evidence, e.g. whether the evaluation used appropriate study design or methods. It is rather to assess whether they ask a broader set of questions relevant for policy making." Here, Adam's critique is quite accurate. If the paper had found 50% of the studies *did* address relevant questions but failed to consider whether those studies were done well or not, the reader would be left with a very skewed sense of the condition of the literature on this subject. I'm sympathetic to the straightjacket argument that Sara references. A well-done complex system study is unlikely to pass muster with a systematic review process tailored for clinical trials. But i don't think the recommendations made by Waddington or Wagstaff preclude systematic reviews that incorporate such studies. All they demand is that the evolving standards for addressing bias and establishing proof are explicitly treated in the inclusion criteria and appropriately incorporated when reaching conclusions. For systems analysis to move forward, we need to have ways of judging the quality of a study. And to do that the *concepts* of bias, validity, and reliability still hold even if, currently, we don't have a consensus on which feasible methods will get us there.