I am very glad the Cochrane Collaboration exists. However, it is important to consider how its activities might be harmful and to take every effort to mitigate this risk.
I periodically re-read Joel Lexchin’s review of outcome in industry supported vs non industry supported studies to remind myself of the powerful and pervasive impact that conflict of interest has on the evidence base. Given most randomised controlled trials are industry funded, there is a risk that Cochrane Reviews lend this biased evidence base legitimacy.
Of course, much of what the collaboration already does limits the chances of this bias being fed through into the findings of their reviews.
The wrong choice of comparator drug or outcome shouldn’t be fed through if inclusion criteria are robust.
Similarly, good inclusion criteria should limit the effects of poor study design. However, in Lexchin’s review, study quality did not explain the bias seen.
Publication bias is more problematic. Freedom of information requests have been needed in recent years to gain access to unpublished negative studies and get a true picture of treatment effect. Not all Cochrane reviewers go to these lengths.
However, I propose the Cochrane Collaboration go further. If all meta-analyses included a separate analysis of both independent and industry funded studies, the most perverse distortions of the evidence base might be visible.
Perhaps more importantly, routinely demanding such analyses from Cochrane reviewers would make it easy for the collaboration to assess periodically how effective their procedures are at preventing funder bias from being fed through into the conclusions of their reviews.
This blog was originally posted at BMJ.com.