People often complain that systematic reviewers make use of arbitrary quality thresholds to select or reject papers for inclusion in meta-analyses and do not allow for bias in any included studies that are not perfect. But how to avoid arbitrary selections and to allow for bias? The answer is provided in a truly beautiful methodological paper from the MRC Biostatistics Unit in Cambridge. The paper explains how a probability density can be elicited for the bias associated with each study included in a meta-analysis. It goes on to show how these probability distributions are incorporated in the analysis. But should it be assumed that the bias is additive or proportional (increases with effect size)? This is a judgement to be used in each case, but an example is given under each assumption in the paper.
Remarkably, this study incorporated estimates for both internal and external bias, and argued that evidence regarding bias could supplement for expert opinion in the former. External bias need not be taken into account if the exercise aims to summarise the literature rather than answer a policy question for a particular target audience.
— Richard Lilford, CLAHRC WM Director
- Turner RM, Spiegelhalter DJ, Smith GCS, Thompson SG. Bias modelling in evidence synthesis. J R Stat Soc A. 2009; 172(1): 21-47.