The impact of stopping rules on heterogeneity of results in overviews of clinical trials

This paper explores the extent to which application of statistical stopping rules in clinical trials can create an artificial heterogeneity of treatment effects in overviews (meta-analyses) of related trials. For illustration, we concentrate on overviews of identically designed group sequential trials, using either fixed nominal or O'Brien and Fleming two- sided boundaries. Some analytic results are obtained for two-group designs and simulation studies are otherwise used, with the following overall findings. The use of stopping rules leads to biased estimates of treatment effect so that the assessment of heterogeneity of results in an overview of trials, some of which have used stopping rules, is confounded by this bias. If the true treatment effect being studied is small, as is often the case, then artificial heterogeneity is introduced, thus increasing the Type I error rate in the test of homogeneity. This could lead to erroneous use of a random effects model, producing exaggerated estimates and confidence intervals. However, if the true mean effect is large, then between-trial heterogeneity may be underestimated. When undertaking or interpreting overviews, one should ascertain whether stopping rules have been used (either formally or informally) and should consider whether their use might account for any heterogeneity found.