Original Article
Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses

https://doi.org/10.1016/j.jclinepi.2007.10.007Get rights and content

Abstract

Objectives

To evaluate meta-analyses with trial sequential analysis (TSA). TSA adjusts for random error risk and provides the required number of participants (information size) in a meta-analysis. Meta-analyses not reaching information size are analyzed with trial sequential monitoring boundaries analogous to interim monitoring boundaries in a single trial.

Study Design and Setting

We applied TSA on meta-analyses performed in Cochrane Neonatal reviews. We calculated information sizes and monitoring boundaries with three different anticipated intervention effects of 30% relative risk reduction (TSA30%), 15% (TSA15%), or a risk reduction suggested by low-bias risk trials of the meta-analysis corrected for heterogeneity (TSALBHIS).

Results

A total of 174 meta-analyses were eligible; 79 out of 174 (45%) meta-analyses were statistically significant (P < 0.05). In the significant meta-analyses, TSA30% showed firm evidence in 61%. TSA15% and TSALBHIS found firm evidence in 33% and 73%, respectively. The remaining significant meta-analyses had potentially spurious evidence of effect. In the 95 statistically nonsignificant (P  0.05) meta-analyses, TSA30% showed absence of evidence in 80% (insufficient information size). TSA15% and TSALBHIS found that 95% and 91% had absence of evidence. The remaining nonsignificant meta-analyses had evidence of lack of effect.

Conclusion

TSA reveals insufficient information size and potentially false positive results in many meta-analyses.

Introduction

Meta-analyses aim to increase the power and precision of the estimated intervention effects [1], [2]. Meta-analyses are, however, criticized because the included trials are inevitably clinical diverse regarding patients, interventions, outcomes, etc. Hence, pooling the potentially heterogeneous trial results is sometimes inappropriate [3], [4]. Meta-analyses may also obtain false positive results (type I errors) or overestimate treatment effects due to systematic errors (bias) and random errors (play of chance). Bias may originate from publication bias [5], [6], [7], inclusion of trials with high-bias risk [8], [9], [10], [11], outcome measure bias [12], premature stopping of “positive” trials [13], and small trial bias [14]. Meta-analyses could also be data driven because they are retrospectively conducted. Random errors may arise due to repetitive testing as data accrue and testing of multiple outcome measures, which inevitably, sooner or later, lead to type I errors [15].

The required number of participants (information size) for a meta-analysis should be at least as large as an adequately powered single trial. Trial sequential analysis (TSA) is an approach that provides the required information size in meta-analyses [16]. To adjust for random error risk, meta-analyses not reaching the required sample size are analyzed with trial sequential monitoring boundaries analogous to interim monitoring boundaries in a single trial [16], [17], [18], [19], [20], [21]. Trial sequential monitoring boundaries adjust the P-value that is required for obtaining a statistical significance according to the number of participants and events in a meta-analysis. The fewer participants and events, the more restrictive the monitoring boundaries are and the lower P-value is required to obtain statistical significance.

The use of TSA in meta-analyses has been debated because the analysis ignores potential bias and heterogeneity [22], but adjustment for these factors seems possible [16]. We recently audited clinical guidelines taking Cochrane Neonatal Group reviews as basis for deciding which intervention to use [23]. Therefore, we have examined meta-analyses in these reviews with TSA with and without bias and heterogeneity adjustment to reassess the evidence they provide.

Section snippets

Material and bias definition

We identified all meta-analyses that included more than two trials reporting on a binary outcome measure from the 188 Cochrane Neonatal Group reviews in The Cochrane Library, Issue 4, 2004 [24]. From each review, whenever possible, we included three meta-analyses. We selected the meta-analyses on mortality outcomes and the first two eligible meta-analyses on clinical outcome measures according to the review authors' priority (or three, in case mortality was not meta-analyzed).

The meta-analyses

Eligible meta-analyses

We identified 188 Cochrane Neonatal Group systematic reviews in The Cochrane Library, Issue 4, 2004 [24]. Of these, we excluded 76 because the review included less than three randomized clinical trials, 29 reviews because they did not report a binary outcome measure, and 6 reviews because all trials had high-bias risk (i.e., had unclear or inadequate allocation concealment). From the remaining 77 reviews, we included a total of 174 eligible meta-analyses.

Characteristics of meta-analyses

The 174 meta-analyses included a median

Discussion

This study is the first to apply TSA on a large cohort of meta-analyses. Applying three different TSAs to Cochrane Neonatal Group meta-analyses revealed that many meta-analyses have insufficient information size and there are several potentially false positive results. The respective TSAs supported only the “traditional” significance (P < 0.05) in 61% (TSA30%), 33% (TSA15%), and 73% (TSALBHIS) of 79 significant meta-analyses. Applying TSA30%, TSA15%, and TSALBHIS on the 95 nonsignificant (P  0.05)

Conclusion

The interpretation of meta-analyses is complex. To adjust for random error risk in meta-analyses, we suggest applying TSA (e.g., TSA with a relevant prespecified intervention effect in combination with TSALBHIS) on meta-analyses. In this way, authors and readers of meta-analyses may reach a more balanced conclusion on the effect of interventions.

References (41)

  • K. Dickersin et al.

    Registering clinical trials

    JAMA

    (2003)
  • F. Song et al.

    Publication and related biases

    Health Technol Assess

    (2000)
  • J.P. Ioannidis

    Contradicted and initially stronger effects in highly cited clinical research

    JAMA

    (2005)
  • K.F. Schulz et al.

    Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment in controlled trials

    JAMA

    (1995)
  • L.L. Kjaergard et al.

    Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses

    Ann Intern Med

    (2001)
  • B. Als-Nielsen et al.

    Methodological quality and treatment effects in randomised trials—a review of six empirical studies

  • A.W. Chan et al.

    Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles

    JAMA

    (2004)
  • V.M. Montori et al.

    Randomised trials stopped early for benefit: a systematic review

    JAMA

    (2005)
  • B. Als-Nielsen et al.

    Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials

  • J.P. Ioannidis

    Why most published research findings are false

    PLoS Med

    (2005)
  • Cited by (0)

    View full text