Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy

Evan R Mayo-Wilson, Tianjing Li, Nicole Fusco, Lorenzo Bertizzolo, Joseph K. Canner, Terrie Cowley, Peter Doshi, Jeffrey Ehmsen, Gillian Gresham, Nan Guo, Jennifer Haythornthwaite, James Heyward, Hwanhee Hong, Diana Pham, Jennifer Payne, Lori Rosman, Elizabeth Stuart, Catalina Suarez-Cuervo, Elizabeth Tolbert, Claire TwoseSwaroop Vedula, Kay Dickersin

Research output: Contribution to journalArticle

Abstract

Objectives: The objective of this study was to determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs). Study Design and Setting: Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic sources (clinical study reports [CSRs] and individual participant data [IPD]). Results: We found 21 gabapentin RCTs (74 reports, 6 IPD) and 7 quetiapine RCTs (50 reports, 1 IPD); most were reported in journal articles (18/21 [86%] and 6/7 [86%], respectively). When available, CSRs contained the most trial design and risk of bias information. CSRs and IPD contained the most results. For the outcome domains "pain intensity" (gabapentin) and "depression" (quetiapine), we found single trials with 68 and 98 different meta-analyzable results, respectively; by purposefully selecting one meta-analyzable result for each RCT, we could change the overall result for pain intensity from effective (standardized mean difference [SMD] = -0.45; 95% confidence interval [CI]: -0.63 to -0.27) to ineffective (SMD = -0.06; 95% CI: -0.24 to 0.12). We could change the effect for depression from a medium effect (SMD = -0.55; 95% CI: -0.85 to -0.25) to a small effect (SMD = -0.26; 95% CI: -0.41 to -0.1). Conclusions: Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.

Original languageEnglish (US)
JournalJournal of Clinical Epidemiology
DOIs
StateAccepted/In press - 2017

Fingerprint

Randomized Controlled Trials
Confidence Intervals
Information Storage and Retrieval
Depression
Pain
Neuralgia
Bipolar Disorder
Meta-Analysis
Drive
gabapentin
Clinical Studies
Quetiapine Fumarate

Keywords

  • Clinical trials
  • Meta-analysis
  • Reporting bias
  • Risk of bias assessment
  • Selective outcome reporting
  • Systematic reviews

ASJC Scopus subject areas

  • Epidemiology

Cite this

Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. / Mayo-Wilson, Evan R; Li, Tianjing; Fusco, Nicole; Bertizzolo, Lorenzo; Canner, Joseph K.; Cowley, Terrie; Doshi, Peter; Ehmsen, Jeffrey; Gresham, Gillian; Guo, Nan; Haythornthwaite, Jennifer; Heyward, James; Hong, Hwanhee; Pham, Diana; Payne, Jennifer; Rosman, Lori; Stuart, Elizabeth; Suarez-Cuervo, Catalina; Tolbert, Elizabeth; Twose, Claire; Vedula, Swaroop; Dickersin, Kay.

In: Journal of Clinical Epidemiology, 2017.

Research output: Contribution to journalArticle

Mayo-Wilson, ER, Li, T, Fusco, N, Bertizzolo, L, Canner, JK, Cowley, T, Doshi, P, Ehmsen, J, Gresham, G, Guo, N, Haythornthwaite, J, Heyward, J, Hong, H, Pham, D, Payne, J, Rosman, L, Stuart, E, Suarez-Cuervo, C, Tolbert, E, Twose, C, Vedula, S & Dickersin, K 2017, 'Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy', Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2017.07.014
Mayo-Wilson, Evan R ; Li, Tianjing ; Fusco, Nicole ; Bertizzolo, Lorenzo ; Canner, Joseph K. ; Cowley, Terrie ; Doshi, Peter ; Ehmsen, Jeffrey ; Gresham, Gillian ; Guo, Nan ; Haythornthwaite, Jennifer ; Heyward, James ; Hong, Hwanhee ; Pham, Diana ; Payne, Jennifer ; Rosman, Lori ; Stuart, Elizabeth ; Suarez-Cuervo, Catalina ; Tolbert, Elizabeth ; Twose, Claire ; Vedula, Swaroop ; Dickersin, Kay. / Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. In: Journal of Clinical Epidemiology. 2017.
@article{c0dfadbf596b41b0b139beb8079b12ba,
title = "Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy",
abstract = "Objectives: The objective of this study was to determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs). Study Design and Setting: Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic sources (clinical study reports [CSRs] and individual participant data [IPD]). Results: We found 21 gabapentin RCTs (74 reports, 6 IPD) and 7 quetiapine RCTs (50 reports, 1 IPD); most were reported in journal articles (18/21 [86{\%}] and 6/7 [86{\%}], respectively). When available, CSRs contained the most trial design and risk of bias information. CSRs and IPD contained the most results. For the outcome domains {"}pain intensity{"} (gabapentin) and {"}depression{"} (quetiapine), we found single trials with 68 and 98 different meta-analyzable results, respectively; by purposefully selecting one meta-analyzable result for each RCT, we could change the overall result for pain intensity from effective (standardized mean difference [SMD] = -0.45; 95{\%} confidence interval [CI]: -0.63 to -0.27) to ineffective (SMD = -0.06; 95{\%} CI: -0.24 to 0.12). We could change the effect for depression from a medium effect (SMD = -0.55; 95{\%} CI: -0.85 to -0.25) to a small effect (SMD = -0.26; 95{\%} CI: -0.41 to -0.1). Conclusions: Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.",
keywords = "Clinical trials, Meta-analysis, Reporting bias, Risk of bias assessment, Selective outcome reporting, Systematic reviews",
author = "Mayo-Wilson, {Evan R} and Tianjing Li and Nicole Fusco and Lorenzo Bertizzolo and Canner, {Joseph K.} and Terrie Cowley and Peter Doshi and Jeffrey Ehmsen and Gillian Gresham and Nan Guo and Jennifer Haythornthwaite and James Heyward and Hwanhee Hong and Diana Pham and Jennifer Payne and Lori Rosman and Elizabeth Stuart and Catalina Suarez-Cuervo and Elizabeth Tolbert and Claire Twose and Swaroop Vedula and Kay Dickersin",
year = "2017",
doi = "10.1016/j.jclinepi.2017.07.014",
language = "English (US)",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier USA",

}

TY - JOUR

T1 - Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy

AU - Mayo-Wilson, Evan R

AU - Li, Tianjing

AU - Fusco, Nicole

AU - Bertizzolo, Lorenzo

AU - Canner, Joseph K.

AU - Cowley, Terrie

AU - Doshi, Peter

AU - Ehmsen, Jeffrey

AU - Gresham, Gillian

AU - Guo, Nan

AU - Haythornthwaite, Jennifer

AU - Heyward, James

AU - Hong, Hwanhee

AU - Pham, Diana

AU - Payne, Jennifer

AU - Rosman, Lori

AU - Stuart, Elizabeth

AU - Suarez-Cuervo, Catalina

AU - Tolbert, Elizabeth

AU - Twose, Claire

AU - Vedula, Swaroop

AU - Dickersin, Kay

PY - 2017

Y1 - 2017

N2 - Objectives: The objective of this study was to determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs). Study Design and Setting: Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic sources (clinical study reports [CSRs] and individual participant data [IPD]). Results: We found 21 gabapentin RCTs (74 reports, 6 IPD) and 7 quetiapine RCTs (50 reports, 1 IPD); most were reported in journal articles (18/21 [86%] and 6/7 [86%], respectively). When available, CSRs contained the most trial design and risk of bias information. CSRs and IPD contained the most results. For the outcome domains "pain intensity" (gabapentin) and "depression" (quetiapine), we found single trials with 68 and 98 different meta-analyzable results, respectively; by purposefully selecting one meta-analyzable result for each RCT, we could change the overall result for pain intensity from effective (standardized mean difference [SMD] = -0.45; 95% confidence interval [CI]: -0.63 to -0.27) to ineffective (SMD = -0.06; 95% CI: -0.24 to 0.12). We could change the effect for depression from a medium effect (SMD = -0.55; 95% CI: -0.85 to -0.25) to a small effect (SMD = -0.26; 95% CI: -0.41 to -0.1). Conclusions: Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.

AB - Objectives: The objective of this study was to determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs). Study Design and Setting: Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic sources (clinical study reports [CSRs] and individual participant data [IPD]). Results: We found 21 gabapentin RCTs (74 reports, 6 IPD) and 7 quetiapine RCTs (50 reports, 1 IPD); most were reported in journal articles (18/21 [86%] and 6/7 [86%], respectively). When available, CSRs contained the most trial design and risk of bias information. CSRs and IPD contained the most results. For the outcome domains "pain intensity" (gabapentin) and "depression" (quetiapine), we found single trials with 68 and 98 different meta-analyzable results, respectively; by purposefully selecting one meta-analyzable result for each RCT, we could change the overall result for pain intensity from effective (standardized mean difference [SMD] = -0.45; 95% confidence interval [CI]: -0.63 to -0.27) to ineffective (SMD = -0.06; 95% CI: -0.24 to 0.12). We could change the effect for depression from a medium effect (SMD = -0.55; 95% CI: -0.85 to -0.25) to a small effect (SMD = -0.26; 95% CI: -0.41 to -0.1). Conclusions: Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.

KW - Clinical trials

KW - Meta-analysis

KW - Reporting bias

KW - Risk of bias assessment

KW - Selective outcome reporting

KW - Systematic reviews

UR - http://www.scopus.com/inward/record.url?scp=85029830783&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029830783&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2017.07.014

DO - 10.1016/j.jclinepi.2017.07.014

M3 - Article

C2 - 28842290

AN - SCOPUS:85029830783

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -