TY - JOUR
T1 - Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy
AU - Mayo-Wilson, Evan R
AU - Li, Tianjing
AU - Fusco, Nicole
AU - Bertizzolo, Lorenzo
AU - Canner, Joseph K.
AU - Cowley, Terrie
AU - Doshi, Peter
AU - Ehmsen, Jeffrey
AU - Gresham, Gillian
AU - Guo, Nan
AU - Haythornthwaite, Jennifer
AU - Heyward, James
AU - Hong, Hwanhee
AU - Pham, Diana
AU - Payne, Jennifer
AU - Rosman, Lori
AU - Stuart, Elizabeth A.
AU - Suarez-Cuervo, Catalina
AU - Tolbert, Elizabeth
AU - Twose, Claire
AU - Vedula, Swaroop
AU - Dickersin, Kay
N1 - Publisher Copyright:
© 2017 The Authors
PY - 2017/11
Y1 - 2017/11
N2 - Objectives The objective of this study was to determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs). Study Design and Setting Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic sources (clinical study reports [CSRs] and individual participant data [IPD]). Results We found 21 gabapentin RCTs (74 reports, 6 IPD) and 7 quetiapine RCTs (50 reports, 1 IPD); most were reported in journal articles (18/21 [86%] and 6/7 [86%], respectively). When available, CSRs contained the most trial design and risk of bias information. CSRs and IPD contained the most results. For the outcome domains “pain intensity” (gabapentin) and “depression” (quetiapine), we found single trials with 68 and 98 different meta-analyzable results, respectively; by purposefully selecting one meta-analyzable result for each RCT, we could change the overall result for pain intensity from effective (standardized mean difference [SMD] = −0.45; 95% confidence interval [CI]: −0.63 to −0.27) to ineffective (SMD = −0.06; 95% CI: −0.24 to 0.12). We could change the effect for depression from a medium effect (SMD = −0.55; 95% CI: −0.85 to −0.25) to a small effect (SMD = −0.26; 95% CI: −0.41 to −0.1). Conclusions Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.
AB - Objectives The objective of this study was to determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs). Study Design and Setting Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and nonpublic sources (clinical study reports [CSRs] and individual participant data [IPD]). Results We found 21 gabapentin RCTs (74 reports, 6 IPD) and 7 quetiapine RCTs (50 reports, 1 IPD); most were reported in journal articles (18/21 [86%] and 6/7 [86%], respectively). When available, CSRs contained the most trial design and risk of bias information. CSRs and IPD contained the most results. For the outcome domains “pain intensity” (gabapentin) and “depression” (quetiapine), we found single trials with 68 and 98 different meta-analyzable results, respectively; by purposefully selecting one meta-analyzable result for each RCT, we could change the overall result for pain intensity from effective (standardized mean difference [SMD] = −0.45; 95% confidence interval [CI]: −0.63 to −0.27) to ineffective (SMD = −0.06; 95% CI: −0.24 to 0.12). We could change the effect for depression from a medium effect (SMD = −0.55; 95% CI: −0.85 to −0.25) to a small effect (SMD = −0.26; 95% CI: −0.41 to −0.1). Conclusions Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.
KW - Clinical trials
KW - Meta-analysis
KW - Reporting bias
KW - Risk of bias assessment
KW - Selective outcome reporting
KW - Systematic reviews
UR - http://www.scopus.com/inward/record.url?scp=85029830783&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85029830783&partnerID=8YFLogxK
U2 - 10.1016/j.jclinepi.2017.07.014
DO - 10.1016/j.jclinepi.2017.07.014
M3 - Article
C2 - 28842290
AN - SCOPUS:85029830783
SN - 0895-4356
VL - 91
SP - 95
EP - 110
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
ER -