TY - JOUR
T1 - Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools
T2 - a simulation study using individual participant data
AU - the Depression Screening Data (DEPRESSD) EPDS Group
AU - Bhandari, Parash Mani
AU - Levis, Brooke
AU - Neupane, Dipika
AU - Patten, Scott B.
AU - Shrier, Ian
AU - Thombs, Brett D.
AU - Benedetti, Andrea
AU - Sun, Ying
AU - He, Chen
AU - Rice, Danielle B.
AU - Krishnan, Ankur
AU - Wu, Yin
AU - Azar, Marleine
AU - Sanchez, Tatiana A.
AU - Chiovitti, Matthew J.
AU - Saadat, Nazanin
AU - Riehm, Kira E.
AU - Imran, Mahrukh
AU - Negeri, Zelalem
AU - Boruff, Jill T.
AU - Cuijpers, Pim
AU - Gilbody, Simon
AU - Ioannidis, John P.A.
AU - Kloda, Lorie A.
AU - Ziegelstein, Roy C.
AU - Comeau, Liane
AU - Mitchell, Nicholas D.
AU - Tonelli, Marcello
AU - Vigod, Simone N.
AU - Aceti, Franca
AU - Alvarado, Rubén
AU - Alvarado-Esquivel, Cosme
AU - Bakare, Muideen O.
AU - Barnes, Jacqueline
AU - Bavle, Amar D.
AU - Beck, Cheryl Tatano
AU - Bindt, Carola
AU - Boyce, Philip M.
AU - Bunevicius, Adomas
AU - Castro e Couto, Tiago
AU - Chaudron, Linda H.
AU - Correa, Humberto
AU - de Figueiredo, Felipe Pinheiro
AU - Eapen, Valsamma
AU - Favez, Nicolas
AU - Felice, Ethel
AU - Fernandes, Michelle
AU - Figueiredo, Barbara
AU - Fisher, Jane R.W.
AU - Tandon, S. Darius
N1 - Publisher Copyright:
© 2021 Elsevier Inc.
PY - 2021/9
Y1 - 2021/9
N2 - Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.
AB - Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.
KW - Accuracy estimates
KW - Bias
KW - Cherry-picking
KW - Data-driven methods
KW - Depression
KW - Optimal cutoff
UR - http://www.scopus.com/inward/record.url?scp=85105570449&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85105570449&partnerID=8YFLogxK
U2 - 10.1016/j.jclinepi.2021.03.031
DO - 10.1016/j.jclinepi.2021.03.031
M3 - Article
C2 - 33838273
AN - SCOPUS:85105570449
SN - 0895-4356
VL - 137
SP - 137
EP - 147
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
ER -