How to interpret a small increase in AUC with an additional risk prediction marker: Decision analysis comes through

Stuart G. Baker, Ewoud Schuit, Ewout W. Steyerberg, Michael J. Pencina, Andew Vickers, Karel G.M. Moons, Ben W.J. Mol, Karen S. Lindeman

Research output: Contribution to journalArticle

Abstract

An important question in the evaluation of an additional risk prediction marker is how to interpret a small increase in the area under the receiver operating characteristic curve (AUC). Many researchers believe that a change in AUC is a poor metric because it increases only slightly with the addition of a marker with a large odds ratio. Because it is not possible on purely statistical grounds to choose between the odds ratio and AUC, we invoke decision analysis, which incorporates costs and benefits. For example, a timely estimate of the risk of later non-elective operative delivery can help a woman in labor decide if she wants an early elective cesarean section to avoid greater complications from possible later non-elective operative delivery. A basic risk prediction model for later non-elective operative delivery involves only antepartum markers. Because adding intrapartum markers to this risk prediction model increases AUC by 0.02, we questioned whether this small improvement is worthwhile. A key decision-analytic quantity is the risk threshold, here the risk of later non-elective operative delivery at which a patient would be indifferent between an early elective cesarean section and usual care. For a range of risk thresholds, we found that an increase in the net benefit of risk prediction requires collecting intrapartum marker data on 68 to 124 women for every correct prediction of later non-elective operative delivery. Because data collection is non-invasive, this test tradeoff of 68 to 124 is clinically acceptable, indicating the value of adding intrapartum markers to the risk prediction model.

Original languageEnglish (US)
Pages (from-to)3946-3959
Number of pages14
JournalStatistics in Medicine
Volume33
Issue number22
DOIs
StatePublished - Sep 28 2014

Keywords

  • AUC
  • Cesarean section
  • Decision curves
  • Receiver operating characteristic curves
  • Relative utility curves

ASJC Scopus subject areas

  • Epidemiology
  • Statistics and Probability

Fingerprint Dive into the research topics of 'How to interpret a small increase in AUC with an additional risk prediction marker: Decision analysis comes through'. Together they form a unique fingerprint.

  • Cite this

    Baker, S. G., Schuit, E., Steyerberg, E. W., Pencina, M. J., Vickers, A., Moons, K. G. M., Mol, B. W. J., & Lindeman, K. S. (2014). How to interpret a small increase in AUC with an additional risk prediction marker: Decision analysis comes through. Statistics in Medicine, 33(22), 3946-3959. https://doi.org/10.1002/sim.6195