A critical review

Kathleen F. Kerr, Zheyu Wang, Holly Janes, Robyn L. McClelland, Bruce M. Psaty, Margaret S. Pepe

Research output: Contribution to journalArticle

Abstract

Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.

Original languageEnglish (US)
Pages (from-to)114-121
Number of pages8
JournalEpidemiology
Volume25
Issue number1
DOIs
StatePublished - Jan 2014
Externally publishedYes

Fingerprint

Research Personnel
Biomarkers
Confidence Intervals
ROC Curve
Equipment and Supplies

ASJC Scopus subject areas

  • Epidemiology
  • Medicine(all)

Cite this

Kerr, K. F., Wang, Z., Janes, H., McClelland, R. L., Psaty, B. M., & Pepe, M. S. (2014). A critical review. Epidemiology, 25(1), 114-121. https://doi.org/10.1097/EDE.0000000000000018

A critical review. / Kerr, Kathleen F.; Wang, Zheyu; Janes, Holly; McClelland, Robyn L.; Psaty, Bruce M.; Pepe, Margaret S.

In: Epidemiology, Vol. 25, No. 1, 01.2014, p. 114-121.

Research output: Contribution to journalArticle

Kerr, KF, Wang, Z, Janes, H, McClelland, RL, Psaty, BM & Pepe, MS 2014, 'A critical review', Epidemiology, vol. 25, no. 1, pp. 114-121. https://doi.org/10.1097/EDE.0000000000000018
Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. A critical review. Epidemiology. 2014 Jan;25(1):114-121. https://doi.org/10.1097/EDE.0000000000000018
Kerr, Kathleen F. ; Wang, Zheyu ; Janes, Holly ; McClelland, Robyn L. ; Psaty, Bruce M. ; Pepe, Margaret S. / A critical review. In: Epidemiology. 2014 ; Vol. 25, No. 1. pp. 114-121.
@article{86da8aefd2aa464ea48f062bbfdbb1eb,
title = "A critical review",
abstract = "Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.",
author = "Kerr, {Kathleen F.} and Zheyu Wang and Holly Janes and McClelland, {Robyn L.} and Psaty, {Bruce M.} and Pepe, {Margaret S.}",
year = "2014",
month = "1",
doi = "10.1097/EDE.0000000000000018",
language = "English (US)",
volume = "25",
pages = "114--121",
journal = "Epidemiology",
issn = "1044-3983",
publisher = "Lippincott Williams and Wilkins",
number = "1",

}

TY - JOUR

T1 - A critical review

AU - Kerr, Kathleen F.

AU - Wang, Zheyu

AU - Janes, Holly

AU - McClelland, Robyn L.

AU - Psaty, Bruce M.

AU - Pepe, Margaret S.

PY - 2014/1

Y1 - 2014/1

N2 - Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.

AB - Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.

UR - http://www.scopus.com/inward/record.url?scp=84890117399&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84890117399&partnerID=8YFLogxK

U2 - 10.1097/EDE.0000000000000018

DO - 10.1097/EDE.0000000000000018

M3 - Article

VL - 25

SP - 114

EP - 121

JO - Epidemiology

JF - Epidemiology

SN - 1044-3983

IS - 1

ER -