A comparison of resampling schemes for estimating model observer performance with small ensembles

Fatma E.A. Elshahaby, Abhinav Kumar Jha, Mickel Ghaly, Eric Frey

Research output: Contribution to journalArticle

Abstract

In objective assessment of image quality, an ensemble of images is used to compute the 1st and 2nd order statistics of the data. Often, only a finite number of images is available, leading to the issue of statistical variability in numerical observer performance. Resampling-based strategies can help overcome this issue. In this paper, we compared different combinations of resampling schemes (the leave-one-out (LOO) and the half-train/half-test (HT/HT)) and model observers (the conventional channelized Hotelling observer (CHO), channelized linear discriminant (CLD) and channelized quadratic discriminant). Observer performance was quantified by the area under the ROC curve (AUC). For a binary classification task and for each observer, the AUC value for an ensemble size of 2000 samples per class served as a gold standard for that observer. Results indicated that each observer yielded a different performance depending on the ensemble size and the resampling scheme. For a small ensemble size, the combination [CHO, HT/HT] had more accurate rankings than the combination [CHO, LOO]. Using the LOO scheme, the CLD and CHO had similar performance for large ensembles. However, the CLD outperformed the CHO and gave more accurate rankings for smaller ensembles. As the ensemble size decreased, the performance of the [CHO, LOO] combination seriously deteriorated as opposed to the [CLD, LOO] combination. Thus, it might be desirable to use the CLD with the LOO scheme when smaller ensemble size is available.

Original languageEnglish (US)
Pages (from-to)7300-7320
Number of pages21
JournalPhysics in Medicine and Biology
Volume62
Issue number18
DOIs
StatePublished - Aug 22 2017

Fingerprint

ROC Curve
Area Under Curve
Sample Size

Keywords

  • channelized model observers
  • Hotelling observe
  • leave-one-out resampling scheme
  • rank correlation coefficient

ASJC Scopus subject areas

  • Radiological and Ultrasound Technology
  • Radiology Nuclear Medicine and imaging

Cite this

A comparison of resampling schemes for estimating model observer performance with small ensembles. / Elshahaby, Fatma E.A.; Jha, Abhinav Kumar; Ghaly, Mickel; Frey, Eric.

In: Physics in Medicine and Biology, Vol. 62, No. 18, 22.08.2017, p. 7300-7320.

Research output: Contribution to journalArticle

@article{d4993e7c28594e0198243de404549a93,
title = "A comparison of resampling schemes for estimating model observer performance with small ensembles",
abstract = "In objective assessment of image quality, an ensemble of images is used to compute the 1st and 2nd order statistics of the data. Often, only a finite number of images is available, leading to the issue of statistical variability in numerical observer performance. Resampling-based strategies can help overcome this issue. In this paper, we compared different combinations of resampling schemes (the leave-one-out (LOO) and the half-train/half-test (HT/HT)) and model observers (the conventional channelized Hotelling observer (CHO), channelized linear discriminant (CLD) and channelized quadratic discriminant). Observer performance was quantified by the area under the ROC curve (AUC). For a binary classification task and for each observer, the AUC value for an ensemble size of 2000 samples per class served as a gold standard for that observer. Results indicated that each observer yielded a different performance depending on the ensemble size and the resampling scheme. For a small ensemble size, the combination [CHO, HT/HT] had more accurate rankings than the combination [CHO, LOO]. Using the LOO scheme, the CLD and CHO had similar performance for large ensembles. However, the CLD outperformed the CHO and gave more accurate rankings for smaller ensembles. As the ensemble size decreased, the performance of the [CHO, LOO] combination seriously deteriorated as opposed to the [CLD, LOO] combination. Thus, it might be desirable to use the CLD with the LOO scheme when smaller ensemble size is available.",
keywords = "channelized model observers, Hotelling observe, leave-one-out resampling scheme, rank correlation coefficient",
author = "Elshahaby, {Fatma E.A.} and Jha, {Abhinav Kumar} and Mickel Ghaly and Eric Frey",
year = "2017",
month = "8",
day = "22",
doi = "10.1088/1361-6560/aa807a",
language = "English (US)",
volume = "62",
pages = "7300--7320",
journal = "Physics in Medicine and Biology",
issn = "0031-9155",
publisher = "IOP Publishing Ltd.",
number = "18",

}

TY - JOUR

T1 - A comparison of resampling schemes for estimating model observer performance with small ensembles

AU - Elshahaby, Fatma E.A.

AU - Jha, Abhinav Kumar

AU - Ghaly, Mickel

AU - Frey, Eric

PY - 2017/8/22

Y1 - 2017/8/22

N2 - In objective assessment of image quality, an ensemble of images is used to compute the 1st and 2nd order statistics of the data. Often, only a finite number of images is available, leading to the issue of statistical variability in numerical observer performance. Resampling-based strategies can help overcome this issue. In this paper, we compared different combinations of resampling schemes (the leave-one-out (LOO) and the half-train/half-test (HT/HT)) and model observers (the conventional channelized Hotelling observer (CHO), channelized linear discriminant (CLD) and channelized quadratic discriminant). Observer performance was quantified by the area under the ROC curve (AUC). For a binary classification task and for each observer, the AUC value for an ensemble size of 2000 samples per class served as a gold standard for that observer. Results indicated that each observer yielded a different performance depending on the ensemble size and the resampling scheme. For a small ensemble size, the combination [CHO, HT/HT] had more accurate rankings than the combination [CHO, LOO]. Using the LOO scheme, the CLD and CHO had similar performance for large ensembles. However, the CLD outperformed the CHO and gave more accurate rankings for smaller ensembles. As the ensemble size decreased, the performance of the [CHO, LOO] combination seriously deteriorated as opposed to the [CLD, LOO] combination. Thus, it might be desirable to use the CLD with the LOO scheme when smaller ensemble size is available.

AB - In objective assessment of image quality, an ensemble of images is used to compute the 1st and 2nd order statistics of the data. Often, only a finite number of images is available, leading to the issue of statistical variability in numerical observer performance. Resampling-based strategies can help overcome this issue. In this paper, we compared different combinations of resampling schemes (the leave-one-out (LOO) and the half-train/half-test (HT/HT)) and model observers (the conventional channelized Hotelling observer (CHO), channelized linear discriminant (CLD) and channelized quadratic discriminant). Observer performance was quantified by the area under the ROC curve (AUC). For a binary classification task and for each observer, the AUC value for an ensemble size of 2000 samples per class served as a gold standard for that observer. Results indicated that each observer yielded a different performance depending on the ensemble size and the resampling scheme. For a small ensemble size, the combination [CHO, HT/HT] had more accurate rankings than the combination [CHO, LOO]. Using the LOO scheme, the CLD and CHO had similar performance for large ensembles. However, the CLD outperformed the CHO and gave more accurate rankings for smaller ensembles. As the ensemble size decreased, the performance of the [CHO, LOO] combination seriously deteriorated as opposed to the [CLD, LOO] combination. Thus, it might be desirable to use the CLD with the LOO scheme when smaller ensemble size is available.

KW - channelized model observers

KW - Hotelling observe

KW - leave-one-out resampling scheme

KW - rank correlation coefficient

UR - http://www.scopus.com/inward/record.url?scp=85029079329&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029079329&partnerID=8YFLogxK

U2 - 10.1088/1361-6560/aa807a

DO - 10.1088/1361-6560/aa807a

M3 - Article

C2 - 28829044

AN - SCOPUS:85029079329

VL - 62

SP - 7300

EP - 7320

JO - Physics in Medicine and Biology

JF - Physics in Medicine and Biology

SN - 0031-9155

IS - 18

ER -