Practical no-gold-standard evaluation framework for quantitative imaging methods: Application to lesion segmentation in positron emission tomography

Abhinav Kumar Jha, Esther Mena, Brian S Caffo, Saeed Ashrafinia, Arman Rahmim, Eric Frey, Rathan M. Subramaniam

Research output: Contribution to journalArticle

Abstract

Recently, a class of no-gold-standard (NGS) techniques have been proposed to evaluate quantitative imaging methods using patient data. These techniques provide figures of merit (FoMs) quantifying the precision of the estimated quantitative value without requiring repeated measurements and without requiring a gold standard. However, applying these techniques to patient data presents several practical difficulties including assessing the underlying assumptions, accounting for patient-sampling-related uncertainty, and assessing the reliability of the estimated FoMs. To address these issues, we propose statistical tests that provide confidence in the underlying assumptions and in the reliability of the estimated FoMs. Furthermore, the NGS technique is integrated within a bootstrap-based methodology to account for patient-sampling-related uncertainty. The developed NGS framework was applied to evaluate four methods for segmenting lesions from F-Fluoro-2-deoxyglucose positron emission tomography images of patients with head-and-neck cancer on the task of precisely measuring the metabolic tumor volume. The NGS technique consistently predicted the same segmentation method as the most precise method. The proposed framework provided confidence in these results, even when gold-standard data were not available. The bootstrap-based methodology indicated improved performance of the NGS technique with larger numbers of patient studies, as was expected, and yielded consistent results as long as data from more than 80 lesions were available for the analysis.

Original languageEnglish (US)
Article number011011
JournalJournal of Medical Imaging
Volume4
Issue number1
DOIs
StatePublished - Jan 1 2017

Fingerprint

Positron emission tomography
Gold
Positron-Emission Tomography
Imaging techniques
Uncertainty
Sampling
Statistical tests
Deoxyglucose
Head and Neck Neoplasms
Tumor Burden
Tumors

Keywords

  • metabolic tumor volume
  • no-gold-standard evaluation
  • positron emission tomography segmentation
  • quantitative imaging biomarkers

ASJC Scopus subject areas

  • Bioengineering
  • Radiology Nuclear Medicine and imaging

Cite this

Practical no-gold-standard evaluation framework for quantitative imaging methods : Application to lesion segmentation in positron emission tomography. / Jha, Abhinav Kumar; Mena, Esther; Caffo, Brian S; Ashrafinia, Saeed; Rahmim, Arman; Frey, Eric; Subramaniam, Rathan M.

In: Journal of Medical Imaging, Vol. 4, No. 1, 011011, 01.01.2017.

Research output: Contribution to journalArticle

@article{99221474525b4532b116871087b314e4,
title = "Practical no-gold-standard evaluation framework for quantitative imaging methods: Application to lesion segmentation in positron emission tomography",
abstract = "Recently, a class of no-gold-standard (NGS) techniques have been proposed to evaluate quantitative imaging methods using patient data. These techniques provide figures of merit (FoMs) quantifying the precision of the estimated quantitative value without requiring repeated measurements and without requiring a gold standard. However, applying these techniques to patient data presents several practical difficulties including assessing the underlying assumptions, accounting for patient-sampling-related uncertainty, and assessing the reliability of the estimated FoMs. To address these issues, we propose statistical tests that provide confidence in the underlying assumptions and in the reliability of the estimated FoMs. Furthermore, the NGS technique is integrated within a bootstrap-based methodology to account for patient-sampling-related uncertainty. The developed NGS framework was applied to evaluate four methods for segmenting lesions from F-Fluoro-2-deoxyglucose positron emission tomography images of patients with head-and-neck cancer on the task of precisely measuring the metabolic tumor volume. The NGS technique consistently predicted the same segmentation method as the most precise method. The proposed framework provided confidence in these results, even when gold-standard data were not available. The bootstrap-based methodology indicated improved performance of the NGS technique with larger numbers of patient studies, as was expected, and yielded consistent results as long as data from more than 80 lesions were available for the analysis.",
keywords = "metabolic tumor volume, no-gold-standard evaluation, positron emission tomography segmentation, quantitative imaging biomarkers",
author = "Jha, {Abhinav Kumar} and Esther Mena and Caffo, {Brian S} and Saeed Ashrafinia and Arman Rahmim and Eric Frey and Subramaniam, {Rathan M.}",
year = "2017",
month = "1",
day = "1",
doi = "10.1117/1.JMI.4.1.011011",
language = "English (US)",
volume = "4",
journal = "Journal of Medical Imaging",
issn = "0720-048X",
publisher = "Elsevier Ireland Ltd",
number = "1",

}

TY - JOUR

T1 - Practical no-gold-standard evaluation framework for quantitative imaging methods

T2 - Application to lesion segmentation in positron emission tomography

AU - Jha, Abhinav Kumar

AU - Mena, Esther

AU - Caffo, Brian S

AU - Ashrafinia, Saeed

AU - Rahmim, Arman

AU - Frey, Eric

AU - Subramaniam, Rathan M.

PY - 2017/1/1

Y1 - 2017/1/1

N2 - Recently, a class of no-gold-standard (NGS) techniques have been proposed to evaluate quantitative imaging methods using patient data. These techniques provide figures of merit (FoMs) quantifying the precision of the estimated quantitative value without requiring repeated measurements and without requiring a gold standard. However, applying these techniques to patient data presents several practical difficulties including assessing the underlying assumptions, accounting for patient-sampling-related uncertainty, and assessing the reliability of the estimated FoMs. To address these issues, we propose statistical tests that provide confidence in the underlying assumptions and in the reliability of the estimated FoMs. Furthermore, the NGS technique is integrated within a bootstrap-based methodology to account for patient-sampling-related uncertainty. The developed NGS framework was applied to evaluate four methods for segmenting lesions from F-Fluoro-2-deoxyglucose positron emission tomography images of patients with head-and-neck cancer on the task of precisely measuring the metabolic tumor volume. The NGS technique consistently predicted the same segmentation method as the most precise method. The proposed framework provided confidence in these results, even when gold-standard data were not available. The bootstrap-based methodology indicated improved performance of the NGS technique with larger numbers of patient studies, as was expected, and yielded consistent results as long as data from more than 80 lesions were available for the analysis.

AB - Recently, a class of no-gold-standard (NGS) techniques have been proposed to evaluate quantitative imaging methods using patient data. These techniques provide figures of merit (FoMs) quantifying the precision of the estimated quantitative value without requiring repeated measurements and without requiring a gold standard. However, applying these techniques to patient data presents several practical difficulties including assessing the underlying assumptions, accounting for patient-sampling-related uncertainty, and assessing the reliability of the estimated FoMs. To address these issues, we propose statistical tests that provide confidence in the underlying assumptions and in the reliability of the estimated FoMs. Furthermore, the NGS technique is integrated within a bootstrap-based methodology to account for patient-sampling-related uncertainty. The developed NGS framework was applied to evaluate four methods for segmenting lesions from F-Fluoro-2-deoxyglucose positron emission tomography images of patients with head-and-neck cancer on the task of precisely measuring the metabolic tumor volume. The NGS technique consistently predicted the same segmentation method as the most precise method. The proposed framework provided confidence in these results, even when gold-standard data were not available. The bootstrap-based methodology indicated improved performance of the NGS technique with larger numbers of patient studies, as was expected, and yielded consistent results as long as data from more than 80 lesions were available for the analysis.

KW - metabolic tumor volume

KW - no-gold-standard evaluation

KW - positron emission tomography segmentation

KW - quantitative imaging biomarkers

UR - http://www.scopus.com/inward/record.url?scp=85014453387&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85014453387&partnerID=8YFLogxK

U2 - 10.1117/1.JMI.4.1.011011

DO - 10.1117/1.JMI.4.1.011011

M3 - Article

C2 - 28331883

AN - SCOPUS:85014453387

VL - 4

JO - Journal of Medical Imaging

JF - Journal of Medical Imaging

SN - 0720-048X

IS - 1

M1 - 011011

ER -