Foibles, follies, and fusion: Web-based collaboration for medical image labeling

Bennett A. Landman, Andrew J. Asman, Andrew G. Scoggins, John A. Bogovic, Joshua A. Stein, Jerry Ladd Prince

Research output: Contribution to journalArticle

Abstract

Labels that identify specific anatomical and functional structures within medical images are essential to the characterization of the relationship between structure and function in many scientific and clinical studies. Automated methods that allow for high throughput have not yet been developed for all anatomical targets or validated for exceptional anatomies, and manual labeling remains the gold standard in many cases. However, manual placement of labels within a large image volume such as that obtained using magnetic resonance imaging (MRI) is exceptionally challenging, resource intensive, and fraught with intra- and inter-rater variability. The use of statistical methods to combine labels produced by multiple raters has grown significantly in popularity, in part, because it is thought that by estimating and accounting for rater reliability estimates of the true labels will be more accurate. This paper demonstrates the performance of a class of these statistical label combination methodologies using real-world data contributed by minimally trained human raters. The consistency of the statistical estimates, the accuracy compared to the individual observations, and the variability of both the estimates and the individual observations with respect to the number of labels are presented. It is demonstrated that statistical fusion successfully combines label information using data from online (Internet-based) collaborations among minimally trained raters. This first successful demonstration of a statistically based approach using minimally trained raters opens numerous possibilities for very large scale efforts in collaboration. Extension and generalization of these technologies for new applications will certainly present fascinating areas for continuing research.

Original languageEnglish (US)
Pages (from-to)530-539
Number of pages10
JournalNeuroImage
Volume59
Issue number1
DOIs
StatePublished - Jan 2 2012

Fingerprint

Internet
Anatomy
Magnetic Resonance Imaging
Technology
Research
Clinical Studies

Keywords

  • Delineation
  • Label fusion
  • Labeling
  • Minimal training
  • Parcellation
  • STAPLE
  • STAPLER

ASJC Scopus subject areas

  • Cognitive Neuroscience
  • Neurology

Cite this

Landman, B. A., Asman, A. J., Scoggins, A. G., Bogovic, J. A., Stein, J. A., & Prince, J. L. (2012). Foibles, follies, and fusion: Web-based collaboration for medical image labeling. NeuroImage, 59(1), 530-539. https://doi.org/10.1016/j.neuroimage.2011.07.085

Foibles, follies, and fusion : Web-based collaboration for medical image labeling. / Landman, Bennett A.; Asman, Andrew J.; Scoggins, Andrew G.; Bogovic, John A.; Stein, Joshua A.; Prince, Jerry Ladd.

In: NeuroImage, Vol. 59, No. 1, 02.01.2012, p. 530-539.

Research output: Contribution to journalArticle

Landman, BA, Asman, AJ, Scoggins, AG, Bogovic, JA, Stein, JA & Prince, JL 2012, 'Foibles, follies, and fusion: Web-based collaboration for medical image labeling', NeuroImage, vol. 59, no. 1, pp. 530-539. https://doi.org/10.1016/j.neuroimage.2011.07.085
Landman, Bennett A. ; Asman, Andrew J. ; Scoggins, Andrew G. ; Bogovic, John A. ; Stein, Joshua A. ; Prince, Jerry Ladd. / Foibles, follies, and fusion : Web-based collaboration for medical image labeling. In: NeuroImage. 2012 ; Vol. 59, No. 1. pp. 530-539.
@article{70fefc59a17c472680cbe6f3d327a346,
title = "Foibles, follies, and fusion: Web-based collaboration for medical image labeling",
abstract = "Labels that identify specific anatomical and functional structures within medical images are essential to the characterization of the relationship between structure and function in many scientific and clinical studies. Automated methods that allow for high throughput have not yet been developed for all anatomical targets or validated for exceptional anatomies, and manual labeling remains the gold standard in many cases. However, manual placement of labels within a large image volume such as that obtained using magnetic resonance imaging (MRI) is exceptionally challenging, resource intensive, and fraught with intra- and inter-rater variability. The use of statistical methods to combine labels produced by multiple raters has grown significantly in popularity, in part, because it is thought that by estimating and accounting for rater reliability estimates of the true labels will be more accurate. This paper demonstrates the performance of a class of these statistical label combination methodologies using real-world data contributed by minimally trained human raters. The consistency of the statistical estimates, the accuracy compared to the individual observations, and the variability of both the estimates and the individual observations with respect to the number of labels are presented. It is demonstrated that statistical fusion successfully combines label information using data from online (Internet-based) collaborations among minimally trained raters. This first successful demonstration of a statistically based approach using minimally trained raters opens numerous possibilities for very large scale efforts in collaboration. Extension and generalization of these technologies for new applications will certainly present fascinating areas for continuing research.",
keywords = "Delineation, Label fusion, Labeling, Minimal training, Parcellation, STAPLE, STAPLER",
author = "Landman, {Bennett A.} and Asman, {Andrew J.} and Scoggins, {Andrew G.} and Bogovic, {John A.} and Stein, {Joshua A.} and Prince, {Jerry Ladd}",
year = "2012",
month = "1",
day = "2",
doi = "10.1016/j.neuroimage.2011.07.085",
language = "English (US)",
volume = "59",
pages = "530--539",
journal = "NeuroImage",
issn = "1053-8119",
publisher = "Academic Press Inc.",
number = "1",

}

TY - JOUR

T1 - Foibles, follies, and fusion

T2 - Web-based collaboration for medical image labeling

AU - Landman, Bennett A.

AU - Asman, Andrew J.

AU - Scoggins, Andrew G.

AU - Bogovic, John A.

AU - Stein, Joshua A.

AU - Prince, Jerry Ladd

PY - 2012/1/2

Y1 - 2012/1/2

N2 - Labels that identify specific anatomical and functional structures within medical images are essential to the characterization of the relationship between structure and function in many scientific and clinical studies. Automated methods that allow for high throughput have not yet been developed for all anatomical targets or validated for exceptional anatomies, and manual labeling remains the gold standard in many cases. However, manual placement of labels within a large image volume such as that obtained using magnetic resonance imaging (MRI) is exceptionally challenging, resource intensive, and fraught with intra- and inter-rater variability. The use of statistical methods to combine labels produced by multiple raters has grown significantly in popularity, in part, because it is thought that by estimating and accounting for rater reliability estimates of the true labels will be more accurate. This paper demonstrates the performance of a class of these statistical label combination methodologies using real-world data contributed by minimally trained human raters. The consistency of the statistical estimates, the accuracy compared to the individual observations, and the variability of both the estimates and the individual observations with respect to the number of labels are presented. It is demonstrated that statistical fusion successfully combines label information using data from online (Internet-based) collaborations among minimally trained raters. This first successful demonstration of a statistically based approach using minimally trained raters opens numerous possibilities for very large scale efforts in collaboration. Extension and generalization of these technologies for new applications will certainly present fascinating areas for continuing research.

AB - Labels that identify specific anatomical and functional structures within medical images are essential to the characterization of the relationship between structure and function in many scientific and clinical studies. Automated methods that allow for high throughput have not yet been developed for all anatomical targets or validated for exceptional anatomies, and manual labeling remains the gold standard in many cases. However, manual placement of labels within a large image volume such as that obtained using magnetic resonance imaging (MRI) is exceptionally challenging, resource intensive, and fraught with intra- and inter-rater variability. The use of statistical methods to combine labels produced by multiple raters has grown significantly in popularity, in part, because it is thought that by estimating and accounting for rater reliability estimates of the true labels will be more accurate. This paper demonstrates the performance of a class of these statistical label combination methodologies using real-world data contributed by minimally trained human raters. The consistency of the statistical estimates, the accuracy compared to the individual observations, and the variability of both the estimates and the individual observations with respect to the number of labels are presented. It is demonstrated that statistical fusion successfully combines label information using data from online (Internet-based) collaborations among minimally trained raters. This first successful demonstration of a statistically based approach using minimally trained raters opens numerous possibilities for very large scale efforts in collaboration. Extension and generalization of these technologies for new applications will certainly present fascinating areas for continuing research.

KW - Delineation

KW - Label fusion

KW - Labeling

KW - Minimal training

KW - Parcellation

KW - STAPLE

KW - STAPLER

UR - http://www.scopus.com/inward/record.url?scp=80054108422&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80054108422&partnerID=8YFLogxK

U2 - 10.1016/j.neuroimage.2011.07.085

DO - 10.1016/j.neuroimage.2011.07.085

M3 - Article

C2 - 21839181

AN - SCOPUS:80054108422

VL - 59

SP - 530

EP - 539

JO - NeuroImage

JF - NeuroImage

SN - 1053-8119

IS - 1

ER -