Longitudinal multiple sclerosis lesion segmentation: Resource and challenge

Aaron Carass, Snehashis Roy, Amod Jog, Jennifer L. Cuzzocreo, Elizabeth Magrath, Adrian Gherman, Julia Button, James Nguyen, Ferran Prados, Carole H. Sudre, Manuel Jorge Cardoso, Niamh Cawley, Olga Ciccarelli, Claudia A M Wheeler-Kingshott, Sébastien Ourselin, Laurence Catanese, Hrishikesh Deshpande, Pierre Maurel, Olivier Commowick, Christian BarillotXavier Tomas-Fernandez, Simon K. Warfield, Suthirth Vaidya, Abhijith Chunduru, Ramanathan Muthuganapathy, Ganapathy Krishnamurthi, Andrew Jesson, Tal Arbel, Oskar Maier, Heinz Handels, Leonardo O. Iheme, Devrim Unay, Saurabh Jain, Diana M. Sima, Dirk Smeets, Mohsen Ghafoorian, Bram Platel, Ariel Birenbaum, Hayit Greenspan, Pierre Louis Bazin, Peter Calabresi, Ciprian M Crainiceanu, Lotta M. Ellingsen, Daniel S. Reich, Jerry Ladd Prince, Dzung L. Pham

Research output: Contribution to journalArticle

Abstract

In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2 The Challenge Evaluation Website is: http://smart-stats-tools.org/lesion-challenge-2015 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.

Original languageEnglish (US)
Pages (from-to)77-102
Number of pages26
JournalNeuroImage
Volume148
DOIs
StatePublished - Mar 1 2017

Fingerprint

Multiple Sclerosis
Research Personnel
Research
Datasets

Keywords

  • Magnetic resonance imaging
  • Multiple sclerosis

ASJC Scopus subject areas

  • Neurology
  • Cognitive Neuroscience

Cite this

Carass, A., Roy, S., Jog, A., Cuzzocreo, J. L., Magrath, E., Gherman, A., ... Pham, D. L. (2017). Longitudinal multiple sclerosis lesion segmentation: Resource and challenge. NeuroImage, 148, 77-102. https://doi.org/10.1016/j.neuroimage.2016.12.064

Longitudinal multiple sclerosis lesion segmentation : Resource and challenge. / Carass, Aaron; Roy, Snehashis; Jog, Amod; Cuzzocreo, Jennifer L.; Magrath, Elizabeth; Gherman, Adrian; Button, Julia; Nguyen, James; Prados, Ferran; Sudre, Carole H.; Jorge Cardoso, Manuel; Cawley, Niamh; Ciccarelli, Olga; Wheeler-Kingshott, Claudia A M; Ourselin, Sébastien; Catanese, Laurence; Deshpande, Hrishikesh; Maurel, Pierre; Commowick, Olivier; Barillot, Christian; Tomas-Fernandez, Xavier; Warfield, Simon K.; Vaidya, Suthirth; Chunduru, Abhijith; Muthuganapathy, Ramanathan; Krishnamurthi, Ganapathy; Jesson, Andrew; Arbel, Tal; Maier, Oskar; Handels, Heinz; Iheme, Leonardo O.; Unay, Devrim; Jain, Saurabh; Sima, Diana M.; Smeets, Dirk; Ghafoorian, Mohsen; Platel, Bram; Birenbaum, Ariel; Greenspan, Hayit; Bazin, Pierre Louis; Calabresi, Peter; Crainiceanu, Ciprian M; Ellingsen, Lotta M.; Reich, Daniel S.; Prince, Jerry Ladd; Pham, Dzung L.

In: NeuroImage, Vol. 148, 01.03.2017, p. 77-102.

Research output: Contribution to journalArticle

Carass, A, Roy, S, Jog, A, Cuzzocreo, JL, Magrath, E, Gherman, A, Button, J, Nguyen, J, Prados, F, Sudre, CH, Jorge Cardoso, M, Cawley, N, Ciccarelli, O, Wheeler-Kingshott, CAM, Ourselin, S, Catanese, L, Deshpande, H, Maurel, P, Commowick, O, Barillot, C, Tomas-Fernandez, X, Warfield, SK, Vaidya, S, Chunduru, A, Muthuganapathy, R, Krishnamurthi, G, Jesson, A, Arbel, T, Maier, O, Handels, H, Iheme, LO, Unay, D, Jain, S, Sima, DM, Smeets, D, Ghafoorian, M, Platel, B, Birenbaum, A, Greenspan, H, Bazin, PL, Calabresi, P, Crainiceanu, CM, Ellingsen, LM, Reich, DS, Prince, JL & Pham, DL 2017, 'Longitudinal multiple sclerosis lesion segmentation: Resource and challenge', NeuroImage, vol. 148, pp. 77-102. https://doi.org/10.1016/j.neuroimage.2016.12.064
Carass, Aaron ; Roy, Snehashis ; Jog, Amod ; Cuzzocreo, Jennifer L. ; Magrath, Elizabeth ; Gherman, Adrian ; Button, Julia ; Nguyen, James ; Prados, Ferran ; Sudre, Carole H. ; Jorge Cardoso, Manuel ; Cawley, Niamh ; Ciccarelli, Olga ; Wheeler-Kingshott, Claudia A M ; Ourselin, Sébastien ; Catanese, Laurence ; Deshpande, Hrishikesh ; Maurel, Pierre ; Commowick, Olivier ; Barillot, Christian ; Tomas-Fernandez, Xavier ; Warfield, Simon K. ; Vaidya, Suthirth ; Chunduru, Abhijith ; Muthuganapathy, Ramanathan ; Krishnamurthi, Ganapathy ; Jesson, Andrew ; Arbel, Tal ; Maier, Oskar ; Handels, Heinz ; Iheme, Leonardo O. ; Unay, Devrim ; Jain, Saurabh ; Sima, Diana M. ; Smeets, Dirk ; Ghafoorian, Mohsen ; Platel, Bram ; Birenbaum, Ariel ; Greenspan, Hayit ; Bazin, Pierre Louis ; Calabresi, Peter ; Crainiceanu, Ciprian M ; Ellingsen, Lotta M. ; Reich, Daniel S. ; Prince, Jerry Ladd ; Pham, Dzung L. / Longitudinal multiple sclerosis lesion segmentation : Resource and challenge. In: NeuroImage. 2017 ; Vol. 148. pp. 77-102.
@article{84c846e79fdb41fa898ce07e87d0b34d,
title = "Longitudinal multiple sclerosis lesion segmentation: Resource and challenge",
abstract = "In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2 The Challenge Evaluation Website is: http://smart-stats-tools.org/lesion-challenge-2015 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.",
keywords = "Magnetic resonance imaging, Multiple sclerosis",
author = "Aaron Carass and Snehashis Roy and Amod Jog and Cuzzocreo, {Jennifer L.} and Elizabeth Magrath and Adrian Gherman and Julia Button and James Nguyen and Ferran Prados and Sudre, {Carole H.} and {Jorge Cardoso}, Manuel and Niamh Cawley and Olga Ciccarelli and Wheeler-Kingshott, {Claudia A M} and S{\'e}bastien Ourselin and Laurence Catanese and Hrishikesh Deshpande and Pierre Maurel and Olivier Commowick and Christian Barillot and Xavier Tomas-Fernandez and Warfield, {Simon K.} and Suthirth Vaidya and Abhijith Chunduru and Ramanathan Muthuganapathy and Ganapathy Krishnamurthi and Andrew Jesson and Tal Arbel and Oskar Maier and Heinz Handels and Iheme, {Leonardo O.} and Devrim Unay and Saurabh Jain and Sima, {Diana M.} and Dirk Smeets and Mohsen Ghafoorian and Bram Platel and Ariel Birenbaum and Hayit Greenspan and Bazin, {Pierre Louis} and Peter Calabresi and Crainiceanu, {Ciprian M} and Ellingsen, {Lotta M.} and Reich, {Daniel S.} and Prince, {Jerry Ladd} and Pham, {Dzung L.}",
year = "2017",
month = "3",
day = "1",
doi = "10.1016/j.neuroimage.2016.12.064",
language = "English (US)",
volume = "148",
pages = "77--102",
journal = "NeuroImage",
issn = "1053-8119",
publisher = "Academic Press Inc.",

}

TY - JOUR

T1 - Longitudinal multiple sclerosis lesion segmentation

T2 - Resource and challenge

AU - Carass, Aaron

AU - Roy, Snehashis

AU - Jog, Amod

AU - Cuzzocreo, Jennifer L.

AU - Magrath, Elizabeth

AU - Gherman, Adrian

AU - Button, Julia

AU - Nguyen, James

AU - Prados, Ferran

AU - Sudre, Carole H.

AU - Jorge Cardoso, Manuel

AU - Cawley, Niamh

AU - Ciccarelli, Olga

AU - Wheeler-Kingshott, Claudia A M

AU - Ourselin, Sébastien

AU - Catanese, Laurence

AU - Deshpande, Hrishikesh

AU - Maurel, Pierre

AU - Commowick, Olivier

AU - Barillot, Christian

AU - Tomas-Fernandez, Xavier

AU - Warfield, Simon K.

AU - Vaidya, Suthirth

AU - Chunduru, Abhijith

AU - Muthuganapathy, Ramanathan

AU - Krishnamurthi, Ganapathy

AU - Jesson, Andrew

AU - Arbel, Tal

AU - Maier, Oskar

AU - Handels, Heinz

AU - Iheme, Leonardo O.

AU - Unay, Devrim

AU - Jain, Saurabh

AU - Sima, Diana M.

AU - Smeets, Dirk

AU - Ghafoorian, Mohsen

AU - Platel, Bram

AU - Birenbaum, Ariel

AU - Greenspan, Hayit

AU - Bazin, Pierre Louis

AU - Calabresi, Peter

AU - Crainiceanu, Ciprian M

AU - Ellingsen, Lotta M.

AU - Reich, Daniel S.

AU - Prince, Jerry Ladd

AU - Pham, Dzung L.

PY - 2017/3/1

Y1 - 2017/3/1

N2 - In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2 The Challenge Evaluation Website is: http://smart-stats-tools.org/lesion-challenge-2015 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.

AB - In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2 The Challenge Evaluation Website is: http://smart-stats-tools.org/lesion-challenge-2015 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.

KW - Magnetic resonance imaging

KW - Multiple sclerosis

UR - http://www.scopus.com/inward/record.url?scp=85009736721&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85009736721&partnerID=8YFLogxK

U2 - 10.1016/j.neuroimage.2016.12.064

DO - 10.1016/j.neuroimage.2016.12.064

M3 - Article

C2 - 28087490

AN - SCOPUS:85009736721

VL - 148

SP - 77

EP - 102

JO - NeuroImage

JF - NeuroImage

SN - 1053-8119

ER -