Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure

Olivier Commowick; Audrey Istace; Michaël Kain; Baptiste Laurent; Florent Leray; Mathieu Simon; Sorina Camarasu Pop; Pascal Girard; Roxana Améli; Jean Christophe Ferré; Anne Kerbrat; Thomas Tourdias; Frédéric Cervenansky; Tristan Glatard; Jérémy Beaumont; Senan Doyle; Florence Forbes; Jesse Knight; April Khademi; Amirreza Mahbod; Chunliang Wang; Richard McKinley; Franca Wagner; John Muschelli; Elizabeth Sweeney; Eloy Roura; Xavier Lladó; Michel M. Santos; Wellington P. Santos; Abel G. Silva-Filho; Xavier Tomas-Fernandez; Hélène Urien; Isabelle Bloch; Sergi Valverde; Mariano Cabezas; Francisco Javier Vera-Olmos; Norberto Malpica; Charles Guttmann; Sandra Vukusic; Gilles Edan; Michel Dojat; Martin Styner; Simon K. Warfield; François Cotton; Christian Barillot

doi:10.1038/s41598-018-31911-7

Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure

Olivier Commowick, Audrey Istace, Michaël Kain, Baptiste Laurent, Florent Leray, Mathieu Simon, Sorina Camarasu Pop, Pascal Girard, Roxana Améli, Jean Christophe Ferré, Anne Kerbrat, Thomas Tourdias, Frédéric Cervenansky, Tristan Glatard, Jérémy Beaumont, Senan Doyle, Florence Forbes, Jesse Knight, April Khademi, Amirreza MahbodChunliang Wang, Richard McKinley, Franca Wagner, John Muschelli, Elizabeth Sweeney, Eloy Roura, Xavier Lladó, Michel M. Santos, Wellington P. Santos, Abel G. Silva-Filho, Xavier Tomas-Fernandez, Hélène Urien, Isabelle Bloch, Sergi Valverde, Mariano Cabezas, Francisco Javier Vera-Olmos, Norberto Malpica, Charles Guttmann, Sandra Vukusic, Gilles Edan, Michel Dojat, Martin Styner, Simon K. Warfield, François Cotton, Christian Barillot

Bloomberg School of Public Health

Research output: Contribution to journal › Article › peer-review

51 Scopus citations

Abstract

We present a study of multiple sclerosis segmentation algorithms conducted at the international MICCAI 2016 challenge. This challenge was operated using a new open-science computing infrastructure. This allowed for the automatic and independent evaluation of a large range of algorithms in a fair and completely automatic manner. This computing infrastructure was used to evaluate thirteen methods of MS lesions segmentation, exploring a broad range of state-of-theart algorithms, against a high-quality database of 53 MS cases coming from four centers following a common definition of the acquisition protocol. Each case was annotated manually by an unprecedented number of seven different experts. Results of the challenge highlighted that automatic algorithms, including the recent machine learning methods (random forests, deep learning, …), are still trailing human expertise on both detection and delineation criteria. In addition, we demonstrate that computing a statistically robust consensus of the algorithms performs closer to human expertise on one score (segmentation) although still trailing on detection scores.

Original language	English (US)
Article number	13650
Journal	Scientific reports
Volume	8
Issue number	1
DOIs	https://doi.org/10.1038/s41598-018-31911-7
State	Published - Dec 1 2018

ASJC Scopus subject areas

General

Access to Document

10.1038/s41598-018-31911-7

Cite this

Commowick, O., Istace, A., Kain, M., Laurent, B., Leray, F., Simon, M., Pop, S. C., Girard, P., Améli, R., Ferré, J. C., Kerbrat, A., Tourdias, T., Cervenansky, F., Glatard, T., Beaumont, J., Doyle, S., Forbes, F., Knight, J., Khademi, A., ... Barillot, C. (2018). Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure. Scientific reports, 8(1), Article 13650. https://doi.org/10.1038/s41598-018-31911-7

Commowick, O, Istace, A, Kain, M, Laurent, B, Leray, F, Simon, M, Pop, SC, Girard, P, Améli, R, Ferré, JC, Kerbrat, A, Tourdias, T, Cervenansky, F, Glatard, T, Beaumont, J, Doyle, S, Forbes, F, Knight, J, Khademi, A, Mahbod, A, Wang, C, McKinley, R, Wagner, F, Muschelli, J, Sweeney, E, Roura, E, Lladó, X, Santos, MM, Santos, WP, Silva-Filho, AG, Tomas-Fernandez, X, Urien, H, Bloch, I, Valverde, S, Cabezas, M, Vera-Olmos, FJ, Malpica, N, Guttmann, C, Vukusic, S, Edan, G, Dojat, M, Styner, M, Warfield, SK, Cotton, F & Barillot, C 2018, 'Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure', Scientific reports, vol. 8, no. 1, 13650. https://doi.org/10.1038/s41598-018-31911-7

@article{1df3855f207048209c9d48ee7efd14d2,

title = "Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure",

abstract = "We present a study of multiple sclerosis segmentation algorithms conducted at the international MICCAI 2016 challenge. This challenge was operated using a new open-science computing infrastructure. This allowed for the automatic and independent evaluation of a large range of algorithms in a fair and completely automatic manner. This computing infrastructure was used to evaluate thirteen methods of MS lesions segmentation, exploring a broad range of state-of-theart algorithms, against a high-quality database of 53 MS cases coming from four centers following a common definition of the acquisition protocol. Each case was annotated manually by an unprecedented number of seven different experts. Results of the challenge highlighted that automatic algorithms, including the recent machine learning methods (random forests, deep learning, …), are still trailing human expertise on both detection and delineation criteria. In addition, we demonstrate that computing a statistically robust consensus of the algorithms performs closer to human expertise on one score (segmentation) although still trailing on detection scores.",

author = "Olivier Commowick and Audrey Istace and Micha{\"e}l Kain and Baptiste Laurent and Florent Leray and Mathieu Simon and Pop, {Sorina Camarasu} and Pascal Girard and Roxana Am{\'e}li and Ferr{\'e}, {Jean Christophe} and Anne Kerbrat and Thomas Tourdias and Fr{\'e}d{\'e}ric Cervenansky and Tristan Glatard and J{\'e}r{\'e}my Beaumont and Senan Doyle and Florence Forbes and Jesse Knight and April Khademi and Amirreza Mahbod and Chunliang Wang and Richard McKinley and Franca Wagner and John Muschelli and Elizabeth Sweeney and Eloy Roura and Xavier Llad{\'o} and Santos, {Michel M.} and Santos, {Wellington P.} and Silva-Filho, {Abel G.} and Xavier Tomas-Fernandez and H{\'e}l{\`e}ne Urien and Isabelle Bloch and Sergi Valverde and Mariano Cabezas and Vera-Olmos, {Francisco Javier} and Norberto Malpica and Charles Guttmann and Sandra Vukusic and Gilles Edan and Michel Dojat and Martin Styner and Warfield, {Simon K.} and Fran{\c c}ois Cotton and Christian Barillot",

note = "Publisher Copyright: {\textcopyright} 2018, The Author(s).",

year = "2018",

month = dec,

day = "1",

doi = "10.1038/s41598-018-31911-7",

language = "English (US)",

volume = "8",

journal = "Scientific reports",

issn = "2045-2322",

publisher = "Nature Publishing Group",

number = "1",

}

TY - JOUR

T1 - Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure

AU - Commowick, Olivier

AU - Istace, Audrey

AU - Kain, Michaël

AU - Laurent, Baptiste

AU - Leray, Florent

AU - Simon, Mathieu

AU - Pop, Sorina Camarasu

AU - Girard, Pascal

AU - Améli, Roxana

AU - Ferré, Jean Christophe

AU - Kerbrat, Anne

AU - Tourdias, Thomas

AU - Cervenansky, Frédéric

AU - Glatard, Tristan

AU - Beaumont, Jérémy

AU - Doyle, Senan

AU - Forbes, Florence

AU - Knight, Jesse

AU - Khademi, April

AU - Mahbod, Amirreza

AU - Wang, Chunliang

AU - McKinley, Richard

AU - Wagner, Franca

AU - Muschelli, John

AU - Sweeney, Elizabeth

AU - Roura, Eloy

AU - Lladó, Xavier

AU - Santos, Michel M.

AU - Santos, Wellington P.

AU - Silva-Filho, Abel G.

AU - Tomas-Fernandez, Xavier

AU - Urien, Hélène

AU - Bloch, Isabelle

AU - Valverde, Sergi

AU - Cabezas, Mariano

AU - Vera-Olmos, Francisco Javier

AU - Malpica, Norberto

AU - Guttmann, Charles

AU - Vukusic, Sandra

AU - Edan, Gilles

AU - Dojat, Michel

AU - Styner, Martin

AU - Warfield, Simon K.

AU - Cotton, François

AU - Barillot, Christian

PY - 2018/12/1

Y1 - 2018/12/1

N2 - We present a study of multiple sclerosis segmentation algorithms conducted at the international MICCAI 2016 challenge. This challenge was operated using a new open-science computing infrastructure. This allowed for the automatic and independent evaluation of a large range of algorithms in a fair and completely automatic manner. This computing infrastructure was used to evaluate thirteen methods of MS lesions segmentation, exploring a broad range of state-of-theart algorithms, against a high-quality database of 53 MS cases coming from four centers following a common definition of the acquisition protocol. Each case was annotated manually by an unprecedented number of seven different experts. Results of the challenge highlighted that automatic algorithms, including the recent machine learning methods (random forests, deep learning, …), are still trailing human expertise on both detection and delineation criteria. In addition, we demonstrate that computing a statistically robust consensus of the algorithms performs closer to human expertise on one score (segmentation) although still trailing on detection scores.

AB - We present a study of multiple sclerosis segmentation algorithms conducted at the international MICCAI 2016 challenge. This challenge was operated using a new open-science computing infrastructure. This allowed for the automatic and independent evaluation of a large range of algorithms in a fair and completely automatic manner. This computing infrastructure was used to evaluate thirteen methods of MS lesions segmentation, exploring a broad range of state-of-theart algorithms, against a high-quality database of 53 MS cases coming from four centers following a common definition of the acquisition protocol. Each case was annotated manually by an unprecedented number of seven different experts. Results of the challenge highlighted that automatic algorithms, including the recent machine learning methods (random forests, deep learning, …), are still trailing human expertise on both detection and delineation criteria. In addition, we demonstrate that computing a statistically robust consensus of the algorithms performs closer to human expertise on one score (segmentation) although still trailing on detection scores.

UR - http://www.scopus.com/inward/record.url?scp=85053246791&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85053246791&partnerID=8YFLogxK

U2 - 10.1038/s41598-018-31911-7

DO - 10.1038/s41598-018-31911-7

M3 - Article

C2 - 30209345

AN - SCOPUS:85053246791

SN - 2045-2322

VL - 8

JO - Scientific reports

JF - Scientific reports

IS - 1

M1 - 13650

ER -

Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this