Evaluation of registration methods on thoracic CT: The EMPIRE10 challenge

Keelin Murphy, Bram Van Ginneken, Joseph M. Reinhardt, Sven Kabus, Kai Ding, Xiang Deng, Kunlin Cao, Kaifang Du, Gary E. Christensen, Vincent Garcia, Tom Vercauteren, Nicholas Ayache, Olivier Commowick, Grgoire Malandain, Ben Glocker, Nikos Paragios, Nassir Navab, Vladlena Gorbunova, Jon Sporring, Marleen De BruijneXiao Han, Mattias P. Heinrich, Julia A. Schnabel, Mark Jenkinson, Cristian Lorenz, Marc Modat, Jamie R. McClelland, Sebastien Ourselin, Sascha E.A. Muenzing, Max A. Viergever, Dante De Nigris, D. Louis Collins, Tal Arbel, Marta Peroni, Rui Li, Gregory C. Sharp, Alexander Schmidt-Richberg, Jan Ehrhardt, René Werner, Dirk Smeets, Dirk Loeckx, Gang Song, Nicholas Tustison, Brian Avants, James C. Gee, Marius Staring, Stefan Klein, Berend C. Stoel, Martin Urschler, Manuel Werlberger, Jef Vandemeulebroucke, Simon Rit, David Sarrut, Josien P.W. Pluim

Research output: Contribution to journalArticle

Abstract

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

Original languageEnglish (US)
Article number5782992
Pages (from-to)1901-1920
Number of pages20
JournalIEEE Transactions on Medical Imaging
Volume30
Issue number11
DOIs
StatePublished - Nov 2011
Externally publishedYes

Fingerprint

Image registration
Thorax
Lung
Research Personnel
Industrial research
Websites
Databases
Research

Keywords

  • Chest
  • computed tomography
  • evaluation
  • registration

ASJC Scopus subject areas

  • Software
  • Radiological and Ultrasound Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

Murphy, K., Van Ginneken, B., Reinhardt, J. M., Kabus, S., Ding, K., Deng, X., ... Pluim, J. P. W. (2011). Evaluation of registration methods on thoracic CT: The EMPIRE10 challenge. IEEE Transactions on Medical Imaging, 30(11), 1901-1920. [5782992]. https://doi.org/10.1109/TMI.2011.2158349

Evaluation of registration methods on thoracic CT : The EMPIRE10 challenge. / Murphy, Keelin; Van Ginneken, Bram; Reinhardt, Joseph M.; Kabus, Sven; Ding, Kai; Deng, Xiang; Cao, Kunlin; Du, Kaifang; Christensen, Gary E.; Garcia, Vincent; Vercauteren, Tom; Ayache, Nicholas; Commowick, Olivier; Malandain, Grgoire; Glocker, Ben; Paragios, Nikos; Navab, Nassir; Gorbunova, Vladlena; Sporring, Jon; De Bruijne, Marleen; Han, Xiao; Heinrich, Mattias P.; Schnabel, Julia A.; Jenkinson, Mark; Lorenz, Cristian; Modat, Marc; McClelland, Jamie R.; Ourselin, Sebastien; Muenzing, Sascha E.A.; Viergever, Max A.; De Nigris, Dante; Collins, D. Louis; Arbel, Tal; Peroni, Marta; Li, Rui; Sharp, Gregory C.; Schmidt-Richberg, Alexander; Ehrhardt, Jan; Werner, René; Smeets, Dirk; Loeckx, Dirk; Song, Gang; Tustison, Nicholas; Avants, Brian; Gee, James C.; Staring, Marius; Klein, Stefan; Stoel, Berend C.; Urschler, Martin; Werlberger, Manuel; Vandemeulebroucke, Jef; Rit, Simon; Sarrut, David; Pluim, Josien P.W.

In: IEEE Transactions on Medical Imaging, Vol. 30, No. 11, 5782992, 11.2011, p. 1901-1920.

Research output: Contribution to journalArticle

Murphy, K, Van Ginneken, B, Reinhardt, JM, Kabus, S, Ding, K, Deng, X, Cao, K, Du, K, Christensen, GE, Garcia, V, Vercauteren, T, Ayache, N, Commowick, O, Malandain, G, Glocker, B, Paragios, N, Navab, N, Gorbunova, V, Sporring, J, De Bruijne, M, Han, X, Heinrich, MP, Schnabel, JA, Jenkinson, M, Lorenz, C, Modat, M, McClelland, JR, Ourselin, S, Muenzing, SEA, Viergever, MA, De Nigris, D, Collins, DL, Arbel, T, Peroni, M, Li, R, Sharp, GC, Schmidt-Richberg, A, Ehrhardt, J, Werner, R, Smeets, D, Loeckx, D, Song, G, Tustison, N, Avants, B, Gee, JC, Staring, M, Klein, S, Stoel, BC, Urschler, M, Werlberger, M, Vandemeulebroucke, J, Rit, S, Sarrut, D & Pluim, JPW 2011, 'Evaluation of registration methods on thoracic CT: The EMPIRE10 challenge', IEEE Transactions on Medical Imaging, vol. 30, no. 11, 5782992, pp. 1901-1920. https://doi.org/10.1109/TMI.2011.2158349
Murphy, Keelin ; Van Ginneken, Bram ; Reinhardt, Joseph M. ; Kabus, Sven ; Ding, Kai ; Deng, Xiang ; Cao, Kunlin ; Du, Kaifang ; Christensen, Gary E. ; Garcia, Vincent ; Vercauteren, Tom ; Ayache, Nicholas ; Commowick, Olivier ; Malandain, Grgoire ; Glocker, Ben ; Paragios, Nikos ; Navab, Nassir ; Gorbunova, Vladlena ; Sporring, Jon ; De Bruijne, Marleen ; Han, Xiao ; Heinrich, Mattias P. ; Schnabel, Julia A. ; Jenkinson, Mark ; Lorenz, Cristian ; Modat, Marc ; McClelland, Jamie R. ; Ourselin, Sebastien ; Muenzing, Sascha E.A. ; Viergever, Max A. ; De Nigris, Dante ; Collins, D. Louis ; Arbel, Tal ; Peroni, Marta ; Li, Rui ; Sharp, Gregory C. ; Schmidt-Richberg, Alexander ; Ehrhardt, Jan ; Werner, René ; Smeets, Dirk ; Loeckx, Dirk ; Song, Gang ; Tustison, Nicholas ; Avants, Brian ; Gee, James C. ; Staring, Marius ; Klein, Stefan ; Stoel, Berend C. ; Urschler, Martin ; Werlberger, Manuel ; Vandemeulebroucke, Jef ; Rit, Simon ; Sarrut, David ; Pluim, Josien P.W. / Evaluation of registration methods on thoracic CT : The EMPIRE10 challenge. In: IEEE Transactions on Medical Imaging. 2011 ; Vol. 30, No. 11. pp. 1901-1920.
@article{abedf4686c1346e18d71006707eaed61,
title = "Evaluation of registration methods on thoracic CT: The EMPIRE10 challenge",
abstract = "EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.",
keywords = "Chest, computed tomography, evaluation, registration",
author = "Keelin Murphy and {Van Ginneken}, Bram and Reinhardt, {Joseph M.} and Sven Kabus and Kai Ding and Xiang Deng and Kunlin Cao and Kaifang Du and Christensen, {Gary E.} and Vincent Garcia and Tom Vercauteren and Nicholas Ayache and Olivier Commowick and Grgoire Malandain and Ben Glocker and Nikos Paragios and Nassir Navab and Vladlena Gorbunova and Jon Sporring and {De Bruijne}, Marleen and Xiao Han and Heinrich, {Mattias P.} and Schnabel, {Julia A.} and Mark Jenkinson and Cristian Lorenz and Marc Modat and McClelland, {Jamie R.} and Sebastien Ourselin and Muenzing, {Sascha E.A.} and Viergever, {Max A.} and {De Nigris}, Dante and Collins, {D. Louis} and Tal Arbel and Marta Peroni and Rui Li and Sharp, {Gregory C.} and Alexander Schmidt-Richberg and Jan Ehrhardt and Ren{\'e} Werner and Dirk Smeets and Dirk Loeckx and Gang Song and Nicholas Tustison and Brian Avants and Gee, {James C.} and Marius Staring and Stefan Klein and Stoel, {Berend C.} and Martin Urschler and Manuel Werlberger and Jef Vandemeulebroucke and Simon Rit and David Sarrut and Pluim, {Josien P.W.}",
year = "2011",
month = "11",
doi = "10.1109/TMI.2011.2158349",
language = "English (US)",
volume = "30",
pages = "1901--1920",
journal = "IEEE Transactions on Medical Imaging",
issn = "0278-0062",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "11",

}

TY - JOUR

T1 - Evaluation of registration methods on thoracic CT

T2 - The EMPIRE10 challenge

AU - Murphy, Keelin

AU - Van Ginneken, Bram

AU - Reinhardt, Joseph M.

AU - Kabus, Sven

AU - Ding, Kai

AU - Deng, Xiang

AU - Cao, Kunlin

AU - Du, Kaifang

AU - Christensen, Gary E.

AU - Garcia, Vincent

AU - Vercauteren, Tom

AU - Ayache, Nicholas

AU - Commowick, Olivier

AU - Malandain, Grgoire

AU - Glocker, Ben

AU - Paragios, Nikos

AU - Navab, Nassir

AU - Gorbunova, Vladlena

AU - Sporring, Jon

AU - De Bruijne, Marleen

AU - Han, Xiao

AU - Heinrich, Mattias P.

AU - Schnabel, Julia A.

AU - Jenkinson, Mark

AU - Lorenz, Cristian

AU - Modat, Marc

AU - McClelland, Jamie R.

AU - Ourselin, Sebastien

AU - Muenzing, Sascha E.A.

AU - Viergever, Max A.

AU - De Nigris, Dante

AU - Collins, D. Louis

AU - Arbel, Tal

AU - Peroni, Marta

AU - Li, Rui

AU - Sharp, Gregory C.

AU - Schmidt-Richberg, Alexander

AU - Ehrhardt, Jan

AU - Werner, René

AU - Smeets, Dirk

AU - Loeckx, Dirk

AU - Song, Gang

AU - Tustison, Nicholas

AU - Avants, Brian

AU - Gee, James C.

AU - Staring, Marius

AU - Klein, Stefan

AU - Stoel, Berend C.

AU - Urschler, Martin

AU - Werlberger, Manuel

AU - Vandemeulebroucke, Jef

AU - Rit, Simon

AU - Sarrut, David

AU - Pluim, Josien P.W.

PY - 2011/11

Y1 - 2011/11

N2 - EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

AB - EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

KW - Chest

KW - computed tomography

KW - evaluation

KW - registration

UR - http://www.scopus.com/inward/record.url?scp=80455145309&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80455145309&partnerID=8YFLogxK

U2 - 10.1109/TMI.2011.2158349

DO - 10.1109/TMI.2011.2158349

M3 - Article

C2 - 21632295

AN - SCOPUS:80455145309

VL - 30

SP - 1901

EP - 1920

JO - IEEE Transactions on Medical Imaging

JF - IEEE Transactions on Medical Imaging

SN - 0278-0062

IS - 11

M1 - 5782992

ER -