TY - JOUR
T1 - The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions
T2 - A Systematic Review and Perspective
AU - Unberath, Mathias
AU - Gao, Cong
AU - Hu, Yicheng
AU - Judish, Max
AU - Taylor, Russell H.
AU - Armand, Mehran
AU - Grupp, Robert
N1 - Funding Information:
We gratefully acknowledge financial support from NIH NIBIB Trailblazer R21 EB028505, and internal funds of the Malone Center for Engineering in Healthcare at Johns Hopkins University.
Publisher Copyright:
© Copyright © 2021 Unberath, Gao, Hu, Judish, Taylor, Armand and Grupp.
PY - 2021/8/30
Y1 - 2021/8/30
N2 - Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.
AB - Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.
KW - artificial intelligence
KW - augmented reality
KW - computer-assisted interventions
KW - deep learning
KW - image registration
KW - robotic surgery
KW - surgical data science
UR - http://www.scopus.com/inward/record.url?scp=85115015985&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85115015985&partnerID=8YFLogxK
U2 - 10.3389/frobt.2021.716007
DO - 10.3389/frobt.2021.716007
M3 - Article
C2 - 34527706
AN - SCOPUS:85115015985
SN - 2296-9144
VL - 8
JO - Frontiers in Robotics and AI
JF - Frontiers in Robotics and AI
M1 - 716007
ER -