Enabling machine learning in X-ray-based procedures via realistic simulation of image formation

Mathias Unberath, Jan Nico Zaech, Cong Gao, Bastian Bier, Florian Goldmann, Sing Chun Lee, Javad Fotouhi, Russell H Taylor, Mehran Armand, Nassir Navab

Research output: Contribution to journalArticle

Abstract

Purpose: Machine learning-based approaches now outperform competing methods in most disciplines relevant to diagnostic radiology. Image-guided procedures, however, have not yet benefited substantially from the advent of deep learning, in particular because images for procedural guidance are not archived and thus unavailable for learning, and even if they were available, annotations would be a severe challenge due to the vast amounts of data. In silico simulation of X-ray images from 3D CT is an interesting alternative to using true clinical radiographs since labeling is comparably easy and potentially readily available. Methods: We extend our framework for fast and realistic simulation of fluoroscopy from high-resolution CT, called DeepDRR, with tool modeling capabilities. The framework is publicly available, open source, and tightly integrated with the software platforms native to deep learning, i.e., Python, PyTorch, and PyCuda. DeepDRR relies on machine learning for material decomposition and scatter estimation in 3D and 2D, respectively, but uses analytic forward projection and noise injection to ensure acceptable computation times. On two X-ray image analysis tasks, namely (1) anatomical landmark detection and (2) segmentation and localization of robot end-effectors, we demonstrate that convolutional neural networks (ConvNets) trained on DeepDRRs generalize well to real data without re-training or domain adaptation. To this end, we use the exact same training protocol to train ConvNets on naïve and DeepDRRs and compare their performance on data of cadaveric specimens acquired using a clinical C-arm X-ray system. Results: Our findings are consistent across both considered tasks. All ConvNets performed similarly well when evaluated on the respective synthetic testing set. However, when applied to real radiographs of cadaveric anatomy, ConvNets trained on DeepDRRs significantly outperformed ConvNets trained on naïve DRRs (p< 0.01). Conclusion: Our findings for both tasks are positive and promising. Combined with complementary approaches, such as image style transfer, the proposed framework for fast and realistic simulation of fluoroscopy from CT contributes to promoting the implementation of machine learning in X-ray-guided procedures. This paradigm shift has the potential to revolutionize intra-operative image analysis to simplify surgical workflows.

Fingerprint

Learning systems
Image processing
X-Rays
Neural networks
X rays
Fluoroscopy
Learning
Image analysis
Boidae
Workflow
Radiology
Computer Simulation
Noise
End effectors
Anatomy
Software
Labeling
Injections
Machine Learning
Robots

Keywords

  • Artificial intelligence
  • Computer assisted surgery
  • Image guidance
  • Monte Carlo simulation
  • Robotic surgery
  • Segmentation

ASJC Scopus subject areas

  • Surgery
  • Biomedical Engineering
  • Radiology Nuclear Medicine and imaging
  • Computer Vision and Pattern Recognition
  • Health Informatics
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Cite this

Enabling machine learning in X-ray-based procedures via realistic simulation of image formation. / Unberath, Mathias; Zaech, Jan Nico; Gao, Cong; Bier, Bastian; Goldmann, Florian; Lee, Sing Chun; Fotouhi, Javad; Taylor, Russell H; Armand, Mehran; Navab, Nassir.

In: International Journal of Computer Assisted Radiology and Surgery, 01.01.2019.

Research output: Contribution to journalArticle

Unberath, Mathias ; Zaech, Jan Nico ; Gao, Cong ; Bier, Bastian ; Goldmann, Florian ; Lee, Sing Chun ; Fotouhi, Javad ; Taylor, Russell H ; Armand, Mehran ; Navab, Nassir. / Enabling machine learning in X-ray-based procedures via realistic simulation of image formation. In: International Journal of Computer Assisted Radiology and Surgery. 2019.
@article{cdb237984419414ebd53c2e872c7a15a,
title = "Enabling machine learning in X-ray-based procedures via realistic simulation of image formation",
abstract = "Purpose: Machine learning-based approaches now outperform competing methods in most disciplines relevant to diagnostic radiology. Image-guided procedures, however, have not yet benefited substantially from the advent of deep learning, in particular because images for procedural guidance are not archived and thus unavailable for learning, and even if they were available, annotations would be a severe challenge due to the vast amounts of data. In silico simulation of X-ray images from 3D CT is an interesting alternative to using true clinical radiographs since labeling is comparably easy and potentially readily available. Methods: We extend our framework for fast and realistic simulation of fluoroscopy from high-resolution CT, called DeepDRR, with tool modeling capabilities. The framework is publicly available, open source, and tightly integrated with the software platforms native to deep learning, i.e., Python, PyTorch, and PyCuda. DeepDRR relies on machine learning for material decomposition and scatter estimation in 3D and 2D, respectively, but uses analytic forward projection and noise injection to ensure acceptable computation times. On two X-ray image analysis tasks, namely (1) anatomical landmark detection and (2) segmentation and localization of robot end-effectors, we demonstrate that convolutional neural networks (ConvNets) trained on DeepDRRs generalize well to real data without re-training or domain adaptation. To this end, we use the exact same training protocol to train ConvNets on na{\"i}ve and DeepDRRs and compare their performance on data of cadaveric specimens acquired using a clinical C-arm X-ray system. Results: Our findings are consistent across both considered tasks. All ConvNets performed similarly well when evaluated on the respective synthetic testing set. However, when applied to real radiographs of cadaveric anatomy, ConvNets trained on DeepDRRs significantly outperformed ConvNets trained on na{\"i}ve DRRs (p< 0.01). Conclusion: Our findings for both tasks are positive and promising. Combined with complementary approaches, such as image style transfer, the proposed framework for fast and realistic simulation of fluoroscopy from CT contributes to promoting the implementation of machine learning in X-ray-guided procedures. This paradigm shift has the potential to revolutionize intra-operative image analysis to simplify surgical workflows.",
keywords = "Artificial intelligence, Computer assisted surgery, Image guidance, Monte Carlo simulation, Robotic surgery, Segmentation",
author = "Mathias Unberath and Zaech, {Jan Nico} and Cong Gao and Bastian Bier and Florian Goldmann and Lee, {Sing Chun} and Javad Fotouhi and Taylor, {Russell H} and Mehran Armand and Nassir Navab",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/s11548-019-02011-2",
language = "English (US)",
journal = "Computer-Assisted Radiology and Surgery",
issn = "1861-6410",
publisher = "Springer Verlag",

}

TY - JOUR

T1 - Enabling machine learning in X-ray-based procedures via realistic simulation of image formation

AU - Unberath, Mathias

AU - Zaech, Jan Nico

AU - Gao, Cong

AU - Bier, Bastian

AU - Goldmann, Florian

AU - Lee, Sing Chun

AU - Fotouhi, Javad

AU - Taylor, Russell H

AU - Armand, Mehran

AU - Navab, Nassir

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Purpose: Machine learning-based approaches now outperform competing methods in most disciplines relevant to diagnostic radiology. Image-guided procedures, however, have not yet benefited substantially from the advent of deep learning, in particular because images for procedural guidance are not archived and thus unavailable for learning, and even if they were available, annotations would be a severe challenge due to the vast amounts of data. In silico simulation of X-ray images from 3D CT is an interesting alternative to using true clinical radiographs since labeling is comparably easy and potentially readily available. Methods: We extend our framework for fast and realistic simulation of fluoroscopy from high-resolution CT, called DeepDRR, with tool modeling capabilities. The framework is publicly available, open source, and tightly integrated with the software platforms native to deep learning, i.e., Python, PyTorch, and PyCuda. DeepDRR relies on machine learning for material decomposition and scatter estimation in 3D and 2D, respectively, but uses analytic forward projection and noise injection to ensure acceptable computation times. On two X-ray image analysis tasks, namely (1) anatomical landmark detection and (2) segmentation and localization of robot end-effectors, we demonstrate that convolutional neural networks (ConvNets) trained on DeepDRRs generalize well to real data without re-training or domain adaptation. To this end, we use the exact same training protocol to train ConvNets on naïve and DeepDRRs and compare their performance on data of cadaveric specimens acquired using a clinical C-arm X-ray system. Results: Our findings are consistent across both considered tasks. All ConvNets performed similarly well when evaluated on the respective synthetic testing set. However, when applied to real radiographs of cadaveric anatomy, ConvNets trained on DeepDRRs significantly outperformed ConvNets trained on naïve DRRs (p< 0.01). Conclusion: Our findings for both tasks are positive and promising. Combined with complementary approaches, such as image style transfer, the proposed framework for fast and realistic simulation of fluoroscopy from CT contributes to promoting the implementation of machine learning in X-ray-guided procedures. This paradigm shift has the potential to revolutionize intra-operative image analysis to simplify surgical workflows.

AB - Purpose: Machine learning-based approaches now outperform competing methods in most disciplines relevant to diagnostic radiology. Image-guided procedures, however, have not yet benefited substantially from the advent of deep learning, in particular because images for procedural guidance are not archived and thus unavailable for learning, and even if they were available, annotations would be a severe challenge due to the vast amounts of data. In silico simulation of X-ray images from 3D CT is an interesting alternative to using true clinical radiographs since labeling is comparably easy and potentially readily available. Methods: We extend our framework for fast and realistic simulation of fluoroscopy from high-resolution CT, called DeepDRR, with tool modeling capabilities. The framework is publicly available, open source, and tightly integrated with the software platforms native to deep learning, i.e., Python, PyTorch, and PyCuda. DeepDRR relies on machine learning for material decomposition and scatter estimation in 3D and 2D, respectively, but uses analytic forward projection and noise injection to ensure acceptable computation times. On two X-ray image analysis tasks, namely (1) anatomical landmark detection and (2) segmentation and localization of robot end-effectors, we demonstrate that convolutional neural networks (ConvNets) trained on DeepDRRs generalize well to real data without re-training or domain adaptation. To this end, we use the exact same training protocol to train ConvNets on naïve and DeepDRRs and compare their performance on data of cadaveric specimens acquired using a clinical C-arm X-ray system. Results: Our findings are consistent across both considered tasks. All ConvNets performed similarly well when evaluated on the respective synthetic testing set. However, when applied to real radiographs of cadaveric anatomy, ConvNets trained on DeepDRRs significantly outperformed ConvNets trained on naïve DRRs (p< 0.01). Conclusion: Our findings for both tasks are positive and promising. Combined with complementary approaches, such as image style transfer, the proposed framework for fast and realistic simulation of fluoroscopy from CT contributes to promoting the implementation of machine learning in X-ray-guided procedures. This paradigm shift has the potential to revolutionize intra-operative image analysis to simplify surgical workflows.

KW - Artificial intelligence

KW - Computer assisted surgery

KW - Image guidance

KW - Monte Carlo simulation

KW - Robotic surgery

KW - Segmentation

UR - http://www.scopus.com/inward/record.url?scp=85067389055&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067389055&partnerID=8YFLogxK

U2 - 10.1007/s11548-019-02011-2

DO - 10.1007/s11548-019-02011-2

M3 - Article

JO - Computer-Assisted Radiology and Surgery

JF - Computer-Assisted Radiology and Surgery

SN - 1861-6410

ER -