Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization

Sing Chun Lee, Bernhard Fuerst, Javad Fotouhi, Marius Fischer, Greg Osgood, Nassir Navab

Research output: Contribution to journalArticle

Abstract

Purpose: This work proposes a novel algorithm to register cone-beam computed tomography (CBCT) volumes and 3D optical (RGBD) camera views. The co-registered real-time RGBD camera and CBCT imaging enable a novel augmented reality solution for orthopedic surgeries, which allows arbitrary views using digitally reconstructed radiographs overlaid on the reconstructed patient’s surface without the need to move the C-arm. Methods: An RGBD camera is rigidly mounted on the C-arm near the detector. We introduce a calibration method based on the simultaneous reconstruction of the surface and the CBCT scan of an object. The transformation between the two coordinate spaces is recovered using Fast Point Feature Histogram descriptors and the Iterative Closest Point algorithm. Results: Several experiments are performed to assess the repeatability and the accuracy of this method. Target registration error is measured on multiple visual and radio-opaque landmarks to evaluate the accuracy of the registration. Mixed reality visualizations from arbitrary angles are also presented for simulated orthopedic surgeries. Conclusion: To the best of our knowledge, this is the first calibration method which uses only tomographic and RGBD reconstructions. This means that the method does not impose a particular shape of the phantom. We demonstrate a marker-less calibration of CBCT volumes and 3D depth cameras, achieving reasonable registration accuracy. This design requires a one-time factory calibration, is self-contained, and could be integrated into existing mobile C-arms to provide real-time augmented reality views from arbitrary angles.

Original languageEnglish (US)
Pages (from-to)1-9
Number of pages9
JournalInternational journal of computer assisted radiology and surgery
DOIs
StateAccepted/In press - Apr 8 2016

Fingerprint

Cone-Beam Computed Tomography
Calibration
Tomography
Cones
Visualization
Cameras
Augmented reality
Orthopedics
Surgery
Industrial plants
Detectors
Imaging techniques
Experiments

Keywords

  • 3D-3D calibration
  • Augmented reality
  • C-arm
  • Cone-beam CT
  • Intra-operative imaging

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging
  • Health Informatics
  • Surgery

Cite this

Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization. / Lee, Sing Chun; Fuerst, Bernhard; Fotouhi, Javad; Fischer, Marius; Osgood, Greg; Navab, Nassir.

In: International journal of computer assisted radiology and surgery, 08.04.2016, p. 1-9.

Research output: Contribution to journalArticle

@article{331277cd07784d5bb72a8f25fc205435,
title = "Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization",
abstract = "Purpose: This work proposes a novel algorithm to register cone-beam computed tomography (CBCT) volumes and 3D optical (RGBD) camera views. The co-registered real-time RGBD camera and CBCT imaging enable a novel augmented reality solution for orthopedic surgeries, which allows arbitrary views using digitally reconstructed radiographs overlaid on the reconstructed patient’s surface without the need to move the C-arm. Methods: An RGBD camera is rigidly mounted on the C-arm near the detector. We introduce a calibration method based on the simultaneous reconstruction of the surface and the CBCT scan of an object. The transformation between the two coordinate spaces is recovered using Fast Point Feature Histogram descriptors and the Iterative Closest Point algorithm. Results: Several experiments are performed to assess the repeatability and the accuracy of this method. Target registration error is measured on multiple visual and radio-opaque landmarks to evaluate the accuracy of the registration. Mixed reality visualizations from arbitrary angles are also presented for simulated orthopedic surgeries. Conclusion: To the best of our knowledge, this is the first calibration method which uses only tomographic and RGBD reconstructions. This means that the method does not impose a particular shape of the phantom. We demonstrate a marker-less calibration of CBCT volumes and 3D depth cameras, achieving reasonable registration accuracy. This design requires a one-time factory calibration, is self-contained, and could be integrated into existing mobile C-arms to provide real-time augmented reality views from arbitrary angles.",
keywords = "3D-3D calibration, Augmented reality, C-arm, Cone-beam CT, Intra-operative imaging",
author = "Lee, {Sing Chun} and Bernhard Fuerst and Javad Fotouhi and Marius Fischer and Greg Osgood and Nassir Navab",
year = "2016",
month = "4",
day = "8",
doi = "10.1007/s11548-016-1396-1",
language = "English (US)",
pages = "1--9",
journal = "Computer-Assisted Radiology and Surgery",
issn = "1861-6410",
publisher = "Springer Verlag",

}

TY - JOUR

T1 - Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization

AU - Lee, Sing Chun

AU - Fuerst, Bernhard

AU - Fotouhi, Javad

AU - Fischer, Marius

AU - Osgood, Greg

AU - Navab, Nassir

PY - 2016/4/8

Y1 - 2016/4/8

N2 - Purpose: This work proposes a novel algorithm to register cone-beam computed tomography (CBCT) volumes and 3D optical (RGBD) camera views. The co-registered real-time RGBD camera and CBCT imaging enable a novel augmented reality solution for orthopedic surgeries, which allows arbitrary views using digitally reconstructed radiographs overlaid on the reconstructed patient’s surface without the need to move the C-arm. Methods: An RGBD camera is rigidly mounted on the C-arm near the detector. We introduce a calibration method based on the simultaneous reconstruction of the surface and the CBCT scan of an object. The transformation between the two coordinate spaces is recovered using Fast Point Feature Histogram descriptors and the Iterative Closest Point algorithm. Results: Several experiments are performed to assess the repeatability and the accuracy of this method. Target registration error is measured on multiple visual and radio-opaque landmarks to evaluate the accuracy of the registration. Mixed reality visualizations from arbitrary angles are also presented for simulated orthopedic surgeries. Conclusion: To the best of our knowledge, this is the first calibration method which uses only tomographic and RGBD reconstructions. This means that the method does not impose a particular shape of the phantom. We demonstrate a marker-less calibration of CBCT volumes and 3D depth cameras, achieving reasonable registration accuracy. This design requires a one-time factory calibration, is self-contained, and could be integrated into existing mobile C-arms to provide real-time augmented reality views from arbitrary angles.

AB - Purpose: This work proposes a novel algorithm to register cone-beam computed tomography (CBCT) volumes and 3D optical (RGBD) camera views. The co-registered real-time RGBD camera and CBCT imaging enable a novel augmented reality solution for orthopedic surgeries, which allows arbitrary views using digitally reconstructed radiographs overlaid on the reconstructed patient’s surface without the need to move the C-arm. Methods: An RGBD camera is rigidly mounted on the C-arm near the detector. We introduce a calibration method based on the simultaneous reconstruction of the surface and the CBCT scan of an object. The transformation between the two coordinate spaces is recovered using Fast Point Feature Histogram descriptors and the Iterative Closest Point algorithm. Results: Several experiments are performed to assess the repeatability and the accuracy of this method. Target registration error is measured on multiple visual and radio-opaque landmarks to evaluate the accuracy of the registration. Mixed reality visualizations from arbitrary angles are also presented for simulated orthopedic surgeries. Conclusion: To the best of our knowledge, this is the first calibration method which uses only tomographic and RGBD reconstructions. This means that the method does not impose a particular shape of the phantom. We demonstrate a marker-less calibration of CBCT volumes and 3D depth cameras, achieving reasonable registration accuracy. This design requires a one-time factory calibration, is self-contained, and could be integrated into existing mobile C-arms to provide real-time augmented reality views from arbitrary angles.

KW - 3D-3D calibration

KW - Augmented reality

KW - C-arm

KW - Cone-beam CT

KW - Intra-operative imaging

UR - http://www.scopus.com/inward/record.url?scp=84964009633&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84964009633&partnerID=8YFLogxK

U2 - 10.1007/s11548-016-1396-1

DO - 10.1007/s11548-016-1396-1

M3 - Article

C2 - 27059022

AN - SCOPUS:84964009633

SP - 1

EP - 9

JO - Computer-Assisted Radiology and Surgery

JF - Computer-Assisted Radiology and Surgery

SN - 1861-6410

ER -