Self-supervised learning for dense depth estimation in monocular endoscopy

Xingtong Liu, Ayushi Sinha, Mathias Unberath, Masaru Ishii, Gregory Hager, Russell H Taylor, Austin Reiter

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires sequential data from monocular endoscopic videos and a multi-view stereo reconstruction method, e.g. structure from motion, that supervises learning in a sparse but accurate manner. Consequently, our method requires neither manual interaction, such as scaling or labeling, nor patient CT in the training and application phases. We demonstrate the performance of our method on sinus endoscopy data from two patients and validate depth prediction quantitatively using corresponding patient CT scans where we found submillimeter residual errors. (Link to the supplementary video: https://camp.lcsr.jhu.edu/miccai-2018-demonstration-videos/).

Original languageEnglish (US)
Title of host publicationOR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018
EditorsAnand Malpani, Marco A. Zenati, Cristina Oyarzun Laura, M. Emre Celebi, Duygu Sarikaya, Noel C. Codella, Allan Halpern, Marius Erdt, Lena Maier-Hein, Luo Xiongbiao, Stefan Wesarg, Danail Stoyanov, Zeike Taylor, Klaus Drechsler, Kristin Dana, Anne Martel, Raj Shekhar, Sandrine De Ribaupierre, Tobias Reichl, Jonathan McLeod, Miguel Angel González Ballester, Toby Collins, Marius George Linguraru
PublisherSpringer Verlag
Pages128-138
Number of pages11
ISBN (Print)9783030012007
DOIs
StatePublished - Jan 1 2018
Event1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018 - Granada, Spain
Duration: Sep 16 2018Sep 20 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11041 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018
CountrySpain
CityGranada
Period9/16/189/20/18

Fingerprint

Depth Estimation
Endoscopy
Supervised learning
Supervised Learning
Computerized tomography
Labeling
Demonstrations
Neural networks
Structure from Motion
Shading
Anatomy
Scaling
Neural Networks
Prediction
Interaction
Modeling
Demonstrate
Training

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Liu, X., Sinha, A., Unberath, M., Ishii, M., Hager, G., Taylor, R. H., & Reiter, A. (2018). Self-supervised learning for dense depth estimation in monocular endoscopy. In A. Malpani, M. A. Zenati, C. Oyarzun Laura, M. E. Celebi, D. Sarikaya, N. C. Codella, A. Halpern, M. Erdt, L. Maier-Hein, L. Xiongbiao, S. Wesarg, D. Stoyanov, Z. Taylor, K. Drechsler, K. Dana, A. Martel, R. Shekhar, S. De Ribaupierre, T. Reichl, J. McLeod, M. A. González Ballester, T. Collins, ... M. G. Linguraru (Eds.), OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018 (pp. 128-138). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11041 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-01201-4_15

Self-supervised learning for dense depth estimation in monocular endoscopy. / Liu, Xingtong; Sinha, Ayushi; Unberath, Mathias; Ishii, Masaru; Hager, Gregory; Taylor, Russell H; Reiter, Austin.

OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. ed. / Anand Malpani; Marco A. Zenati; Cristina Oyarzun Laura; M. Emre Celebi; Duygu Sarikaya; Noel C. Codella; Allan Halpern; Marius Erdt; Lena Maier-Hein; Luo Xiongbiao; Stefan Wesarg; Danail Stoyanov; Zeike Taylor; Klaus Drechsler; Kristin Dana; Anne Martel; Raj Shekhar; Sandrine De Ribaupierre; Tobias Reichl; Jonathan McLeod; Miguel Angel González Ballester; Toby Collins; Marius George Linguraru. Springer Verlag, 2018. p. 128-138 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11041 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Liu, X, Sinha, A, Unberath, M, Ishii, M, Hager, G, Taylor, RH & Reiter, A 2018, Self-supervised learning for dense depth estimation in monocular endoscopy. in A Malpani, MA Zenati, C Oyarzun Laura, ME Celebi, D Sarikaya, NC Codella, A Halpern, M Erdt, L Maier-Hein, L Xiongbiao, S Wesarg, D Stoyanov, Z Taylor, K Drechsler, K Dana, A Martel, R Shekhar, S De Ribaupierre, T Reichl, J McLeod, MA González Ballester, T Collins & MG Linguraru (eds), OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11041 LNCS, Springer Verlag, pp. 128-138, 1st International Workshop on OR 2.0 Context-Aware Operating Theaters, OR 2.0 2018, 5th International Workshop on Computer Assisted Robotic Endoscopy, CARE 2018, 7th International Workshop on Clinical Image-Based Procedures, CLIP 2018, and 1st International Workshop on Skin Image Analysis, ISIC 2018, held in conjunction with the 21st International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018, Granada, Spain, 9/16/18. https://doi.org/10.1007/978-3-030-01201-4_15
Liu X, Sinha A, Unberath M, Ishii M, Hager G, Taylor RH et al. Self-supervised learning for dense depth estimation in monocular endoscopy. In Malpani A, Zenati MA, Oyarzun Laura C, Celebi ME, Sarikaya D, Codella NC, Halpern A, Erdt M, Maier-Hein L, Xiongbiao L, Wesarg S, Stoyanov D, Taylor Z, Drechsler K, Dana K, Martel A, Shekhar R, De Ribaupierre S, Reichl T, McLeod J, González Ballester MA, Collins T, Linguraru MG, editors, OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. Springer Verlag. 2018. p. 128-138. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-01201-4_15
Liu, Xingtong ; Sinha, Ayushi ; Unberath, Mathias ; Ishii, Masaru ; Hager, Gregory ; Taylor, Russell H ; Reiter, Austin. / Self-supervised learning for dense depth estimation in monocular endoscopy. OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018. editor / Anand Malpani ; Marco A. Zenati ; Cristina Oyarzun Laura ; M. Emre Celebi ; Duygu Sarikaya ; Noel C. Codella ; Allan Halpern ; Marius Erdt ; Lena Maier-Hein ; Luo Xiongbiao ; Stefan Wesarg ; Danail Stoyanov ; Zeike Taylor ; Klaus Drechsler ; Kristin Dana ; Anne Martel ; Raj Shekhar ; Sandrine De Ribaupierre ; Tobias Reichl ; Jonathan McLeod ; Miguel Angel González Ballester ; Toby Collins ; Marius George Linguraru. Springer Verlag, 2018. pp. 128-138 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{c8ef5142c15347d9a317810cda6fd8d3,
title = "Self-supervised learning for dense depth estimation in monocular endoscopy",
abstract = "We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires sequential data from monocular endoscopic videos and a multi-view stereo reconstruction method, e.g. structure from motion, that supervises learning in a sparse but accurate manner. Consequently, our method requires neither manual interaction, such as scaling or labeling, nor patient CT in the training and application phases. We demonstrate the performance of our method on sinus endoscopy data from two patients and validate depth prediction quantitatively using corresponding patient CT scans where we found submillimeter residual errors. (Link to the supplementary video: https://camp.lcsr.jhu.edu/miccai-2018-demonstration-videos/).",
author = "Xingtong Liu and Ayushi Sinha and Mathias Unberath and Masaru Ishii and Gregory Hager and Taylor, {Russell H} and Austin Reiter",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-01201-4_15",
language = "English (US)",
isbn = "9783030012007",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "128--138",
editor = "Anand Malpani and Zenati, {Marco A.} and {Oyarzun Laura}, Cristina and Celebi, {M. Emre} and Duygu Sarikaya and Codella, {Noel C.} and Allan Halpern and Marius Erdt and Lena Maier-Hein and Luo Xiongbiao and Stefan Wesarg and Danail Stoyanov and Zeike Taylor and Klaus Drechsler and Kristin Dana and Anne Martel and Raj Shekhar and {De Ribaupierre}, Sandrine and Tobias Reichl and Jonathan McLeod and {Gonz{\'a}lez Ballester}, {Miguel Angel} and Toby Collins and Linguraru, {Marius George}",
booktitle = "OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018",

}

TY - GEN

T1 - Self-supervised learning for dense depth estimation in monocular endoscopy

AU - Liu, Xingtong

AU - Sinha, Ayushi

AU - Unberath, Mathias

AU - Ishii, Masaru

AU - Hager, Gregory

AU - Taylor, Russell H

AU - Reiter, Austin

PY - 2018/1/1

Y1 - 2018/1/1

N2 - We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires sequential data from monocular endoscopic videos and a multi-view stereo reconstruction method, e.g. structure from motion, that supervises learning in a sparse but accurate manner. Consequently, our method requires neither manual interaction, such as scaling or labeling, nor patient CT in the training and application phases. We demonstrate the performance of our method on sinus endoscopy data from two patients and validate depth prediction quantitatively using corresponding patient CT scans where we found submillimeter residual errors. (Link to the supplementary video: https://camp.lcsr.jhu.edu/miccai-2018-demonstration-videos/).

AB - We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires sequential data from monocular endoscopic videos and a multi-view stereo reconstruction method, e.g. structure from motion, that supervises learning in a sparse but accurate manner. Consequently, our method requires neither manual interaction, such as scaling or labeling, nor patient CT in the training and application phases. We demonstrate the performance of our method on sinus endoscopy data from two patients and validate depth prediction quantitatively using corresponding patient CT scans where we found submillimeter residual errors. (Link to the supplementary video: https://camp.lcsr.jhu.edu/miccai-2018-demonstration-videos/).

UR - http://www.scopus.com/inward/record.url?scp=85054850438&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054850438&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01201-4_15

DO - 10.1007/978-3-030-01201-4_15

M3 - Conference contribution

AN - SCOPUS:85054850438

SN - 9783030012007

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 128

EP - 138

BT - OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis - 1st International Workshop, OR 2.0 2018 5th International Workshop, CARE 2018, 7th International Workshop, CLIP 2018, 3rd International Workshop, ISIC 2018 Held in Conjunction with MICCAI 2018

A2 - Malpani, Anand

A2 - Zenati, Marco A.

A2 - Oyarzun Laura, Cristina

A2 - Celebi, M. Emre

A2 - Sarikaya, Duygu

A2 - Codella, Noel C.

A2 - Halpern, Allan

A2 - Erdt, Marius

A2 - Maier-Hein, Lena

A2 - Xiongbiao, Luo

A2 - Wesarg, Stefan

A2 - Stoyanov, Danail

A2 - Taylor, Zeike

A2 - Drechsler, Klaus

A2 - Dana, Kristin

A2 - Martel, Anne

A2 - Shekhar, Raj

A2 - De Ribaupierre, Sandrine

A2 - Reichl, Tobias

A2 - McLeod, Jonathan

A2 - González Ballester, Miguel Angel

A2 - Collins, Toby

A2 - Linguraru, Marius George

PB - Springer Verlag

ER -