Unsupervised learning for surgical motion by learning to predict the future

Robert DiPietro, Gregory D. Hager

Research output: Contribution to journalArticlepeer-review


We show that it is possible to learn meaningful representations of surgical motion, without supervision, by learning to predict the future. An architecture that combines an RNN encoder-decoder and mixture density networks (MDNs) is developed to model the conditional distribution over future motion given past motion. We show that the learned encodings naturally cluster according to high-level activities, and we demonstrate the usefulness of these learned encodings in the context of information retrieval, where a database of surgical motion is searched for suturing activity using a motion-based query. Future prediction with MDNs is found to significantly outperform simpler baselines as well as the best previously-published result for this task, advancing state-of-the-art performance from an F1 score of 0.60 ± 0.14 to 0.77 ± 0.05.

Original languageEnglish (US)
JournalUnknown Journal
StatePublished - Jun 8 2018

ASJC Scopus subject areas

  • General

Fingerprint Dive into the research topics of 'Unsupervised learning for surgical motion by learning to predict the future'. Together they form a unique fingerprint.

Cite this