Query-by-example surgical activity detection

Yixin Gao, S. Swaroop Vedula, Gyusung I. Lee, Mija R. Lee, Sanjeev Khudanpur, Gregory Hager

Research output: Contribution to journalArticle

Abstract

Purpose: Easy acquisition of surgical data opens many opportunities to automate skill evaluation and teaching. Current technology to search tool motion data for surgical activity segments of interest is limited by the need for manual pre-processing, which can be prohibitive at scale. We developed a content-based information retrieval method, query-by-example (QBE), to automatically detect activity segments within surgical data recordings of long duration that match a query. Methods: The example segment of interest (query) and the surgical data recording (target trial) are time series of kinematics. Our approach includes an unsupervised feature learning module using a stacked denoising autoencoder (SDAE), two scoring modules based on asymmetric subsequence dynamic time warping (AS-DTW) and template matching, respectively, and a detection module. A distance matrix of the query against the trial is computed using the SDAE features, followed by AS-DTW combined with template scoring, to generate a ranked list of candidate subsequences (substrings). To evaluate the quality of the ranked list against the ground-truth, thresholding conventional DTW distances and bipartite matching are applied. We computed the recall, precision, F1-score, and a Jaccard index-based score on three experimental setups. We evaluated our QBE method using a suture throw maneuver as the query, on two tool motion datasets (JIGSAWS and MISTIC-SL) captured in a training laboratory. Results: We observed a recall of 93, 90 and 87 % and a precision of 93, 91, and 88 % with same surgeon same trial (SSST), same surgeon different trial (SSDT) and different surgeon (DS) experiment setups on JIGSAWS, and a recall of 87, 81 and 75 % and a precision of 72, 61, and 53 % with SSST, SSDT and DS experiment setups on MISTIC-SL, respectively. Conclusion: We developed a novel, content-based information retrieval method to automatically detect multiple instances of an activity within long surgical recordings. Our method demonstrated adequate recall across different complexity datasets and experimental conditions.

Original languageEnglish (US)
Pages (from-to)1-10
Number of pages10
JournalInternational journal of computer assisted radiology and surgery
DOIs
StateAccepted/In press - Apr 12 2016

Keywords

  • Asymmetric subsequence dynamic time warping
  • Query-by-example
  • Stacked denoising autoencoder
  • Surgical activity detection
  • Surgical data indexing

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging
  • Health Informatics
  • Surgery

Fingerprint Dive into the research topics of 'Query-by-example surgical activity detection'. Together they form a unique fingerprint.

  • Cite this