Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks

Robert DiPietro, Narges Ahmidi, Anand Malpani, Madeleine Waldram, Gyusung I. Lee, Mija R. Lee, S. Swaroop Vedula, Gregory D. Hager

Research output: Contribution to journalArticlepeer-review

10 Scopus citations


Purpose: Automatically segmenting and classifying surgical activities is an important prerequisite to providing automated, targeted assessment and feedback during surgical training. Prior work has focused almost exclusively on recognizing gestures, or short, atomic units of activity such as pushing needle through tissue, whereas we also focus on recognizing higher-level maneuvers, such as suture throw. Maneuvers exhibit more complexity and variability than the gestures from which they are composed, however working at this granularity has the benefit of being consistent with existing training curricula. Methods: Prior work has focused on hidden Markov model and conditional-random-field-based methods, which typically leverage unary terms that are local in time and linear in model parameters. Because maneuvers are governed by long-term, nonlinear dynamics, we argue that the more expressive unary terms offered by recurrent neural networks (RNNs) are better suited for this task. Four RNN architectures are compared for recognizing activities from kinematics: simple RNNs, long short-term memory, gated recurrent units, and mixed history RNNs. We report performance in terms of error rate and edit distance, and we use a functional analysis-of-variance framework to assess hyperparameter sensitivity for each architecture. Results: We obtain state-of-the-art performance for both maneuver recognition from kinematics (4 maneuvers; error rate of 8.6 ± 3.4 % ; normalized edit distance of 9.3 ± 4.3 %) and gesture recognition from kinematics (10 gestures; error rate of 15.2 ± 6.0 % ; normalized edit distance of 8.4 ± 6.3 %). Conclusions: Automated maneuver recognition is feasible with RNNs, an exciting result which offers the opportunity to provide targeted assessment and feedback at a higher level of granularity. In addition, we show that multiple hyperparameters are important for achieving good performance, and our hyperparameter analysis serves to aid future work in RNN-based activity recognition.

Original languageEnglish (US)
Pages (from-to)2005-2020
Number of pages16
JournalInternational Journal of Computer Assisted Radiology and Surgery
Issue number11
StatePublished - Nov 1 2019


  • Gesture recognition
  • Maneuver recognition
  • Recurrent neural networks
  • Robot-assisted surgery
  • Surgical activity recognition

ASJC Scopus subject areas

  • Surgery
  • Biomedical Engineering
  • Radiology Nuclear Medicine and imaging
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Health Informatics
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks'. Together they form a unique fingerprint.

Cite this