Pairwise comparison-based objective score for automated skill assessment of segments in a surgical task

Anand Malpani, S. Swaroop Vedula, Chi Chiung Grace Chen, Gregory Hager

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Current methods for manual evaluation of surgical skill yield a global score for the entire task. The global score does not inform surgical trainees about where in the task they need to improve. We developed and evaluated a framework to automatically generate an objective score for assessing skill in maneuvers (circumscribed segments) within a surgical task. We used an existing video and kinematic data set (with manual annotation for maneuvers) of a suturing and knot-tying task performed by 18 surgeons on a bench-top model using a da Vinci® Surgical System (Intuitive Surgical, Inc., CA). We collected crowd annotations of preferences, for which of the maneuver in a presented pair appeared to have been performed with greater skill and their confidence in the annotation. We trained a classifier to automatically predict preferences using quantitative metrics of time and motion. We generated an objective percentile score for skill assessment by comparing each maneuver sample to all remaining samples in the data set. Accuracy of the classifier for assigning a preference to pairs of maneuvers was at least 80.06% against a single individual (with a larger training data set) and at least 68.0% against each of the seven individuals (with a smaller training data set). Our reliability analyses indicate that automated preference annotations by the classifier are consistent with those by the seven individuals. Trial-level scores computed from maneuver-level scores generated using our framework were moderately correlated with global rating scores assigned by an experienced surgeon (Spearman correlation = 0.47; P-value <0.0001).

Original languageEnglish (US)
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer Verlag
Pages138-147
Number of pages10
Volume8498 LNCS
ISBN (Print)9783319075204
DOIs
StatePublished - 2014
Event5th International Conference on Information Processing in Computer-Assisted Interventions, IPCAI 2014 - Fukuoka, Japan
Duration: Jun 28 2014Jun 28 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8498 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other5th International Conference on Information Processing in Computer-Assisted Interventions, IPCAI 2014
CountryJapan
CityFukuoka
Period6/28/146/28/14

    Fingerprint

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Malpani, A., Vedula, S. S., Chen, C. C. G., & Hager, G. (2014). Pairwise comparison-based objective score for automated skill assessment of segments in a surgical task. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8498 LNCS, pp. 138-147). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8498 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-07521-1_15