Pairwise comparison-based objective score for automated skill assessment of segments in a surgical task

Anand Malpani, S. Swaroop Vedula, Chi Chiung Grace Chen, Gregory D. Hager

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Current methods for manual evaluation of surgical skill yield a global score for the entire task. The global score does not inform surgical trainees about where in the task they need to improve. We developed and evaluated a framework to automatically generate an objective score for assessing skill in maneuvers (circumscribed segments) within a surgical task. We used an existing video and kinematic data set (with manual annotation for maneuvers) of a suturing and knot-tying task performed by 18 surgeons on a bench-top model using a da Vinci® Surgical System (Intuitive Surgical, Inc., CA). We collected crowd annotations of preferences, for which of the maneuver in a presented pair appeared to have been performed with greater skill and their confidence in the annotation. We trained a classifier to automatically predict preferences using quantitative metrics of time and motion. We generated an objective percentile score for skill assessment by comparing each maneuver sample to all remaining samples in the data set. Accuracy of the classifier for assigning a preference to pairs of maneuvers was at least 80.06% against a single individual (with a larger training data set) and at least 68.0% against each of the seven individuals (with a smaller training data set). Our reliability analyses indicate that automated preference annotations by the classifier are consistent with those by the seven individuals. Trial-level scores computed from maneuver-level scores generated using our framework were moderately correlated with global rating scores assigned by an experienced surgeon (Spearman correlation = 0.47; P-value < 0.0001).

Original languageEnglish (US)
Title of host publicationInformation Processing in Computer-Assisted Interventions - 5th International Conference, IPCAI 2014, Proceedings
PublisherSpringer Verlag
Pages138-147
Number of pages10
ISBN (Print)9783319075204
DOIs
StatePublished - Jan 1 2014
Event5th International Conference on Information Processing in Computer-Assisted Interventions, IPCAI 2014 - Fukuoka, Japan
Duration: Jun 28 2014Jun 28 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8498 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other5th International Conference on Information Processing in Computer-Assisted Interventions, IPCAI 2014
CountryJapan
CityFukuoka
Period6/28/146/28/14

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Pairwise comparison-based objective score for automated skill assessment of segments in a surgical task'. Together they form a unique fingerprint.

  • Cite this

    Malpani, A., Vedula, S. S., Chen, C. C. G., & Hager, G. D. (2014). Pairwise comparison-based objective score for automated skill assessment of segments in a surgical task. In Information Processing in Computer-Assisted Interventions - 5th International Conference, IPCAI 2014, Proceedings (pp. 138-147). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8498 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-07521-1_15