Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment

S. Swaroop Vedula, Anand Malpani, Narges Ahmidi, Sanjeev Khudanpur, Gregory Hager, Chi Chiung Grace Chen

Research output: Contribution to journalArticle

Abstract

Objective: Task-level metrics of time and motion efficiency are valid measures of surgical technical skill. Metrics may be computed for segments (maneuvers and gestures) within a task after hierarchical task decomposition. Our objective was to compare task-level and segment (maneuver and gesture)-level metrics for surgical technical skill assessment. Design: Our analyses include predictive modeling using data from a prospective cohort study. We used a hierarchical semantic vocabulary to segment a simple surgical task of passing a needle across an incision and tying a surgeon's knot into maneuvers and gestures. We computed time, path length, and movements for the task, maneuvers, and gestures using tool motion data. We fit logistic regression models to predict experience-based skill using the quantitative metrics. We compared the area under a receiver operating characteristic curve (AUC) for task-level, maneuver-level, and gesture-level models. Setting: Robotic surgical skills training laboratory. Participants: In total, 4 faculty surgeons with experience in robotic surgery and 14 trainee surgeons with no or minimal experience in robotic surgery. Results: Experts performed the task in shorter time (49.74. s; 95% CI = 43.27-56.21 vs. 81.97; 95% CI = 69.71-94.22), with shorter path length (1.63. m; 95% CI = 1.49-1.76 vs. 2.23; 95% CI = 1.91-2.56), and with fewer movements (429.25; 95% CI = 383.80-474.70 vs. 728.69; 95% CI = 631.84-825.54) than novices. Experts differed from novices on metrics for individual maneuvers and gestures. The AUCs were 0.79; 95% CI = 0.62-0.97 for task-level models, 0.78; 95% CI = 0.6-0.96 for maneuver-level models, and 0.7; 95% CI = 0.44-0.97 for gesture-level models. There was no statistically significant difference in AUC between task-level and maneuver-level (p = 0.7) or gesture-level models (p = 0.17). Conclusions: Maneuver-level and gesture-level metrics are discriminative of surgical skill and can be used to provide targeted feedback to surgical trainees.

Original languageEnglish (US)
JournalJournal of Surgical Education
DOIs
StateAccepted/In press - 2016

Fingerprint

Gestures
Robotics
Area Under Curve
trainee
surgery
Logistic Models
expert
experience
Vocabulary
Semantics
ROC Curve
Needles
vocabulary
Cohort Studies
recipient
logistics
semantics
Prospective Studies
regression
efficiency

Keywords

  • Objective skill assessment
  • Practice-Based Learning and Improvement
  • Robotic surgical skills
  • Segment-level skill metrics
  • Task decomposition
  • Task-level skill metrics

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment. / Vedula, S. Swaroop; Malpani, Anand; Ahmidi, Narges; Khudanpur, Sanjeev; Hager, Gregory; Chen, Chi Chiung Grace.

In: Journal of Surgical Education, 2016.

Research output: Contribution to journalArticle

@article{3251002294fc4f9ea00428cfd8f6d05e,
title = "Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment",
abstract = "Objective: Task-level metrics of time and motion efficiency are valid measures of surgical technical skill. Metrics may be computed for segments (maneuvers and gestures) within a task after hierarchical task decomposition. Our objective was to compare task-level and segment (maneuver and gesture)-level metrics for surgical technical skill assessment. Design: Our analyses include predictive modeling using data from a prospective cohort study. We used a hierarchical semantic vocabulary to segment a simple surgical task of passing a needle across an incision and tying a surgeon's knot into maneuvers and gestures. We computed time, path length, and movements for the task, maneuvers, and gestures using tool motion data. We fit logistic regression models to predict experience-based skill using the quantitative metrics. We compared the area under a receiver operating characteristic curve (AUC) for task-level, maneuver-level, and gesture-level models. Setting: Robotic surgical skills training laboratory. Participants: In total, 4 faculty surgeons with experience in robotic surgery and 14 trainee surgeons with no or minimal experience in robotic surgery. Results: Experts performed the task in shorter time (49.74. s; 95{\%} CI = 43.27-56.21 vs. 81.97; 95{\%} CI = 69.71-94.22), with shorter path length (1.63. m; 95{\%} CI = 1.49-1.76 vs. 2.23; 95{\%} CI = 1.91-2.56), and with fewer movements (429.25; 95{\%} CI = 383.80-474.70 vs. 728.69; 95{\%} CI = 631.84-825.54) than novices. Experts differed from novices on metrics for individual maneuvers and gestures. The AUCs were 0.79; 95{\%} CI = 0.62-0.97 for task-level models, 0.78; 95{\%} CI = 0.6-0.96 for maneuver-level models, and 0.7; 95{\%} CI = 0.44-0.97 for gesture-level models. There was no statistically significant difference in AUC between task-level and maneuver-level (p = 0.7) or gesture-level models (p = 0.17). Conclusions: Maneuver-level and gesture-level metrics are discriminative of surgical skill and can be used to provide targeted feedback to surgical trainees.",
keywords = "Objective skill assessment, Practice-Based Learning and Improvement, Robotic surgical skills, Segment-level skill metrics, Task decomposition, Task-level skill metrics",
author = "Vedula, {S. Swaroop} and Anand Malpani and Narges Ahmidi and Sanjeev Khudanpur and Gregory Hager and Chen, {Chi Chiung Grace}",
year = "2016",
doi = "10.1016/j.jsurg.2015.11.009",
language = "English (US)",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",

}

TY - JOUR

T1 - Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment

AU - Vedula, S. Swaroop

AU - Malpani, Anand

AU - Ahmidi, Narges

AU - Khudanpur, Sanjeev

AU - Hager, Gregory

AU - Chen, Chi Chiung Grace

PY - 2016

Y1 - 2016

N2 - Objective: Task-level metrics of time and motion efficiency are valid measures of surgical technical skill. Metrics may be computed for segments (maneuvers and gestures) within a task after hierarchical task decomposition. Our objective was to compare task-level and segment (maneuver and gesture)-level metrics for surgical technical skill assessment. Design: Our analyses include predictive modeling using data from a prospective cohort study. We used a hierarchical semantic vocabulary to segment a simple surgical task of passing a needle across an incision and tying a surgeon's knot into maneuvers and gestures. We computed time, path length, and movements for the task, maneuvers, and gestures using tool motion data. We fit logistic regression models to predict experience-based skill using the quantitative metrics. We compared the area under a receiver operating characteristic curve (AUC) for task-level, maneuver-level, and gesture-level models. Setting: Robotic surgical skills training laboratory. Participants: In total, 4 faculty surgeons with experience in robotic surgery and 14 trainee surgeons with no or minimal experience in robotic surgery. Results: Experts performed the task in shorter time (49.74. s; 95% CI = 43.27-56.21 vs. 81.97; 95% CI = 69.71-94.22), with shorter path length (1.63. m; 95% CI = 1.49-1.76 vs. 2.23; 95% CI = 1.91-2.56), and with fewer movements (429.25; 95% CI = 383.80-474.70 vs. 728.69; 95% CI = 631.84-825.54) than novices. Experts differed from novices on metrics for individual maneuvers and gestures. The AUCs were 0.79; 95% CI = 0.62-0.97 for task-level models, 0.78; 95% CI = 0.6-0.96 for maneuver-level models, and 0.7; 95% CI = 0.44-0.97 for gesture-level models. There was no statistically significant difference in AUC between task-level and maneuver-level (p = 0.7) or gesture-level models (p = 0.17). Conclusions: Maneuver-level and gesture-level metrics are discriminative of surgical skill and can be used to provide targeted feedback to surgical trainees.

AB - Objective: Task-level metrics of time and motion efficiency are valid measures of surgical technical skill. Metrics may be computed for segments (maneuvers and gestures) within a task after hierarchical task decomposition. Our objective was to compare task-level and segment (maneuver and gesture)-level metrics for surgical technical skill assessment. Design: Our analyses include predictive modeling using data from a prospective cohort study. We used a hierarchical semantic vocabulary to segment a simple surgical task of passing a needle across an incision and tying a surgeon's knot into maneuvers and gestures. We computed time, path length, and movements for the task, maneuvers, and gestures using tool motion data. We fit logistic regression models to predict experience-based skill using the quantitative metrics. We compared the area under a receiver operating characteristic curve (AUC) for task-level, maneuver-level, and gesture-level models. Setting: Robotic surgical skills training laboratory. Participants: In total, 4 faculty surgeons with experience in robotic surgery and 14 trainee surgeons with no or minimal experience in robotic surgery. Results: Experts performed the task in shorter time (49.74. s; 95% CI = 43.27-56.21 vs. 81.97; 95% CI = 69.71-94.22), with shorter path length (1.63. m; 95% CI = 1.49-1.76 vs. 2.23; 95% CI = 1.91-2.56), and with fewer movements (429.25; 95% CI = 383.80-474.70 vs. 728.69; 95% CI = 631.84-825.54) than novices. Experts differed from novices on metrics for individual maneuvers and gestures. The AUCs were 0.79; 95% CI = 0.62-0.97 for task-level models, 0.78; 95% CI = 0.6-0.96 for maneuver-level models, and 0.7; 95% CI = 0.44-0.97 for gesture-level models. There was no statistically significant difference in AUC between task-level and maneuver-level (p = 0.7) or gesture-level models (p = 0.17). Conclusions: Maneuver-level and gesture-level metrics are discriminative of surgical skill and can be used to provide targeted feedback to surgical trainees.

KW - Objective skill assessment

KW - Practice-Based Learning and Improvement

KW - Robotic surgical skills

KW - Segment-level skill metrics

KW - Task decomposition

KW - Task-level skill metrics

UR - http://www.scopus.com/inward/record.url?scp=84958206682&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84958206682&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2015.11.009

DO - 10.1016/j.jsurg.2015.11.009

M3 - Article

C2 - 26896147

AN - SCOPUS:84958206682

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

ER -