Crowd-sourced assessment of technical skills: Differentiating animate surgical skill through the wisdom of crowds

Daniel Holst, Timothy M. Kowalewski, Lee W. White, Timothy C. Brand, Jonathan D. Harper, Mathew D. Sorensen, Mireille Truong, Khara Simpson, Alyssa Tanaka, Roger Smith, Thomas S. Lendvay

Research output: Contribution to journalArticle

Abstract

Background: Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing. Methods: Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement. Results: Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89. Conclusion: Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.

Original languageEnglish (US)
Pages (from-to)1183-1188
Number of pages6
JournalJournal of Endourology
Volume29
Issue number10
DOIs
StatePublished - Oct 1 2015
Externally publishedYes

Fingerprint

Crowdsourcing
Robotics
Quality Improvement
Surgical Instruments
Urinary Bladder
Swine
Surgeons
Delivery of Health Care
Costs and Cost Analysis

ASJC Scopus subject areas

  • Urology

Cite this

Holst, D., Kowalewski, T. M., White, L. W., Brand, T. C., Harper, J. D., Sorensen, M. D., ... Lendvay, T. S. (2015). Crowd-sourced assessment of technical skills: Differentiating animate surgical skill through the wisdom of crowds. Journal of Endourology, 29(10), 1183-1188. https://doi.org/10.1089/end.2015.0104

Crowd-sourced assessment of technical skills : Differentiating animate surgical skill through the wisdom of crowds. / Holst, Daniel; Kowalewski, Timothy M.; White, Lee W.; Brand, Timothy C.; Harper, Jonathan D.; Sorensen, Mathew D.; Truong, Mireille; Simpson, Khara; Tanaka, Alyssa; Smith, Roger; Lendvay, Thomas S.

In: Journal of Endourology, Vol. 29, No. 10, 01.10.2015, p. 1183-1188.

Research output: Contribution to journalArticle

Holst, D, Kowalewski, TM, White, LW, Brand, TC, Harper, JD, Sorensen, MD, Truong, M, Simpson, K, Tanaka, A, Smith, R & Lendvay, TS 2015, 'Crowd-sourced assessment of technical skills: Differentiating animate surgical skill through the wisdom of crowds', Journal of Endourology, vol. 29, no. 10, pp. 1183-1188. https://doi.org/10.1089/end.2015.0104
Holst, Daniel ; Kowalewski, Timothy M. ; White, Lee W. ; Brand, Timothy C. ; Harper, Jonathan D. ; Sorensen, Mathew D. ; Truong, Mireille ; Simpson, Khara ; Tanaka, Alyssa ; Smith, Roger ; Lendvay, Thomas S. / Crowd-sourced assessment of technical skills : Differentiating animate surgical skill through the wisdom of crowds. In: Journal of Endourology. 2015 ; Vol. 29, No. 10. pp. 1183-1188.
@article{21756794470840b4b8704ad85d260668,
title = "Crowd-sourced assessment of technical skills: Differentiating animate surgical skill through the wisdom of crowds",
abstract = "Background: Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing. Methods: Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement. Results: Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89. Conclusion: Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.",
author = "Daniel Holst and Kowalewski, {Timothy M.} and White, {Lee W.} and Brand, {Timothy C.} and Harper, {Jonathan D.} and Sorensen, {Mathew D.} and Mireille Truong and Khara Simpson and Alyssa Tanaka and Roger Smith and Lendvay, {Thomas S.}",
year = "2015",
month = "10",
day = "1",
doi = "10.1089/end.2015.0104",
language = "English (US)",
volume = "29",
pages = "1183--1188",
journal = "Journal of Endourology",
issn = "0892-7790",
publisher = "Mary Ann Liebert Inc.",
number = "10",

}

TY - JOUR

T1 - Crowd-sourced assessment of technical skills

T2 - Differentiating animate surgical skill through the wisdom of crowds

AU - Holst, Daniel

AU - Kowalewski, Timothy M.

AU - White, Lee W.

AU - Brand, Timothy C.

AU - Harper, Jonathan D.

AU - Sorensen, Mathew D.

AU - Truong, Mireille

AU - Simpson, Khara

AU - Tanaka, Alyssa

AU - Smith, Roger

AU - Lendvay, Thomas S.

PY - 2015/10/1

Y1 - 2015/10/1

N2 - Background: Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing. Methods: Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement. Results: Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89. Conclusion: Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.

AB - Background: Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing. Methods: Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement. Results: Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89. Conclusion: Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.

UR - http://www.scopus.com/inward/record.url?scp=84933513900&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84933513900&partnerID=8YFLogxK

U2 - 10.1089/end.2015.0104

DO - 10.1089/end.2015.0104

M3 - Article

C2 - 25867006

AN - SCOPUS:84933513900

VL - 29

SP - 1183

EP - 1188

JO - Journal of Endourology

JF - Journal of Endourology

SN - 0892-7790

IS - 10

ER -