Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills

Nazema Y. Siddiqui, Megan E. Tarr, Elizabeth J. Geller, Arnold P. Advincula, Michael L. Galloway, Isabel C. Green, Hye Chun Hur, Michael C. Pitter, Emily E. Burke, Martin A. Martino

Research output: Contribution to journalArticle

Abstract

The Robotic Objective Structured Assessment of Technical Skills (R-OSATS) is a previously validated assessment tool that is used to assess 5 standardized inanimate robotic surgery drills. R-OSATS is used to evaluate performance on surgical drills, with scores of 0 to 20 for each drill. Our objective was to establish the minimum threshold score that denotes competence on these drills. Thus, we performed a standard setting study using data from surgeons and trainees in 8 academic medical centers. Cutoff scores for the minimal level of competence using R-OSATS were established using 2 techniques: the modified Angoff and the contrasting groups methods. For the modified Angoff method, 8 content experts met and, in an iterative process, derived the scores that a minimally competent trainee should receive. After 2 iterative rounds of scoring and discussion with the modified Angoff method, we established a minimum competence score per drill with high agreement (rWG range, 0.92-0.98). There was unanimous consensus that a trainee needs to achieve competence on each independent drill. A second method, the contrasting groups method, was used to verify our results. In this method, we compared R-OSATS scores from "inexperienced" (34 postgraduate year 1 and 2 trainees) with "experienced" (22 faculty and fellow) robotic surgeons. The distributions of scores from both groups were plotted, and a cutoff score for each drill was determined from the intersection of the 2 curves. Using this method, the minimum score for competence would be 14 per drill, which is slightly more stringent but confirms the results obtained from the modified Angoff approach. In conclusion, using 2 well-described standard setting techniques, we have established minimum benchmarks designating trainee competence for 5 dry lab robotic surgery drills.

Original languageEnglish (US)
Pages (from-to)633-638
Number of pages6
JournalJournal of Minimally Invasive Gynecology
Volume23
Issue number4
DOIs
StatePublished - May 1 2016

Fingerprint

Benchmarking
Mandrillus
Robotics
Mental Competency
Consensus

Keywords

  • Objective Structured Assessment of Technical Skills
  • Robotic Objective Structured Assessment of Technical Skills
  • Robotic surgery
  • Simulation
  • Surgical education
  • Technical skills

ASJC Scopus subject areas

  • Obstetrics and Gynecology

Cite this

Siddiqui, N. Y., Tarr, M. E., Geller, E. J., Advincula, A. P., Galloway, M. L., Green, I. C., ... Martino, M. A. (2016). Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills. Journal of Minimally Invasive Gynecology, 23(4), 633-638. https://doi.org/10.1016/j.jmig.2016.03.014

Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills. / Siddiqui, Nazema Y.; Tarr, Megan E.; Geller, Elizabeth J.; Advincula, Arnold P.; Galloway, Michael L.; Green, Isabel C.; Hur, Hye Chun; Pitter, Michael C.; Burke, Emily E.; Martino, Martin A.

In: Journal of Minimally Invasive Gynecology, Vol. 23, No. 4, 01.05.2016, p. 633-638.

Research output: Contribution to journalArticle

Siddiqui, NY, Tarr, ME, Geller, EJ, Advincula, AP, Galloway, ML, Green, IC, Hur, HC, Pitter, MC, Burke, EE & Martino, MA 2016, 'Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills', Journal of Minimally Invasive Gynecology, vol. 23, no. 4, pp. 633-638. https://doi.org/10.1016/j.jmig.2016.03.014
Siddiqui, Nazema Y. ; Tarr, Megan E. ; Geller, Elizabeth J. ; Advincula, Arnold P. ; Galloway, Michael L. ; Green, Isabel C. ; Hur, Hye Chun ; Pitter, Michael C. ; Burke, Emily E. ; Martino, Martin A. / Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills. In: Journal of Minimally Invasive Gynecology. 2016 ; Vol. 23, No. 4. pp. 633-638.
@article{7f906c6083ff4358a4917f0d7d82ce51,
title = "Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills",
abstract = "The Robotic Objective Structured Assessment of Technical Skills (R-OSATS) is a previously validated assessment tool that is used to assess 5 standardized inanimate robotic surgery drills. R-OSATS is used to evaluate performance on surgical drills, with scores of 0 to 20 for each drill. Our objective was to establish the minimum threshold score that denotes competence on these drills. Thus, we performed a standard setting study using data from surgeons and trainees in 8 academic medical centers. Cutoff scores for the minimal level of competence using R-OSATS were established using 2 techniques: the modified Angoff and the contrasting groups methods. For the modified Angoff method, 8 content experts met and, in an iterative process, derived the scores that a minimally competent trainee should receive. After 2 iterative rounds of scoring and discussion with the modified Angoff method, we established a minimum competence score per drill with high agreement (rWG range, 0.92-0.98). There was unanimous consensus that a trainee needs to achieve competence on each independent drill. A second method, the contrasting groups method, was used to verify our results. In this method, we compared R-OSATS scores from {"}inexperienced{"} (34 postgraduate year 1 and 2 trainees) with {"}experienced{"} (22 faculty and fellow) robotic surgeons. The distributions of scores from both groups were plotted, and a cutoff score for each drill was determined from the intersection of the 2 curves. Using this method, the minimum score for competence would be 14 per drill, which is slightly more stringent but confirms the results obtained from the modified Angoff approach. In conclusion, using 2 well-described standard setting techniques, we have established minimum benchmarks designating trainee competence for 5 dry lab robotic surgery drills.",
keywords = "Objective Structured Assessment of Technical Skills, Robotic Objective Structured Assessment of Technical Skills, Robotic surgery, Simulation, Surgical education, Technical skills",
author = "Siddiqui, {Nazema Y.} and Tarr, {Megan E.} and Geller, {Elizabeth J.} and Advincula, {Arnold P.} and Galloway, {Michael L.} and Green, {Isabel C.} and Hur, {Hye Chun} and Pitter, {Michael C.} and Burke, {Emily E.} and Martino, {Martin A.}",
year = "2016",
month = "5",
day = "1",
doi = "10.1016/j.jmig.2016.03.014",
language = "English (US)",
volume = "23",
pages = "633--638",
journal = "Journal of Minimally Invasive Gynecology",
issn = "1553-4650",
publisher = "Elsevier",
number = "4",

}

TY - JOUR

T1 - Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills

AU - Siddiqui, Nazema Y.

AU - Tarr, Megan E.

AU - Geller, Elizabeth J.

AU - Advincula, Arnold P.

AU - Galloway, Michael L.

AU - Green, Isabel C.

AU - Hur, Hye Chun

AU - Pitter, Michael C.

AU - Burke, Emily E.

AU - Martino, Martin A.

PY - 2016/5/1

Y1 - 2016/5/1

N2 - The Robotic Objective Structured Assessment of Technical Skills (R-OSATS) is a previously validated assessment tool that is used to assess 5 standardized inanimate robotic surgery drills. R-OSATS is used to evaluate performance on surgical drills, with scores of 0 to 20 for each drill. Our objective was to establish the minimum threshold score that denotes competence on these drills. Thus, we performed a standard setting study using data from surgeons and trainees in 8 academic medical centers. Cutoff scores for the minimal level of competence using R-OSATS were established using 2 techniques: the modified Angoff and the contrasting groups methods. For the modified Angoff method, 8 content experts met and, in an iterative process, derived the scores that a minimally competent trainee should receive. After 2 iterative rounds of scoring and discussion with the modified Angoff method, we established a minimum competence score per drill with high agreement (rWG range, 0.92-0.98). There was unanimous consensus that a trainee needs to achieve competence on each independent drill. A second method, the contrasting groups method, was used to verify our results. In this method, we compared R-OSATS scores from "inexperienced" (34 postgraduate year 1 and 2 trainees) with "experienced" (22 faculty and fellow) robotic surgeons. The distributions of scores from both groups were plotted, and a cutoff score for each drill was determined from the intersection of the 2 curves. Using this method, the minimum score for competence would be 14 per drill, which is slightly more stringent but confirms the results obtained from the modified Angoff approach. In conclusion, using 2 well-described standard setting techniques, we have established minimum benchmarks designating trainee competence for 5 dry lab robotic surgery drills.

AB - The Robotic Objective Structured Assessment of Technical Skills (R-OSATS) is a previously validated assessment tool that is used to assess 5 standardized inanimate robotic surgery drills. R-OSATS is used to evaluate performance on surgical drills, with scores of 0 to 20 for each drill. Our objective was to establish the minimum threshold score that denotes competence on these drills. Thus, we performed a standard setting study using data from surgeons and trainees in 8 academic medical centers. Cutoff scores for the minimal level of competence using R-OSATS were established using 2 techniques: the modified Angoff and the contrasting groups methods. For the modified Angoff method, 8 content experts met and, in an iterative process, derived the scores that a minimally competent trainee should receive. After 2 iterative rounds of scoring and discussion with the modified Angoff method, we established a minimum competence score per drill with high agreement (rWG range, 0.92-0.98). There was unanimous consensus that a trainee needs to achieve competence on each independent drill. A second method, the contrasting groups method, was used to verify our results. In this method, we compared R-OSATS scores from "inexperienced" (34 postgraduate year 1 and 2 trainees) with "experienced" (22 faculty and fellow) robotic surgeons. The distributions of scores from both groups were plotted, and a cutoff score for each drill was determined from the intersection of the 2 curves. Using this method, the minimum score for competence would be 14 per drill, which is slightly more stringent but confirms the results obtained from the modified Angoff approach. In conclusion, using 2 well-described standard setting techniques, we have established minimum benchmarks designating trainee competence for 5 dry lab robotic surgery drills.

KW - Objective Structured Assessment of Technical Skills

KW - Robotic Objective Structured Assessment of Technical Skills

KW - Robotic surgery

KW - Simulation

KW - Surgical education

KW - Technical skills

UR - http://www.scopus.com/inward/record.url?scp=84971635961&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84971635961&partnerID=8YFLogxK

U2 - 10.1016/j.jmig.2016.03.014

DO - 10.1016/j.jmig.2016.03.014

M3 - Article

AN - SCOPUS:84971635961

VL - 23

SP - 633

EP - 638

JO - Journal of Minimally Invasive Gynecology

JF - Journal of Minimally Invasive Gynecology

SN - 1553-4650

IS - 4

ER -