Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy

Stacey L. Ishman, James R. Benke, Kaalan Erik Johnson, Karen B. Zur, Ian N. Jacobs, Marc C. Thorne, David J. Brown, Sandra Y Lin, Nasir Islam Bhatti, Ellen S. Deutsch

Research output: Contribution to journalArticle

Abstract

Objectives: Toconfirm interrater reliability using blinded evaluation of a skills-assessment instrument to assess the surgical performance of resident and fellow trainees performing pediatric direct laryngoscopy and rigid bronchoscopy in simulated models. Design: Prospective, paired, blinded observational validation study. Subjects: Paired observers from multiple institutions simultaneously evaluated residents and fellows who were performing surgery in an animal laboratory or using high-fidelity manikins. The evaluators had no previous affiliation with the residents and fellows and did not know their year of training. Interventions: One- and 2-page versions of an objective structured assessment of technical skills (OSATS) assessment instrument composed of global and a taskspecific surgical items were used to evaluate surgical performance. Results: Fifty-two evaluations were completed by 17 attending evaluators. The instrument agreement for the 2-page assessment was 71.4% when measured as a binary variable (ie, competent vs not competent) (κ = 0.38; P =.08). Evaluation as a continuous variable revealed a 42.9% percentage agreement (κ =0.18; P =.14). The intraclass correlation was 0.53, considered substantial/good interrater reliability (69% reliable). For the 1-page instrument, agreement was 77.4% when measured as a binary variable (κ = 0.53, P =. 0015). Agreement when evaluated as a continuous measure was 71.0% (κ =0.54, P <.001). The intraclass correlation was 0.73, considered high interrater reliability (85% reliable). Conclusions: The OSATS assessment instrument is an effective tool for evaluating surgical performance among trainees with acceptable interrater reliability in a simulator setting. Reliability was good for both the 1- and 2-page OSATS checklists, and both serve as excellent tools to provide immediate formative feedback on operational competency.

Original languageEnglish (US)
Pages (from-to)916-922
Number of pages7
JournalArchives of Otolaryngology--Head and Neck Surgery
Volume138
Issue number10
DOIs
StatePublished - 2012

Fingerprint

Laryngoscopy
Bronchoscopy
Manikins
Validation Studies
Laboratory Animals
Checklist
Observational Studies
Pediatrics

ASJC Scopus subject areas

  • Otorhinolaryngology
  • Surgery
  • Medicine(all)

Cite this

Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy. / Ishman, Stacey L.; Benke, James R.; Johnson, Kaalan Erik; Zur, Karen B.; Jacobs, Ian N.; Thorne, Marc C.; Brown, David J.; Lin, Sandra Y; Bhatti, Nasir Islam; Deutsch, Ellen S.

In: Archives of Otolaryngology--Head and Neck Surgery, Vol. 138, No. 10, 2012, p. 916-922.

Research output: Contribution to journalArticle

Ishman, Stacey L. ; Benke, James R. ; Johnson, Kaalan Erik ; Zur, Karen B. ; Jacobs, Ian N. ; Thorne, Marc C. ; Brown, David J. ; Lin, Sandra Y ; Bhatti, Nasir Islam ; Deutsch, Ellen S. / Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy. In: Archives of Otolaryngology--Head and Neck Surgery. 2012 ; Vol. 138, No. 10. pp. 916-922.
@article{6a0f2d8724d441569cebda929388d196,
title = "Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy",
abstract = "Objectives: Toconfirm interrater reliability using blinded evaluation of a skills-assessment instrument to assess the surgical performance of resident and fellow trainees performing pediatric direct laryngoscopy and rigid bronchoscopy in simulated models. Design: Prospective, paired, blinded observational validation study. Subjects: Paired observers from multiple institutions simultaneously evaluated residents and fellows who were performing surgery in an animal laboratory or using high-fidelity manikins. The evaluators had no previous affiliation with the residents and fellows and did not know their year of training. Interventions: One- and 2-page versions of an objective structured assessment of technical skills (OSATS) assessment instrument composed of global and a taskspecific surgical items were used to evaluate surgical performance. Results: Fifty-two evaluations were completed by 17 attending evaluators. The instrument agreement for the 2-page assessment was 71.4{\%} when measured as a binary variable (ie, competent vs not competent) (κ = 0.38; P =.08). Evaluation as a continuous variable revealed a 42.9{\%} percentage agreement (κ =0.18; P =.14). The intraclass correlation was 0.53, considered substantial/good interrater reliability (69{\%} reliable). For the 1-page instrument, agreement was 77.4{\%} when measured as a binary variable (κ = 0.53, P =. 0015). Agreement when evaluated as a continuous measure was 71.0{\%} (κ =0.54, P <.001). The intraclass correlation was 0.73, considered high interrater reliability (85{\%} reliable). Conclusions: The OSATS assessment instrument is an effective tool for evaluating surgical performance among trainees with acceptable interrater reliability in a simulator setting. Reliability was good for both the 1- and 2-page OSATS checklists, and both serve as excellent tools to provide immediate formative feedback on operational competency.",
author = "Ishman, {Stacey L.} and Benke, {James R.} and Johnson, {Kaalan Erik} and Zur, {Karen B.} and Jacobs, {Ian N.} and Thorne, {Marc C.} and Brown, {David J.} and Lin, {Sandra Y} and Bhatti, {Nasir Islam} and Deutsch, {Ellen S.}",
year = "2012",
doi = "10.1001/2013.jamaoto.115",
language = "English (US)",
volume = "138",
pages = "916--922",
journal = "Archives of Otolaryngology",
issn = "2168-6181",
publisher = "American Medical Association",
number = "10",

}

TY - JOUR

T1 - Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy

AU - Ishman, Stacey L.

AU - Benke, James R.

AU - Johnson, Kaalan Erik

AU - Zur, Karen B.

AU - Jacobs, Ian N.

AU - Thorne, Marc C.

AU - Brown, David J.

AU - Lin, Sandra Y

AU - Bhatti, Nasir Islam

AU - Deutsch, Ellen S.

PY - 2012

Y1 - 2012

N2 - Objectives: Toconfirm interrater reliability using blinded evaluation of a skills-assessment instrument to assess the surgical performance of resident and fellow trainees performing pediatric direct laryngoscopy and rigid bronchoscopy in simulated models. Design: Prospective, paired, blinded observational validation study. Subjects: Paired observers from multiple institutions simultaneously evaluated residents and fellows who were performing surgery in an animal laboratory or using high-fidelity manikins. The evaluators had no previous affiliation with the residents and fellows and did not know their year of training. Interventions: One- and 2-page versions of an objective structured assessment of technical skills (OSATS) assessment instrument composed of global and a taskspecific surgical items were used to evaluate surgical performance. Results: Fifty-two evaluations were completed by 17 attending evaluators. The instrument agreement for the 2-page assessment was 71.4% when measured as a binary variable (ie, competent vs not competent) (κ = 0.38; P =.08). Evaluation as a continuous variable revealed a 42.9% percentage agreement (κ =0.18; P =.14). The intraclass correlation was 0.53, considered substantial/good interrater reliability (69% reliable). For the 1-page instrument, agreement was 77.4% when measured as a binary variable (κ = 0.53, P =. 0015). Agreement when evaluated as a continuous measure was 71.0% (κ =0.54, P <.001). The intraclass correlation was 0.73, considered high interrater reliability (85% reliable). Conclusions: The OSATS assessment instrument is an effective tool for evaluating surgical performance among trainees with acceptable interrater reliability in a simulator setting. Reliability was good for both the 1- and 2-page OSATS checklists, and both serve as excellent tools to provide immediate formative feedback on operational competency.

AB - Objectives: Toconfirm interrater reliability using blinded evaluation of a skills-assessment instrument to assess the surgical performance of resident and fellow trainees performing pediatric direct laryngoscopy and rigid bronchoscopy in simulated models. Design: Prospective, paired, blinded observational validation study. Subjects: Paired observers from multiple institutions simultaneously evaluated residents and fellows who were performing surgery in an animal laboratory or using high-fidelity manikins. The evaluators had no previous affiliation with the residents and fellows and did not know their year of training. Interventions: One- and 2-page versions of an objective structured assessment of technical skills (OSATS) assessment instrument composed of global and a taskspecific surgical items were used to evaluate surgical performance. Results: Fifty-two evaluations were completed by 17 attending evaluators. The instrument agreement for the 2-page assessment was 71.4% when measured as a binary variable (ie, competent vs not competent) (κ = 0.38; P =.08). Evaluation as a continuous variable revealed a 42.9% percentage agreement (κ =0.18; P =.14). The intraclass correlation was 0.53, considered substantial/good interrater reliability (69% reliable). For the 1-page instrument, agreement was 77.4% when measured as a binary variable (κ = 0.53, P =. 0015). Agreement when evaluated as a continuous measure was 71.0% (κ =0.54, P <.001). The intraclass correlation was 0.73, considered high interrater reliability (85% reliable). Conclusions: The OSATS assessment instrument is an effective tool for evaluating surgical performance among trainees with acceptable interrater reliability in a simulator setting. Reliability was good for both the 1- and 2-page OSATS checklists, and both serve as excellent tools to provide immediate formative feedback on operational competency.

UR - http://www.scopus.com/inward/record.url?scp=84867821340&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84867821340&partnerID=8YFLogxK

U2 - 10.1001/2013.jamaoto.115

DO - 10.1001/2013.jamaoto.115

M3 - Article

C2 - 22986639

AN - SCOPUS:84867821340

VL - 138

SP - 916

EP - 922

JO - Archives of Otolaryngology

JF - Archives of Otolaryngology

SN - 2168-6181

IS - 10

ER -