Blinded evaluation of interrater reliability of an operative competency assessment tool for direct laryngoscopy and rigid bronchoscopy

Stacey L. Ishman, James R. Benke, Kaalan Erik Johnson, Karen B. Zur, Ian N. Jacobs, Marc C. Thorne, David J. Brown, Sandra Y Lin, Nasir Islam Bhatti, Ellen S. Deutsch

Research output: Contribution to journalArticle


Objectives: Toconfirm interrater reliability using blinded evaluation of a skills-assessment instrument to assess the surgical performance of resident and fellow trainees performing pediatric direct laryngoscopy and rigid bronchoscopy in simulated models. Design: Prospective, paired, blinded observational validation study. Subjects: Paired observers from multiple institutions simultaneously evaluated residents and fellows who were performing surgery in an animal laboratory or using high-fidelity manikins. The evaluators had no previous affiliation with the residents and fellows and did not know their year of training. Interventions: One- and 2-page versions of an objective structured assessment of technical skills (OSATS) assessment instrument composed of global and a taskspecific surgical items were used to evaluate surgical performance. Results: Fifty-two evaluations were completed by 17 attending evaluators. The instrument agreement for the 2-page assessment was 71.4% when measured as a binary variable (ie, competent vs not competent) (κ = 0.38; P =.08). Evaluation as a continuous variable revealed a 42.9% percentage agreement (κ =0.18; P =.14). The intraclass correlation was 0.53, considered substantial/good interrater reliability (69% reliable). For the 1-page instrument, agreement was 77.4% when measured as a binary variable (κ = 0.53, P =. 0015). Agreement when evaluated as a continuous measure was 71.0% (κ =0.54, P <.001). The intraclass correlation was 0.73, considered high interrater reliability (85% reliable). Conclusions: The OSATS assessment instrument is an effective tool for evaluating surgical performance among trainees with acceptable interrater reliability in a simulator setting. Reliability was good for both the 1- and 2-page OSATS checklists, and both serve as excellent tools to provide immediate formative feedback on operational competency.

Original languageEnglish (US)
Pages (from-to)916-922
Number of pages7
JournalArchives of Otolaryngology--Head and Neck Surgery
Issue number10
Publication statusPublished - 2012


ASJC Scopus subject areas

  • Otorhinolaryngology
  • Surgery
  • Medicine(all)

Cite this