Reliability and Validity of 3 Methods of Assessing Orthopedic Resident Skill in Shoulder Surgery

Johnathan A. Bernard, Jonathan R. Dattilo, Uma Srikumaran, Bashir Zikria, Amit Jain, Dawn Mitzner Laporte

Research output: Contribution to journalArticle


Objective: Traditional measures for evaluating resident surgical technical skills (e.g., case logs) assess operative volume but not level of surgical proficiency. Our goal was to compare the reliability and validity of 3 tools for measuring surgical skill among orthopedic residents when performing 3 open surgical approaches to the shoulder. Methods: A total of 23 residents at different stages of their surgical training were tested for technical skill pertaining to 3 shoulder surgical approaches using the following measures: Objective Structured Assessment of Technical Skills (OSATS) checklists, the Global Rating Scale (GRS), and a final pass/fail assessment determined by 3 upper extremity surgeons. Adverse events were recorded. The Cronbach α coefficient was used to assess reliability of the OSATS checklists and GRS scores. Interrater reliability was calculated with intraclass correlation coefficients. Correlations among OSATS checklist scores, GRS scores, and pass/fail assessment were calculated with Spearman ρ. Validity of OSATS checklists was determined using analysis of variance with postgraduate year (PGY) as a between-subjects factor. Significance was set at p <0.05 for all tests. Results: Criterion validity was shown between the OSATS checklists and GRS for the 3 open shoulder approaches. Checklist scores showed superior interrater reliability compared with GRS and subjective pass/fail measurements. GRS scores were positively correlated across training years. The incidence of adverse events was significantly higher among PGY-1 and PGY-2 residents compared with more experienced residents. Conclusion: OSATS checklists are a valid and reliable assessment of technical skills across 3 surgical shoulder approaches. However, checklist scores do not measure quality of technique. Documenting adverse events is necessary to assess quality of technique and ultimate pass/fail status. Multiple methods of assessing surgical skill should be considered when evaluating orthopedic resident surgical performance.

Original languageEnglish (US)
JournalJournal of Surgical Education
StateAccepted/In press - 2016


  • Competency evaluation
  • Global Rating Scale
  • Objective Structured Assessment of Technical Skills
  • Patient Care
  • Residency training

ASJC Scopus subject areas

  • Surgery
  • Education

Fingerprint Dive into the research topics of 'Reliability and Validity of 3 Methods of Assessing Orthopedic Resident Skill in Shoulder Surgery'. Together they form a unique fingerprint.

  • Cite this