Performance of 'standardized examinees' in a standardized-patient examination of clinical skills

Louis N. Pangaro, Heidi Worth-Dickstein, Mary K. Macmillan, Daniel J. Klass, John H. Shatzer

Research output: Contribution to journalArticle

Abstract

Purpose. As a first step in testing the utility of using trained 'standardized examinees' (SEs) as a quality-assurance measure for the scoring process in a standardized-patient (SP) examination, to test whether medical residents could simulate students in an SP examination and perform consistently to specified levels under test conditions. Method. Fourth-year students from 'the Baltimore-Washington Consortium for SPs participated in a National Board of Medical Examiners Prototype Examination of clinical skills consisting of twelve 15-minute student-patient encounters in 1994-95. For this examination, internal medicine residents were trained to act as ordinary candidates and to achieve target scores by performing to a set level on specific checklist items used by SPs for recording interviewing, physical- examination, and communication skills: The 'strong SEs' were trained to score 80% correct on six of the examination's 12 cases (study cases), and the 'weak SEs' were trained to score 40% correct on the same six cases. The strong and weak SEs' checklist scores on the study cases were compared through independent, two-tailed t-tests. When there was less than 85% agreement on specific checklist items in each case between the SE training and the SP recording, videotapes of the cases were reviewed; in such cases an SE's performance was the final score agreed upon after review. Results. Seven SEs took the SP examination and were not detected by the SPs. There was a total of 84 discrepancies between predicted and recorded checklist scores across 659 checklist items in 40 encounters scored by the SPs. After correcting the discrepancies based on video-tape review, the estimated actual mean score was 77.3% for the strong SEs and 44.0% for the weak SEs, and was higher for the strong SEs in each study case. The overall fidelity of the SEs to their training was estimated to be 97%, and the overall SP accuracy was estimated to be 91%. The videotape review revealed 47 training-scoring discrepancies, most in the area of communication skills. Conclusion. This study suggests that SEs can be trained to specific performance levels and may be an effective internal control for a high-stakes SP examination. They may also provide a mechanism for refining scoring checklists and for exploring the validity of SP examinations.

Original languageEnglish (US)
Pages (from-to)1008-1011
Number of pages4
JournalAcademic Medicine
Volume72
Issue number11
Publication statusPublished - Nov 1997

    Fingerprint

ASJC Scopus subject areas

  • Nursing(all)
  • Public Health, Environmental and Occupational Health
  • Education

Cite this

Pangaro, L. N., Worth-Dickstein, H., Macmillan, M. K., Klass, D. J., & Shatzer, J. H. (1997). Performance of 'standardized examinees' in a standardized-patient examination of clinical skills. Academic Medicine, 72(11), 1008-1011.