Performance of 'standardized examinees' in a standardized-patient examination of clinical skills

Louis N. Pangaro, Heidi Worth-Dickstein, Mary K. Macmillan, Daniel J. Klass, John H. Shatzer

Research output: Contribution to journalArticle

Abstract

Purpose. As a first step in testing the utility of using trained 'standardized examinees' (SEs) as a quality-assurance measure for the scoring process in a standardized-patient (SP) examination, to test whether medical residents could simulate students in an SP examination and perform consistently to specified levels under test conditions. Method. Fourth-year students from 'the Baltimore-Washington Consortium for SPs participated in a National Board of Medical Examiners Prototype Examination of clinical skills consisting of twelve 15-minute student-patient encounters in 1994-95. For this examination, internal medicine residents were trained to act as ordinary candidates and to achieve target scores by performing to a set level on specific checklist items used by SPs for recording interviewing, physical- examination, and communication skills: The 'strong SEs' were trained to score 80% correct on six of the examination's 12 cases (study cases), and the 'weak SEs' were trained to score 40% correct on the same six cases. The strong and weak SEs' checklist scores on the study cases were compared through independent, two-tailed t-tests. When there was less than 85% agreement on specific checklist items in each case between the SE training and the SP recording, videotapes of the cases were reviewed; in such cases an SE's performance was the final score agreed upon after review. Results. Seven SEs took the SP examination and were not detected by the SPs. There was a total of 84 discrepancies between predicted and recorded checklist scores across 659 checklist items in 40 encounters scored by the SPs. After correcting the discrepancies based on video-tape review, the estimated actual mean score was 77.3% for the strong SEs and 44.0% for the weak SEs, and was higher for the strong SEs in each study case. The overall fidelity of the SEs to their training was estimated to be 97%, and the overall SP accuracy was estimated to be 91%. The videotape review revealed 47 training-scoring discrepancies, most in the area of communication skills. Conclusion. This study suggests that SEs can be trained to specific performance levels and may be an effective internal control for a high-stakes SP examination. They may also provide a mechanism for refining scoring checklists and for exploring the validity of SP examinations.

Original languageEnglish (US)
Pages (from-to)1008-1011
Number of pages4
JournalAcademic Medicine
Volume72
Issue number11
StatePublished - Nov 1997

Fingerprint

Clinical Competence
Checklist
examination
performance
Videotape Recording
Students
communication skills
recording
Communication
Coroners and Medical Examiners
Baltimore
resident
Process Assessment (Health Care)
medical examiner
Internal Medicine
student
Physical Examination
quality assurance
candidacy
video

ASJC Scopus subject areas

  • Nursing(all)
  • Public Health, Environmental and Occupational Health
  • Education

Cite this

Pangaro, L. N., Worth-Dickstein, H., Macmillan, M. K., Klass, D. J., & Shatzer, J. H. (1997). Performance of 'standardized examinees' in a standardized-patient examination of clinical skills. Academic Medicine, 72(11), 1008-1011.

Performance of 'standardized examinees' in a standardized-patient examination of clinical skills. / Pangaro, Louis N.; Worth-Dickstein, Heidi; Macmillan, Mary K.; Klass, Daniel J.; Shatzer, John H.

In: Academic Medicine, Vol. 72, No. 11, 11.1997, p. 1008-1011.

Research output: Contribution to journalArticle

Pangaro, LN, Worth-Dickstein, H, Macmillan, MK, Klass, DJ & Shatzer, JH 1997, 'Performance of 'standardized examinees' in a standardized-patient examination of clinical skills', Academic Medicine, vol. 72, no. 11, pp. 1008-1011.
Pangaro LN, Worth-Dickstein H, Macmillan MK, Klass DJ, Shatzer JH. Performance of 'standardized examinees' in a standardized-patient examination of clinical skills. Academic Medicine. 1997 Nov;72(11):1008-1011.
Pangaro, Louis N. ; Worth-Dickstein, Heidi ; Macmillan, Mary K. ; Klass, Daniel J. ; Shatzer, John H. / Performance of 'standardized examinees' in a standardized-patient examination of clinical skills. In: Academic Medicine. 1997 ; Vol. 72, No. 11. pp. 1008-1011.
@article{5acc8601bc824937bb660abc432d9620,
title = "Performance of 'standardized examinees' in a standardized-patient examination of clinical skills",
abstract = "Purpose. As a first step in testing the utility of using trained 'standardized examinees' (SEs) as a quality-assurance measure for the scoring process in a standardized-patient (SP) examination, to test whether medical residents could simulate students in an SP examination and perform consistently to specified levels under test conditions. Method. Fourth-year students from 'the Baltimore-Washington Consortium for SPs participated in a National Board of Medical Examiners Prototype Examination of clinical skills consisting of twelve 15-minute student-patient encounters in 1994-95. For this examination, internal medicine residents were trained to act as ordinary candidates and to achieve target scores by performing to a set level on specific checklist items used by SPs for recording interviewing, physical- examination, and communication skills: The 'strong SEs' were trained to score 80{\%} correct on six of the examination's 12 cases (study cases), and the 'weak SEs' were trained to score 40{\%} correct on the same six cases. The strong and weak SEs' checklist scores on the study cases were compared through independent, two-tailed t-tests. When there was less than 85{\%} agreement on specific checklist items in each case between the SE training and the SP recording, videotapes of the cases were reviewed; in such cases an SE's performance was the final score agreed upon after review. Results. Seven SEs took the SP examination and were not detected by the SPs. There was a total of 84 discrepancies between predicted and recorded checklist scores across 659 checklist items in 40 encounters scored by the SPs. After correcting the discrepancies based on video-tape review, the estimated actual mean score was 77.3{\%} for the strong SEs and 44.0{\%} for the weak SEs, and was higher for the strong SEs in each study case. The overall fidelity of the SEs to their training was estimated to be 97{\%}, and the overall SP accuracy was estimated to be 91{\%}. The videotape review revealed 47 training-scoring discrepancies, most in the area of communication skills. Conclusion. This study suggests that SEs can be trained to specific performance levels and may be an effective internal control for a high-stakes SP examination. They may also provide a mechanism for refining scoring checklists and for exploring the validity of SP examinations.",
author = "Pangaro, {Louis N.} and Heidi Worth-Dickstein and Macmillan, {Mary K.} and Klass, {Daniel J.} and Shatzer, {John H.}",
year = "1997",
month = "11",
language = "English (US)",
volume = "72",
pages = "1008--1011",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "11",

}

TY - JOUR

T1 - Performance of 'standardized examinees' in a standardized-patient examination of clinical skills

AU - Pangaro, Louis N.

AU - Worth-Dickstein, Heidi

AU - Macmillan, Mary K.

AU - Klass, Daniel J.

AU - Shatzer, John H.

PY - 1997/11

Y1 - 1997/11

N2 - Purpose. As a first step in testing the utility of using trained 'standardized examinees' (SEs) as a quality-assurance measure for the scoring process in a standardized-patient (SP) examination, to test whether medical residents could simulate students in an SP examination and perform consistently to specified levels under test conditions. Method. Fourth-year students from 'the Baltimore-Washington Consortium for SPs participated in a National Board of Medical Examiners Prototype Examination of clinical skills consisting of twelve 15-minute student-patient encounters in 1994-95. For this examination, internal medicine residents were trained to act as ordinary candidates and to achieve target scores by performing to a set level on specific checklist items used by SPs for recording interviewing, physical- examination, and communication skills: The 'strong SEs' were trained to score 80% correct on six of the examination's 12 cases (study cases), and the 'weak SEs' were trained to score 40% correct on the same six cases. The strong and weak SEs' checklist scores on the study cases were compared through independent, two-tailed t-tests. When there was less than 85% agreement on specific checklist items in each case between the SE training and the SP recording, videotapes of the cases were reviewed; in such cases an SE's performance was the final score agreed upon after review. Results. Seven SEs took the SP examination and were not detected by the SPs. There was a total of 84 discrepancies between predicted and recorded checklist scores across 659 checklist items in 40 encounters scored by the SPs. After correcting the discrepancies based on video-tape review, the estimated actual mean score was 77.3% for the strong SEs and 44.0% for the weak SEs, and was higher for the strong SEs in each study case. The overall fidelity of the SEs to their training was estimated to be 97%, and the overall SP accuracy was estimated to be 91%. The videotape review revealed 47 training-scoring discrepancies, most in the area of communication skills. Conclusion. This study suggests that SEs can be trained to specific performance levels and may be an effective internal control for a high-stakes SP examination. They may also provide a mechanism for refining scoring checklists and for exploring the validity of SP examinations.

AB - Purpose. As a first step in testing the utility of using trained 'standardized examinees' (SEs) as a quality-assurance measure for the scoring process in a standardized-patient (SP) examination, to test whether medical residents could simulate students in an SP examination and perform consistently to specified levels under test conditions. Method. Fourth-year students from 'the Baltimore-Washington Consortium for SPs participated in a National Board of Medical Examiners Prototype Examination of clinical skills consisting of twelve 15-minute student-patient encounters in 1994-95. For this examination, internal medicine residents were trained to act as ordinary candidates and to achieve target scores by performing to a set level on specific checklist items used by SPs for recording interviewing, physical- examination, and communication skills: The 'strong SEs' were trained to score 80% correct on six of the examination's 12 cases (study cases), and the 'weak SEs' were trained to score 40% correct on the same six cases. The strong and weak SEs' checklist scores on the study cases were compared through independent, two-tailed t-tests. When there was less than 85% agreement on specific checklist items in each case between the SE training and the SP recording, videotapes of the cases were reviewed; in such cases an SE's performance was the final score agreed upon after review. Results. Seven SEs took the SP examination and were not detected by the SPs. There was a total of 84 discrepancies between predicted and recorded checklist scores across 659 checklist items in 40 encounters scored by the SPs. After correcting the discrepancies based on video-tape review, the estimated actual mean score was 77.3% for the strong SEs and 44.0% for the weak SEs, and was higher for the strong SEs in each study case. The overall fidelity of the SEs to their training was estimated to be 97%, and the overall SP accuracy was estimated to be 91%. The videotape review revealed 47 training-scoring discrepancies, most in the area of communication skills. Conclusion. This study suggests that SEs can be trained to specific performance levels and may be an effective internal control for a high-stakes SP examination. They may also provide a mechanism for refining scoring checklists and for exploring the validity of SP examinations.

UR - http://www.scopus.com/inward/record.url?scp=0030829974&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030829974&partnerID=8YFLogxK

M3 - Article

VL - 72

SP - 1008

EP - 1011

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 11

ER -