Moving the United States Medical Licensing Examination Step 1 after Core Clerkships: An Outcomes Analysis

Daniel Jurich, Michelle Daniel, Miguel Paniagua, Amy Fleming, Victoria Harnik, Arnyce Pock, Aubrie Swan-Sein, Michael A. Barone, Sally A. Santen

Research output: Contribution to journalArticle

Abstract

Purpose Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships. Method Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period. Results The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95% confidence interval: 1.50-3.83, P <.001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87% (n = 48) pre change to 0.39% (n = 6) post change (P <.001). Conclusions Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of "noninferiority" may allow schools to implement significant curricular reforms.

Original languageEnglish (US)
Pages (from-to)371-377
Number of pages7
JournalAcademic Medicine
Volume94
Issue number3
DOIs
StatePublished - Mar 1 2019
Externally publishedYes

Fingerprint

examination
school
reform
psychometrics
confidence
curriculum
ability
science

ASJC Scopus subject areas

  • Education

Cite this

Jurich, D., Daniel, M., Paniagua, M., Fleming, A., Harnik, V., Pock, A., ... Santen, S. A. (2019). Moving the United States Medical Licensing Examination Step 1 after Core Clerkships: An Outcomes Analysis. Academic Medicine, 94(3), 371-377. https://doi.org/10.1097/ACM.0000000000002458

Moving the United States Medical Licensing Examination Step 1 after Core Clerkships : An Outcomes Analysis. / Jurich, Daniel; Daniel, Michelle; Paniagua, Miguel; Fleming, Amy; Harnik, Victoria; Pock, Arnyce; Swan-Sein, Aubrie; Barone, Michael A.; Santen, Sally A.

In: Academic Medicine, Vol. 94, No. 3, 01.03.2019, p. 371-377.

Research output: Contribution to journalArticle

Jurich, D, Daniel, M, Paniagua, M, Fleming, A, Harnik, V, Pock, A, Swan-Sein, A, Barone, MA & Santen, SA 2019, 'Moving the United States Medical Licensing Examination Step 1 after Core Clerkships: An Outcomes Analysis', Academic Medicine, vol. 94, no. 3, pp. 371-377. https://doi.org/10.1097/ACM.0000000000002458
Jurich, Daniel ; Daniel, Michelle ; Paniagua, Miguel ; Fleming, Amy ; Harnik, Victoria ; Pock, Arnyce ; Swan-Sein, Aubrie ; Barone, Michael A. ; Santen, Sally A. / Moving the United States Medical Licensing Examination Step 1 after Core Clerkships : An Outcomes Analysis. In: Academic Medicine. 2019 ; Vol. 94, No. 3. pp. 371-377.
@article{064295ba5cf24b2e8792d0ea7c84d475,
title = "Moving the United States Medical Licensing Examination Step 1 after Core Clerkships: An Outcomes Analysis",
abstract = "Purpose Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships. Method Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period. Results The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95{\%} confidence interval: 1.50-3.83, P <.001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87{\%} (n = 48) pre change to 0.39{\%} (n = 6) post change (P <.001). Conclusions Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of {"}noninferiority{"} may allow schools to implement significant curricular reforms.",
author = "Daniel Jurich and Michelle Daniel and Miguel Paniagua and Amy Fleming and Victoria Harnik and Arnyce Pock and Aubrie Swan-Sein and Barone, {Michael A.} and Santen, {Sally A.}",
year = "2019",
month = "3",
day = "1",
doi = "10.1097/ACM.0000000000002458",
language = "English (US)",
volume = "94",
pages = "371--377",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "3",

}

TY - JOUR

T1 - Moving the United States Medical Licensing Examination Step 1 after Core Clerkships

T2 - An Outcomes Analysis

AU - Jurich, Daniel

AU - Daniel, Michelle

AU - Paniagua, Miguel

AU - Fleming, Amy

AU - Harnik, Victoria

AU - Pock, Arnyce

AU - Swan-Sein, Aubrie

AU - Barone, Michael A.

AU - Santen, Sally A.

PY - 2019/3/1

Y1 - 2019/3/1

N2 - Purpose Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships. Method Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period. Results The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95% confidence interval: 1.50-3.83, P <.001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87% (n = 48) pre change to 0.39% (n = 6) post change (P <.001). Conclusions Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of "noninferiority" may allow schools to implement significant curricular reforms.

AB - Purpose Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships. Method Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period. Results The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95% confidence interval: 1.50-3.83, P <.001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87% (n = 48) pre change to 0.39% (n = 6) post change (P <.001). Conclusions Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of "noninferiority" may allow schools to implement significant curricular reforms.

UR - http://www.scopus.com/inward/record.url?scp=85064826632&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064826632&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000002458

DO - 10.1097/ACM.0000000000002458

M3 - Article

C2 - 30211755

AN - SCOPUS:85064826632

VL - 94

SP - 371

EP - 377

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 3

ER -