Are Scores from NBME Subject Examinations Valid Measures of Knowledge Acquired during Clinical Clerkships?

Purpose The National Board of Medical Examiners' Clinical Science Subject Examinations are a component used by most U.S. medical schools to determine clerkship grades. The purpose of this study was to examine the validity of this practice. Method This was a retrospective cohort study of medical students at the Virginia Commonwealth University School of Medicine who completed clerkships in 2012 through 2014. Linear regression was used to determine how well United States Medical Licensing Examination Step 1 scores predicted Subject Examination scores in seven clerkships. The authors then substituted each student's Subject Examination standard scores with his or her Step 1 standard score. Clerkship grades based on the Step 1 substitution were compared with actual grades with the Wilcoxon rank test. Results A total of 2,777 Subject Examination scores from 432 students were included in the analysis. Step 1 scores significantly predicted between 23% and 44% of the variance in Subject Examination scores, P <.001 for all clerkship regression equations. Mean differences between expected and actual Subject Examination scores were small (≤ 0.2 points). There was a match between 73% of Step 1 substituted final clerkship grades and actual final clerkship grades. Conclusions The results of this study suggest that performance on Step 1 can be used to identify and counsel students at risk for poor performance on the Subject Examinations. In addition, these findings call into the question the validity of using scores from Subject Examinations as a high-stakes assessment of learning in individual clerkships.