Do end-of-rotation and end-of-shift assessments inform clinical competency committees’ (CCC) decisions?

Research output: Contribution to journalArticle

Abstract

Introduction: Clinical Competency Committees (CCC) require reliable, objective data to inform decisions regarding assignment of milestone proficiency levels, which must be reported to the Accreditation Council for Graduate Medical Education. After the development of two new assessment methods, the end-of-shift (EOS) assessment and the end-of-rotation (EOR) assessment, we sought to evaluate their performance. We report data on the concordance between these assessments, as well as how each informs the final proficiency level determined in biannual CCC meetings. We hypothesized that there would be a high concordance level between the two assessment methods, including concordance of both the EOS and EOR with the final proficiency level designation by the CCC. Methods: The residency program is an urban academic four-year emergency medicine residency with 48 residents. After their shifts in the emergency department (ED), residents handed out EOS assessment forms asking about individual milestones from 15 subcompetencies to supervising physicians, as well as triggered electronic EOR-doctor (EORd) assessments to supervising doctors and EOR-nurse (EORn) to nurses they had worked with after each two-week ED block. EORd assessments contained the full proficiency level scale from 16 subcompetencies, while EORn assessments contained four subcompetencies. Data reports were generated after each six-month assessment period and data was aggregated. We calculated Spearman’s rank order correlations for correlations between assessment types and between assessments and final CCC proficiency levels. Results: Over 24 months, 5,234 assessments were completed. The strongest correlations with CCC proficiency levels were the EORd for the immediate six-month assessment period prior (rs 0.71-0.84), and the CCC proficiency levels from the previous six-months (rs 0.83-0.92). EOS assessments had weaker correlations (rs 0.49 to 0.62), as did EORn (rs 0.4 to 0.73). Conclusion: End-of-rotation assessments completed by supervising doctors are most highly correlated with final CCC proficiency level designations, while end-of-shift assessments and end-of-rotation assessments by nurses did not correlate strongly with final CCC proficiency levels, both with overestimation of levels noted. Every level of proficiency the CCC assigned appears to be highly correlated with the designated level in the immediate six-month period, perhaps implying CCC members are biased by previous level assignments.

Original languageEnglish (US)
Pages (from-to)121-127
Number of pages7
JournalWestern Journal of Emergency Medicine
Volume19
Issue number1
DOIs
StatePublished - Jan 1 2018

Fingerprint

Clinical Competence
Nurses
Internship and Residency
Hospital Emergency Service
Committee Membership
Graduate Medical Education
Emergency Medicine
Accreditation
Physicians

ASJC Scopus subject areas

  • Emergency Medicine

Cite this

@article{11f96daf138349eb858463317a9f4395,
title = "Do end-of-rotation and end-of-shift assessments inform clinical competency committees’ (CCC) decisions?",
abstract = "Introduction: Clinical Competency Committees (CCC) require reliable, objective data to inform decisions regarding assignment of milestone proficiency levels, which must be reported to the Accreditation Council for Graduate Medical Education. After the development of two new assessment methods, the end-of-shift (EOS) assessment and the end-of-rotation (EOR) assessment, we sought to evaluate their performance. We report data on the concordance between these assessments, as well as how each informs the final proficiency level determined in biannual CCC meetings. We hypothesized that there would be a high concordance level between the two assessment methods, including concordance of both the EOS and EOR with the final proficiency level designation by the CCC. Methods: The residency program is an urban academic four-year emergency medicine residency with 48 residents. After their shifts in the emergency department (ED), residents handed out EOS assessment forms asking about individual milestones from 15 subcompetencies to supervising physicians, as well as triggered electronic EOR-doctor (EORd) assessments to supervising doctors and EOR-nurse (EORn) to nurses they had worked with after each two-week ED block. EORd assessments contained the full proficiency level scale from 16 subcompetencies, while EORn assessments contained four subcompetencies. Data reports were generated after each six-month assessment period and data was aggregated. We calculated Spearman’s rank order correlations for correlations between assessment types and between assessments and final CCC proficiency levels. Results: Over 24 months, 5,234 assessments were completed. The strongest correlations with CCC proficiency levels were the EORd for the immediate six-month assessment period prior (rs 0.71-0.84), and the CCC proficiency levels from the previous six-months (rs 0.83-0.92). EOS assessments had weaker correlations (rs 0.49 to 0.62), as did EORn (rs 0.4 to 0.73). Conclusion: End-of-rotation assessments completed by supervising doctors are most highly correlated with final CCC proficiency level designations, while end-of-shift assessments and end-of-rotation assessments by nurses did not correlate strongly with final CCC proficiency levels, both with overestimation of levels noted. Every level of proficiency the CCC assigned appears to be highly correlated with the designated level in the immediate six-month period, perhaps implying CCC members are biased by previous level assignments.",
author = "Linda Regan and Leslie Cope and Rodney Omron and Bright, {Leah Elaine} and Bayram, {Jamil Daoud}",
year = "2018",
month = "1",
day = "1",
doi = "10.5811/westjem.2017.10.35290",
language = "English (US)",
volume = "19",
pages = "121--127",
journal = "Western Journal of Emergency Medicine",
issn = "1936-900X",
publisher = "University of California",
number = "1",

}

TY - JOUR

T1 - Do end-of-rotation and end-of-shift assessments inform clinical competency committees’ (CCC) decisions?

AU - Regan, Linda

AU - Cope, Leslie

AU - Omron, Rodney

AU - Bright, Leah Elaine

AU - Bayram, Jamil Daoud

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Introduction: Clinical Competency Committees (CCC) require reliable, objective data to inform decisions regarding assignment of milestone proficiency levels, which must be reported to the Accreditation Council for Graduate Medical Education. After the development of two new assessment methods, the end-of-shift (EOS) assessment and the end-of-rotation (EOR) assessment, we sought to evaluate their performance. We report data on the concordance between these assessments, as well as how each informs the final proficiency level determined in biannual CCC meetings. We hypothesized that there would be a high concordance level between the two assessment methods, including concordance of both the EOS and EOR with the final proficiency level designation by the CCC. Methods: The residency program is an urban academic four-year emergency medicine residency with 48 residents. After their shifts in the emergency department (ED), residents handed out EOS assessment forms asking about individual milestones from 15 subcompetencies to supervising physicians, as well as triggered electronic EOR-doctor (EORd) assessments to supervising doctors and EOR-nurse (EORn) to nurses they had worked with after each two-week ED block. EORd assessments contained the full proficiency level scale from 16 subcompetencies, while EORn assessments contained four subcompetencies. Data reports were generated after each six-month assessment period and data was aggregated. We calculated Spearman’s rank order correlations for correlations between assessment types and between assessments and final CCC proficiency levels. Results: Over 24 months, 5,234 assessments were completed. The strongest correlations with CCC proficiency levels were the EORd for the immediate six-month assessment period prior (rs 0.71-0.84), and the CCC proficiency levels from the previous six-months (rs 0.83-0.92). EOS assessments had weaker correlations (rs 0.49 to 0.62), as did EORn (rs 0.4 to 0.73). Conclusion: End-of-rotation assessments completed by supervising doctors are most highly correlated with final CCC proficiency level designations, while end-of-shift assessments and end-of-rotation assessments by nurses did not correlate strongly with final CCC proficiency levels, both with overestimation of levels noted. Every level of proficiency the CCC assigned appears to be highly correlated with the designated level in the immediate six-month period, perhaps implying CCC members are biased by previous level assignments.

AB - Introduction: Clinical Competency Committees (CCC) require reliable, objective data to inform decisions regarding assignment of milestone proficiency levels, which must be reported to the Accreditation Council for Graduate Medical Education. After the development of two new assessment methods, the end-of-shift (EOS) assessment and the end-of-rotation (EOR) assessment, we sought to evaluate their performance. We report data on the concordance between these assessments, as well as how each informs the final proficiency level determined in biannual CCC meetings. We hypothesized that there would be a high concordance level between the two assessment methods, including concordance of both the EOS and EOR with the final proficiency level designation by the CCC. Methods: The residency program is an urban academic four-year emergency medicine residency with 48 residents. After their shifts in the emergency department (ED), residents handed out EOS assessment forms asking about individual milestones from 15 subcompetencies to supervising physicians, as well as triggered electronic EOR-doctor (EORd) assessments to supervising doctors and EOR-nurse (EORn) to nurses they had worked with after each two-week ED block. EORd assessments contained the full proficiency level scale from 16 subcompetencies, while EORn assessments contained four subcompetencies. Data reports were generated after each six-month assessment period and data was aggregated. We calculated Spearman’s rank order correlations for correlations between assessment types and between assessments and final CCC proficiency levels. Results: Over 24 months, 5,234 assessments were completed. The strongest correlations with CCC proficiency levels were the EORd for the immediate six-month assessment period prior (rs 0.71-0.84), and the CCC proficiency levels from the previous six-months (rs 0.83-0.92). EOS assessments had weaker correlations (rs 0.49 to 0.62), as did EORn (rs 0.4 to 0.73). Conclusion: End-of-rotation assessments completed by supervising doctors are most highly correlated with final CCC proficiency level designations, while end-of-shift assessments and end-of-rotation assessments by nurses did not correlate strongly with final CCC proficiency levels, both with overestimation of levels noted. Every level of proficiency the CCC assigned appears to be highly correlated with the designated level in the immediate six-month period, perhaps implying CCC members are biased by previous level assignments.

UR - http://www.scopus.com/inward/record.url?scp=85041052247&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041052247&partnerID=8YFLogxK

U2 - 10.5811/westjem.2017.10.35290

DO - 10.5811/westjem.2017.10.35290

M3 - Article

C2 - 29383066

AN - SCOPUS:85041052247

VL - 19

SP - 121

EP - 127

JO - Western Journal of Emergency Medicine

JF - Western Journal of Emergency Medicine

SN - 1936-900X

IS - 1

ER -