TY - JOUR
T1 - Evaluation of a measurement system to assess ICU team performance
AU - Dietz, Aaron S.
AU - Salas, Eduardo
AU - Pronovost, Peter J.
AU - Jentsch, Florian
AU - Wyskiel, Rhonda
AU - Mendez-Tellez, Pedro Alejandro
AU - Dwyer, Cynthia
AU - Rosen, Michael A.
N1 - Funding Information:
1The Armstrong Institute for Patient Safety and Quality, The Johns Hopkins University School of Medicine, Baltimore, MD. 2Department of Anesthesiology and Critical Care Medicine, The Johns Hopkins University School of Medicine, Baltimore, MD. 3Department of Psychology, Rice University, Houston, TX. 4Department of Psychology, and Institute for Simulation & Training, University of Central Florida, Orlando, FL. 5Surgical Intensive Care Unit, Johns Hopkins Hospital, Baltimore, MD. This work was performed at the Johns Hopkins University. Portions of the data collection and analyses that are reported were a part of Dr. Dietz’s dissertation work. The views presented in this article are those of the authors and do not necessarily reflective of the Johns Hopkins University, Johns Hopkins Hospital, Rice University, the University of Central Florida, or the Gordon and Betty Moore Foundation. Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website (http://journals.lww.com/ ccmjournal). Supported, in part, by grants from the Gordon and Betty Moore Founda- Teamwork is a salient topic in clinical research and prac-tion (grant number: 3186.01). tice because healthcare is increasingly specialized and Dr. Dietz and Ms. Dwyer’s institutions received funding from the Gordon many hands care for one patient. Breakdowns in team-cle research from the National Institutes of Health. Ms. Dwyer received and Betty Moore Foundation. Dr. Mendez-Tellez received support for arti- work have linked to patient harm (1, 2), whereas interventions support for article research from the Gordon and Betty Moore Foundation. to promote teamwork have improved safety and efficiency Dr. Rosen’s institution received funding from the Gordon and Betty Moore (1–4). In the ICU, care teams have significantly impacted the and Quality, Centers for Disease Control and Prevention, and Jhpiego - Foundation (grant number: 3186.01), Agency for Healthcare Research outcomes and experiences of patients (5), and poor teamwork Global Health Services, Treatment & Prevention; and he disclosed that he has contributed to preventable harm and staff burnout (6, 7). is a co-investigator on a project funded through the National Aeronautics Given that teamwork competencies can be effectively trained do not have any potential conflicts of interest.and Space Administration. The remaining authors have disclosed that they (4, 8) resulting in reductions in mortality (9), healthcare For information regarding this article, E-mail: mrosen44@jhmi.edu should prioritize ensuring clinicians have and practice team-Copyright © 2018 by the Society of Critical Care Medicine and Wolters work skills. This requires training and competency assessment. Kluwer Health, Inc. All Rights Reserved. However, valid measures for evaluating team skills DOI: 10.1097/CCM.0000000000003431 are needed to 1) advance teamwork science and our
Publisher Copyright:
© 2018 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc.
Copyright:
Copyright 2019 Elsevier B.V., All rights reserved.
PY - 2018
Y1 - 2018
N2 - Objective: Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs. Design: Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects. Setting: A 15-bed surgical ICU at a large academic hospital. Subjects: One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises. Interventions: No intervention was conducted for this study. Measurements and Main Results: Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64-0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity. Conclusions: Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.
AB - Objective: Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs. Design: Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects. Setting: A 15-bed surgical ICU at a large academic hospital. Subjects: One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises. Interventions: No intervention was conducted for this study. Measurements and Main Results: Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64-0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity. Conclusions: Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.
KW - Group processes
KW - Intensive care unit
KW - Interdisciplinary communication
KW - Patient safety
KW - Quality improvement
KW - Teamwork
UR - http://www.scopus.com/inward/record.url?scp=85056606261&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85056606261&partnerID=8YFLogxK
U2 - 10.1097/CCM.0000000000003431
DO - 10.1097/CCM.0000000000003431
M3 - Article
C2 - 30247242
AN - SCOPUS:85056606261
SN - 0090-3493
VL - 46
SP - 1898
EP - 1905
JO - Critical care medicine
JF - Critical care medicine
IS - 12
ER -