Inter-Rater Agreement of Intensivists Evaluating the Goal Concordance of Preference-Sensitive ICU Interventions

Context: Goal-concordant care has been identified as an important outcome of advance care planning and shared decision-making initiatives. However, validated methods for measuring goal concordance are needed. Objectives: To estimate the inter-rater reliability of senior critical care fellows rating the goal concordance of preference-sensitive interventions performed in intensive care units (ICUs) while considering patient-specific circumstances as described in a previously proposed methodology. Methods: We identified ICU patients receiving preference-sensitive interventions in three adult ICUs at Johns Hopkins Hospital. A simulated cohort was created by randomly assigning each patient one of 10 sets of goals and preferences about limiting life support. Critical care fellows then independently reviewed patient charts and answered two questions: 1) Is this patient's goal achievable? and 2) Will performing this intervention help achieve the patient's goal? When the answer to both questions was yes, the intervention was rated as goal concordant. Inter-rater agreement was summarized by estimating intraclass correlation coefficient using mixed-effects models. Results: Six raters reviewed the charts of 201 patients. Interventions were rated as goal concordant 22%–92% of the time depending on the patient's goal-limitation combination. Percent agreement between pairs of raters ranged from 59% to 86%. The intraclass correlation coefficient for ratings of goal concordance was 0.50 (95% CI 0.31–0.69) and was robust to patient age, gender, ICU, severity of illness, and lengths of stay. Conclusion: Inter-rater agreement between intensivists using a standardized methodology to evaluate the goal concordance of preference-sensitive ICU interventions was moderate. Further testing is needed before this methodology can be recommended as a clinical research outcome.