The bronchiolitis severity score: An assessment of face validity, construct validity, and interobserver reliability

Objective: To assess face validity, interobserver reliability, and the ability to discriminate escalations of care within 24-h of admission (late rescues) for the bronchiolitis severity score (BSS) for children hospitalized for acute bronchiolitis. Hypotheses: The BSS will yield variable face validity, have clinically relevant interobserver reliability (kappa > 0.7), and distinguish late rescues during hospitalization. Methods: We performed a combined retrospective and prospective, mixed methods study where (1) interobserver agreement was prospectively assessed by overall and subcategory congruence (kappa) calculations, (2) face value were qualitatively assessed from aggregate questionnaire responses, and (3) construct validity for late rescues were assessed using receiver operator characteristic (ROC) curve analyses. Results: Face validity, assessed from 39 questionnaire respondents, were generally positive for BSS utility, reliability, and usability. The BSS exhibited weak interobserver reliability (kappa = 0.22, 95% confidence interval [CI]: 0.11–0.31) calculated from 72 sequential, blinded calculations. Retrospectively, 181 children less than 2 years of age admitted to the general pediatric ward for acute bronchiolitis from November 2017 to April 2019 were identified of which 18 (9.9%) experienced late rescues. Admission BSS values were no different for children with and without late rescues (6[3,6] vs. 4[3,6]; p =.09). An ROC curve analysis revealed an area under the curve of 0.61 (95% CI: 0.48–0.75; threshold ≥6 with sensitivity = 56%, specificity = 69%) for BSS to discriminate late rescues. Conclusion: Although clinicians expressed favorable perceptions of BSS face and content validity, we noted weak interobserver reliability and limited construct validity. Further development and validation are needed to strengthen the BSS before routine use.