TY - JOUR
T1 - Comparing the accuracy of syndrome surveillance systems in detecting influenza-like illness
T2 - GUARDIAN vs. RODS vs. electronic medical record reports
AU - Silva, Julio C.
AU - Shah, Shital C.
AU - Rumoro, Dino P.
AU - Bayram, Jamil D.
AU - Hallock, Marilyn M.
AU - Gibbs, Gillian S.
AU - Waddell, Michael J.
N1 - Funding Information:
The authors conducted the research through funding from the Telemedicine & Advanced Technology Research Center, US Army Medical Research and Materiel Command, US Department of Defense. The funder had no role in the study design, collection, analysis or interpretation of the data, nor in the writing of the report or decision to submit the paper for publication.
Copyright:
Copyright 2013 Elsevier B.V., All rights reserved.
PY - 2013/11
Y1 - 2013/11
N2 - Background: A highly sensitive real-time syndrome surveillance system is critical to detect, monitor, and control infectious disease outbreaks, such as influenza. Direct comparisons of diagnostic accuracy of various surveillance systems are scarce. Objective: To statistically compare sensitivity and specificity of multiple proprietary and open source syndrome surveillance systems to detect influenza-like illness (ILI). Methods: A retrospective, cross-sectional study was conducted utilizing data from 1122 patients seen during November 1-7, 2009 in the emergency department of a single urban academic medical center. The study compared the Geographic Utilization of Artificial Intelligence in Real-time for Disease Identification and Alert Notification (GUARDIAN) system to the Complaint Coder (CoCo) of the Real-time Outbreak Detection System (RODS), the Symptom Coder (SyCo) of RODS, and to a standardized report generated via a proprietary electronic medical record (EMR) system. Sensitivity, specificity, and accuracy of each classifier's ability to identify ILI cases were calculated and compared to a manual review by a board-certified emergency physician. Chi-square and McNemar's tests were used to evaluate the statistical difference between the various surveillance systems. Results: The performance of GUARDIAN in detecting ILI in terms of sensitivity, specificity, and accuracy, as compared to a physician chart review, was 95.5%, 97.6%, and 97.1%, respectively. The EMR-generated reports were the next best system at identifying disease activity with a sensitivity, specificity, and accuracy of 36.7%, 99.3%, and 83.2%, respectively. RODS (CoCo and SyCo) had similar sensitivity (35.3%) but slightly different specificity (CoCo=98.9%; SyCo=99.3%). The GUARDIAN surveillance system with its multiple data sources performed significantly better compared to CoCo (χ2=130.6, p<0.05), SyCo (χ2=125.2, p<0.05), and EMR-based reports (χ2=121.3, p<0.05). In addition, similar significant improvements in the accuracy (>12%) and sensitivity (>47%) were observed for GUARDIAN with only chief complaint data as compared to RODS (CoCo and SyCo) and EMR-based reports. Conclusion: In our study population, the GUARDIAN surveillance system, with its ability to utilize multiple data sources from patient encounters and real-time automaticity, demonstrated a more robust performance when compared to standard EMR-based reports and the RODS systems in detecting ILI. More large-scale studies are needed to validate the study findings, and to compare the performance of GUARDIAN in detecting other infectious diseases.
AB - Background: A highly sensitive real-time syndrome surveillance system is critical to detect, monitor, and control infectious disease outbreaks, such as influenza. Direct comparisons of diagnostic accuracy of various surveillance systems are scarce. Objective: To statistically compare sensitivity and specificity of multiple proprietary and open source syndrome surveillance systems to detect influenza-like illness (ILI). Methods: A retrospective, cross-sectional study was conducted utilizing data from 1122 patients seen during November 1-7, 2009 in the emergency department of a single urban academic medical center. The study compared the Geographic Utilization of Artificial Intelligence in Real-time for Disease Identification and Alert Notification (GUARDIAN) system to the Complaint Coder (CoCo) of the Real-time Outbreak Detection System (RODS), the Symptom Coder (SyCo) of RODS, and to a standardized report generated via a proprietary electronic medical record (EMR) system. Sensitivity, specificity, and accuracy of each classifier's ability to identify ILI cases were calculated and compared to a manual review by a board-certified emergency physician. Chi-square and McNemar's tests were used to evaluate the statistical difference between the various surveillance systems. Results: The performance of GUARDIAN in detecting ILI in terms of sensitivity, specificity, and accuracy, as compared to a physician chart review, was 95.5%, 97.6%, and 97.1%, respectively. The EMR-generated reports were the next best system at identifying disease activity with a sensitivity, specificity, and accuracy of 36.7%, 99.3%, and 83.2%, respectively. RODS (CoCo and SyCo) had similar sensitivity (35.3%) but slightly different specificity (CoCo=98.9%; SyCo=99.3%). The GUARDIAN surveillance system with its multiple data sources performed significantly better compared to CoCo (χ2=130.6, p<0.05), SyCo (χ2=125.2, p<0.05), and EMR-based reports (χ2=121.3, p<0.05). In addition, similar significant improvements in the accuracy (>12%) and sensitivity (>47%) were observed for GUARDIAN with only chief complaint data as compared to RODS (CoCo and SyCo) and EMR-based reports. Conclusion: In our study population, the GUARDIAN surveillance system, with its ability to utilize multiple data sources from patient encounters and real-time automaticity, demonstrated a more robust performance when compared to standard EMR-based reports and the RODS systems in detecting ILI. More large-scale studies are needed to validate the study findings, and to compare the performance of GUARDIAN in detecting other infectious diseases.
KW - Biosurveillance
KW - GUARDIAN
KW - Influenza-like illness
KW - Public health informatics
KW - RODS
KW - Syndromic surveillance systems
UR - http://www.scopus.com/inward/record.url?scp=84887616028&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84887616028&partnerID=8YFLogxK
U2 - 10.1016/j.artmed.2013.09.001
DO - 10.1016/j.artmed.2013.09.001
M3 - Article
C2 - 24369035
AN - SCOPUS:84887616028
VL - 59
SP - 169
EP - 174
JO - Artificial Intelligence in Medicine
JF - Artificial Intelligence in Medicine
SN - 0933-3657
IS - 3
ER -