Suchi Saria, PhD
Associate Professor
Johns Hopkins University, United States
Disclosure information not submitted.
Katharine Henry, PhD
Postdoctoral Fellow
Johns Hopkins University, United States
Disclosure information not submitted.
Hossein Soleimani, PhD
Data Scientist
University of California, San Francisco, United States
Disclosure information not submitted.
Andong Zhan, PhD
Research Assistant
Johns Hopkins University, United States
Disclosure information not submitted.
Nishi Rawat, MD, MBA
Assistant Professor
Johns Hopkins University, United States
Disclosure information not submitted.
Edward Chen, MD
Assistant Professor
Johns Hopkins University School of Medicine, United States
Disclosure information not submitted.
Albert Wu, MD
Professor
Johns Hopkins University, United States
Disclosure information not submitted.
Title: Lead Time and Accuracy of TREWS, a Machine Learning-based Sepsis Alert
INTRODUCTION/HYPOTHESIS: Not all sepsis patients have equal opportunity to benefit from an early warning system. We evaluated accuracy and lead time across different subgroups for a sepsis early warning system, Targeted Real-time Early Warning Score (TREWS).
Methods: On a retrospective cohort of emergency, medical, and surgical patients admitted between January 1, 2016 and March 31, 2018 to any of two academic and one community hospital, we evaluated TREWS based on alert accuracy (per encounter AUC, PPV, and sensitivity) and lead time relative to standard of care among sepsis cases with an alert, no sepsis treatment within 3 hours of arrival (defined as the earlier of ED triage or admission to a care unit), and limited opportunity for earlier intervention. Lead time was measured as median time between the alert and first antibiotic order. We report lead-time among all patients, patients identified by a second alert-time risk score as having high risk of mortality without timely treatment, and patients who died in-hospital. Results were computed at a pre-specified configuration which delayed alerts until significant symptoms were apparent. During the study, providers used a rule-based alert based on the SEP-1 criteria which may have influenced antibiotics timing in the data.
Results: The cohort included 173,931 patient encounters (3,858 sepsis cases); 1,552 patients who died in-hospital (558 sepsis cases). The model identified sepsis with an AUC of 0.97. At a sensitivity of 0.81, alerts occurred on 7% of patient encounters and had a PPV of 0.27. Among patients who died in-hospital, the alert had higher sensitivity (0.93) and higher PPV (0.53).
Among sepsis patients, 1,622 (42%) did not receive antibiotics within 3 hours of arrival, of which 1,182 had an alert, 415 had a high-risk alert, and 205 had an alert and died in-hospital. Median lead-time was 3.6 hours in sepsis patients, 4.2 hours in sepsis patients with a high-risk alert, and 5.7 hours in sepsis patients who died in-hospital.
Conclusions: On the full cohort, TREWS produced highly sensitive and precise alerts. On sepsis patients who were not treated immediately upon arrival, TREWS alerted with significant lead time, especially on patients flagged as high-risk or who died, indicating high potential clinical utility of the system on such patients.