Caitlin Burke, MD,
Acute Care Surgery Fellow
Hartford Hospital
Farmington, Connecticut, United States
Disclosure information not submitted.
Jakob Wollborn, n/a
Clinical Fellow in Cardiothoracic Anesthesiology
Brigham & Women's Hospital, United States
Disclosure information not submitted.
Saniya Sami, n/a
Critical Care Medicine Fellow
Brigham & Women's Hospital, United States
Disclosure information not submitted.
David Buric, n/a
Critical Care Medicine Fellow
Brigham & Women's Hospital, United States
Disclosure information not submitted.
Jeffrey Carness, MD
Critical Care Medicine Fellow
Brigham & Women's Hospital, United States
Disclosure information not submitted.
Marcio Aloisio Bezerra Cavalcanti Rockenbach, MD
Massachusetts General Hospital
Boston, Massachusetts
Disclosure information not submitted.
Nir Neumark, n/a
Product Manager
Mass General Brigham, United States
Disclosure information not submitted.
Varun Buch
Director of AI Product Development
Massachusetts General Hospital, United States
Disclosure information not submitted.
Quanzheng Li
Associate Professor
Harvard Medical School, United States
Disclosure information not submitted.
Romane Gauriau, n/a
Senior Machine Learning Scientist
Center for Clinical Data Science, United States
Disclosure information not submitted.
Dufan Wu
Instructor in Investigation
Massachusetts General Hospital, United States
Disclosure information not submitted.
Hariharan Ravishankar
Senior Scientist
GE Healthcare, United States
Disclosure information not submitted.
Rohit Pardasani
Senior Data Scientist
GE Healthcare, United States
Disclosure information not submitted.
Abhijit Patil
Senior Director - AI & Analytics
GE Healthcare, United States
Disclosure information not submitted.
Pankaj Sarin, n/a
Anesthesiologist
Brigham & Women's Hospital, United States
Disclosure information not submitted.
Dirk Varelmann, MD, EDIC, DESA
Anesthesiologist
Brigham & Women's Hospital, United States
Disclosure information not submitted.
Title: Evaluating Agreement Across Clinicians in Predicting Oxygen Requirements for Critically Ill Patients
Introduction/Hypothesis: Machine learning may be used to help guide oxygen device management in the ICU. To assess the accuracy of an algorithm designed to predict oxygen requirements, clinicians predicted oxygen requirements for patients at numerous timepoints based on clinical data from the electronic health records (EHR). In this work, we evaluate the recommendation's agreement among clinicians.
Methods: 7 intensivists (5 fellows, 2 attendings) evaluated data from 10 respiratory disease ICU patients, including demographics, comorbidities, vital signs, labs, and current oxygenation extracted retrospectively over the ICU stay. Each timepoint represented 1 hour and clinicians predicted the next hour's oxygenation requirement (room air, low-flow, high-flow, or mechanical ventilation) with a corresponding step-down, maintain, or step-up recommendation and respective confidence level (low, somewhat, high). The"ground truth" was based on the oxygen device used in the actual clinical setting. Timepoints involving mechanical ventilation and low confidence recommendations were removed from the analysis.
Results: Among the 556 timepoints in the final analysis, current oxygenation devices included 239 (43%) room air, 224 (40.2%) low-flow, 93 (16.8%) high-flow. Ground truth included 520 (93.6%) maintain, 19 (3.4%) step-up and 17 (3.0%) step-down decisions. All 7 clinicians agreed at 222 (39.9%) timepoints, 6 out of 7 at 159 (28.6%) timepoints, 5/7 at 98 (17.6%) timepoints, 4/7 at 74 (13.4%) timepoints, and 3/7 at 3 (0.5%) timepoints. Of 19 step-up cases (patient required oxygen device escalation in the real clinical setting), all 7 clinicians agreed at 9 (47.4%) timepoints, 6/7 at 7 (36.8%), 5/7 at 2 (10.5%), and 4/7 at 1 (5.3%) timepoints. Of 19 step-down cases, 7 clinicians agreed at 1 (5.9%) timepoint, 6/7 at 4 (23.5%), 5/7 at 5 (29.4%), and 4/7 at 7 (41.2%) timepoints.
Conclusions: The majority of clinicians (7/7 or 6/7) made the same recommendation for 68.5% of the timepoints. For actual step-up cases, most clinicians made the same recommendation for 84.2% of the timepoints. 7 intensivists comprised a homogeneous group in using EHR data for clinical predictions about oxygen requirements for critically ill patients and, in particular, regarding patients requiring oxygen escalation.