Mark Mai, MD, MHS
Fellow in Pediatric Critical Care
Childrens Hospital of Philadelphia
Philadelphia, Pennsylvania
Disclosure information not submitted.
Adam Dziorny, MD, PhD,
Assistant Professor
Golisano Children's Hospital University of Rochester
Rochester, NY
Disclosure information not submitted.
Donald Boyer, MD, MSEd
Associate Professor of Clinical Anesthesiology and Critical Care
The Children's Hospital of Philadelphia, United States
Disclosure information not submitted.
Title: A Model to Predict PICU Fellow Clinical Experiences from Electronic Health Record Data
INTRODUCTION/HYPOTHESIS: Clinical competency is associated with exposure breadth and depth; however, quantification of exposure during pediatric critical care training is lacking, making attestation of competence challenging for individual trainees and those responsible for certification and credentialing processes. Manual case-logging is time-consuming, unreliable, and poorly scaled. Existing models using electronic health record (EHR) data may approximate PICU fellow experiences, but are based on trainee providers in different care settings, with different workflows. The aim of this project was to demonstrate feasibility of predicting PICU fellow clinical experiences from EHR data.
Methods: We surveyed PICU fellows in a 65-bed PICU immediately following shifts (11/2020-3/2021) and asked them to identify patient interactions from a list of all potential PICU patients during that shift. We identified routine EHR variables based on trainee workflow, including audit logs, note contribution, order placement, and care team assignment, and extracted these for each shift. Total time-in-chart was calculated using a previously published algorithm. Using k-fold cross validation, we trained a multivariate logistic regression classifier on extracted EHR data to predict the self-reported patient interactions. We report model performance, per-variable odds ratios (OR) and 95% confidence intervals (95% CI).
Results: Self-reported patient interactions were collected from 9 PICU fellows (PGY-4: 3; PGY-5: 3; PGY-6: 3) across 13 total shifts. Fellows reported 206 patient interactions and 754 patient non-interactions. Orders placed (OR: 14.3, 95% CI: 6.4-34.8), time in chart (OR: 9.9, 95% CI: 4.1-26.9), and note contribution (OR: 1.5, 95% CI: 0.2-9.8) were the best predictors in the model. The final model achieved a c-statistic of 0.953, sensitivity of 87.6%, and a specificity of 95.7%.
Conclusions: Prediction of PICU fellow patient interactions with classification models based on EHR data is feasible. Without reliance on case logs, this automated system provides objective insight into the training experiences of PICU fellows. This type of system can inform curriculum planning, individualized learning plans, and competency based medical education assessment systems within a fellowship program.