Performance Degradation between Development and Deployment of a Predictive Model for Central Line-Associated Bloodstream Infections in Hospitalized Children

Abstract

Background

Central line-associated bloodstream infections (CLABSIs) are associated with substantial pediatric morbidity and mortality. The capacity to predict which children with central lines are at greatest risk of CLABSI could inform surveillance and prevention efforts. Our team previously published in silico predictive models for CLABSI.

Objective

To prospectively implement a pediatric CLABSI predictive model and achieve adequate performance in offline validation for implementation in clinical practice.

Methods

Most performant predictive models were deep learning models requiring substantial pre-processing of many features into 8-hour windows including the current day and up to 56 days prior for the current admission. To replicate this pre-processing, we created a novel infrastructure to (1) organize current-day data for all the relevant features and (2) create a staged historical data store for those same features with application programming interfaces to connect the two. We compared predictive performance of these scores for CLABSI in the next 48 hours with two labels, one based on manual review of positive blood cultures in children with central lines and another based on positive blood culture and receipt of at least 4 days of new IV antibiotics.

Results

The area under the receiver-operating characteristic (AUROC) fell from 0.97 from retrospective data to <0.60 despite multiple iterations of troubleshooting. Primary root causes included train/serve skew, feature leakage, and overfitting. Hypothesized secondary drivers were complex model specification, poor data governance, inadequate testing, challenging feature translation between real-time and historical data models, limited monitoring and logging infrastructure for troubleshooting, and suboptimal handoff between the model development and deployment teams.

Conclusion

Bridging the gap from predictive model development to clinical deployment requires early and close coordination between data governance, data science, clinical informatics, and implementation engineers. Balancing predictive performance with implementation feasibility can accelerate the adoption of predictive clinical decision support systems.

Keywords

artificial intelligence - electronic health records and systems - clinical decision support - implementation and deployment - deep learning

Protection of Human and Animal Subjects

CLABSI model development was approved by the Children's Healthcare of Atlanta IRB (approval no.: STUDY 19–012). Implementation and evaluation were deemed by the Children's Healthcare of Atlanta IRB to constitute non-human subjects research as a quality improvement activity (approval no.: STUDY00001526).

Publication History

Received: 12 January 2025

Accepted: 09 May 2025

Accepted Manuscript online:
12 May 2025

Article published online:
26 September 2025

• References

• 1 CDC. Current HAI Progress Report. Healthcare-Associated Infections (HAIs). November 25, 2024. Accessed December 29, 2024 at: https://www.cdc.gov/healthcare-associated-infections/php/data/progress-report.html
• 2 Guidelines for the Prevention of Intravascular Catheter-Related Infections. Accessed December 29, 2024 at: https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5110a1.htm
• 3 Karagiannidou S, Kourlaba G, Zaoutis T, Maniadakis N, Papaevangelou V. Attributable mortality for pediatric and neonatal central line-associated bloodstream infections in Greece. J Pediatr Intensive Care 2021; 13 (02) 174-183
• 4 Goudie A, Dynan L, Brady PW, Rettiganti M. Attributable cost and length of stay for central line-associated bloodstream infections. Pediatrics 2014; 133 (06) e1525-e1532
• 5 Bundy DG, Gaur AH, Billett AL, He B, Colantuoni EA, Miller MR. Children's Hospital Association Hematology/Oncology CLABSI Collaborative. Preventing CLABSIs among pediatric hematology/oncology inpatients: national collaborative results. Pediatrics 2014; 134 (06) e1678-e1685
• 6 Willis DN, Looper K, Malone RA. et al. Eliminating central line associated bloodstream infections in pediatric oncology patients: a quality improvement effort. Pediatr Qual Saf 2023; 8 (03) e660
• 7 Carter JH, Langley JM, Kuhle S, Kirkland S. Risk factors for central venous catheter-associated bloodstream infection in pediatric patients: a cohort study. Infect Control Hosp Epidemiol 2016; 37 (08) 939-945
• 8 Paioni P, Kuhn S, Strässle Y, Seifert B, Berger C. Risk factors for central line-associated bloodstream infections in children with tunneled central venous catheters. Am J Infect Control 2020; 48 (01) 33-39
• 9 Tabaie A, Orenstein EW, Nemati S. et al. Predicting presumed serious infection among hospitalized children on central venous lines with machine learning. Comput Biol Med 2021; 132: 104289
• 10 Tabaie A, Orenstein EW, Nemati S, Basu RK, Clifford GD, Kamaleswaran R. Deep learning model to predict serious infection among children with central venous lines. Front Pediatr 2021; 9: 726870
• 11 Tabaie A, Orenstein EW, Kandaswamy S, Kamaleswaran R. Integrating structured and unstructured data for timely prediction of bloodstream infection among children. Pediatr Res 2023; 93 (04) 969-975
• 12 Bloodstream Infection (BSI) Events. | PSC | NHSN | CDC. December 9, 2024. Accessed December 29, 2024 at: https://www.cdc.gov/nhsn/psc/bsi/index.html
• 13 Harris PA, Taylor R, Minor BL. et al; REDCap Consortium. The REDCap consortium: building an international community of software platform partners. J Biomed Inform 2019; 95: 103208
• 14 Tennekes M, de Jonge E, Daas PJH. Visualizing and inspecting large datasets with tableplots. J Data Sci 2013; 11: 43-58
• 15 Corbin CK, Maclay R, Acharya A. et al. DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record. J Am Med Inform Assoc 2023; 30 (09) 1532-1542