Appl Clin Inform 2022; 13(02): 431-438
DOI: 10.1055/s-0042-1746168
Research Article

Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model

Keith E. Morse
1   Division of Pediatric Hospital Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
,
Conner Brown
2   Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
,
Scott Fleming
3   Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
,
Irene Todd
2   Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
,
Austin Powell
2   Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
,
Alton Russell
4   Harvard Medical School, Boston, Massachusetts, United States
,
David Scheinker
2   Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
,
Scott M. Sutherland
5   Division of Nephrology, Department of Pediatrics, Stanford University, Stanford, California, United States
,
Jonathan Lu
3   Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
,
Brendan Watkins
2   Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
,
Nigam H. Shah
3   Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
,
Natalie M. Pageler
6   Division of Pediatric Critical Care Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
7   Division of Systems Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
,
Jonathan P. Palma
8   Division of Neonatology, Department of Pediatrics, Orlando Health, Orlando, Florida, United States
› Author Affiliations
Funding None.

Abstract

Objective The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital.

Methods The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a “membership model”; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes.

Results The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings.

Conclusion This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.

Projection of Human and Animal Subjects

This project was reviewed and approved by the Stanford University Institutional Review Board.


Supplementary Material



Publication History

Received: 13 September 2021

Accepted: 01 March 2022

Publication Date:
04 May 2022 (online)

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany