Methods Inf Med
DOI: 10.1055/s-0042-1742672
Original Article

A Comparison of Methods to Detect Changes in Prediction Models

Erin M. Schnellinger
1   Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Wei Yang
1   Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Michael O. Harhay
1   Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Stephen E. Kimmel
2   Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, Florida, United States
› Author Affiliations
Funding S.K. was supported by US National Institutes of Health, grant number NIH R01HL14129. E.S. was supported by US National Institutes of Health (NIH) National Heart, Lung, and Blood Institute (NHLBI), grant number F31HL19433. M.H. was supported by US National Institutes of Health (grant number: R00HL141678) and Patient-Centered Outcomes Research Institute (grant number: ME-2020C1-19220). W.Y. was supported by US National Institutes of Health (NIH) National Heart, Lung, and Blood Institute (NHLBI), grant number: R01-HL141294.

Abstract

Background Prediction models inform decisions in many areas of medicine. Most models are fitted once and then applied to new (future) patients, despite the fact that model coefficients can vary over time due to changes in patients' clinical characteristics and disease risk. However, the optimal method to detect changes in model parameters has not been rigorously assessed.

Methods We simulated data, informed by post–lung transplant mortality data and tested the following two approaches for detecting model change: (1) the “Direct Approach,” it compares coefficients of the model refit on recent data to those at baseline; and (2) “Calibration Regression,” it fits a logistic regression model of the log-odds of the observed outcomes versus the linear predictor from the baseline model (i.e., the log-odds of the predicted probabilities obtained from the baseline model) and tests whether the intercept and slope differ from 0 and 1, respectively. Four scenarios were simulated using logistic regression for binary outcomes as follows: (1) we fixed all model parameters, (2) we varied the outcome prevalence between 0.1 and 0.2, (3) we varied the coefficient of one of the ten predictors between 0.2 and 0.4, and (4) we varied the outcome prevalence and coefficient of one predictor simultaneously.

Results Calibration regression tended to detect changes sooner than the Direct Approach, with better performance (e.g., larger proportion of true claims). When the sample size was large, both methods performed well. When two parameters changed simultaneously, neither method performed well.

Conclusion Neither change detection method examined here proved optimal under all circumstances. However, our results suggest that if one is interested in detecting a change in overall incidence of an outcome (e.g., intercept), the Calibration Regression method may be superior to the Direct Approach. Conversely, if one is interested in detecting a change in other model covariates (e.g., slope), the Direct Approach may be superior.

Data and Code

This simulation study was informed by data from the United Network for Organ Sharing (UNOS). The authors do not have the authority to share UNOS data; researchers interested in accessing this data must submit a request to UNOS directly. The code used to conduct the simulations is available from the authors on request.


Authors' Contributions

All authors contributed to the study's conception and design. E.M.S. performed the statistical analysis, drafted the initial manuscript, and revised it based on the critical review and scientific input of W.Y., M.O.H., and S.E.K. All authors contributed to the design of the simulations, programmed by E.M.S. under the guidance of W.Y., M.O.H., and S.E.K.


Supplementary Material



Publication History

Received: 20 October 2021

Accepted: 31 December 2021

Article published online:
12 February 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany