Download PDF
CC BY-NC-ND 4.0 · Methods Inf Med 2022; 61(S 01): e35-e44
DOI: 10.1055/s-0042-1743170
Original Article

Evaluating Prediction of Continuous Clinical Values: A Glucose Case Study

George Hripcsak
1   Department of Biomedical Informatics, Columbia University, New York, New York, United States
2   Medical Informatics Services, NewYork-Presbyterian Hospital, New York, New York, United States
,
David J. Albers
1   Department of Biomedical Informatics, Columbia University, New York, New York, United States
3   Department of Pediatrics, University of Colorado Denver—Anschutz Medical Campus, Denver, Colorado, United States
› Author Affiliations Funding This work was funded by grants from the National Institutes of Health R01 LM006910 “Discovering and applying knowledge in clinical databases” and R01 LM012734 “Mechanistic machine learning.”
› Further Information
   Permissions and Reprints

Abstract

Background It would be useful to be able to assess the utility of predictive models of continuous values before clinical trials are performed.

Objective The aim of the study is to compare metrics to assess the potential clinical utility of models that produce continuous value forecasts.

Methods We ran a set of data assimilation forecast algorithms on time series of glucose measurements from neurological intensive care unit patients. We evaluated the forecasts using four sets of metrics: glucose root mean square (RMS) error, a set of metrics on a transformed glucose value, the estimated effect on clinical care based on an insulin guideline, and a glucose measurement error grid (Parkes grid). We assessed correlation among the metrics and created a set of factor models.

Results The metrics generally correlated with each other, but those that estimated the effect on clinical care correlated with others the least and were generally associated with their own independent factors. The other metrics appeared to separate into those that emphasized errors in low glucose versus errors in high glucose. The Parkes grid was well correlated with the transformed glucose but not the estimation of clinical care.

Discussion Our results indicate that we need to be careful before we assume that commonly used metrics like RMS error in raw glucose or even metrics like the Parkes grid that are designed to measure importance of differences will correlate well with actual effect on clinical care processes. A combination of metrics appeared to explain the most variance between cases. As prediction algorithms move into practice, it will be important to measure actual effects.

Keywords

metrics - predictive models - data assimilation - machine learning

Authors' Contributions

All authors made substantial contributions to the conception and design of the work; drafted the work or revised it critically for important intellectual content; had final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.




Publication History

Received: 18 June 2021

Accepted: 28 December 2021

Article published online:
23 February 2022

© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Albers DJ, Levine ME, Stuart A, Mamykina L, Gluckman B, Hripcsak G. Mechanistic machine learning: how data assimilation leverages physiologic knowledge using Bayesian inference to forecast the future, infer the present, and phenotype. J Am Med Inform Assoc 2018; 25 (10) 1392-1401
    CrossrefPubMedGoogle Scholar
  • 2 Law K, Stuart A, Zygalakis K. Data Assimilation. New York, NY: Springer; 2015
    CrossrefGoogle Scholar
  • 3 Sturis J, Polonsky KS, Mosekilde E, Van Cauter E. Computer model for mechanisms underlying ultradian oscillations of insulin and glucose. Am J Physiol 1991; 260 (5 Pt 1): E801-E809
    PubMedGoogle Scholar
  • 4 Dalla Man C, Rizza RA, Cobelli C. Meal simulation model of the glucose-insulin system. IEEE Trans Biomed Eng 2007; 54 (10) 1740-1749
    CrossrefPubMedGoogle Scholar
  • 5 Albers DJ, Levine M, Gluckman B, Ginsberg H, Hripcsak G, Mamykina L. Personalized glucose forecasting for type 2 diabetes using data assimilation. PLOS Comput Biol 2017; 13 (04) e1005232
    CrossrefPubMedGoogle Scholar
  • 6 Albers DJ, Elhadad N, Claassen J, Perotte R, Goldstein A, Hripcsak G. Estimating summary statistics for electronic health record laboratory data for use in high-throughput phenotyping algorithms. J Biomed Inform 2018; 78: 87-101
    CrossrefPubMedGoogle Scholar
  • 7 Albers DJ, Blancquart P-A, Levine ME, Seylabi EE, Stuart A. Ensemble Kalman methods with constraints. Inverse Probl 2019; 35 (09) 095007
    CrossrefPubMedGoogle Scholar
  • 8 Hripcsak G, Albers DJ, Perotte A. Parameterizing time in electronic health record studies. J Am Med Inform Assoc 2015; 22 (04) 794-804
    CrossrefPubMedGoogle Scholar
  • 9 Albers DJ, Levine ME, Sirlanci M, Stuart AM. A simple modeling framework for prediction in the human glucose-insulin system. arXiv preprint arXiv:1910.14193
    PubMedGoogle Scholar
  • 10 Xiao C, Choi E, Sun J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. J Am Med Inform Assoc 2018; 25 (10) 1419-1428
    CrossrefPubMedGoogle Scholar
  • 11 Jolliffe IT, Stephenson DB. eds. Forecast Verification: A Practitioner's Guide in Atmospheric Science, 2nd ed. Chichester, UK: John Wiley & Sons; 2012
    Google Scholar
  • 12 Malouf R, Brust JCM. Hypoglycemia: causes, neurological manifestations, and outcome. Ann Neurol 1985; 17 (05) 421-430
    CrossrefPubMedGoogle Scholar
  • 13 Adeyinka A, Kondamudi NP. Hyperosmolar Hyperglycemic Nonketotic Coma (HHNC, Hyperosmolar Hyperglycemic Nonketotic Syndrome). StatPearls. Treasure Island, FL: StatPearls Publishing; 2019
    Google Scholar
  • 14 Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care 2000; 23 (08) 1143-1148
    CrossrefPubMedGoogle Scholar
  • 15 Pfützner A, Klonoff DC, Pardo S, Parkes JL. Technical aspects of the Parkes error grid. J Diabetes Sci Technol 2013; 7 (05) 1275-1281
    CrossrefPubMedGoogle Scholar
  • 16 Wilson M, Weinreb J, Hoo GW. Intensive insulin therapy in critical care: a review of 12 protocols. Diabetes Care 2007; 30 (04) 1005-1011
    CrossrefPubMedGoogle Scholar