A Machine Learning-Based Clinical Decision Support System to Improve End-of-Life Care

Abstract

Background

End-of-life care (EoLC), such as advance care planning, advance directives, hospice, and palliative care consults, can improve patient quality of life and reduce costs, yet such interventions are underused. Machine learning-based prediction models show promise in identifying patients who may be candidates for EoLC based on increased risk of short-term (less than 1 year) mortality. Clinical decision support systems using these models can identify candidate patients at a time during their care when care teams can increase the provision of EoLC.

Objectives

Evaluate changes in the provision of EoLC with implementation of a machine learning-based mortality prediction model in an academic health center.

Methods

A clinical decision support system based on a random forest machine learning mortality prediction model is described. The system was implemented in an academic health system, first in the medical intensive care unit, then house-wide. An interrupted time series analysis was performed over the 16 weeks prior to and 43 weeks after the implementations. Primary outcomes were the rates of documentation of advance directives, palliative care consultations, and do not attempt resuscitation (DNAR) orders among encounters with an alert for PRISM score over 50% (PRISM positive) compared with those without an alert (PRISM negative).

Results

Following a steep preintervention decline, the rate of advance directive documentation improved immediately after implementation. However, the implementations were not associated with improvements in any of the other primary outcomes. The model discrimination was substantially worse than that observed in model development, and after 16 months, it was withdrawn from production.

Conclusion

A clinical decision support system based on a machine learning mortality prediction model failed to provide clinically meaningful improvements in EoLC measures. Possible causes for the failure include system-level factors, clinical decision support system design, and poor model performance.

Keywords

clinical decision support - machine learning - random forest - palliative care - advance directives

Protection of Human and Animal Subjects

The University of Missouri Institutional Review Board reviewed and approved the study as exempt and approved a waiver of the requirement to obtain informed consent.

Publikationsverlauf

Eingereicht: 02. Januar 2025

Angenommen: 06. Juni 2025

Artikel online veröffentlicht:
07. November 2025

• References

• 1 Temel JS, Greer JA, Muzikansky A. et al. Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med 2010; 363 (08) 733-742
• 2 Khandelwal N, Kross EK, Engelberg RA, Coe NB, Long AC, Curtis JR. Estimating the effect of palliative care interventions and advance care planning on ICU utilization: a systematic review. Crit Care Med 2015; 43 (05) 1102-1111
• 3 Diop MS, Rudolph JL, Zimmerman KM, Richter MA, Skarf LM. Palliative care interventions for patients with heart failure: A systematic review and meta-analysis. J Palliat Med 2017; 20 (01) 84-92
• 4 Kavalieratos D, Corbelli J, Zhang D. et al. Association between palliative care and patient and caregiver outcomes: a systematic review and meta-analysis. JAMA 2016; 316 (20) 2104-2114
• 5 Dalal S, Bruera E. End-of-life care matters: Palliative cancer care results in better care and lower costs. Oncologist 2017; 22 (04) 361-368
• 6 May P, Normand C, Cassel JB. et al. Economics of palliative care for hospitalized adults with serious illness: A meta-analysis. JAMA Intern Med 2018; 178 (06) 820-829
• 7 Nguyen M-T, Feeney T, Kim C. et al. Differential utilization of palliative care consultation between medical and surgical services. Am J Hosp Palliat Care 2020; 37 (04) 250-257
• 8 Kathpalia P, Smith A, Lai JC. Underutilization of palliative care services in the liver transplant population. World J Transplant 2016; 6 (03) 594-598
• 9 Emanuel EJ. The status of end-of-life care in the United States: The glass is half full. JAMA 2018; 320 (03) 239-241
• 10 Shalev A, Phongtankuel V, Kozlov E, Shen MJ, Adelman RD, Reid MC. Awareness and misperceptions of hospice and palliative care: A population-based survey study. Am J Hosp Palliat Care 2018; 35 (03) 431-439
• 11 Osheroff JA. Ed. Improving Medication Use and Outcome with Clinical Decision Support: A Step-by-Step Guide. Healthcare Information and Management Systems Society; Chicago, IL: 2009
• 12 Inouye SK, Bogardus Jr ST, Vitagliano G. et al. Burden of illness score for elderly persons: risk adjustment incorporating the cumulative impact of diseases, physiologic abnormalities, and functional impairments. Med Care 2003; 41 (01) 70-83
• 13 Levine SK, Sachs GA, Jin L, Meltzer D. A prognostic model for 1-year mortality in older adults after hospital discharge. Am J Med 2007; 120 (05) 455-460
• 14 Walter LC, Brand RJ, Counsell SR. et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA 2001; 285 (23) 2987-2994
• 15 Parikh RB, Manz C, Chivers C. et al. Machine learning approaches to predict 6-month mortality among patients with cancer. JAMA Netw Open 2019; 2 (10) e1915997
• 16 Makar M, Ghassemi M, Cutler DM, Obermeyer Z. Short-term mortality prediction for elderly patients using Medicare claims data. Int J Mach Learn Comput 2015; 5 (03) 192-197
• 17 Pilotto A, Ferrucci L, Franceschi M. et al. Development and validation of a multidimensional prognostic index for one-year mortality from comprehensive geriatric assessment in hospitalized older patients. Rejuvenation Res 2008; 11 (01) 151-161
• 18 Teno JM, Harrell Jr FE, Knaus W. et al. Prediction of survival for older hospitalized patients: the HELP survival model. Hospitalized Elderly Longitudinal Project. J Am Geriatr Soc 2000; 48 (S1): S16-S24
• 19 Dramé M, Novella JL, Mahmoudi R. et al. Derivation and validation of a mortality-risk index from a cohort of frail elderly patients hospitalised in medical wards via emergencies: The SAFES study. J Nutr Health Aging 2009; 13: S429
• 20 van Walraven C, Forster AJ. The HOMR-now! Model accurately predicts 1-year death risk for hospitalized patients on admission. Am J Med 2017; 130 (08) 991.e9-991.e16
• 21 Gallego González E, Ortiz Alonso FJ, Vidán Astiz MT. et al. Development and validation of a prognostic index for 6- and 12-month mortality in hospitalized older adults. Arch Gerontol Geriatr 2017; 73: 269-278
• 22 Weng SF, Vaz L, Qureshi N, Kai J. Prediction of premature all-cause mortality: A prospective general population cohort study comparing machine-learning and standard epidemiological approaches. PLoS ONE 2019; 14 (03) e0214365
• 23 Taylor RA, Pare JR, Venkatesh AK. et al. Prediction of in-hospital mortality in emergency department patients with sepsis: A local big data-driven, machine learning approach. Acad Emerg Med 2016; 23 (03) 269-278
• 24 Pierce RP, Raithel S, Brandt L, Clary KW, Craig K. A comparison of models predicting one-year mortality at time of admission. J Pain Symptom Manage 2022; 63 (03) e287-e293
• 25 Wegier P, Koo E, Ansari S. et al. mHOMR: A feasibility study of an automated system for identifying inpatients having an elevated risk of 1-year mortality. BMJ Qual Saf 2019; 28 (12) 971-979
• 26 Wegier P, Kurahashi A, Saunders S. et al. mHOMR: A prospective observational study of an automated mortality prediction model to identify patients with unmet palliative needs. BMJ Support Palliat Care 2024; 14: e969-e975
• 27 Pierce RP, Eskridge BR, Ross B, Day MA, Dean B, Belden JL. Improving the user experience with discount site-specific user testing. Appl Clin Inform 2022; 13 (05) 1040-1052
• 28 Sauro J, Dumas JS. Comparison of three one-question, post-task usability questionnaires. Paper presented at: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2009 ; Boston, MA. Accessed June 11, 2025 at: https://doi.org/10.1145/1518701.1518946
• 29 Sendak MP, Gao M, Brajer N, Balu S. Presenting machine learning model information to clinical end users with model facts labels. NPJ Digit Med 2020; 3 (01) 41
• 30 Taseen R, Ethier J-F. Expected clinical utility of automatable prediction models for improving palliative and end-of-life care outcomes: Toward routine decision analysis before implementation. J Am Med Inform Assoc 2021; 28 (11) 2366-2378
• 31 Hategeka C, Ruton H, Karamouzian M, Lynd LD, Law MR. Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review. BMJ Glob Health 2020; 5 (10) e003567
• 32 Lopez Bernal J, Cummins S, Gasparrini A. The use of controls in interrupted time series studies of public health interventions. Int J Epidemiol 2018; 47 (06) 2082-2093
• 33 Cumby RE, Huizinga J. Testing the autocorrelation structure of disturbances in ordinary least squares and instrumental variables regressions. Econometrica 1992; 60 (01) 185-195
• 34 Linden A. Conducting interrupted time-series analysis for single- and multiple-group comparisons. Stata J 2015; 15 (02) 480-500
• 35 Baum C, Schaffer M. . ACTEST: Stata module to perform Cumby-Huizinga general test for autocorrelation in time series. Statistical Software Components S457668. Boston College Department of Economics: 2024
• 36 Avati A, Jung K, Harman S, Downing L, Ng A, Shah NH. Improving palliative care with deep learning. BMC Med Inform Decis Mak 2018; 18 (Suppl. 04) 122
• 37 Jung K, Kashyap S, Avati A. et al. A framework for making predictive models useful in practice. J Am Med Inform Assoc 2021; 28 (06) 1149-1158
• 38 Courtright KR, Chivers C, Becker M. et al. Electronic health record mortality prediction model for targeted palliative care among hospitalized medical patients: A pilot quasi-experimental study. J Gen Intern Med 2019; 34 (09) 1841-1847
• 39 Manz CR, Parikh RB, Small DS. et al. Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer: A Stepped-Wedge Cluster Randomized Clinical Trial. JAMA Oncol 2020; 6 (12) e204759
• 40 Seneviratne MG, Shah NH, Chu L. Bridging the implementation gap of machine learning in healthcare. BMJ Innov 2020; 6 (02) 45-47
• 41 Speiser JL, Miller ME, Tooze J, Ip E. A comparison of random forest variable selection methods for classification prediction modeling. Expert Syst Appl 2019; 134: 93-101
• 42 Johns Hopkins University and Medicine. Coronavirus Resource Center. United States: Missouri; 2024. . Accessed December 17, 2024 at: https://coronavirus.jhu.edu/region/us/missouri
• 43 Kagerbauer SM, Ulm B, Podtschaske AH. et al. Susceptibility of AutoML mortality prediction algorithms to model drift caused by the COVID pandemic. BMC Med Inform Decis Mak 2024; 24 (01) 34
• 44 Chi C-Y, Ao S, Winkler A. et al. Predicting the mortality and readmission of in-hospital cardiac arrest patients with electronic health records: A machine learning approach. J Med Internet Res 2021; 23 (09) e27798
• 45 Hameed MAB, Alamgir Z. Improving mortality prediction in acute pancreatitis by machine learning and data augmentation. Comput Biol Med 2022; 150: 106077
• 46 Turkson AJ, Ayiah-Mensah F, Nimoh V. Handling censoring and censored data in survival analysis: A standalone systematic literature review. Int J Math Math Sci 2021; 2021 (01) 9307475