Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients' Palliative Care Needs Predicted by a Machine Learning Tool

Abstract

Objectives To assess primary care physicians' (PCPs) perception of the need for serious illness conversations (SIC) or other palliative care interventions in patients flagged by a machine learning tool for high 1-year mortality risk.

Methods We surveyed PCPs from four Brigham and Women's Hospital primary care practice sites. Multiple mortality prediction algorithms were ensembled to assess adult patients of these PCPs who were either enrolled in the hospital's integrated care management program or had one of several chronic conditions. The patients were classified as high or low risk of 1-year mortality. A blinded survey had PCPs evaluate these patients for palliative care needs. We measured PCP and machine learning tool agreement regarding patients' need for an SIC/elevated risk of mortality.

Results Of 66 PCPs, 20 (30.3%) participated in the survey. Out of 312 patients evaluated, 60.6% were female, with a mean (standard deviation [SD]) age of 69.3 (17.5) years, and a mean (SD) Charlson Comorbidity Index of 2.80 (2.89). The machine learning tool identified 162 (51.9%) patients as high risk. Excluding deceased or unfamiliar patients, PCPs felt that an SIC was appropriate for 179 patients; the machine learning tool flagged 123 of these patients as high risk (68.7% concordance). For 105 patients whom PCPs deemed SIC unnecessary, the tool classified 83 as low risk (79.1% concordance). There was substantial agreement between PCPs and the tool (Gwet's agreement coefficient of 0.640).

Conclusions A machine learning mortality prediction tool offers promise as a clinical decision aid, helping clinicians pinpoint patients needing palliative care interventions.

Keywords

machine learning - patient–provider - decision support algorithm - ambulatory care - outpatient care

^* Co-first authors.

Protection of Human and Animal Subjects

This study was approved by the Mass General Brigham Institutional Review Board.

Publication History

Received: 13 December 2023

Accepted: 17 April 2024

Accepted Manuscript online:
18 April 2024

Article published online:
12 June 2024

• References

• 1 Morrison RS, Meier DE. Clinical practice. Palliative care. N Engl J Med 2004; 350 (25) 2582-2590
• 2 Morrison RS, Dietrich J, Ladwig S. et al. Palliative care consultation teams cut hospital costs for Medicaid beneficiaries. Health Aff (Millwood) 2011; 30 (03) 454-463
• 3 Kavalieratos D, Corbelli J, Zhang D. et al. Association between palliative care and patient and caregiver outcomes: a systematic review and meta-analysis. JAMA 2016; 316 (20) 2104-2114
• 4 Ma J, Chi S, Buettner B. et al. Early palliative care consultation in the medical ICU: a cluster randomized crossover trial. Crit Care Med 2019; 47 (12) 1707-1715
• 5 Temel JS, Greer JA, El-Jawahri A. et al. Effects of early integrated palliative care in patients with lung and GI cancer: a randomized clinical trial. J Clin Oncol 2017; 35 (08) 834-841
• 6 Temel JS, Greer JA, Muzikansky A. et al. Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med 2010; 363 (08) 733-742
• 7 Meier DE, Back AL, Berman A, Block SD, Corrigan JM, Morrison RS. A national strategy for palliative care. Health Aff (Millwood) 2017; 36 (07) 1265-1273
• 8 Meier DE, Beresford L. Outpatient clinics are a new frontier for palliative care. J Palliat Med 2008; 11 (06) 823-828
• 9 Spetz J, Dudley N, Trupin L, Rogers M, Meier DE, Dumanovsky T. Few hospital palliative care programs meet national staffing recommendations. Health Aff (Millwood) 2016; 35 (09) 1690-1697
• 10 Lupu D. American Academy of Hospice and Palliative Medicine Workforce Task Force. Estimate of current hospice and palliative medicine physician workforce shortage. J Pain Symptom Manage 2010; 40 (06) 899-911
• 11 Kelley AS, Bollens-Lund E. Identifying the population with serious illness: the “denominator” challenge. J Palliat Med 2017; 21 (S2): S7-S16
• 12 Kelley AS, Covinsky KE, Gorges RJ. et al. Identifying older adults with serious illness: a critical step toward improving the value of health care. Health Serv Res 2017; 52 (01) 113-131
• 13 Ahluwalia SC, Chen C, Raaen L. et al. A systematic review in support of the national consensus project clinical practice guidelines for quality palliative care, fourth edition. J Pain Symptom Manage 2018; 56 (06) 831-870
• 14 Quill TE, Abernethy AP. Generalist plus specialist palliative care—creating a more sustainable model. N Engl J Med 2013; 368 (13) 1173-1175
• 15 Rosa WE, Izumi S, Sullivan DR. et al. Advance care planning in serious illness: a narrative review. J Pain Symptom Manage 2023; 65 (01) e63-e78
• 16 Paladino J, Brannen E, Benotti E. et al. Implementing serious illness communication processes in primary care: a qualitative study. Am J Hosp Palliat Care 2021; 38 (05) 459-466
• 17 Lakin JR, Block SD, Billings JA. et al. Improving communication about serious illness in primary care: a review. JAMA Intern Med 2016; 176 (09) 1380-1387
• 18 Jacobsen J, Bernacki R, Paladino J. Shifting to serious illness communication. JAMA 2022; 327 (04) 321-322
• 19 Bernacki RE, Block SD. American College of Physicians High Value Care Task Force. Communication about serious illness care goals: a review and synthesis of best practices. JAMA Intern Med 2014; 174 (12) 1994-2003
• 20 Paladino J, Koritsanszky L, Neal BJ. et al. Effect of the serious illness care program on health care utilization at the end of life for patients with cancer. J Palliat Med 2020; 23 (10) 1365-1369
• 21 Paladino J, Bernacki R, Neville BA. et al. Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: a cluster randomized clinical trial of the serious illness care program. JAMA Oncol 2019; 5 (06) 801-809
• 22 Lakin JR, Neal BJ, Maloney FL. et al. A systematic intervention to improve serious illness communication in primary care: effect on expenses at the end of life. Healthc (Amst) 2020; 8 (02) 100431
• 23 Lakin JR, Koritsanszky LA, Cunningham R. et al. A systematic intervention to improve serious illness communication in primary care. Health Aff (Millwood) 2017; 36 (07) 1258-1264
• 24 Lakin JR, Arnold CG, Catzen HZ. et al. Early serious illness communication in hospitalized patients: a study of the implementation of the Speaking About Goals and Expectations (SAGE) program. Healthc (Amst) 2021; 9 (02) 100510
• 25 Bernacki R, Paladino J, Neville BA. et al. Effect of the serious illness care program in outpatient oncology: a cluster randomized clinical trial. JAMA Intern Med 2019; 179 (06) 751-759
• 26 Starfield B, Shi L, Macinko J. Contribution of primary care to health systems and health. Milbank Q 2005; 83 (03) 457-502
• 27 Lakin JR, Desai M, Engelman K. et al. Earlier identification of seriously ill patients: an implementation case series. BMJ Support Palliat Care 2020; 10 (04) e31
• 28 Lakin JR, Robinson MG, Obermeyer Z. et al. Prioritizing primary care patients for a communication intervention using the “surprise question”: a prospective cohort study. J Gen Intern Med 2019; 34 (08) 1467-1474
• 29 Baxter R, Fromme EK, Sandgren A. Patient identification for serious illness conversations: a scoping review. Int J Environ Res Public Health 2022; 19 (07) 4162
• 30 Allen A, Mataraso S, Siefkas A. et al. A racially unbiased, machine learning approach to prediction of mortality: algorithm development study. JMIR Public Health Surveill 2020; 6 (04) e22400
• 31 Chi S, Kim S, Reuter M. et al. Advanced care planning for hospitalized patients following clinician notification of patient mortality by a machine learning algorithm. JAMA Netw Open 2023; 6 (04) e238795
• 32 Deardorff WJ, Barnes DE, Jeon SY. et al. Development and external validation of a mortality prediction model for community-dwelling older adults with dementia. JAMA Intern Med 2022; 182 (11) 1161-1170
• 33 Wang E, Major VJ, Adler N. et al. Supporting acute advance care planning with precise, timely mortality risk predictions. NEJM Catal 2021; 2 (03) DOI: 10.1056/CAT.20.0655.
• 34 Zachariah FJ, Rossi LA, Roberts LM, Bosserman LD. Prospective comparison of medical oncologists and a machine learning model to predict 3-month mortality in patients with metastatic solid tumors. JAMA Netw Open 2022; 5 (05) e2214514
• 35 Wang L, Wang Y, Laurentiev J. et al Mortality risk prediction for patients with chronic diseases using electronic health records. In Proceedings of the AMIA Informatics Summit; March 13, 2023; Seattle, WA
• 36 Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 2008; 61 (Pt 1): 29-48
• 37 Manz CR, Parikh RB, Small DS. et al. Effect of integrating machine learning mortality estimates with behavioral nudges to clinicians on serious illness conversations among patients with cancer: a stepped-wedge cluster randomized clinical trial. JAMA Oncol 2020; 6 (12) e204759
• 38 Manz CR, Zhang Y, Chen K. et al. Long-term effect of machine learning-triggered behavioral nudges on serious illness conversations and end-of-life outcomes among patients with cancer: a randomized clinical trial. JAMA Oncol 2023; 9 (03) 414-418
• 39 Wilson PM, Ramar P, Philpot LM. et al. Effect of an artificial intelligence decision support tool on palliative care referral in hospitalized patients: a randomized clinical trial. J Pain Symptom Manage 2023; 66 (01) 24-32
• 40 Footracer KG. Alert fatigue in electronic health records. JAAPA 2015; 28 (07) 41-42
• 41 Rotenstein LS, Holmgren AJ, Downing NL, Bates DW. Differences in total and after-hours electronic health record time across ambulatory specialties. JAMA Intern Med 2021; 181 (06) 863-865
• 42 DeCamp M, Lindvall C. Latent bias and the implementation of artificial intelligence in medicine. J Am Med Inform Assoc 2020; 27 (12) 2020-2023
• 43 Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med 2018; 178 (11) 1544-1547
• 44 Porter AS, Harman S, Lakin JR. Power and perils of prediction in palliative care. Lancet 2020; 395 (10225): 680-681