Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients' Palliative Care Needs Predicted by a Machine Learning Tool

Abstract

Objectives To assess primary care physicians' (PCPs) perception of the need for serious illness conversations (SIC) or other palliative care interventions in patients flagged by a machine learning tool for high 1-year mortality risk.

Methods We surveyed PCPs from four Brigham and Women's Hospital primary care practice sites. Multiple mortality prediction algorithms were ensembled to assess adult patients of these PCPs who were either enrolled in the hospital's integrated care management program or had one of several chronic conditions. The patients were classified as high or low risk of 1-year mortality. A blinded survey had PCPs evaluate these patients for palliative care needs. We measured PCP and machine learning tool agreement regarding patients' need for an SIC/elevated risk of mortality.

Results Of 66 PCPs, 20 (30.3%) participated in the survey. Out of 312 patients evaluated, 60.6% were female, with a mean (standard deviation [SD]) age of 69.3 (17.5) years, and a mean (SD) Charlson Comorbidity Index of 2.80 (2.89). The machine learning tool identified 162 (51.9%) patients as high risk. Excluding deceased or unfamiliar patients, PCPs felt that an SIC was appropriate for 179 patients; the machine learning tool flagged 123 of these patients as high risk (68.7% concordance). For 105 patients whom PCPs deemed SIC unnecessary, the tool classified 83 as low risk (79.1% concordance). There was substantial agreement between PCPs and the tool (Gwet's agreement coefficient of 0.640).

Conclusions A machine learning mortality prediction tool offers promise as a clinical decision aid, helping clinicians pinpoint patients needing palliative care interventions.

 

MissingLinkText

Keywords

machine learning - patient–provider - decision support algorithm - ambulatory care - outpatient care

Background and Significance

Palliative care involves interprofessional physical and psychological symptom control, social and spiritual care, expert communication, and care coordination for those living with serious illnesses. It is delivered by teams of specialty-trained clinicians along with other types of medical and surgical care and focuses on improving quality of life for patients and their caregivers.[1] [2] Palliative care improves physical, psychological, and quality-of-life outcomes for patients and caregivers dealing with serious illness.[3] [4] [5] [6] While palliative care delivery has expanded notably in the past decade, it is still not delivered equally or universally, and there is considerably more demand than supply.[7] [8] [9] [10] As such, identifying which patients have unmet palliative care needs and thus might benefit from palliative care interventions is a well-documented problem without a known solution.[11] [12]

Unmet palliative care needs can include any individual or combination of several of the key domains of palliative care—physical, psychological, social, spiritual, communication, end of life, and care coordination.[13] Depending on the individual clinical scenario, unmet needs can be addressed by activities performed by all clinicians—such as advance care planning (ACP) or serious illness conversations (SICs)—or require the involvement of specialty palliative care or hospice teams.[14] [15] [16] [17] SICs, which are proactive conversations about patients' goals and values in the context of serious illness,[18] [19] are an important intervention that can improve health care outcomes such as anxiety, depression, and bereavement, while also aligning medical care with patients' evolving preferences.[20] [21] [22] [23] [24] [25] Effective communication about what matters most to patients can enhance health care delivery by improving patient and family outcomes and facilitating more efficient use of health care resources. Primary care physicians (PCPs) have a key and unique role in identifying and addressing unmet palliative care needs, especially serious illness communication.[17] As central components of the care team, PCPs leverage their longitudinal relationships with patients and caregivers to meet a broad array of needs directly, while also coordinating referral to specialist care, such as palliative care, when indicated.[26]

Proposed solutions for identifying unmet palliative care needs have included both clinician screens and analytic models.[22] [27] [28] [29] One option used with increasing frequency for identifying unmet palliative care needs is the use of predictive analytics-based platforms for clinical decision support.[27] In addition to their commonly described uses for predicting serious, nonmortality clinical outcomes,[1] [2] [3] these platforms often use mortality prediction as a proxy for the broad paradigms that define likely unmet palliative care needs—risk of a shortened lifespan, impaired functional status, degrading quality of life, or caregiver impact.[11] [12] While a significant body of work has explored the ability of machine learning algorithms to predict mortality among high-risk patients,[30] [31] [32] [33] [34] there is limited evidence about the extent to which mortality prediction can serve as a proxy predictor for unmet clinical palliative care needs. Therefore, in this study, we assessed the extent to which PCPs' perceptions of which patients may benefit from palliative care and serious illness communication in the primary care setting overlapped with assessments of mortality risk made by a machine learning-based mortality prediction tool.

 

MissingLinkText

Methods

Settings and Data Sources

The study was carried out at Brigham and Women's Hospital (BWH), an academic medical center within the Mass General Brigham (MGB) health care system in Boston, Massachusetts, United States. We recruited PCPs from four BWH primary care practices sites: Phyllis Jen Center for Primary Care, South Huntington Primary Care Associates, Brookline Primary Care, and Brookside Community Health Center. PCPs at these clinics have had access to training in conducting SICs with appropriate patients and thus possess experience and working knowledge of palliative care, serious illness communication, and identifying unmet palliative care needs. Consistent with prior evidence regarding effective SIC training methods,[4] [5] training delivered to the PCPs in these clinics included didactic content, reflection on clinician practice, and actor-based skills practice focusing on using a structured approach to SICs. We contacted all PCPs at each practice site, and participation was voluntary.

 

MissingLinkText

Study Participants

Adult patients of the PCPs at the four clinic sites were selected if they were part of BWH's integrated care management program (iCMP) or had serious illnesses: congestive heart failure, chronic obstructive pulmonary disease, chronic kidney disease, chronic liver disease, and dementia. Patients with cancer were not included in the study, as they are typically followed at BWH's affiliate cancer center, the Dana-Farber Cancer Institute. Their disease status was determined either by an active disease diagnosis on their electronic health record (EHR) problem list or by ≥2 separate encounters before July 11, 2022, with an International Statistical Classification of Diseases and Related Health Problems, Ninth Revision and/or Tenth Revision (ICD-9/10) code for one of the above conditions as a primary or secondary diagnosis.

 

MissingLinkText

Machine Learning Models

We developed eight machine learning models (AdaBoost, XGBoost, random forest, extra trees, linear support vector machine, logistic regression, bagging, and multilayer perceptron) to predict 1-year mortality.[35] These models were trained using data from more than 200,000 patients with the aforementioned conditions or those enrolled in the iCMP. A total of 183 predictors were extracted from MGB's EHRs and utilized in the models, which are themselves not integrated into the EHR. The predictors used in the models include patient sociodemographic data (such as age, sex, race, education, marital status, and smoking history), body mass index, a preselected list of abnormal laboratory tests, counts of medication prescriptions in the 6 months prior to the prediction, the number of encounter diagnoses for individual comorbid diseases, and Charlson Comorbidity Index (CCI) using ICD codes within the past 2 years prior to the prediction. The models attained an area under the receiver operating characteristic curve ranging from 0.54 to 0.88. We used sample weighting to accommodate potential selection bias during preparation of the training dataset. Additionally, we applied previously described methods for fairness evaluation and bias assessment in the model development and testing phase to evaluate and control for potential bias.[6] [7] The models were then employed to estimate patients' risk of 1-year mortality. Specifically, these models were run individually, and patients were considered as “high risk” if any model predicted a 50% or greater mortality chance within a year; otherwise, they were classified as “low risk.” The combination of these eight proprietary machine learning models is hereafter referred to as the “machine learning tool.” Of note, this tool is currently in the research and development phase and is not available beyond our institution.

 

MissingLinkText

Survey Questions

To assess the alignment between PCP perspectives on unmet palliative care needs and the machine learning tool's mortality risk estimates, we conducted a structured two-part survey between August 12, 2022, and October 12, 2022. This survey was designed by authors L.R. and J.R.L. and was tested among volunteer PCPs (who were not involved in the formal study) prior to survey distribution. The first part of the survey collected PCPs' consent, demographics, and experience details, including age, gender, race, ethnicity, years in practice, SIC training completion status, and the self-reported number of SICs conducted in the preceding year. The second part listed patients for PCPs to evaluate for appropriate interventions. We identified up to 10 patients for each PCP among the “high-risk” group and a matched number of patients from the PCP's “low-risk” group. Therefore, each PCP was presented with a maximum of 20 patients per survey. These were randomly distributed in the survey, with PCPs blinded to the risk level. We then asked the PCPs about their perceptions of their patients' need for SICs and specific palliative care services. For each patient in the survey, the PCPs answered questions about the appropriateness, need, and feasibility of conducting an SIC, without specific definitions for these concepts, allowing respondents to reply on their own interpretation. They also provided reasons if they deemed certain patients not appropriate for an SIC. Furthermore, the PCPs rated the benefit of addressing other unmet needs, such as ACP conversations, code status discussions, completing Massachusetts Medical Orders for Life-Sustaining Treatment (MOLST) forms, specialty palliative care consultations, home-based palliative care interventions, or hospice discussions. They also indicated whether patients presented had other unmet palliative care needs. Each of these terms was defined in the survey tool ([Supplementary Appendix 1], available in the online version), with definitions based on current literature and expert opinion.

 

MissingLinkText

Statistical Analysis

We calculated the response rates of PCPs and described the demographic and clinical characteristics of both PCPs and their patients. We analyzed the proportion of patients for whom a PCP deemed an SIC conversation necessary compared with those who did not, and the proportion of patients which the machine learning tool classified as high risk versus not. We delineated the demographic and clinical characteristics of patients in each group.

Next, we evaluated the concordance between the tool's high-risk designation for a patient and the PCPs' perception of the need for an SIC. For patients whom PCPs felt needed an SIC, we quantified the proportion for which PCPs felt SIC feasibility was high, medium, or low. To formally assess the relationship between PCPs perceptions of SIC necessity and the tool's flagging patients as high risk, we used logistic regression models where the binary dependent variable indicated the PCPs opinion on the need for an SIC and the independent variable indicated whether the patient was flagged by the machine learning tool as high risk. Since PCPs provided opinions on multiple patients, we clustered the models by PCP. Furthermore, we calculated Gwet's agreement coefficient to formally assess the agreement between PCP evaluations and the machine learning tool's patient classification.[36]

We also described the characteristics of patients for whom there was a mismatch between PCP opinions and the tool's risk categorization. Specifically, we described instances where PCPs felt an SIC was indicated, but the tool did not flag the patient as high risk, and vice versa.

Lastly, we calculated the proportion of patients deemed high risk by the tool for whom PCPs believed other conversations or interventions, such as ACP, code status, MOLST form discussion, specialty palliative care consultation, home-based palliative care, or hospice discussion, were necessary. To determine the association between PCPs' opinions on these interventions and the tool's mortality risk classification, we used the chi-square test or Fisher's exact test.

We used a 95% confidence threshold to assess statistical significance. Presurvey power calculations indicated that to estimate 80% association between the tool's predictions and the PCP's opinions with 95% confidence, we needed PCP responses for 264 patients. All analyses were performed using SAS 9.3 (SAS Institute, Cary, NC).

 

MissingLinkText

 

MissingLinkText

Results

We distributed the survey to all 66 PCPs with longitudinal patient panels at the four study sites. Of these, 43 PCPs did not start the survey, and three PCPs consented but did not answer any patient-specific questions. A total of 20 PCPs (30.3%) responded to survey questions regarding at least one designated patient. Most of these respondents (75%) were female, and over half (65%) were aged 40 or above. Over half of the respondents (55%) had a practice history of 10 years or more. Thirteen PCPs (65%) had previously undergone the Serious Illness Care Program training, averaging 2.1 (standard deviation [SD]: 2.9) years since completion. A significant majority (87.0%) conducted 6 or fewer SICs in the preceding year ([Table 1]).

Table 1 Characteristics of primary care physician respondents

Characteristics	PCP respondents (n = 20)
Practice site (n, %)
BWH Advanced Primary Care Associates at South Huntington	6 (30%)
Primary Care at Brookline	2 (10%)
Jen Center	8 (40%)
Brookside Community Health Center	4 (20%)
Gender (n, %)
Male	5 (25%)
Female	15 (75%)
Nonbinary	0 (0%)
Other	0 (0%)
Age (n, %)
<30 y old	0 (0%)
30–39 y old	7 (35%)
40–49 y old	8 (40%)
50–59 y old	1 (5%)
60 y old or older	4 (20%)
Years in practice since clinical training (n, %)
Less than 5 y	5 (25%)
5–<10 y	4 (20%)
10–<20 y	5 (25%)
20–<30 y	2 (10%)
30–<40 y	3 (15%)
40–<50 y	1 (5%)
50+ y	0 (0%)
Self-identified race (n, %)
African American or Black	0 (0%)
American Indian or Alaska Native American	0 (0%)
Asian	2 (10%)
Native Hawaiian or Pacific Islander	0 (0%)
White (European, Middle Eastern, other)	17 (85%)
Prefer to self-describe as	1 (5%)
Self-identified ethnicity (n, %)
Hispanic/Latino	0 (0%)
Not Hispanic/Latino	20 (100%)
Previously completed serious illness care program training (n, %)
Yes	13 (65%)
No	4 (20%)
Maybe, I don't know	3 (15%)
Years since training (mean, SD)	2.1 (2.9)
Self-estimated number of serious illness conversations in past year (n, %)
None	2 (10%)
1–3	8 (40%)
4–6	8 (40%)
6–10	0 (0.0%)
> 10	2 (10%)

Abbreviations: BWH, Brigham and Women's Hospital; PCP, primary care physician; SD, standard deviation.

Among the 312 patients under these PCPs who were eligible for risk stratification, the mean age was 69.3 years (SD: 17.5). The mean CCI for these patients was 2.80 (SD: 2.89). Most (60.6%) of these risk-stratified patients were female, and 41.7% were married. In terms of racial distribution, 48.7% were White, 21.2% were Black, 2.9% were Asian, and for 26.3%, their race was unknown ([Table 2]).

Table 2 Characteristics of patients groups categorized by predicted 1-year mortality and identified by primary care physicians for serious illness conversations

Patient characteristics	Overall	High predicted 1-year mortality (n = 162)	SIC indicated or appropriate per PCP (n = 179)
Age at prediction, year
Mean (SD)	69.3 (17.5)	77.6 (15.2)	74.1 (15.5)
Gender, female, n (%)	189 (60.6)	92 (56.8%)	106 (59.2)
Race, n (%)
White	152 (48.7)	82 (50.6%)	92 (51.4)
Black	66 (21.2)	31 (19.1%)	31 (17.3)
Asian	9 (2.9)	3 (1.9%)	2 (1.1)
Mixed	3 (1.0)	2 (1.2%)	2 (1.1)
Unknown	82 (26.3)	44 (27.2%)	52 (29.1)
Education level, n (%)
College and above	118 (37.8)	49 (30.2%)	62 (34.6)
High school or equivalent	116 (37.2)	66 (40.7%)	60 (33.5)
Did not complete high school	56 (17.9)	37 (22.8%)	43 (24.0)
Unknown	22 (7.1)	10 (6.2%)	14 (7.8)
Smoking status, n (%)
Never smoker	167 (53.5)	76 (46.9%)	83 (46.4)
Former smoker	122 (39.1)	71 (43.8%)	80 (44.7)
Current smoker	23 (7.4)	15 (9.3%)	16 (8.9)
Marital status, n (%)
Married	130 (41.7)	58 (35.8%)	71 (39.7)
Single	90 (28.8)	46 (28.4%)	50 (27.9)
Divorced	44 (14.1)	22 (13.6%)	24 (13.4)
Widow	46 (14.7)	35 (21.6%)	33 (18.4)
Unknown	2 (0.6)	1 (0.6%)	1 (0.6)
BMI (n, %)
Mean (SD)	28.8 (6.73)	26.9 (5.97)	28.4 (6.55)
Median [min, max]	27.5 [14.2, 52.7]	25.6 [14.2, 45.2]	26.8 [17.9, 50.5]
Missing	2 (0.6%)	0 (0%)	0 (0%)
Charlson comorbidity index, mean (SD)
Mean (SD)	2.80 (2.89)	3.79 (2.94)	3.41 (2.9)
Median [min, max]	2.00 [0, 14.0]	3.00 [0, 12.0]	3.00 [0, 12.0]
PCP location, n (%)
Jen Center	113 (36.2)	60 (37.0%)	57 (31.8)
BWH Advanced Primary Care Associates at South Huntington	95 (30.4)	49 (30.2%)	47 (26.3)
Brookside Community Health Center	77 (24.7)	39 (24.1%)	55 (30.7)
BWH Primary Care at Brookline	27 (8.7)	14 (8.6%)	20 (11.2)

Abbreviations: BMI, body mass index; BWH, Brigham and Women's Hospital; PCP, primary care physician; SIC, serious illness conversation; SD, standard deviation.

The machine learning tool identified 162 of these 312 patients (51.9%) as high risk for mortality within the next year. Those flagged as high risk had an average age of 60.2 years (SD: 15.3) and an average CCI of 3.79 (SD: 2.94). Among this high-risk group, 56.8% were female, and 35.8% were married. A majority (50.6%) were White, 19.1% were Black, and the race of 27.2% remained unknown ([Table 2]).

Of the full cohort of 312 patients, PCPs identified eight patients who had already died, while an additional 20 were not recognized by the PCPs. As shown in [Table 2], from the remaining 284 patients, PCPs felt that an SIC was appropriate for 179 patients. These patients had an average age of 74.1 years (SD: 15.5) and a mean CCI of 3.41 (SD: 2.9). Among them, 59.2% were female and 39.7% were married. Over half (51.4%) of patients were White, 17.3% were Black, and the race of 29.1% was unknown.

Among the 179 patients for whom PCPs felt an SIC was necessary, the machine learning tool identified 123 patients as high risk, reflecting PCP algorithm concordance for 68.7% of patients deemed suitable for SICs ([Table 3]). Among all 284 patients evaluated by both PCPs and the tool, this indicates a 43.3% concordance rate regarding patients who both required an SIC and were considered high risk.

Table 3 Association of patients flagged by primary care physicians as needing a serious illness conversation and patients identified by machine learning tool as high risk

Patients flagged by PCPs as needing SIC	Machine learning tool flagged patient as high risk
	Yes (n, %)	No (n, %)	Total (n, %)
Yes (n, %)	123 (43.3)	56 (19.7)	179 (63.0)
No (n, %)	22 (7.8)	83 (29.2)	105 (37.0)
Total	145 (51.1)	139 (48.9)	284
p-Value[a]	<0.0001

Abbreviations: PCP, primary care physician; SIC, serious illness conversation.

^a Adjusted for PCP.

Among the 105 patients whom PCPs deemed didn't currently require an SIC, the tool classified 83 (79.1%) as low risk. This represented 29.2% of patients among all 284 patients considered by both PCPs and the tool, for whom the PCP and tool agreed were lower risk and did not currently need an SIC. In terms of disagreements, PCPs thought that an SIC would be beneficial for 56 patients (19.7% of patients) that the tool did not flag as high risk. On the other hand, the tool flagged 22 patients (7.8% of patients) as high risk that the PCP did not think needed an SIC at this time ([Table 3]).

The characteristics of patients with PCP and tool discordance were largely similar, as detailed in [Supplementary Appendix 2], available in the online version. Notably, patients whom the PCPs didn't consider as needing an SIC but were labeled as high risk by the tool were older, with an average age of 60.7 years (SD: 19.7) versus 65.6 years (SD: 14.6), and they had a higher CCI, with a mean of 3.00 (SD: 3.19) versus 1.73 (SD: 2.10). Overall, there was a significant association between the PCP's classification and the tool's classification of patients (p < 0.0001), and Gwet's agreement coefficient was 0.640.

Of the 22 patients identified as high risk by the machine learning tool whom PCPs did not deem appropriate for SICs, 14 were considered clinically stable or relatively robust given their age or illnesses, two already had palliative care involved, two were no longer under the care of the PCP, and for one patient, an SIC was deemed too emotionally challenging (summarized from free text survey responses). PCPs did not provide a reason for the appropriateness of an SIC for three patients.

Among the 123 patients whom both PCPs and the tool agreed were high risk, PCPs deemed 68 (55.3%) had a high need for a conversation, 42 (34.1%) a medium need, and 13 (10.6%) a low need. For these 123 patients, PCPs deemed an SIC as highly feasible for 36 (29.3%), somewhat feasible for 58 (47.2%), and less feasible for 29 (23.6%). In terms of other unmet palliative care needs, PCPs felt that 117 patients (95.1%) would benefit from an ACP conversation, 105 (85.4%) from a code status conversation, and 93 (75.6%) from a MOLST form discussion and completion. Thirty-five (28.5%) patients were flagged as needing specialty palliative care consultation, 27 (22.0%) a home-based palliative care intervention, 17 (13.8%) a hospice discussion, and 8 (6.5%) as having other unmet needs ([Table 4]). When considering only patients identified by PCPs as needing an SIC (n = 179) and compared to patients identified as high risk by the algorithm, PCPs identified a greater proportion of this group as needing an ACP conversation (91.1 vs. 68.7%; p = 0.005), a code status conversation (79.9 vs. 68.7%; p = 0.007), and a MOLST form discussion and completion (70.4 vs. 68.7%; p = 0.02). However, compared with the patients in this group flagged as high risk by the tool, a significantly lower proportion were flagged by the PCPs as needing a specialty palliative care consultation (24.0 vs. 68.7%; p = 0.004), a home-based palliative care discussion (17.3 vs. 68.7%; p = 0.0), or a hospice discussion (10.6 vs. 68.7%; p = 0.04; [Table 4]).

Table 4 Concordance between primary care physician-suggested interventions and machine learning tool flagging patient as high risk

		Algorithm flagged patient as high risk from among N = 179 patients identified as needing an SIC by PCP		p-value
		Yes	No
PCP flagged patient as needing advance care planning conversation	Yes	117	46	0.005
	No	6	10
PCP flagged patient as needing code status conversation	Yes	105	38	0.007
	No	18	18
PCP flagged patient as needing MOLST form discussion and completion	Yes	93	33	0.02
	No	30	23
PCP flagged patient as needing specialty palliative care consultation	Yes	35	8	0.04
	No	88	48
PCP flagged patient as needing home based palliative care intervention	Yes	27	4	0.02
	No	96	52
PCP flagged patient as needing hospice discussion	Yes	17	2	0.02
	No	106	54
PCP flagged patient as having other unmet needs	Yes	8	0	0.06
	No	115	56
PCP flagged patient as none of the above	Yes	5	7	0.05
	No	118	49

Abbreviations: PCP, primary care physician; MOLST, Medical Orders for Life-Sustaining Treatment; SIC, serious illness conversation.

 

MissingLinkText

Discussion

In this study, we assessed clinicians' perceptions of the outputs of a machine learning tool in a blinded fashion. We found substantial agreement between the mortality prediction tool's classification of patients at elevated risk of 1-year mortality and PCPs' perceptions of their patients' unmet palliative care needs, including the need for an SIC. Among those patients identified by the machine learning tool as potentially benefiting from an SIC, more than half were rated as having a high need for an SIC by their PCPs.

Our study, which uniquely focuses on PCPs' perceptions of a machine learning tool's predictions rather than its impact on downstream process or clinical outcomes, suggests that a machine learning-based tool aimed at predicting 1-year mortality may potentially serve as a valuable decision support tool. It could aid busy clinicians in identifying patients who would benefit from SICs and help clinical teams prioritize patients for palliative care interventions. Specifically, the tool could be used to prompt PCPs to start SICs with patients (even if these conversations are completed over time), be leveraged by other members of the care team who have longitudinal relationships with patients in addition to PCPs (e.g., care coordinators, social workers), or and be used to prioritize patients in panel reviews that PCPs conduct with other members of the care team (e.g., population health personnel).

Notably, our results suggest that a machine learning-based tool aimed at predicting 1-year mortality may not accurately detect all the cases in which a physician's intuition flags patients as needing an SIC. Specifically, for 56 patients (19.7% of the sample), the PCP felt an SIC was appropriate, but the machine learning tool did not flag these patients as high risk. However, we observed relatively few (n = 22, 7.8%) false-positive results—instances where the tool flagged a patient as high risk, but the PCP disagreed. In three of the 22 false-positive cases, the physician noted that either palliative care was already in progress, or an SIC was considered but ultimately deemed too emotionally difficult based on what was known about the patient.

Although we found that the machine learning tool's prediction of 1-year mortality was associated with a substantial level of clinician agreement for interventions that target earlier unmet palliative care needs—such as SIC and ACP conversations—the tool was less effective in identifying needs for interventions closer to the end of life, such as hospice discussions and specialized palliative care support. This finding is significant, as screening tools used for decision support are designed to identify patients most in need of specific interventions for prioritization in clinical practice. Our data suggest that a machine learning tool aimed at predicting 1-year mortality may be well suited for flagging needs for early interventions like SICs. However, it may be less effective for predicting the need for end-of-life interventions such as hospice discussions, for which a model that predicts near-term mortality (i.e., within 3 months)[34] may offer better utility for clinicians. This consideration is critical, as overly broad applications of 1-year mortality prediction algorithms could lead to unwarranted or premature clinical actions. Fundamentally, neither clinician judgement nor machine prediction may serve as a gold standard for identifying unmet palliative care needs. In most such situations, best performance is achieved by supplementing clinical judgment with machine prediction. Therefore, the output of a machine learning tool may best function as a decision-support aid for treating clinicians, especially through highlighting an algorithm's “concern” that a patient may be at high risk and subsequently prompting clinicians to consider the most appropriate interventions. This is especially relevant because clinicians are generally very busy and may miss or postpone such conversations when under stress.

A key question emanating from this work is how best to inform PCPs about which patients may be at high risk and could therefore benefit from palliative care interventions like SICs. Previous studies have demonstrated the value of EHR-based nudges to facilitate completion of SICs in cancer settings, with positive downstream impacts on end-of-life treatment.[37] [38] [39] However, given the already substantial administrative burden faced by clinicians, including frequent EHR alerts[40] and documentation requirments,[41] there is a pressing need to prioritize these alerts and tasks given to physicians. In this context, a machine learning tool that predicts 1-year mortality with substantial agreement from clinicians could serve to prioritize which patients should be considered for palliative care interventions amid a heavy existing workload.

This study has several limitations. It was performed in one academic institution with which all clinicians were affiliated, and the results may not be generalizable to other settings. The study population was diverse but did not represent all ethnicities or ethnic backgrounds, and preferences for end-of-life care may vary among different groups and cultures. Furthermore, we experienced a relatively low response rate to the PCP survey. In the future, this study should be expanded to additional practice settings and geographies to enhance the generalizability of the results. While the response rate met our predetermined statistical power requirements, the results could be biased if, for example, the respondents were among the most engaged in the system or the preponderance of female respondents. In terms of identifying high-risk patients, multiple mortality prediction models were utilized because they tended to select different patient groups. The performance of individual models was not evaluated as they were collectively used to determine the survey population. An additional limitation lies in the use of pre-coronavirus disease 2019 (COVID-19) data to train these models, potentially affecting their efficacy when applied to patients during the COVID-19 pandemic.

Strengths of the study include its use of a robust set of previously studied machine learning algorithms that demonstrated relatively high predictive performance[35] over a 1-year prediction window across a diverse range of comorbid diseases. In this study, we also assessed from PCPs their detailed views regarding how a machine learning-based mortality prediction tool may apply to their patient population.

Future research should focus on refining the predictive tool based on the study's findings, for example by examining the patients who were deemed appropriate for SICs by PCPs but were overlooked by the algorithm. Additionally, it will be valuable to test the tool's efficacy in aiding real-world decision-making as well as to design and continually refine a user interface integrated with existing EHRs that guides palliative care interventions and SICs at the point of care. From a research perspective, the team is actively working on enhancing visualization of algorithm results and recommendations as well as working to ensure model fairness and minimize potential embedded algorithm bias, which is a well-known and serious risk to implementation of such models.[30] [42] [43] [44] Finally, it will be important to monitor for model drift over time by regularly evaluating the models' predictions against new data and periodically updating the models with new and updated training datasets.

 

MissingLinkText

Conclusion

We found substantial agreement between PCPs' perceptions of who could benefit from an SIC and a machine-learning based tool's prediction of patients as having a high 1-year risk of mortality. Our findings suggest that such tools that predict mortality could potentially serve as a valuable decision support tool for PCPs in daily practice in identifying and prioritizing patients who would benefit from palliative care interventions, including SICs.

 

MissingLinkText

Clinical Relevance Statement

This study can inform machine-learning informed decision support tools used in clinical practice to help PCPs consider which of their patients could most benefit from serious illness interventions.

 

MissingLinkText

Multiple Choice Questions

1. How do physicians' perceptions of the need for a serious illness conversation compare to the predictions of a mortality prediction tool?

   • Little agreement

   • Substantial agreement

   • Perfect agreement

   Correct Answer: The correct answer is option b. There was substantial agreement between the primary care physicians and the machine learning tool, as quantified by Gwet's agreement coefficient of 0.640. This level of agreement supports selecting “substantial agreement” as the answer.

2. What are the characteristics of a patient whom a mortality prediction tool is likely to identify as high risk, but a primary care physician does not believe needs a serious illness conversation?

   • Older age, higher CCI

   • Younger age, lower CCI

   • Younger age, higher CCI

   • Older age, lower CCI

   Correct Answer: The correct answer is option a. Patients whom the PCPs didn't consider as needing an SIC but were labeled as high risk by the tool were older, with an average age of 60.7 years (SD: 19.7) versus 65.6 years (SD: 14.6), and they had a higher CCI, with a mean of 3.00 (SD: 3.19) versus 1.73 (SD: 2.10).

 

MissingLinkText

 

MissingLinkText

Conflict of Interest

L.R. reports serving on the AI Advisory Board for Augmedix Inc, receiving grants from the Agency for Healthcare Research and Quality (AHRQ), the American Medical Association, and FeelBetter Inc, outside the submitted work. D.W.B. reports receiving grants and personal fees from EarlySense, personal fees from CDI Negev, equity from ValeraHealth, equity from Clew, equity from MDClone, personal fees and equity from AESOP, personal fees and equity from FeelBetter, personal fees and equity from Guided Clinical Solutions and grants from IBM Watson Health, outside the submitted work. L.Z. reported receiving grants from the AHRQ, CRICO, IBM Watson Health, and the National Institutes of Health (NIH).

^* Co-first authors.

Protection of Human and Animal Subjects

This study was approved by the Mass General Brigham Institutional Review Board.

• References

• 1 Morrison RS, Meier DE. Clinical practice. Palliative care. N Engl J Med 2004; 350 (25) 2582-2590
• 2 Morrison RS, Dietrich J, Ladwig S. et al. Palliative care consultation teams cut hospital costs for Medicaid beneficiaries. Health Aff (Millwood) 2011; 30 (03) 454-463
• 3 Kavalieratos D, Corbelli J, Zhang D. et al. Association between palliative care and patient and caregiver outcomes: a systematic review and meta-analysis. JAMA 2016; 316 (20) 2104-2114
• 4 Ma J, Chi S, Buettner B. et al. Early palliative care consultation in the medical ICU: a cluster randomized crossover trial. Crit Care Med 2019; 47 (12) 1707-1715
• 5 Temel JS, Greer JA, El-Jawahri A. et al. Effects of early integrated palliative care in patients with lung and GI cancer: a randomized clinical trial. J Clin Oncol 2017; 35 (08) 834-841
• 6 Temel JS, Greer JA, Muzikansky A. et al. Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med 2010; 363 (08) 733-742
• 7 Meier DE, Back AL, Berman A, Block SD, Corrigan JM, Morrison RS. A national strategy for palliative care. Health Aff (Millwood) 2017; 36 (07) 1265-1273
• 8 Meier DE, Beresford L. Outpatient clinics are a new frontier for palliative care. J Palliat Med 2008; 11 (06) 823-828
• 9 Spetz J, Dudley N, Trupin L, Rogers M, Meier DE, Dumanovsky T. Few hospital palliative care programs meet national staffing recommendations. Health Aff (Millwood) 2016; 35 (09) 1690-1697
• 10 Lupu D. American Academy of Hospice and Palliative Medicine Workforce Task Force. Estimate of current hospice and palliative medicine physician workforce shortage. J Pain Symptom Manage 2010; 40 (06) 899-911
• 11 Kelley AS, Bollens-Lund E. Identifying the population with serious illness: the “denominator” challenge. J Palliat Med 2017; 21 (S2): S7-S16
• 12 Kelley AS, Covinsky KE, Gorges RJ. et al. Identifying older adults with serious illness: a critical step toward improving the value of health care. Health Serv Res 2017; 52 (01) 113-131
• 13 Ahluwalia SC, Chen C, Raaen L. et al. A systematic review in support of the national consensus project clinical practice guidelines for quality palliative care, fourth edition. J Pain Symptom Manage 2018; 56 (06) 831-870
• 14 Quill TE, Abernethy AP. Generalist plus specialist palliative care—creating a more sustainable model. N Engl J Med 2013; 368 (13) 1173-1175
• 15 Rosa WE, Izumi S, Sullivan DR. et al. Advance care planning in serious illness: a narrative review. J Pain Symptom Manage 2023; 65 (01) e63-e78
• 16 Paladino J, Brannen E, Benotti E. et al. Implementing serious illness communication processes in primary care: a qualitative study. Am J Hosp Palliat Care 2021; 38 (05) 459-466
• 17 Lakin JR, Block SD, Billings JA. et al. Improving communication about serious illness in primary care: a review. JAMA Intern Med 2016; 176 (09) 1380-1387
• 18 Jacobsen J, Bernacki R, Paladino J. Shifting to serious illness communication. JAMA 2022; 327 (04) 321-322
• 19 Bernacki RE, Block SD. American College of Physicians High Value Care Task Force. Communication about serious illness care goals: a review and synthesis of best practices. JAMA Intern Med 2014; 174 (12) 1994-2003
• 20 Paladino J, Koritsanszky L, Neal BJ. et al. Effect of the serious illness care program on health care utilization at the end of life for patients with cancer. J Palliat Med 2020; 23 (10) 1365-1369
• 21 Paladino J, Bernacki R, Neville BA. et al. Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: a cluster randomized clinical trial of the serious illness care program. JAMA Oncol 2019; 5 (06) 801-809
• 22 Lakin JR, Neal BJ, Maloney FL. et al. A systematic intervention to improve serious illness communication in primary care: effect on expenses at the end of life. Healthc (Amst) 2020; 8 (02) 100431
• 23 Lakin JR, Koritsanszky LA, Cunningham R. et al. A systematic intervention to improve serious illness communication in primary care. Health Aff (Millwood) 2017; 36 (07) 1258-1264
• 24 Lakin JR, Arnold CG, Catzen HZ. et al. Early serious illness communication in hospitalized patients: a study of the implementation of the Speaking About Goals and Expectations (SAGE) program. Healthc (Amst) 2021; 9 (02) 100510
• 25 Bernacki R, Paladino J, Neville BA. et al. Effect of the serious illness care program in outpatient oncology: a cluster randomized clinical trial. JAMA Intern Med 2019; 179 (06) 751-759
• 26 Starfield B, Shi L, Macinko J. Contribution of primary care to health systems and health. Milbank Q 2005; 83 (03) 457-502
• 27 Lakin JR, Desai M, Engelman K. et al. Earlier identification of seriously ill patients: an implementation case series. BMJ Support Palliat Care 2020; 10 (04) e31
• 28 Lakin JR, Robinson MG, Obermeyer Z. et al. Prioritizing primary care patients for a communication intervention using the “surprise question”: a prospective cohort study. J Gen Intern Med 2019; 34 (08) 1467-1474
• 29 Baxter R, Fromme EK, Sandgren A. Patient identification for serious illness conversations: a scoping review. Int J Environ Res Public Health 2022; 19 (07) 4162
• 30 Allen A, Mataraso S, Siefkas A. et al. A racially unbiased, machine learning approach to prediction of mortality: algorithm development study. JMIR Public Health Surveill 2020; 6 (04) e22400
• 31 Chi S, Kim S, Reuter M. et al. Advanced care planning for hospitalized patients following clinician notification of patient mortality by a machine learning algorithm. JAMA Netw Open 2023; 6 (04) e238795
• 32 Deardorff WJ, Barnes DE, Jeon SY. et al. Development and external validation of a mortality prediction model for community-dwelling older adults with dementia. JAMA Intern Med 2022; 182 (11) 1161-1170
• 33 Wang E, Major VJ, Adler N. et al. Supporting acute advance care planning with precise, timely mortality risk predictions. NEJM Catal 2021; 2 (03)
• 34 Zachariah FJ, Rossi LA, Roberts LM, Bosserman LD. Prospective comparison of medical oncologists and a machine learning model to predict 3-month mortality in patients with metastatic solid tumors. JAMA Netw Open 2022; 5 (05) e2214514
• 35 Wang L, Wang Y, Laurentiev J. et al Mortality risk prediction for patients with chronic diseases using electronic health records. In Proceedings of the AMIA Informatics Summit; March 13, 2023; Seattle, WA
• 36 Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 2008; 61 (Pt 1): 29-48
• 37 Manz CR, Parikh RB, Small DS. et al. Effect of integrating machine learning mortality estimates with behavioral nudges to clinicians on serious illness conversations among patients with cancer: a stepped-wedge cluster randomized clinical trial. JAMA Oncol 2020; 6 (12) e204759
• 38 Manz CR, Zhang Y, Chen K. et al. Long-term effect of machine learning-triggered behavioral nudges on serious illness conversations and end-of-life outcomes among patients with cancer: a randomized clinical trial. JAMA Oncol 2023; 9 (03) 414-418
• 39 Wilson PM, Ramar P, Philpot LM. et al. Effect of an artificial intelligence decision support tool on palliative care referral in hospitalized patients: a randomized clinical trial. J Pain Symptom Manage 2023; 66 (01) 24-32
• 40 Footracer KG. Alert fatigue in electronic health records. JAAPA 2015; 28 (07) 41-42
• 41 Rotenstein LS, Holmgren AJ, Downing NL, Bates DW. Differences in total and after-hours electronic health record time across ambulatory specialties. JAMA Intern Med 2021; 181 (06) 863-865
• 42 DeCamp M, Lindvall C. Latent bias and the implementation of artificial intelligence in medicine. J Am Med Inform Assoc 2020; 27 (12) 2020-2023
• 43 Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med 2018; 178 (11) 1544-1547
• 44 Porter AS, Harman S, Lakin JR. Power and perils of prediction in palliative care. Lancet 2020; 395 (10225): 680-681

Publication History

Received: 13 December 2023

Accepted: 17 April 2024

Accepted Manuscript online:
18 April 2024

Article published online:
12 June 2024