Evaluating Equity in Usage and Effectiveness of the CONCERN Early Warning System

Background: The CONCERN Early Warning System (CONCERN EWS) is an artificial intelligence based clinical decision support system (AI-CDSS) for prediction of clinical deterioration leveraging signals from nursing documentation patterns. While a recent multi-site randomized controlled trial demonstrated its effectiveness in reducing inpatient mortality and length of stay, evaluating implementation outcomes is essential to ensure equitable results across patient populations. Objectives: 1) assess whether clinicians’ usage of the CONCERN EWS, as measured by CONCERN Detailed Prediction Screen launches, varied by patient demographic characteristics, including sex, race, ethnicity, and primary language; 2) evaluate whether CONCERN EWS’s effectiveness in reducing the risk of in-hospital mortality varied across patient demographic groups. Methods: We conducted a retrospective observational analysis of electronic health records log files and clinical outcomes from a multi-site pragmatic cluster-randomized controlled trial involving four hospitals across two healthcare systems. Equity in usage was assessed by comparing CONCERN Detailed Prediction Screen launches across demographic groups, and effectiveness was examined by comparing the risk of in-hospital mortality between intervention and usual care groups using Cox proportional hazards models adjusted for patient characteristics. Results: Clinicians’ CONCERN Detailed Prediction Screen launches did not significantly differ by patients’ demographic characteristics, suggesting equitable usage. The CONCERN EWS significantly associated with reduced risk of in-hospital mortality overall (adjusted hazard ratio [HR] = 0.644, 95% CI: 0.532–0.778, p < 0.0001), with consistent effectiveness across most groups. Notably, patients whose primary language was not English experienced a greater reduction of mortality risk compared to patients whose primary language was English (adjusted HR = 0.419, 95% CI: 0.287–0.610, p = 0.0082). Conclusions: This study presents a case of evaluating equity in AI-CDSS usage and effectiveness, contributing to the limited literature. While findings suggest equitable engagement and effectiveness, ongoing evaluations are needed to understand the observed variability and ensure responsible implementation.

Publikationsverlauf

Eingereicht: 21. Februar 2025

Angenommen nach Revision: 07. Juni 2025

Accepted Manuscript online:
10. Juni 2025