Resident-Driven Clinical Decision Support Governance to Improve the Utility of Clinical Decision Support

Abstract

Objectives This resident-driven quality improvement project aimed to better understand the known problem of a misaligned clinical decision support (CDS) strategy and improve CDS utilization.

Methods An internal survey was sent to all internal medicine (IM) residents to identify the most bothersome CDS alerts. Survey results were supported by electronic health record (EHR) data of CDS firing rates and response rates which were collected for each of the three most bothersome CDS tools. Changes to firing criteria were created to increase utilization and to better align with the five rights of CDS. Findings and proposed changes were presented to our institution's CDS Governance Committee. Changes were approved and implemented. Postintervention firing rates were then collected for 1 week.

Results Twenty nine residents participated in the CDS survey and identified sepsis alerts, lipid profile reminders, and telemetry renewals to be the most bothersome alerts. EHR data showed action rates for these CDS as low as 1%. We implemented changes to focus emergency department (ED)-based sepsis alerts to the right provider, better address the right information for lipid profile reminders, and select the right time in workflow for telemetry renewals to be most effective. With these changes we successfully eliminated ED-based sepsis CDS reminders for IM providers, saw a 97% reduction in firing rates for the lipid profile CDS, and noted a 55% reduction in firing rates for telemetry CDS.

Conclusion This project highlighted that alert improvements spearheaded by resident teams can be completed successfully using robust CDS governance strategies and can effectively optimize interruptive alerts.

 

Scroll to top

Keywords

quality improvement - electronic health records - clinical decision support - alert fatigue

Background and Significance

Clinical decision support (CDS) tools include interruptive alerts built into an electronic health record (EHR). These “pop-up” style alerts aim to promote situationally relevant and evidence-based decision-making by reminding clinicians to perform certain tasks, such as order physical therapy, or to alert them of potential consequences associated with performing certain tasks, such as drug–drug interactions.[1] [2] [3] Work by McDonald demonstrated early evidence for the successful implementation of CDS to improve patient care and serve as educational tools.[4] Despite multiple subsequent studies continuing to support the efficacy of CDS, a growing body of data suggests that up to 96% of CDS may be overridden or ignored in clinical practice.[5] [6] [7] [8] [9] [10] [11] One possible contribution to high CDS override rates is alert fatigue, the phenomenon that occurs when an overwhelming number of alerts, or frequently repeating alerts, result in decreased engagement with future alerts.[12] [13] [14] [15] Alert fatigue can contribute to overall clinician burnout from frequent interruptions in workflow, poor usability, and a wearing out of mental energy leading to increased frustration.[16] [17] Currently, only limited data exist to establish a connection between the volume of CDS alerts and frequency of alert override.[5] [12] [18] However, one recent study demonstrated that with each additional CDS per patient encounter, clinicians were 30% less likely to accept a future CDS.[19]

Although CDS applications have been demonstrated to reduce risk for patient harm, poor clinician interactions with CDS tools have the potential to increase risk for patients. As part of efforts to avoid this, we embarked on a resident-driven quality improvement (QI) project within the Wake Forest Internal Medicine Residency to better understand CDS utilization among internal medicine (IM) residents at our tertiary care academic medical center.

 

Scroll to top

Objectives

The goal of this study was to improve the utilization of CDS pop-up alerts by reducing unnecessary firing. Working closely with our hospital's CDS governance committee, we sought to modify the trigger criteria for the most underutilized and subjectively bothersome CDS in efforts to optimize CDS integrity and decrease physician burnout.

 

Scroll to top

Methods

This study was conducted at Atrium Health Wake Forest Baptist Medical Center (AHWFBMC), an 885-bed, regional academic medical center which uses EPIC as the EHR. IM residents were anonymously polled using a Research Electronic Data Capture (REDCap version 12.4.12) survey to identify the most bothersome CDS, the most helpful CDS, and assess for alert fatigue. Residents were chosen for the survey because of their extensive EHR-facing responsibilities. IM residents most often hold the position of primary inpatient team provider and are therefore likely to encounter CDS almost daily. The REDCap survey was distributed to IM residents by email. The top three most bothersome CDS were chosen for further evaluation and intervention by the resident QI project team. The following data were collected from the EHR as part of the evaluation: the number of CDS alerts for each subject matter (i.e., telemetry renewals), CDS firing rates, and the user actions taken for each firing. Our evaluation also involved reviewing the criteria that triggered each CDS to fire and the exclusions/inclusion practice environments and departments.

The data were compiled and presented to the CDS Governance committee at AHWFBMC in order to discuss improvements to CDS design and implementation. The CDS Governance Committee meets monthly and is a multidisciplinary team composed of approximately 10 physician informaticists, nurse informaticists, and analysts. Each of the three CDS tools was further discussed in relation to the five rights of CDS and which elements of the five rights were violated or strained. The five rights of CDS include providing the right information, to the right person, in the right format, through the right channel, in the right time in workflow.[20] Keeping in mind the context of physician workflows, the purpose of the CDS, and the pain points voiced by the IM residents in the anonymous survey, the resident QI project team made recommendations to the CDS Governance team. Expert opinion was voiced by the governance team which aided in shaping and implementing the changes to the alerts. All three implementation plans were then carried out successfully and evaluated in the postimplementation environment.

 

Scroll to top

Results

A total of 29 out of 115 residents responded to the anonymous survey and identified the most burdensome CDS to be sepsis alerts (83%), lipid profile order alerts (69%), and telemetry renewal alerts (62%) as seen in [Fig. 1]. Residents were then asked how often these CDS alerts affected their management by rating how often they read and act upon CDS appropriately. In total, 31% of residents responded that they sometimes read and act upon CDS, 55% responded rarely, and 14% responded never ([Fig. 2]). Free response answers for suggestions on what would help decrease alert fatigue were also assessed. Common themes included decreasing frequency of alerts (31%), limiting alerts to only fire for the primary team responsible for the patient (34%), and converting intrusive to nonintrusive alerts (24%).

Fig. 1 Results of the poll on 29 residents' opinions on which of the listed CDS were most bothersome and most helpful to their workflows. Sepsis alerts, lipid profile, and telemetry reminders were found to be most bothersome, while PT/OT was the only CDS that was voted more helpful than bothersome. “Other” indicates a free response for a CDS that was not otherwise listed. “None” indicates that the respondent found there were no CDS alerts that were bothersome or helpful. PT, physical therapy; OT, occupational therapy; CPAP, continuous positive airway pressure.

Fig. 2 Recorded responses of the 29 residents polled on whether CDS affect their daily practice. 0% (n = 0) responded always, 31% (n = 9) responded sometimes, 55% (n = 16) responded rarely, and 14% (n = 4) responded never.

Prior to evaluating the data for the top three alerts, history was collected regarding the creation, implementation, and maintenance of each alert. All three were custom alerts made by the institution within 3 years of this project. None of the alerts used predictive analytics or predictive models. All three had been implemented without long-term follow-up on outcome metrics.

Data collected from the EHR on the top three alerts revealed excessive numbers of CDS related to the same subject matter with low action rates. For example, there were 22 different telemetry reminders with action rates as low as 1%. A sample of some of these CDS is provided in [Table 1]. Low action rates were also seen with the lipid profile CDS with action taken only 14% of the time ([Table 2]).

Table 1 Provider selection of the options presented for telemetry CDS

Telemetry CDS	Number of alert firings	Follow-up action was taken, n (%)	Provider selected “I am not a member of the patient's primary team,” n (%)
Base has active expiring continuous telemetry monitoring order, chest pain protocol, dysrhythmia protocol	42,085	693 (1.65)	39,781 (94.5)
Base has expiring continuous telemetry monitoring order, no protocols	28,159	1,000 (3.55)	25,205 (89.5)
Base has active continuous telemetry monitoring order, chest pain protocol, dysrhythmia protocol (not on continuous telemetry monitoring unit)	17,759	2,564 (14.4)	14,946 (84.2)
Base has active continuous telemetry monitoring order (not on continuous telemetry monitoring unit)	10,693	2,573 (24.1)	7,863 (73.5)
Base has active chest pain and dysrhythmia protocols, but no continuous telemetry monitoring order (on continuous telemetry monitoring unit)	7,702	100 (1.3)	7,226 (93.8)
Base has expiring continuous telemetry monitoring order, dysrhythmia protocol	639	8 (1.25)	603 (94.4)

Abbreviation: CDS, clinical decision support.

Note: Examples of exact breakdowns are listed below based upon the indication for the telemetry ordered and the unit.

Table 2 Provider selection of the lipid profile popup advisory. 74.8% of providers chose they were not a member of the primary care team, 11.0% chose appropriate labs had already been drawn, and 14.2% ordered a lipid panel in response to the CDS alert

Lipid profile CDS	Number of alert firings	Provider ordered lipid profile, n (%)	Provider selected “I am not a member of the patient's primary care team,” n (%)	Provider selected “Appropriate labs drawn,” n (%)
Base lipid profile for acute myocardial infarction (NSTEMI/STEMI)	31,455	4,468 (14.2)	23,541 (74.8)	3,446 (11.0)

Abbreviations: CDS, clinical decision support; NSTEMI, non-ST elevation myocardial infarction; STEMI, ST elevation myocardial infarction.

Based on survey responses and input from the CDS Governance Committee, changes were made to each of the above CDS alerts. For emergency department (ED)-based sepsis alerts (those that fire for patients physically located in the ED), we changed the target audience and only allowed the CDS to fire when a provider was logged onto the EHR under an emergency medicine context ([Appendix Fig. A1], available in the online version). The lipid profile CDS criteria were changed to better align with the definition of acute coronary syndrome. The original firing criteria required the following: STEMI or NSTEMI listed on hospital problem list or a high-sensitivity troponin value of greater than 18 and no lipid profile results in the last 24 hours ([Appendix Fig. A2], available in the online version). These criteria were changed to the addition of chest pain to the elevated troponin and expansion of last lipid profile to be within 1 month rather than 1 day. Telemetry reminders were limited to firing only during daytime hours ([Appendix Fig. A3], available in the online version).

Data were collected 1 week after implementation of these changes. Lipid profile prior to alteration over a 1-week period totaled 889 instances decreasing to 27 for a 97% reduction after changes to the CDS. Telemetry notifications prior to alteration to firing criteria totaled 910 over 1 week, which decreased to 496 instances postchanges resulting in a 55% reduction ([Fig. 3]). Changing the target audience to exclude IM login contexts for ED-based sepsis alerts successfully prevented all alerts (100%) to IM providers.

Fig. 3 The number of times the alert was triggered in 1 week for an internal medicine provider before and after implemented changes. There was a 97% reduction in lipid profile CDS firing and 55% reduction in telemetry CDS firing.

 

Scroll to top

Discussion

The utilization of interruptive alerts, although one of many tools in a CDS tool kit, has steadily become a prominent mechanism in aiding providers with making clinical decisions.[21] While these alerts are aimed to improve patient care, reduce poor outcomes, and decrease health care costs, the number of reminders currently implemented in the electronic medical record creates a barrier to meeting these goals while also leading to alert fatigue and burnout.[22] [23] These findings of frustration and disillusion with intrusive alerts were present among our IM residents. We used the question, “how often do you read and act on CDS appropriately,” as a surrogate marker for assessing alert fatigue and found that none of the residents surveyed were always acting on CDS appropriately. Free response answers regarding the utilization of CDS also supported our hypothesis of frustration among users. By examining the infrastructure of select CDS, we were able to address some of the suggestions made by the residents. We used a survey that had residents identify both the most burdensome and the most helpful CDS tools so that we could target our interventions to achieve the greatest effect in enhancing CDS effectiveness. We identified that the most burdensome CDS to IM residents were sepsis alerts. A variety of sepsis alerts exist and were originally intended to target certain provider populations. Sepsis alerts for patients being treated in the emergency room are most useful to emergency room providers who must take early action to identify and treat patients with sepsis. However, the CDS fires across all specialties who open an ED patient's chart, missing its ideal target audience. By narrowing the provider selected to view this alert, we are able to better target it to those who are most likely to act on the information. Similarly, telemetry reminders are another high-volume CDS with extremely low action rates. Residents are often bothered by these alerts at inopportune times, such as during night shift. By implementing a restriction on the firing time of telemetry CDS, we focused the target audience on the appropriate day team providers. Lastly, the lipid profile CDS was designed to help meet the American Heart Association guideline recommending a fasting lipid profile in patients with acute coronary syndromes (ACS) within 24 hours of presentation.[24] However, the previous CDS inclusion guidelines were so broad that the alert fired for many patients without an ACS event. By narrowing the inclusion criteria, we were able to more accurately select the cohort of patients with ACS presentations. Collaboration with our institution's Alert Governance Committee was key to successfully implementing these changes.

This project highlighted that alert improvements can be spearheaded by resident teams and can be completed successfully using robust CDS governance strategies described by several well-established governance entities.[25] [26] Resident physicians can and should be involved in CDS governance given their extensive patient-facing and EHR-facing experiences. It is also reasonable to conclude that other clinicians including attending physicians, nurses, and therapists can be useful advocates in this field for similar reasons.

This study has several important limitations. First, the primary outcome studied was CDS firing rates. We do not know the impact that these CDS criteria changes had on patient outcomes or rates of appropriate CDS dismissals and further study is needed in this area. Furthermore, advanced CDS performance measures, such as sensitivity, specificity, and positive predictive values were not performed, but in the future may provide additional support for our findings.

Further studies in this field are needed to determine more effective means for delivering CDS tools including balancing the extent of interruptive alerts with maximizing workflow efficiency. Thought should be given to better designed opt-outs to improve response accuracy. We should continue to involve residents and other clinicians in these improvement projects as well as work to integrate them into all parts of CDS design, build, implementation, and follow-up.

 

Scroll to top

Conclusion

In this resident-driven QI project, we focused on alerts that were subjectively identified by IM residents to cause the most disruption to their workflows. By examining the criteria that makes each of our targeted CDS alerts fire, we were able to identify ways to enhance the appropriateness of each alert. Working closely with our institution's CDS Governance Committee, we were able to successfully implement these changes and saw an effective decrease in alert firing rates. Our success supports that resident physicians are assets to CDS governance panels.

 

Scroll to top

Clinical Relevance Statement

The number of intrusive alerts used in electronic medical records creates a barrier to meeting the original aims while also leading to alert fatigue and burnout. However, by examining the infrastructure of these alerts, we can better align them with clinical decision support governance strategies to reduce these negative effects. Resident physicians can be an asset in the creation and implementation of these changes.

 

Scroll to top

Multiple Choice Questions

1. Which of the following are part of the five rights of clinical decision support?

   • The right information

   • The right person

   • The right time in workflow

   • All of the above

2. Which of the following has been a consistent unintended consequence of CDS tools if not constructed appropriately?

   • Alert fatigue

   • Patient safety

   • Increased resource utilization

   • Cost

3. Clinical decision support tools can be used for which of the following?

   • Drug interactions

   • Lab monitoring

   • Consult suggestions

   • All of the above

 

Scroll to top

Multiple Choice Answers

1. d

   The five rights of clinical decision support consist of providing the right information (answer choice a) to the right person (answer choice b) in the right format, through the right channel, in the right time in workflow (answer choice c), thus the best answer is all of the above (answer choice d).

2. a

   Clinical decision support tools can be intentionally used to improve patient care and safety, improve resource utilization, and decrease cost. However, when not constructed appropriately, they can lead to the unintended consequence of alert fatigue (answer choice a).

3. d

   Clinical decision support tools can be used for a variety of reasons including reminding clinicians to perform certain tasks, such as ordering physical therapy, monitoring labs, renewing orders, or requesting consults (answer choices b and c), or alerting them to potential consequences associated with performing certain tasks, such as drug-drug interactions (answer choice a). Thus, the best answer is all of the above (answer choice d).

 

Scroll to top

 

Scroll to top

Conflict of Interest

None declared.

Ethical Approval

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by AHWFBMC Institutional Review Board.

• References

• 1 Fry C. Development and evaluation of best practice alerts: methods to optimize care quality and clinician communication. AACN Adv Crit Care 2021; 32 (04) 468-472
• 2 Baird A, Kibbe B, Lesandrini J. Stakeholder bias in best practice advisories: an ethical perspective. JAMIA Open 2020; 3 (02) 142-145
• 3 Backman R, Bayliss S, Moore D, Litchfield I. Clinical reminder alert fatigue in healthcare: a systematic literature review protocol using qualitative evidence. Syst Rev 2017; 6 (01) 255
• 4 McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med 1976; 295 (24) 1351-1355
• 5 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 6 Greene RA, Zullo AR, Mailloux CM, Berard-Collins C, Levy MM, Amass T. Effect of best practice advisories on sedation protocol compliance and drug-related hazardous condition mitigation among critical care patients. Crit Care Med 2020; 48 (02) 185-191
• 7 Presti Jr J, Alexeeff S, Horton B, Prausnitz S, Avins AL. Changing provider PSA screening behavior using best practice advisories: interventional study in a multispecialty group practice. J Gen Intern Med 2020; 35 (Suppl. 02) 796-801
• 8 Chanas T, Volles D, Sawyer R, Mallow-Corbett S. Analysis of a new best-practice advisory on time to initiation of antibiotics in surgical intensive care unit patients with septic shock. J Intensive Care Soc 2019; 20 (01) 34-39
• 9 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 10 Schedlbauer A, Prasad V, Mulvaney C. et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior?. J Am Med Inform Assoc 2009; 16 (04) 531-538
• 11 Bedoya AD, Clement ME, Phelan M, Steorts RC, O'Brien C, Goldstein BA. Minimal impact of implemented early warning score and best practice alert for patient deterioration. Crit Care Med 2019; 47 (01) 49-55
• 12 Murad DA, Tsugawa Y, Elashoff DA, Baldwin KM, Bell DS. Distinct components of alert fatigue in physicians' responses to a noninterruptive clinical decision support alert. J Am Med Inform Assoc 2022; 30 (01) 64-72
• 13 Jankovic I, Chen JH. Clinical decision support and implications for the clinician burnout crisis. Yearb Med Inform 2020; 29 (01) 145-154
• 14 Li C, Parpia C, Sriharan A, Keefe DT. Electronic medical record-related burnout in healthcare providers: a scoping review of outcomes and interventions. BMJ Open 2022; 12 (08) e060865
• 15 Chaparro JD, Beus JM, Dziorny AC. et al. Clinical decision support stewardship: best practices and techniques to monitor and improve interruptive alerts. Appl Clin Inform 2022; 13 (03) 560-568
• 16 Tajirian T, Stergiopoulos V, Strudwick G. et al. The influence of electronic health record use on physician burnout: cross-sectional survey. J Med Internet Res 2020; 22 (07) e19274
• 17 Tai-Seale M, Baxter S, Millen M. et al. Association of physician burnout with perceived EHR work stress and potentially actionable factors. J Am Med Inform Assoc 2023; 30 (10) 1665-1672
• 18 Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the Meaningful Use era: no evidence of progress. Appl Clin Inform 2014; 5 (03) 802-813
• 19 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 20 Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag 2009; 23 (04) 38-45
• 21 Kwan JL, Lo L, Ferguson J. et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020; 370: m3216
• 22 Embi PJ, Leonard AC. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Inform Assoc 2012; 19 (e1): e145-e148
• 23 Hussain MI, Reynolds TL, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc 2019; 26 (10) 1141-1149
• 24 Amsterdam EA, Wenger NK, Brindis RG. et al. 2014 AHA/ACG guideline for the management of patients with non-ST-elevation acute coronary syndromes: executive summary. Circulation 2014; 130 (25) 2354-2394
• 25 McCoy AB, Russo EM, Johnson KB. et al. Clinician collaboration to improve clinical decision support: the Clickbusters initiative. J Am Med Inform Assoc 2022; 29 (06) 1050-1059
• 26 Kawamanto K, Flynn MC, Kukhareva P. et al. A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: a case study. AMIA Annu Symp Proc 2018; 2018: 624-633

Publication History

Received: 16 October 2023

Accepted: 12 March 2024

Article published online:
01 May 2024