Lessons Learned from Creating Alert Governance during an Electronic Health Record Transition

Abstract

Background Our multihospital transition to a single electronic health record (EHR) provided an opportunity to transform alert governance. Our case provides insights into the unique challenges and opportunities of creating governance during a transition to meet both implementation and future alert management needs.

ObjectivesThis case report describes the efforts of UW Medicine Information Technology Services to advance alert governance during EHR transition and highlights the opportunities to improve care quality and provider experience within a changing environment.

Methods We used a multidisciplinary approach and external evidence to define governance for provider-facing interruptive alerts. We established the context for our governance efforts with a systemic environmental scan. We used literature review and expert consultation to determine alert design and performance best practices, sought to design postimplementation evaluation tools, and engaged clinical stakeholders to help with decision-making.

Results We created alert design and implementation tools and an alert approval process, eliminating 21 unnecessary alerts prior to implementation. We developed prototype evaluation metrics and enlisted clinical owners for postimplementation optimization of 221 alerts.

Conclusion We leveraged the fluid environment of our EHR implementation to rapidly build a provider-led governance infrastructure to meet immediate transitional needs and to facilitate future alert maintenance and improvement.

Keywords

clinical decision support - alerting - governance

Background and Significance

UW Medicine health system includes a quaternary care hospital, trauma center, two community hospitals, and 15 neighborhood clinics, providing more than 1.6 million total annual patient visits. In March 2021, we replaced a hybrid of two inpatient electronic health record (EHR) systems (Cerner Millennium and Soarian) with the EHR already deployed in our ambulatory clinics, Hyperspace (Epic Systems). This transition from hybrid to a single system required decisions about how inpatient alerts would be designed and monitored. Previously, alert oversight was inconsistent and fragmented. The system-wide goal of improving provider experience by minimizing workflow interruptions prompted the clinical informatics team to transform clinical decision support (CDS) governance during Epic implementation.

A basic approach to CDS implementation, including alerts, consists of installing vendor-recommended default features, followed by post-go-live optimization. However, this approach may require significant rework, and change fatigue typically limits ongoing improvements after go-live. Also, alerts needed for clinical reasons from legacy systems still require thoughtful adaptation. Furthermore, EHR vendor-provided and contracted implementation teams are transient. These provisional teams increase the risk of inconsistent decision-making, variable engagement with clinical subject matter experts (SMEs), and the loss of information important to postimplementation alert management. To address these, we invested in early planning, new governance structures, and consistent interactions with SMEs.

Objectives

This case report describes the efforts of UW Medicine Information Technology Services to advance alert governance during an EHR transition and highlights the opportunities to improve care quality and provider experience within a changing environment.

Methods

Our project timeline was from March 2020 to EHR go-live in March 2021. During this period, our work progressed from analysis of the current state and determining best practices, to development of tools and processes implemented to both address current-state challenges and ensure consistent post-go-live alert governance.

Analysis

We convened a multidisciplinary physician-led EHR alert workgroup, including physicians, nurses, pharmacists, compliance and health information management representatives, and analysts. The project scope was limited to provider-targeted interruptive inpatient alerts; alerts involving drug–drug interactions (DDIs) were considered out of scope. We designed an approach that established informatics oversight of alerts, including: (1) developing and documenting alert design standards, (2) evaluating alerts not yet approved with the help of clinical stakeholders, (3) prototyping and validating alert evaluation metrics in our current ambulatory EHR, and (4) identifying and assigning clinical owners to new alerts.

The team conducted a review of the literature to identify best practices, examined practices reported at conferences,[1] [2] and recruited external experts in the field to assist with decision-making (A.W.). Applying quality improvement methodology,[3] [4] we created a key driver diagram to articulate our aims with primary and secondary drivers ([Fig. 1]).

Fig. 1 Key driver diagram for optimizing implementation of interruptive provider-facing alerts.

We obtained a baseline alert volume from our legacy EHR data dashboard (Cerner LightsOn Network), including the total number of alerts fired and the percent of alerts overridden (of those that could be overridden). We engaged our new vendor and local experts to better understand alert evaluation tools. We then assessed the state of the vendor's emerging alert build, including determining stakeholders and identifying historical decisions related to the new alerts.

Data Synthesis and Tool Development

Approximately 150 inpatient alerts had been built by vendor analysts in an Epic test environment when the alert workgroup was formed. These included Epic versions of existing Cerner alerts and Epic “Foundation” alerts, selected from stock alerts used for many clients. Some alerts had not yet been reviewed and approved by clinical specialty project groups. The team recognized that with many of the alerts already built in the testing environment, the most productive approach to implementing governance mid-project would be to focus on those alerts still needing review and approval. Additionally, throughout the project additional alerts built by siloed vendor-supported local application teams were brought to our awareness. We incorporated any provider-facing interruptive alerts into our project scope, increasing the total to approximately 250 alerts managed. Our workgroup met weekly to develop alert design standards informed by our environmental scan and applied these standards to the unapproved alerts.

We developed clinical ownership standards informed by the roles and expertise needed to provide clinical oversight for each alert. We built a framework to enlist owners by engaging divisional leadership and individuals to educate them on our workgroup's role and solicit designations for alert ownership. We then identified clinical owners for all approved alerts. Clinical owners were selected so that front-line recipients of the alerts could directly influence their design and performance.

Based on our baseline environmental scan of alerts and our survey of the literature, we developed methods to evaluate the alert frequency, responses, and firing patterns in both the implementation stage and in the future postimplementation period. As our organization had previously implemented Epic in our ambulatory clinics, we had a ready data source to pilot our alert evaluation metrics before rollout in the inpatient setting. We explored alert configurations in Epic's test environment to map how back-end metrics related to front-end usability and functionality.

Results

Alert Design Standards

We developed an algorithm that takes into account clinical functionality, usability, and safety/regulatory needs, as shown in [Fig. 2]. We used this algorithm to evaluate existing and vendor-proposed alerts ([Table 1] and [Fig. 2]). Then, we developed a prototype evaluation and improvement model to manage the alert life cycle after go-live ([Fig. 3]).

Fig. 2 New alert implementation decision algorithm.

Fig. 3 Existing alerts evaluation and improvement schema for postimplementation governance.

Table 1 Alert standards checklist

1. Confirm that an alert is needed

Can the desired outcome be accomplished without using an alert?

Is there an effective noninterruptive alternative mechanism to achieve the same action (e.g., order set modifications or information displayed elsewhere in the chart)?

Is the desired effect better achieved outside the EHR altogether (by addressing workflow of communication gaps)

2. Determine if the context of the alert request is fully understood

Is the alert purpose for patient safety/risk management?

Is the alert a regulatory driver?

3. Optimize the technical and clinical alert design

What are the proposed criteria and triggers for the alert?

Is the alert clinically valid (promoting actions based in standard of care)?

Is the alert actionable and the information/decision mechanism included in the alert design (e.g., actionable laboratory values and a medication order embedded in the alert)?

4. Estimate the positive predictive value of the alert

What is the likelihood of the alert firing inaccurately or imprecisely?

What are the consequences of poor alert targeting (e.g., alert fatigue, unintended actions)?

5. Is an alert still the best option after answering questions 1–4? If yes: plan to build the alert, and consider:

Has the alert interface itself been optimized (e.g., intuitive design, understandable text)?

Would it be beneficial to fire the alert silently at first and monitor for unintended effects?

Abbreviation: EHR, electronic health record.

Alert Evaluation

Using standards shown in [Table 1], we evaluated new proposed alerts brought to the group during implementation, made recommendations for alert design and triggers, and developed a rubric to approve or reject build proposals ([Fig. 2]). Based on this evaluation and by engaging with clinical and administrative stakeholders, we removed 21 additional alerts from the implementation queue.

Individual and Overall Alert Performance Reports

Our analysis of baseline alerts using Cerner LightsOn Network showed a mean monthly burden of 604,029 (±22,708) inpatient alerts per month in the 12-month period between March 1, 2020 and March 1, 2021. Most alerts were of the “open chart” non-overridable type. Of the alerts that could be overridden (46,579 ± 1,263 per month), the percentage overridden was never below 90% (91.2 ± 0.19).

We queried the enterprise data warehouse to analyze attributes of ambulatory alerts, and designed prototype reports for monitoring alerts. Our variables were selected to reflect alert technical performance and functional user interactions. We prototyped three broad metric categories: basic descriptive data, alert firing contextual data, and user interaction data ([Table 2]).

Table 2 Prototype alerts evaluation metrics

Basic information

● Time period analyzed  ● Alert ID  ● Alert description  ● Total # times fired  ● Average # times fired per day  ● Distinct # of alerted encounters  ● Distinct # of alerted patients  ● Distinct # of alerted EHR users  ● # of times EHR users interacted with the alert interface  ● Percentage of times EHR users interacted with the alert interface

Breakdown of alert firings

● By trigger (where in the EHR workflow did the user see the alert?)  ● By specialty  ● By EHR user role (and clinician title if applicable)  ● By facility  ● By encounter type

Breakdown of user interactions with alerts

● By user action type (e.g., accept, cancel, override, follow a link, add, or remove an order, etc.)  ● By override reason  ● Free text comment where applicable

Abbreviation: EHR, electronic health record.

Alert Owners

As part of our future alert life cycle, we identified the need for a maintenance phase for alerts and continuous engagement of clinical stakeholders. We engaged SMEs for targeted consultation on 221 interruptive provider-facing alerts being implemented at go-live. Each alert was assigned to one or more SMEs to serve as the future “owner” of the alert after implementation ([Table 3]). Alert owners were selected based on their role and context in the EHR, clinical expertise in the workflow impacted by the alert, and user role within the organization (e.g., designation by the departmental chair). Owners will be responsible for monitoring alerts metrics and usability/clinical feedback to further refine or retire alerts ([Fig. 3]).

Table 3 Alert owners checklist

1. Determine the ownership model for the alert: single owner or co-ownership

Will the proposed owner be an expert in the alert's clinical workflow?

Do the alert trigger criteria include multiple user roles and specialty domains (e.g., all providers from multiple specialties)?

Does the alert content encompass detailed specialty knowledge from more than one clinical domain (e.g., pharmacy/pathology and a clinical subspecialty like infectious disease?)

Does the alert address a nonclinical function for which clinical context is still important (e.g., a Joint Commission regulatory reporting requirement)

2. Determine the optimal individual(s) to own the alert

Will the proposed owner be a recipient of the alert?

Will the proposed owner be an expert in the alert's clinical workflow?

Do the alert trigger criteria include multiple user roles and specialty domains (e.g., all providers from multiple specialties)?

3. Determine the appropriate pathway to designate an owner(s)

Is there a departmental or divisional oversight committee for the alert's domain?

If an individual rather than a committee is appropriate, who should recommend that person?

Discussion

Interruptive alerts in our legacy commercial EHR suffered from three key problems: excessive use of “open chart” alerts, very high clinician override rates, and a lack of regular review resulting in outdated alerts. Ad hoc decision-making, a lack of systematic alert performance analysis, variable involvement of front-line clinicians, and a lack of structured alerts life cycle management contributed to inefficient use of alerts. Recent work by Orenstein et al describes a framework for evaluating the maturity of CDS governance by domains of content, analytics, and management. Within this framework, our institution was functioning at the lower levels of the 5-stage model.[5] To overcome these gaps and to avoid introducing new ineffective alerts, we redesigned alert governance during the build phase of our EHR implementation project. We transferred the transient oversight infrastructure into our system's new integrated information technology (IT) services governance to preserve domain-specific expertise. Our intention to rapidly advance CDS maturity was also loosely inspired by the Lewin's theory of change, in which established organizational processes must unfreeze before changing and refreezing.[6] The EHR conversion inherently unfroze CDS design decisions and governance, creating a unique opportunity to comprehensively and rapidly redesign processes at a higher level of CDS maturity.

A growing body of literature suggests that EHR conversions pose inherent risks to CDS performance.[7] The new EHR we were in the process of adopting has been shown elsewhere to worsen DDI alert burden,[8] to displace helpful homegrown CDS,[9] to disrupt safe dosing of pediatric medications,[10] to interfere with severe allergy alerts,[11] and to lower physician satisfaction with CDS.[12] We sought to apply the lessons from these studies and anticipated introducing alert governance before go-live would have twofold benefits. First, we could instill a new culture of provider ownership of alerts. Second, we hoped to avoid the unintended consequences of installing new vendor-default alert configurations not suited to our health system.

During an EHR transition and postimplementation optimization, the cross-domain nature of EHR alerts calls for broad provider engagement. Unlike order sets and documentation templates, many alerts affect large user groups across multiple specialties and pose the risk of low positive predictive value (PPV) if not mindfully designed.[13] Likewise, the purpose of alerts may be several-fold. Alerts may be informational, may promote desired actions, or provide regulatory notifications. Carefully considering alternative noninterruptive CDS and targeting alerts is crucial to avoid alert fatigue and to ensure alerts promote desired actions.[14] [15]

While vendor implementation experts bring expertise in their system's capabilities and experience with other institutions, one-size-fits-all solutions do not often meet the needs of individual health systems. Local IT and clinical experts are essential partners, as they bring an understanding of institutional systems and practices and an appreciation of the unique needs of patient population subsets. Specialized local governance mechanisms are needed to engage with vendor-based technical teams to ensure that alerts help the organization meet its objectives and that there is a smooth transfer of knowledge after the implementation.

A growing body of work supports that alerts contribute to EHR user burnout.[16] Once implemented, alerts need to be periodically evaluated to determine if they should remain in place, be altered, or be eliminated/replaced with a different tool. Once alerts are created, organizations may be hesitant to turn them off due to safety or regulatory concerns, despite high burden on clinicians.[17] Another pitfall to avoid is the tendency of one party to target another by designing alerts “at” them instead of addressing problems that would be better handled outside of the EHR. Governance is needed to provide evidence-based perspective to such concerns and drive consistent decision-making, and the process can benefit from an application of quality improvement methodologies.[14]

We were able to prospectively prototype alert metrics using our current ambulatory implementation of the same EHR. This provided valuable insights into current state alert utilization to inform our postimplementation monitoring. Interpretation of alerts' PPV and negative predictive value can be time consuming due to the need to validate with chart review. Furthermore, clinical owners require training to understand how to use alert metrics to improve alert performance.[18] Metric prototyping also revealed that alert comments (such as override reasons) are not routinely monitored despite providing a valuable source for alert feedback.

An important limitation of our work is that our EHR implementation went live on March 27, 2021, and we are in the process of collecting our performance measures. Time and future evaluation will give additional insights into the strengths and opportunities for improvement in our approach to creating alert governance during an EHR transition.

Conclusion

In conclusion, we found that accomplishing alert governance during an EHR implementation enabled the establishment of standards, reduced unnecessary implementation of alerts, and engaged clinical stakeholders in ways that can support successful long-term alert maintenance and improvement.

Clinical Relevance Statement

This article provides lessons learned and guidance for hospital IT leaders to consider when planning a transition to a new EHR, or for other times of a change environment in the health system that may present an opportunity to implement new or updated governance infrastructure.

Conflict of Interest

None declared.

Acknowledgments

We would like to thank our alert workgroup members, advisors, and the institutional sponsors of this project: Elena Meeker, Marnell Crane, Adam Wilcox, Susanna Myers, Andrew M. Walters, Todd L. Burstain, Matt Smitherman, Anna Marti, Daniel B. Martin, B. Jane Fellner, Eric Neil, Sally Beahan, Jose Luis Mejia, Nick Phillips, Evan Lineberger, Kristi Raschka, and all the clinical subject matter experts and future alert owners enlisted.

Author Contributions

All authors contributed to the work in accordance with the criteria established by the International Committee of Medical Journal Editors.

Protection of Human and Animal Subjects

Human subjects were not included in this project.

• References

• 1 Chaumeton A, Clune C, Tarrago R. Unleashing the Healthcare Revolution. Presentation presented at the: AMIA Clinical Informatics Conference; May 2020; Virtual Conference
• 2 Lourie E, Stevens L, Webber EC, Hribar MR. Addressing EHR-Related Burnout and Measuring Success. Presentation presented at the: AMIA Clinical Informatics Conference; May 2020; Virtual Conference
• 3 Teich ST, Faddoul FF. Lean management-the journey from Toyota to healthcare. Rambam Maimonides Med J 2013; 4 (02) e0007
• 4 Driver Diagram | IHI - Institute for Healthcare Improvement.. Accessed March 3, 2021 at: http://www.ihi.org:80/resources/Pages/Tools/Driver-Diagram.aspx
• 5 Orenstein EW, Muthu N, Weitkamp AO. et al. Towards a maturity model for clinical decision support operations. Appl Clin Inform 2019; 10 (05) 810-819
• 6 Hussain ST, Lei S, Akram T, Haider MJ, Hussain SH, Ali M. Kurt Lewin's change model: a critical review of the role of leadership and employee involvement in organizational change. J Innovat Knowledge 2018; 3 (03) 123-127
• 7 Huang C, Koppel R, McGreevey III JD, Craven CK, Schreiber R. Transitions from one electronic health record to another: challenges, pitfalls, and recommendations. Appl Clin Inform 2020; 11 (05) 742-754
• 8 Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record.. J Gen Intern Med 2018; 33 (11) 1868-1876
• 9 Chaudhry AP, Samudrala S.. Provider survey on automated clinical decision support system for cardiovascular risk assessment.. AMIA Joint Summits On Translational Science Proceedings 2019; 64-71
• 10 Whalen K, Lynch E, Moawad I, John T, Lozowski D, Cummings BM. Transition to a new electronic health record and pediatric medication safety: lessons learned in pediatrics within a large academic health system.. J Am Med Inform Assoc 2018; 25 (07) 848-854
• 11 Rukasin CRF, Henderlight S, Bosen T, Nelson SD, Phillips EJ. Implications of electronic health record transition on drug allergy labels. J Allergy Clin Immunol Pract 2020; 8 (02) 764-766
• 12 Krousel-Wood M, McCoy AB, Ahia C. et al. Implementing electronic health records (EHRs): health care provider perceptions before and after transition from a local basic EHR to a commercial comprehensive EHR. J Am Med Inform Assoc 2018; 25 (06) 618-626
• 13 Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of decision support in computerized provider order entry: systematic literature review. JMIR Med Inform 2018; 6 (01) e3
• 14 Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020; 11 (01) 46-58
• 15 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 16 Jankovic I, Chen JH. Clinical decision support and implications for the clinician burnout crisis. Yearb Med Inform 2020; 29 (01) 145-154
• 17 Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce ‘alert fatigue’ while still minimizing the risk of litigation. Health Aff (Millwood) 2011; 30 (12) 2310-2317
• 18 McGreevey III JD, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing alert burden in electronic health records: state of the art recommendations from four health systems. Appl Clin Inform 2020; 11 (01) 1-12

Publication History

Received: 09 April 2021

Accepted: 11 February 2022

Article published online:
29 June 2022