Drug Alert Experience and Salience during Medical Residency at Two Healthcare Institutions

Abstract

Background Drug alerts are clinical decision support tools intended to prevent medication misadministration. In teaching hospitals, residents encounter the majority of the drug alerts while learning under variable workloads and responsibilities that may have an impact on drug-alert response rates.

Objectives This study was aimed to explore drug-alert experience and salience among postgraduate year 1 (PGY-1), postgraduate year 2 (PGY-2), and postgraduate year 3 (PGY-3) internal medicine resident physicians at two different institutions.

Methods Drug-alert information was queried from the electronic health record (EHR) for 47 internal medicine residents at the University of Pennsylvania Medical Center (UPMC) Pinnacle in Pennsylvania, and 79 internal medicine residents at the MetroHealth System (MHS) in Ohio from December 2018 through February 2019. Salience was defined as the percentage of drug alerts resulting in removal or modification of the triggering order. Comparisons were made across institutions, residency training year, and alert burden.

Results A total of 126 residents were exposed to 52,624 alerts over a 3-month period. UPMC Pinnacle had 15,574 alerts with 47 residents and MHS had 37,050 alerts with 79 residents. At MHS, salience was 8.6% which was lower than UPMC Pinnacle with 15%. The relatively lower salience (42% lower) at MHS corresponded to a greater number of alerts-per-resident (41% higher) compared with UPMC Pinnacle. Overall, salience was 11.6% for PGY-1, 10.5% for PGY-2, and 8.9% for PGY-3 residents.

Conclusion Our results are suggestive of long-term drug-alert desensitization during progressive residency training. A higher number of alerts-per-resident correlating with a lower salience suggests alert fatigue; however, other factors should also be considered including differences in workload and culture.

Keywords

drug alerts - drug fatigue - desensitization - salience rate - clinical decision support system

Background and Significance

The use of electronic health records (EHRs) has substantially transformed the practice of medicine for all disciplines and subspecialties. The digitalization of medical records has enabled timely medical documentation and has access to large volumes of patient health information. Healthcare providers can now place orders, consult, and communicate treatment plans with other specialists from local or remote workstations. The U.S. government has incentivized the adoption of health information technology (HIT), with legislations, such as the Health Information Technology for Economic and Clinical Health (HITECH, 2009), which have further augmented the use of HIT.[1] [2] [3] Currently, more than 70% of healthcare systems in the United States have implemented EHRs among their institutions.[4] [5]

The increasing adoption of HIT has preceded the development of clinical decision support (CDS) systems which encompass an array of software tools aiming to improve physician performance.[6] CDS systems, now largely available across healthcare institutions, incorporate various features such as documentation templates, clinical guidelines, diagnostic support, drug alerts, and reminders.[7] [8] [9] [10] The utilization of CDS has demonstrated different advantages for improving practitioner performance and clinical care, including prevention of harmful prescriptions[11] and reducing adverse drug events.[8] [10] These advantages, however, do not come without associated risks of legal liability pointed out by physician making clinical diagnosis,[12] risks of alert fatigue, and negative impact on user skills by reliance on alert system, as well as limitation of identifying appropriate alerts.[13] [14] Drug alerts are triggered when a medical order is incompatible with a set of parameters which may include dose ranges, frequency of administration, associated allergies, and medical conditions, among others.[15] Healthcare providers may respond to drug alerts either by overriding or acknowledging the alert via modification or cancellation of an order. Drug alerts, for example, can be inappropriate, or may hamper workflow when excessive. Alerts may not account for the complete clinical context in which a medical order is placed in the computerized physician order entry system (CPOE).[16]

CDS acceptance is defined as compliance with the displayed CDS recommendations. Salience is the percentage of alerts that a provider acknowledged by either a change or removal of the order[17] and is often used as a primary outcome measure to estimate appropriateness of the CDS systems. Prior studies have shown suboptimal salience rates among large healthcare institutions, with average override rates of 46.2 to 96.2%.[18] [19] [20] Several authors have proposed different causes for lower salience rate such as “drug-alert fatigue”[4] [21] [22] caused by excessive or inappropriate drug alerts, or “desensitization,”[23] secondary to prolonged or repetitive exposure to alerts, with some suggesting that these may lead to an increased risk for medical events.

Salience rates vary between different providers based on multiple factors, such as degree of medical training, experience, and workload.[21] Resident physicians in primary care specialties comprise a substantial fraction of the healthcare workforce and regularly utilize the CPOE. Throughout training, resident physicians are subject to different work environments and variable workload intensities across the various subspecialties pertaining to a specific residency program. These circumstances are unique to resident physicians, as compared with established medical staff working under less variable conditions or who may have subspecialized functions within an institution. In this study, we explore salience rates among internal medicine resident physicians at two integrated healthcare institutions.

Objective

Our objective is to retrospectively evaluate the association between residence year and drug-alert prescriber response. We hypothesize that residents further along in the program have lower salience rates, possibly reflecting long-term desensitization.

Methods

Institutions

The University of Pennsylvania Medical Center (UPMC) Pinnacle is a healthcare system which employs over 2,900 physicians across seven acute care hospitals and 160 ambulatory clinic sites. In 2014, UPMC Pinnacle adopted the Epic (Epic Systems Corporation, Verona, Wisconsin, United States) EHR system along with CPOE, and has achieved Healthcare Information Management and Systems Society (HIMSS, Chicago, Illinois, United States) Electronic Medical Record Adoption Model Stage-6 certification. The UPMC Pinnacle uses the drug information vendor First Data Bank (Hearst Health Network, New York, New York, United States) to provide drug information for CDS.

The MetroHealth System (MHS) is an academic healthcare system employing over 550 physicians, and more than 350 physicians in training across 21 health centers, 4 emergency departments, and 13 schools. In 1999, the MHS adopted the Epic (Epic Systems Corporation) EHR system along with CPOE. MHS has achieved Healthcare Information Management and Systems Society (HIMSS) Electronic Medical Record Adoption Model Stage-7 certification. MHS uses the drug information vendor MediSpan (Wolters Kluwer Clinical Drug Information, Hudson, Ohio, United States) to provide drug information for CDS.

Both institutions offer dedicated classrooms for in-person and computer-based EHR training at the start of a residency program. Training lasts approximately 2 weeks, following which user proficiency is tested. Ongoing refresher training sessions are mandatory on a regular basis with computer-based EHR sessions. Additionally, resident physicians are involved in advisory committees that discuss CDS tools. Both institutions employ Chief Medical Information Officers who are supported by pharmacy informatics staff for constant monitoring and evaluation of drug alerts.

Residency Programs

Internal Medicine Residency Programs consist of 3 years of training, postgraduate year 1 (PGY-1) to postgraduate year 3 (PGY-3). The training program at UPMC Pinnacle consists of 20 PGY-1 (interns), 14 PGY-2, and 13 PGY-3 residents. The MHS consists of 34 PGY-1, 23 PGY-2, and 22 PGY-3 residents ([Table 1]).

Table 1 Drug alert and salience rates across resident training years at the two institutions

Both institutions
Year	Residents	Alerts (3 months)	Changes/removals	Salience (%)	Alerts/resident/month
PGY-1	54	22,290	2,589	11.6	138
PGY-2	37	19,530	2,043	10.5	176
PGY-3	35	10,804	965	8.9	102
Total	126	52,624	5,597	10.6	158
p < 0.05 for decrease in salience among the three years of residency
UPMC Pinnacle
PGY-1	20	5,558	1,053	18.9	93
PGY-2	14	6,177	954	15.4	147
PGY-3	13	3,839	400	10.4	98
Total	47	15,574	2,407	15.4	110
p < 0.05 for decrease in salience among the 3 years of residency
MHS
PGY-1	34	16,732	1,536	9.2	164
PGY-2	23	13,353	1,089	8.15	193
PGY-3	22	6,965	565	8.11	105
Total	79	37,050	3,190	8.6	156.3
p < 0.05 for decrease in salience from PGY-1 to PGY2 or PGY-3. p >0.05 for difference in salience from PGY-2 to PGY-3

Abbreviations: MHS, the MetroHealth System; PGY, postgraduate year; UPMC, the University of Pennsylvania Medical Center.

Drug Alerts and Responses Elicited

Among the 13 drug-alert categories, this study looked at the most common types of drug alerts, corresponding to the following categories: duplicate medications, drug interactions and compatibility issues, allergies, and misadministrations in terms of dosage and frequency. Resident physicians had three different options to respond to each drug alert ([Table 2]): (1) adjusting medication settings, such as dose or interval of administration; (2) cancelling the order, or (3) overriding the drug alert and continue placing the order with original settings.

Table 2 Categories of responses to drug alerts

Categories of drug-alert response	Salience type
Adjusting the medication setting	Salience type—adjusting the medication setting including dose or interval administration
Cancel the order	Salience type—allows residents to essentially restart by cancelling the original order
Overriding the drug alert	Nonsalience response, choosing to ignore the clinical decision support recommendation

Study Design

Investigators at each academic healthcare institution performed a retrospective cross-sectional review of drug alerts encountered by resident physicians from their corresponding Internal Medicine Residency Programs. The EHR database (Epic Systems) was queried to collect drug-alert information corresponding to Internal Medicine residents at the UPMC Pinnacle and MHS, from December 2018 through February 2019. Multiple variables were collected and classified, including number of drug orders placed per year of residency training, number of alerts triggered, and response elicited.

Analysis

The primary endpoint analyzed was salience rate, defined as percentage of drug alerts that elicited either removal or change of order parameters (i.e., change of dose, frequency, or timing) as opposed to overriding the drug alerts. Comparisons were made across residency training levels and between organizations for the number of alerts as per residency training year and number of alerts generated as per resident per month. Salience rates for alerts were compared across each residency program and between organizations.

Statistical analyses were performed utilizing SPSS software (version 1.0.0.1327; IBM) to calculate salience and override rates as per year of residency, in addition to correlations and associations and descriptive statistics for percentages, frequencies, and rates. Chi-square test was conducted to look for differences in the percentage of removed/changed orders among the three levels of PGY. To compare the means of number of removed/changed orders, one-way analysis of variance (ANOVA) was used to detect differences among the levels of PGY. To determine which groups were different from one another, Bonferroni's method was utilized.

Institutional Review Board (IRB) was consulted as per guidelines at each institution, an institutional review board (IRB) protocol approval was obtained as required by each institution. Program director approval was obtained from each residency program.

Results

Drug-Alert Exposure

A total of 126 residents were exposed to 52,624 alerts over a 3-month period. At UPMC Pinnacle, 15,574 alerts were generated corresponding to 47 residents (average of 331 alerts per resident during the 3-month study period), and at MHS 37,050 alerts were generated corresponding to 79 residents (average of 469 alerts per resident during the 3-month study period). The difference in mean values of number of alerts per resident between the two institutions was significant (p < 0.01).

Salience Rates

Overall, 54 PGY-1 residents were exposed to 22,290 alerts, eliciting 2,589 changes or removals, corresponding to a salience of 11.6%; 37 PGY-2 residents were exposed to 19,530 alerts, eliciting 2,043 changes or removals, corresponding to a salience rate of 10.46%; 35 PGY-3 residents were exposed to 10,804 alerts, eliciting 965 changes or removals, corresponding to a salience of 8.9%. The mean number of drug alerts seen per resident per month were 138, 176, and 102, for PGY-1, PGY-2, and PGY-3, respectively ([Table 1]).

The University of Pennsylvania Medical Center Pinnacle

Twenty PGY-1 residents were exposed to 5,558 alerts, eliciting 1,053 changes or removals, corresponding to a salience rate of 19%. Fourteen PGY-2 residents were exposed to 6,177 alerts, eliciting 954 changes or removals, corresponding to a salience rate of 15%. Thirteen PGY-3 residents were exposed to 3,839 alerts, eliciting 400 changes or removals, corresponding to a salience rate of 10%. The mean number of alerts seen per resident per month was 93, 147, and 98, for PGY-1, PGY-2, and PGY-3, respectively ([Table 1]).

The MetroHealth System

Thirty-four PGY-1 residents were exposed to 16,732 alerts, eliciting 1,536 changes or removals, corresponding to a salience rate of 9.2%. Twenty-three PGY-2 residents were exposed to 13,353 alerts, eliciting 1,089 changes or removals, corresponding to a salience rate of 8.2%. Twenty-two PGY-3 residents were exposed to a total of 6,965 alerts, eliciting 565 changes or removals, corresponding to a salience rate of 8.1%. The average number of alerts per resident per month was 164, 193, and 105 for PGY-1, PGY-2, and PGY-3, respectively. Over a 3-month period, the average number of alerts seen per resident at MHS was 468 which was 41% higher than the average of 331 seen at UPMC Pinnacle ([Table 1]).

At UPMC Pinnacle, saliency rates were significantly different among PGY-1, PGY-2, and PGY-3 (p < 0.001). At MHS, saliency rates were significantly higher for PGY-1 residents (p < 0.05), as compared with PGY-2 and PGY-3, but not significantly different when comparing PGY-2 and PGY-3 (p > 0.05). In the overall sample, including both institutions, salience rate was significantly different among all three groups (p < 0.001).

Discussion

We conducted a cross-institutional retrospective study where we observed and compared drug-alert rates and salience between PGY1, PGY-2, and PGY-3 at two different institutions. We demonstrated a significant variation in response rates between different residency training years. The number of triggered drug alerts was significantly fewer among PGY-3 compared with PGY-1 and PGY-2. This trend was insignificant when comparing PGY-2 and PGY-3 at MHS. Overall, PGY-3 residents were exposed to 45% fewer alerts than PGY-2 and 26% fewer than PGY-1.

A higher number of alerts-per-resident was seen in PGY-2 as compared with PGY-1 and PGY-3. This phenomenon may reflect the increased clinical responsibilities of PGY-2 during the first 6 months of the academic year (July–December) where PGY-1 learnt to utilize the EHR and CPOE and consequently placed fewer orders. An additional consideration is that PGY-2 may have heavier clinical workloads, typically evaluating more patients and placing more orders as compared with junior residents.

PGY-1 were more likely to remove or change orders than senior residents based on drug-alert exposure. This tendency of decreasing salience from PGY-1 to PGY-3 despite a variable number of alerts-per-resident in each year, may be multifactorial. While the trend may suggest long-term desensitization, as PGY-2 and PGY-3 were more likely to override a drug alert, it may also reflect an increased experience in PGY-2 and PGY-3 utilizing the CPOE: PGY-2 and PGY-3 may better identify inappropriate drug alerts, or drug alerts shown to these groups may be more likely inappropriate than those triggered by orders from less knowledgeable residents (PGY-1), and could be therefore more likely overridden. It should also be noted that the tendency of decreasing saliency from PGY-2 to PGY-3 was minimal in MHS as compared with UPMC Pinnacle.

Comparing results between the two institutions, residents at MHS showed a 41% higher number of alerts-per-resident which was associated with a 42% decrease in overall salience rate. This finding may support two hypotheses. First, there may be a degree of drug-alert fatigue due to repetitive or overall higher numbers of alerts at MHS which was evident when comparing the two institutions, and second, resident physicians may be subject to long-term desensitization while progressing through residency. Due to substantial differences in both institutions, however, including vendor, culture, and training, among others, a direct comparison of salience rates can be misleading.

By the final years of residency, trainees of primary care specialties have likely been exposed to a large number of medication safety alerts, which raises a concern for possible drug-alert desensitization which has been previously suggested as a cause of decreased salience in medical providers.[21] [22] [23] [24] [25] [26]

Some authors who have included resident physicians while investigating salience have found variable response rates. Knight et al, for example, found that alerts were more likely to be overridden when encountered by residents,[27] which seems inconsistent with findings reported by Weingart et al[24] who reported that residents were less likely to override medication alerts, and Long et al suggested that physicians with longer years in practice were more resistant to innovation and less likely to accept CDS reminders.[25] However, most studies analyzing salience rates during residency do not account for many variables that may affect drug-alert experience from a residency standpoint, including year of training, specialty, and workload. These variables should be considered when comparing salience not only between residents, but between different providers.

As previously shown, the UPMC Pinnacle demonstrated an inverse relationship between salience and year of training. At MHS, this trend was demonstrated only when comparing PGY-1 to PGY-2 or PGY-3, but not significantly different between PGY-2 and PGY-3. PGY-3 at MHS was exposed to a 54% lower number of alerts per resident per month (105) than PGY-2 (193) ([Table 1]). Despite this difference, however, salience was similar. Based on a premise of drug-alert fatigue, where a higher drug-alert burden leads to lower salience rates, PGY-3 at MHS would have been expected to have a higher salience rate than PGY-2, the fact that this number did not change may suggest long-term desensitization.

Also noted was an increased number of alerts per resident in PGY-2 at both institutions. We believe that this is secondary to two main factors. First, PGY-1 residents place relatively fewer medical orders in their first months of residency, while PGY-2 and PGY-3 have relatively more elective rotations where the number of orders placed is generally lower than in core rotations (i.e., medical wards and intensive care unit).

Residents at MHS were exposed to an average to five alerts per day, while those at UPMC Pinnacle saw approximately four alerts per day. Dexheimer et al analyzed results from 4,575 providers over a 24-month period showing that providers exposed to 49 alerts/day showed maximum saliency. However, this sample included various types of providers including attending physicians and fellows of different specialties. When analyzing only residents, however, Dexheimer et al found a relatively constant salience trend overtime since the beginning of residency which contrasts results from our study.[4] Additionally, Dexheimer et al found a higher alert/day rate which could be related multiple variables including patient background (e.g., pediatric- versus nonpediatric-based healthcare institution) and out-of-box vendor-alert customization differences.

Other authors have found a decreasing saliency with increasing numbers of alerts per provider per day, suggesting drug-alert fatigue.[4] [7] [25] While this trend may be secondary to cumulative exposure to drug alerts, another possibility may be that senior residents were more experienced utilizing the CPOE, thus the smaller number of alerts they triggered were most likely inappropriate and, therefore most commonly overridden. Steele et al compared different types of drug-laboratory interactions in the outpatient setting, with the majority of orders being placed by faculty members, rather than residents, showing that providers would generally continue ordering a medication despite drug alerts.[8]

In a retrospective study, Zenziper Straichman et al found a drug-alert acceptance rate (salience) of 5.3% (p < 0.001) from providers at an internal medicine department. While their sample included faculty members, the majority was composed of resident physicians (64 out of 92). In their prospective study that included two internal medicine departments, Zenziper Straichman et al reported a drug-alert acceptance rate of 4.2%, of which 89.3% corresponded to orders placed by residents and 10.7% to faculty members.[26] Similar to Zenziper Straichman et al, Knight et al found a salience of 4%, less than half of what was found in our study (11%). Knight et al also found that alerts were more likely to be overridden when encountered by residents or providers younger than 40 year of age[27]; however, drug-alert burden was not taken into consideration which may partly contribute to lower salience rates among resident physicians. Unlike Knight et al, Weingart et al found that residents were less likely to override medication alerts,[24] and Long et al showed that physicians with longer years in practice were less likely to accept reminders and suggested that older providers may be more resistant to innovation.[25] When comparing salience rates among different providers, multiple variables should be taken in consideration including number of alerts per provider per unit of time, work hours, overall workload, and other factors which may otherwise confound direct comparisons.

Systematic approaches for optimizing drug alerts have been suggested by other authors to improve salience rates. Saiyed et al highlighted the importance of reviewing reported data and receiving feedback from end users who can offer suggestions for improving the utility of drug alerts.[15] Stutman et al proposed an iterative approach to modify drug-alert display options by tracking their frequency and elicited responses in providers.[28]

Nevertheless, inappropriate drug alerts overriding should also be considered, which may occur due to a variety of reasons such as cognitive overload, inadequate understanding of drug alerts, or desensitization,[4] all of which are possible during residency training.

The overall number of drug alerts-per-resident in the 3-month period was 42% higher in MHS, as compared with UPMC Pinnacle, which was associated with a 41% decrease in overall salience rates, suggesting that there could be some degree of drug-alert fatigue due to excessive or repetitive drug alerts; a phenomenon that has been suggested in prior studies,[4] [7] [20] [21] [26] may be associated with cognitive overload.[21] Ancker et al conducted a retrospective cohort including 112 ambulatory primary care providers and suggested that alert fatigue can be associated with cognitive overload; however, they found no associations with workload or evidence of time-sensitive desensitization.[21] Our study was limited because we did not measure time. Nonadherence to drug alerts, manifested as high override rates, has been reported in 49 to 96% of alerts in other samples,[15] and may be secondary to multiple causes including poor implementation or acceptance of CDS tools, drug-alert fatigue, and possibly long-term desensitization. Continued optimization of drug alerts is needed to decrease the number of inappropriate alerts. Communication with medical providers, including resident physicians, is necessary to recognize obstacles that may lead to nonadherence to CDS tools.

Limitations

This study had some limitations. It was restricted to internal medicine residency programs. Other specialties have different levels of clinical workload and patient demographics which may impact drug-alert response rates. The study compared two different residency programs with multiple variables that were not considered including differences in culture, subspecialty rotations, and drug information vendors, among other factors. The study only included data from 3 months, a 3-year prospective study would allow exploring salience trends, and responses to new CDS implementations throughout residency training. Finally, the study lacked assessment of alert appropriateness, such as recurrent inappropriate duplicate alerts to single users despite prior reasonable overrides.

Conclusion

This is one of the first cross-institutional studies examining drug-alert rates and comparing salience rates between years of medical training and medical experience between two academic institutions utilizing the same EHR. We demonstrate a drug-alert salience of 11% in 126 trainees from two large academic-based residency programs, with significantly different response rates between different years of training. Override rates increased progressively from junior to senior years of residency training. This trend, along with the lack of a significant difference in salience between PGY-2 and PGY-3 at MHS despite nearly half the rate of drug alerts-per-resident, favors long-term desensitization. However, other factors should also be considered.

Resident physicians may be at risk for long-term desensitization to drug alerts from cumulative exposure and drug alert fatigue from repetitive exposure which may lead to inappropriate overrides and medical events. Continued optimization of CDS tools and appropriate training on CPOE systems, starting from medical school and continuing through residency, could hypothetically decrease the risk of inappropriate overrides and, consequently, serious medical events.

Clinical Relevance Statement

Drug alerts and clinical decision support tools have demonstrated decreased risk of medical events including medication misadministration. High override rates in resident physicians may be secondary to various factors, including adequacy of drug alerts, nonadherence, drug-alert fatigue, and chronic desensitization. Identifying these obstacles will facilitate strategies for drug-alert optimization.

Multiple Choice Questions

1. Which of the following has/have been proposed as a cause for increasing override rates?

   • Drug-alert fatigue

   • Desensitization

   • Inappropriate drug alerts

   • Time of the day

   • All of the above

   Correct Answer: The correct answer is option e.

2. Improvement of saliency rates in physician response to medication alerts, may lead to

   • Improved patient outcome

   • Increase in override rates

   • Decrease in adverse drug events

   • Increase in work burden

   • Options a and c are true

   • None of the above

   Correct Answer: The correct answer is option e.

Conflict of Interest

None declared.

Acknowledgment

Authors would like to thank Emmanual Kelechi and Isha Shrimanker for drafting initial data collection sheet.

Protection of Human and Animal Subjects

No human subjects were involved in this project. The Institutional Review Board (IRB) was consulted, per guidelines at each institution, an IRB protocol approval was obtained as required by each institution. Program director approval was obtained from each residency program.

• References

• 1 Slight SP, Berner ES, Galanter W. et al. Meaningful use of electronic health records: experiences from the field and future opportunities. JMIR Med Inform 2015; 3 (03) e30
• 2 Centers for Medicare and Medicaid Services. CMS Quality Measure Development Plan Supporting the Transition to the Merit-Based Incentive Payment System (MIPS) and Alternative Payment Models. (APMs) (DRAFT). Accessed April 6, 2021 at: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/Draft-CMS-Quality-Measure-Development-Plan-MDP.pdf
• 3 Lyman JA, Cohn WF, Bloomrosen M, Detmer DE. Clinical decision support: progress and opportunities. J Am Med Inform Assoc 2010; 17 (05) 487-492
• 4 Dexheimer JW, Kirkendall ES, Kouril M. et al. The effects of medication alerts on prescriber response in a pediatric hospital. Appl Clin Inform 2017; 8 (02) 491-501
• 5 Jamoom E, Beatty P, Bercovitz A, Woodwell D, Palso K, Rechtsteiner E. Physician adoption of electronic health record systems: United States, 2011. NCHS Data Brief 2012; (98) 1-8
• 6 Evans RS. Electronic health records: then, now, and in the future. Yearb Med Inform 2016; (Suppl. 01) S48-S61
• 7 Shah NR, Seger AC, Seger DL. et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006; 13 (01) 5-11
• 8 Steele AW, Eisert S, Witter J. et al. The effect of automated alerts on provider ordering behavior in an outpatient setting. PLoS Med 2005; 2 (09) e255
• 9 McCullagh LJ, Sofianou A, Kannry J, Mann DM, McGinn TG. User centered clinical decision support tools: adoption across clinician training level. Appl Clin Inform 2014; 5 (04) 1015-1025
• 10 Ibáñez-Garcia S, Rodriguez-Gonzalez C, Escudero-Vilaplana V. et al. Development and evaluation of a clinical decision support system to improve medication safety. Appl Clin Inform 2019; 10 (03) 513-520
• 11 Lyell D, Magrabi F, Coiera E. Reduced verification of medication alerts increases prescribing errors. Appl Clin Inform 2019; 10 (01) 66-76
• 12 Segal MM, Rahm AK, Hulse NC. et al. Experience with integrating diagnostic decision support software with electronic health records: benefits versus risks of information sharing. EGEMS (Wash DC) 2017; 5 (01) 23
• 13 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17
• 14 McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 (02) 195-202
• 15 Saiyed SM, Davis KR, Kaelber DC. Differences, opportunities, and strategies in drug alert optimization-experiences of two different integrated health care systems. Appl Clin Inform 2019; 10 (05) 777-782
• 16 Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc 2009; 2009: 548-552
• 17 Kirkendall ES, Kouril M, Minich T, Spooner SA. Analysis of electronic medication orders with large overdoses: opportunities for mitigating dosing errors. Appl Clin Inform 2014; 5 (01) 25-45
• 18 Poly TN, Islam MM, Yang H-C, Li YJ. Appropriateness of overridden alerts in computerized physician order entry: systematic review. JMIR Med Inform 2020; 8 (07) e15653
• 19 Topaz M, Seger DL, Slight SP. et al. Rising drug allergy alert overrides in electronic health records: an observational retrospective study of a decade of experience. J Am Med Inform Assoc 2016; 23 (03) 601-608
• 20 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 21 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17: 36
• 22 Rush JL, Ibrahim J, Saul K, Brodell RT. Improving patient safety by combating alert fatigue. J Grad Med Educ 2016; 8 (04) 620-621
• 23 Cash JJ. Alert fatigue. Am J Health Syst Pharm 2009; 66 (23) 2098-2101
• 24 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21) 2625-2631
• 25 Long A-J, Chang P, Li Y-C, Chiu W-T. The use of a CPOE log for the analysis of physicians' behavior when responding to drug-duplication reminders. Int J Med Inform 2008; 77 (08) 499-506
• 26 Zenziper Straichman Y, Kurnik D, Matok I. et al. Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients. Int J Med Inform 2017; 107: 70-75
• 27 Knight AM, Falade O, Maygers J, Sevransky JE. Factors associated with medication warning acceptance for hospitalized adults. J Hosp Med 2015; 10 (01) 19-25
• 28 Stutman HR, Fineman R, Meyer K, Jones D. Optimizing the acceptance of medication-based alerts by physicians during CPOE implementation in a community hospital environment. AMIA Annu Symp Proc 2007; 2007: 701-705

Publication History

Received: 17 August 2020

Accepted: 22 March 2021

Article published online:
28 April 2021