Low Efficacy of Medication Shortage Clinical Decision Support Alerts

Abstract

Objective We examined clinical decision support (CDS) alerts designed specifically for medication shortages to characterize and assess provider behavior in response to these short-term clinical situations.

Materials and Methods We conducted a retrospective analysis of the usage of medication shortage alerts (MSAs) that included at least one alternative medication suggestion and were active for 60 or more days during the 2-year study period, January 1, 2018 to December 31, 2019, in a large health care system. We characterized ordering provider behavior in response to inpatient MSAs. We then developed a linear regression model to predict provider response to alerts using the characteristics of the ordering provider and alert frequency groupings.

Results During the study period, there were 67 MSAs in use that focused on 42 distinct medications in shortage. The MSAs suggested an average of 3.9 alternative medications. Adjusting for the different alerts, fellows (p = 0.004), residents (p = 0.03), and physician assistants (p = 0.02) were less likely to accept alerts on average compared with attending physicians. Further, female ordering clinicians (p < 0.001) were more likely to accept alerts on average compared with male ordering clinicians.

Conclusion Our findings demonstrate that providers tended to reject MSAs, even those who were sometimes flexible about their responses. The low overall acceptance rate supports the theory that alerts appearing at the time of order entry may have limited value, as they may be presented too late in the decision-making process. Though MSAs are designed to be attention-grabbing and higher impact than traditional CDS, our findings suggest that providers rarely change their clinical decisions when presented with these alerts.

Keywords

medication shortage - quality of care - cost of care - electronic health records

Background and Significance

Across the United States, medication shortages have been increasing over the last few decades.[1] The reasons for these shortages are complex, but can suggest a disruption in the supply chain (e.g., intravenous fluids shortage after Hurricane Maria in 2017),[2] decreased production,[3] change in production priorities by pharmaceutical companies,[4] or increased demand for a particular medication (e.g., increased demand for norepinephrine and hydroxychloroquine during the coronavirus 2019 pandemic).[5] [6] The types of medication shortage and duration of the shortage vary. During periods when medications are in short supply, patients are more likely to experience adverse events, including mortality and medication errors.[7] In this setting, clinicians must decide how to allocate the limited supply of medication and consider when an alternative medication can be administered instead.

Computerized physician order entry and clinical decision support (CDS) systems are designed to facilitate care delivery by presenting ordering providers with relevant and timely information to enhance quality of care.[8] [9] [10] Though these systems have perhaps most often been used to promote adherence to medication treatment guidelines or best practices, they can also be deployed to convey information about short-term, high acuity clinical situations, such as periods of medication shortage.[11] [12] [13] [14] One form of CDS used in such settings are medication shortage alerts (MSAs), which provide specialized guidance to encourage conservation of resources in a responsible way by suggesting alternative medications or treatments. Given the urgency of the medication shortages, MSAs are usually rolled out in a way that would maximize their effectiveness. For instance, these alerts may be designed to be visually distinct from other medication-related CDS alerts, can occur at different points in the ordering process, and their implementation is often accompanied by widespread messaging (e.g., system-wide emails about the ongoing medication shortage, news coverage, training through grand rounds and other educational meetings, etc.; see [Fig. 1]).[15] While medication-related CDS is designed with good intentions, provider response rates are inconsistent with variation among providers[16] [17] [18] [19] and concerns exist that too many alerts within the electronic health record (EHR) may contribute to alert fatigue.[20] [21] [22]

Fig. 1 Example medication shortage alert displayed prior to order finalization. (Note: this image is copyrighted by Epic Systems Corporation 2021.)

While there are many studies focused on CDS alerts, data are scarce on the efficacy of alerts that are only active for a short time to address a temporary situation, as well as the susceptibility of these short-term alerts to alert fatigue. In this study, we examined CDS alerts designed specifically for medication shortages (MSAs) to characterize and assess provider behavior in response to these short-term clinical situations.

Materials and Methods

We conducted a retrospective analysis of the usage of inpatient MSAs where an alternative medication was able to be selected and that were active for 60 or more days during the 2-year study period, January 1, 2018 to December 31, 2019, in a large health care system. We chose 60 or more days to allow for enough time for a variety of ordering providers to encounter the alert. Medications administered in the inpatient setting are ordered through the EHR and the medication orders, regardless of ordering provider type, are reviewed and verified by a pharmacist. We examined ordering provider behavior in response to MSAs and characterized their responses. Through a query of the enterprise data warehouse, we identified all MSAs that met this inclusion criterion. Using audit logs and order entry data, we identified each order entry event that resulted in activation of an MSA and recorded the final choice made by that provider. The outcome choices were categorized as: (1) proceeding with ordering the original medication, (2) selecting an alternative medication suggested by the MSA, or (3) abandoning the originally intended medication order altogether ([Fig. 2]). We used these data to characterize provider response to MSAs and examine how provider behavior changed over the study period and across the lifetime of the alert.

Fig. 2 Schematic of workflow with example clinical scenarios.

Ordering providers often encountered individual alerts multiple times. For each alert, we examined the proportion of times each provider accepted the alternative for the alert, abandoned the order, or rejected the suggested alternative. We used descriptive analytics (e.g., graphs and tables) to visualize the trends of providers' response to an alert over time. For each MSA, we characterized the ordering providers who interacted with it at least two times into three distinct groups: (1) accepters – those who accepted an alternative suggested by the MSA more than two-thirds (≥ 67%) of the time, (2) rejectors – those who accepted an alternative less than one-third (≤ 33%) of the time, and (3) flexible – those accepting an alternative suggested by the alert between 33 and 67% of the time.

For each ordering provider, we extracted characteristics about the ordering provider, including sex and provider type. Using the average acceptance rate to an alert for each provider, we then developed a linear regression model to predict average provider response to alerts with the characteristics of the ordering provider (sex, age, provider type) and alerts. In sensitivity analyses, we included additional factors with respect to orders, including order time of day (night shift vs. day shift)[23] and clinical service (surgical vs. nonsurgical). Separately in additional sensitivity analyses, we averaged the provider response categories across all alerts and categorized the frequency of each alert, grouping them into low (the third of alerts with the lowest frequency), medium (the third of alerts with the next lowest frequency), and high (the third of alerts with the highest frequency), and included the categorization in the sensitivity analyses. We also used data from the first time a provider encountered an alert (i.e., whether they accepted the alert during the first exposure) to inform a regression model. Only those providers who were residents, fellows, attendings, nurse practitioners, or physician assistants were included in the regression analysis. In our main analyses, we used a linear regression model recognizing that there may be justification to use a nonlinear approach. Then, for sensitivity analyses, we employed a negative binomial regression and found that differences in provider types were no longer significant. The standard errors were large, however, and thus we used the linear approach for our main analyses. Further, we considered employing an alternative approach that included examining “order sessions” to evaluate if there were differences when orders were placed and signed together. However, from prior work we know that the average number of orders per session is low (median and mean of less than five orders in the emergency department),[23] making it difficult to observe meaningful differences. All analyses were conducted in R 3.6.1. The Mass General Brigham Institutional Review Board approved this study.

Results

During the study period, there were 67 MSAs in use that focused on 42 distinct medications in shortage (see list in [Supplementary Table S1], available in the online version). The MSAs suggested an average of 3.9 alternative medications and were active for an average of 249 days (range 62–728 days). Of the 67 alerts studied, there were 4 in which no provider continued with the original order. The frequency with which alerts were triggered diminished after 30 days but remained stable for the remaining time of the alert. Twenty percent of activations of the alerts were in the first 30 days after an alert became active for the first time. This decreased to 18% over the next 30 days, as well as ensuing 30-day periods.

A total of 214,373 orders were placed that prompted an MSA. There were 8,917 unique providers who encountered at least one of the MSAs and 5,553 who encountered at least one MSA more than once ([Table 1]). During the study period, ordering providers responded to the alert by choosing a suggested alternative medication 17.2% of the time.

Table 1 Characteristics of ordering providers who encountered one MSA at least twice

	Attending physician	Fellow	Resident	Nurse practitioner	Physician assistant
Total unique ordering providers	3,082	298	996	652	525
Female (%)	45.7%	45.6%	47.5%	94%	79.2%
Total orders placed	84,784	14,205	55,519	15,322	28,105
Average number of MSAs encountered	3.4	6.9	8.3	3.2	6.4
Proportion who consistently accepted MSAs (%)	5.5%	2.3%	3.8%	1.7%	4.2%
Proportion who consistently rejected MSAs (%)	62.1%	43.6%	33.1%	61.7%	42.9%
Proportion who varied in response to MSAs (%)	32.4%	54%	63.1%	36.7%	53%

Abbreviation: MSA, medication shortage alert.

For those providers encountering an MSA at least twice, most ordering provider responses were to reject the suggestion and disregard the alternative medication (3,000, 54.0%), although some providers did show varied responses across different alerts. A small fraction of providers tended to consistently accept the alternatives suggested by nearly all MSAs (251, 4.5%). The remaining 41.5% changed their response across alerts but tended toward more frequent rejection.

Adjusting for the different alerts, fellows (estimate = –0.016, 95% confidence interval [CI] = –0.026, –0.005, p = 0.004), residents (estimate = –0.007, 95% CI = –0.014, –0.001, p = 0.03), and physician assistants (estimate = –0.010, 95% CI = –0.019, –0.002, p = 0.02) were less likely to accept alerts on average compared with attending physicians. Female ordering clinicians were more likely than male ordering clinicians to accept alerts (estimate = 0.012, 95% CI = 0.006–0.018, p < 0.001) ([Table 2]). Results were similar when incorporating order time of day and clinical service as additional covariates into the model. Results were similar when incorporating the alert frequency groupings rather than the alerts and demonstrated that the highest frequency alerts were least likely to be accepted (estimate = –0.27, 95% CI = –0.31, –0.23, p < 0.001) compared with low or medium frequency alerts.

Table 2 Linear regression results of ordering provider response to alerts

Variable	Estimate	95% confidence interval	p-Value
Female ordering clinician	0.012		< 0.001
Provider type
Attending physician	reference	reference	reference
Fellows	–0.016	(–0.026, –0.005)	0.004
Residents	–0.007	(–0.014, –0.001)	0.03
Physician assistants	–0.010	(–0.019, –0.002)	0.02

Note: Linear regression controlled for different alert types. Please note, this analysis was limited due to the inability to control for other confounding factors (e.g., ordering clinician clinical service). Low efficacy of medication shortage clinical decision support alerts.

We then examined the impact of an ordering provider's first response to a specific alert on their overall acceptance of alerts, adjusting for provider type, sex, and alert. We found that those who were more likely to accept an alert the first time they encountered it were significantly more likely to accept more alerts on average (estimate = 0.422, 95% CI = 0.415–0.429, p < 0.001).

Discussion

In our study, CDS alerts designed to address short-term situations were accepted by providers only about one time in six. Across the lifetime of the alert, the acceptance rate was less than 20%. For some medications, for example, if the inventory level of the medication is at a critical low or is truly not available, “hard stop” alerts can be effective.[24]

Our findings demonstrated that providers most often rejected MSAs, even those who were sometimes flexible about their responses. The low overall acceptance rate supports the theory that alerts for MSAs appearing at the time of order entry may have limited value, as they may be presented too late in the clinical decision-making process, particularly if they alert after the decision has been made.[25] [26]

MSAs are designed with the assumption of high efficacy, particularly as they are implemented during a period of stress for the health care system. They may be visually distinct from other CDS alerts and their implementation is often accompanied by widespread messaging (e.g., system-wide emails about the ongoing medication shortage, news coverage). Because MSAs are short term, we would not expect those providers who continued to order alert-triggering medications to develop “muscle memory” to ignore them.

We found that fellows, residents, and physician assistants were less likely to accept the alternative when compared with attending physicians, though the coefficients were small. This suggests that attending physicians may be more open to changing their practice than some other provider types, who may have fixed ideas about how to treat their patients, not have the authority to make the change, or be less confident in prescribing a suggested medication that might be new. Given that we were unable to control for patient location or clinical service, these findings may also indicate patient-level variation, particularly if attending physicians work with different types of patients than trainees or physician assistants. Regardless, the clinical significance of these differences should be further evaluated.

Our study has several limitations. First, we were unable to evaluate characteristics of the patients or clinical situations when the alert was activated to determine the optimal response to the alert. Second, this study did not have a control group to determine how much the alert itself changed overall behavior. Third, we queried the institutional Enterprise Data Warehouse for these specialized CDS alerts which may not fully capture all MSAs in our institution's EHR vendor, Epic Systems. Fourth, we were not able capture the individual site-specific medication supply or the site-specific decisions for utilizing the MSAs during the time of the medication shortages. Fifth, we were not able to determine the extent to which communication outside of the EHR and CDS alerts (e.g., emails about medication shortage) impacted provider workflow. Sixth, we were unable to account for all potential confounding variables, limiting the conclusions associated with alert rejection. Lastly, the approach we used for combining the acceptance rates is rudimentary as it only looks at point estimates for each provider. A more sophisticated approach incorporating weighted averages or shrinkage methods (such as the EB05 method used in pharmacovigilance)[27] could be used in future research to validate the robustness of these findings.

Conclusion

Though MSAs are designed to be attention-grabbing and higher impact than traditional CDS, our findings suggest that providers rarely change their clinical decisions when presented with these alerts. However, it appears that some providers did learn to avoid certain medications that would trigger these types of alerts, decreasing the frequency of the alerts after the first 30 days. The overall low acceptance rate, however, suggests the need to consider innovative ways to improve these alerts and the response rates to them.

Clinical Relevance Statement

Medication shortage alerts are designed with the assumption of high efficacy, particularly as they are implemented during a period of stress for the health care system. Our findings suggest that providers rarely change their clinical decisions when presented with these alerts. The overall low acceptance rate suggests the need to consider innovative ways to improve these alerts and the response rates to them.

Multiple Choice Questions

1. What are medication shortage alerts?

   • Alerts sent to patients regarding their current prescriptions and need to refill.

   • Alerts that provide specialized guidance to encourage conservation of resources in a responsible way by suggesting alternative medications or treatments.

   • Alerts that the ordering provider is prescribing a medication at too low a dose or quantity for the patient.

   • Alerts that the patient's local pharmacy is out of a medication.

   Correct Answer: Option b is the correct answer. Computerized physician order entry (CPOE) and clinical decision support (CDS) systems are designed to facilitate care delivery by presenting ordering providers with relevant and timely information to enhance quality of care. Though these systems have perhaps most often been used to promote adherence to medication treatment guidelines or best practices, they can also be deployed to convey information about short-term, high acuity clinical situations, such as periods of medication shortage. One form of CDS used in such settings are medication shortage alerts (MSAs), which provide specialized guidance that encourage conservation of resources in a responsible way by suggesting alternative medications or treatments.

2. After adjusting for sex of the ordering provider and the different alerts, when compared with attending physicians, what types of clinicians were less likely to accept medication shortage alerts?

   • Fellows

   • Residents

   • Physician assistants

   • All the above

   Correct Answer: Option d is the correct answer. Adjusting for sex of the ordering provider and the different alerts, fellows (estimate = –0.016, 95% confidence interval [CI] = –0.026, –0.005, p = 0.004), residents (estimate = –0.007, 95% CI = –0.014, –0.001, p = 0.03), and physician assistants (estimate = –0.010, 95% CI = –0.019, –0.002, p = 0.02) were less likely to accept alerts on average compared with attending physicians.

Conflict of Interest

N.M.B. volunteers on the Epic Systems Behavioral Health Subspecialty Steering Board. D.W.B. reports grants and personal fees from EarlySense, personal fees from CDI Negev, equity from ValeraHealth, equity from Clew, equity from MDClone, personal fees and equity from AESOP, and grants from IBM Watson Health, outside the submitted work.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the Mass General Brigham Institutional Review Board.

• References

• 1 Ventola CL. The drug shortage crisis in the United States: causes, impact, and management strategies. P&T 2011; 36 (11) 740-757
• 2 United States Food & Drug Administration. FDA Commissioner Scott Gottlieb, M.D., updates on some ongoing shortages related to IV fluids. Published January 16, 2018. Accessed December 15, 2020 at: https://www.fda.gov/news-events/press-announcements/fda-commissioner-scott-gottlieb-md-updates-some-ongoing-shortages-related-iv-fluids
• 3 Phend C. COVID-19 Puts ICU Sedatives into Shortage. Published April 3, 2020. Accessed December 15, 2020 at: https://www.medpagetoday.com/infectiousdisease/covid19/85796
• 4 Plackett B. Why big pharma has abandoned antibiotics. Nature 2020; 2020: 586
• 5 Business Wire. Premier Inc. Data: As COVID-19 Cases Spike, Demand for Drugs Used to Treat Related, Co-Morbid Conditions Also Soars. Published April 30, 2020. Accessed December 15, 2020 at: https://www.businesswire.com/news/home/20200430005788/en/Premier-Inc.-Data-As-COVID-19-Cases-Spike-Demand-for-Drugs-Used-to-Treat-Related-Co-Morbid-Conditions-Also-Soars
• 6 American College of Rheumatology. Patients reported international hydroxychloroquine shortages due to COVID-19. ScienceDaily. Published November 6, 2020. Accessed January 26, 2021 at: https://www.sciencedaily.com/releases/2020/11/201106103103.htm
• 7 Phuong JM, Penm J, Chaar B, Oldfield LD, Moles R. The impacts of medication shortages on patient outcomes: a scoping review. PLoS One 2019; 14 (05) e0215837
• 8 Tsapepas D, Der-Nigoghossian C, Patel K, Berger K, Vawdrey DK, Salmasian H. Medication stewardship using computerized clinical decision support: a case study on intravenous immunoglobulins. Pharmacol Res Perspect 2019; 7 (05) e00508
• 9 Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform 2016; ;Suppl 1 S103-S116
• 10 Yan L, Reese T, Nelson SD. A narrative review of clinical decision support for inpatient clinical pharmacists. Appl Clin Inform 2021; 12 (02) 199-207
• 11 Fox ER, Sweet BV, Jensen V. Drug shortages: a complex health care crisis. Mayo Clin Proc 2014; 89 (03) 361-373
• 12 Joglekar NN, Patel Y, Keller MS. Evaluation of clinical decision support to reduce sedative-hypnotic prescribing in older adults. Appl Clin Inform 2021; 12 (03) 436-444
• 13 Friebe MP, LeGrand JR, Shepherd BE, Breeden EA, Nelson SD. Reducing inappropriate outpatient medication prescribing in older adults across electronic health record systems. Appl Clin Inform 2020; 11 (05) 865-872
• 14 Hendrickson CD, McLemore MF, Dahir KM. et al. Is the climb worth the view? The savings/alert ratio for reducing vitamin D testing. Appl Clin Inform 2020; 11 (01) 160-165
• 15 Vo P, Sylvia DA, Milibari L. et al. Management of a parenteral opioid shortage using ASHP guidelines. Am J Health Syst Pharm 2021; 78 (05) 426-435
• 16 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 17 Beeler PE, Orav EJ, Seger DL, Dykes PC, Bates DW. Provider variation in responses to warnings: do the same providers run stop signs repeatedly?. J Am Med Inform Assoc 2016; 23 (e1): e93-e98
• 18 Trinkley KE, Pell JM, Martinez DD, Maude NR, Hale G, Rosenberg MA. Assessing prescriber behavior with a clinical decision support tool to prevent drug-induced long QT syndrome. Appl Clin Inform 2021; 12 (01) 190-197
• 19 Bhat S, Derington CG, Trinkley KE. Clinicians' values and preferences for medication adherence and cost clinical decision support in primary care: a qualitative study. Appl Clin Inform 2020; 11 (03) 405-414
• 20 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 21 Carroll AE. Averting alert fatigue to prevent adverse drug reactions. JAMA 2019; 322 (07) 601
• 22 Shah SN, Amato MG, Garlo KG, Seger DL, Bates DW. Renal medication-related clinical decision support (CDS) alerts and overrides in the inpatient setting following implementation of a commercial electronic health record: implications for designing more effective alerts. J Am Med Inform Assoc 2021; 28 (06) 1081-1087
• 23 Salmasian H, Blanchfield BB, Joyce K. et al. Association of display of patient photographs in the electronic health record with wrong-patient order entry errors. JAMA Netw Open 2020; 3 (11) e2019652
• 24 Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc 2018; 25 (11) 1556-1566
• 25 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 26 Murphy EV. Clinical decision support: effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J Biol Med 2014; 87 (02) 187-197
• 27 Harpaz R, Vilar S, Dumouchel W. et al. Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions. J Am Med Inform Assoc 2013; 20 (03) 413-419

Publication History

Received: 23 June 2021

Accepted: 20 October 2021

Article published online:
01 December 2021