Appl Clin Inform 2017; 08(03): 866-879
DOI: 10.4338/ACI-2017-04-RA-0059
Research Article
Schattauer GmbH

Comparison of Overridden Medication-related Clinical Decision Support in the Intensive Care Unit between a Commercial System and a Legacy System

Adrian Wong
1  Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA
2  MCPHS University, Boston, MA
,
Adam Wright
1  Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA
3  Harvard Medical School, Boston, MA
,
Diane L. Seger
1  Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA
4  Partners HealthCare, Clinical and Quality Analysis, Somerville, MA
,
Mary G. Amato
1  Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA
2  MCPHS University, Boston, MA
,
Julie M. Fiskio
1  Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA
4  Partners HealthCare, Clinical and Quality Analysis, Somerville, MA
,
David Bates
1  Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA
3  Harvard Medical School, Boston, MA
› Author Affiliations
Further Information

Correspondence to:

David Bates
Division of General Internal Medicine and Primary
Care, Brigham and Women‘s Hospital
Boston/USA

Publication History

10 April 2017

02 June 2017

Publication Date:
20 December 2017 (online)

 

Summary

Background: Electronic health records (EHRs) with clinical decision support (CDS) have shown to be effective at improving patient safety. Despite this, alerts delivered as part of CDS are overridden frequently, which is of concern in the critical care population as this group may have an increased risk of harm. Our organization recently transitioned from an internally-developed EHR to a commercial system. Data comparing various EHR systems, especially after transitions between EHRs, are needed to identify areas for improvement.

Objectives: To compare the two systems and identify areas for potential improvement with the new commercial system at a single institution.

Methods: Overridden medication-related CDS alerts were included from October to December of the systems’ respective years (legacy, 2011; commercial, 2015), restricted to three intensive care units. The two systems were compared with regards to CDS presentation and override rates for four types of CDS: drug-allergy, drug-drug interaction (DDI), geriatric and renal alerts. A post hoc analysis to evaluate for adverse drug events (ADEs) potentially resulting from overridden alerts was performed for ‘contraindicated’ DDIs via chart review.

Results: There was a significant increase in provider exposure to alerts and alert overrides in the commercial system (commercial: n=5,535; legacy: n=1,030). Rates of overrides were higher for the allergy and DDI alerts (p<0.001) in the commercial system. Geriatric and renal alerts were significantly different in incidence and presentation between the two systems. No ADEs were identified in an analysis of 43 overridden contraindicated DDI alerts.

Conclusions: The vendor system had much higher rates of both alerts and overrides, although we did not find evidence of harm in a review of DDIs which were overridden. We propose recommendations for improving our current system which may be helpful to other similar institutions; improving both alert presentation and the underlying knowledge base appear important.

Citation: Wong A, Wright A, Seger DL, Amato MG, Fiskio JM, Bates D. Comparison of Overridden Medication-related Clinical Decision Support in the Intensive Care Unit between a Commercial System and a Legacy System. Appl Clin Inform 2017; 8: 866–879 https://doi.org/10.4338/ACI-2017-04-RA-0059


#

1. Background and Significance

Electronic health records (EHRs) with clinical decision support (CDS) have been found in some studies to reduce hospital length of stay and costs, and also to decrease the incidence of adverse drug events (ADEs) [[1]–[3]]. Despite these clinical benefits, medication-related CDS alerts are common and often annoying to clinicians, and they are often overridden, sometimes inappropriately [[4]]. Patients in the intensive care unit (ICU) are vulnerable to ADEs both because they are ill and because they are receiving many medications [[5]]. Limited data exist regarding the benefits of medication-related CDS specifically in the ICU [[6]].

Hospitals across the United States are rapidly adopting EHRs because of federal incentives to do so, and most of these systems are vendor systems. With the recent transition of our EHR from a legacy, homegrown system to a commercial, vendor system, significant differences in experience with CDS have been noted by providers. Although comparisons between CDS systems and recommendations for successful implementation of CDS with new systems have been reported, information regarding the most effective presentation (e.g., timing of CDS appearance in workflow, display features, clinical content) of CDS is limited [[7], [8]]. These studies addressed how to improve CDS presentation through expert panels. They identified that common weaknesses in EHR systems include lack of alert prioritization, clarity of CDS recommendations and the rationale of why the alert is present (i.e., consequence of overriding the alert) [[7]]. Addressing these weaknesses may be limited by the capability of the institution to modify available CDS and the ability to maintain and update available CDS [[8]].

One key aspect of our legacy system was the ability to tailor our CDS based on the underlying literature and institutional needs, which was effective at reducing the incidence of overrides [[9]]. Another major change was the presentation of all alerts at the same time in the commercial system at the time of signing, allowing providers to override multiple alerts at the same time, despite the potential for differences in significance of the alerts, compared to at the time of order in the commercial system. This study allows for the opportunity to compare two implemented EHR and CDS systems at a single institution, which will allow for greater understanding of what other institutions that may be changing EHR systems should look for at implementation.


#

2. Objectives

The purpose of this study was to compare the CDS presentation and override rate between our institution’s legacy and commercial EHR systems and to identify potential areas for improvement. Based on clinical experience, we expected the override rate to be higher in the commercial system than the legacy system.


#

3. Methods

We performed a retrospective study comparing two CDS systems at Brigham and Women’s Hospital: a legacy, homegrown system and a commercial, vendor system that was implemented in May 2015. Brigham and Women’s Hospital is an urban, tertiary care center that implemented their legacy EHR system in 1993. Efforts have been continually made within our institution to implement the use of technology to improve patient outcomes. Further details about this system have been detailed elsewhere [[10]]. Medication-related CDS overrides were obtained from the legacy system (2011) and the commercial system (2015) from October to December of their respective years. This time period was chosen as a convenience sample of available data from the legacy system. Alerts were restricted to adult patients in the medical, neurology, and surgical ICUs. Both systems utilized a similar knowledge base for CDS, although the legacy system had been modified for certain aspects of alerts based on institutional needs. The bed count for the selected ICUs was similar between the years in the comparison (2011: n=60; 2015: n=58).

3.1 Alert Characteristics

Targeted alerts focused on alert types that fire frequently and that most clinicians agree are of significance for critically ill patients: allergy, drug-drug interaction (DDI), geriatric and renal alerts. The proprietary Partners Knowledge Base was used as the basis of the DDI, geriatric and renal alerts in the legacy system, which had been customized over years based on end-user feedback and review of literature [[1], [11]]. Allergy alert logic in the legacy system was initially sourced from First DataBank (First DataBank, South San Francisco, CA, USA) and then reviewed and adapted for local use by an expert committee. First DataBank provided the knowledge bank for the commercial system. ►[Table 1] details differences between the CDS and their presentation in the legacy and commercial systems.

For DDI alerts, the legacy system used a filtered DDI knowledge base, limiting the DDIs shown to the provider. Only level 2 DDIs from the legacy system were included in the study analysis, while all DDIs were included from the commercial system. This was because only level 2 DDIs in the legacy system were able to be overridden, while all DDIs in the commercial system were displayed to the provider and able to be overridden. Therefore, only alerts that could have been overridden in each system were evaluated (i.e., level 2 DDI in the legacy system and all DDIs alerts in the commercial system). DDIs in the commercial system had tiering in alert presentation according to severity based on First DataBank’s database, but there were no hard stops. Geriatric alerts and renal alerts from the commercial database focused on medications that were not recommended given a patient’s age or renal function. Only a limited number of geriatric alerts are included in this analysis as most alerts were presented to only the provider and not to other healthcare practitioners (e.g., pharmacist, nurse). Only those presented to all healthcare practitioners were included in this study as they were linked to an override having occurred in the medication order.

Table 1

Differences in medication-related clinical decision support between two systems

Alert Type

Legacy System

Commercial System

Allergy

Provides information on last override reason, if available

Documentation as only an ‘allergy’

Documentation allows for differentiation between types (e.g., allergy, intolerance)

Coded and free-text options for reaction

Drop-down list for 38 reactions, including ‘Other (see comments)’

Provides information on the severity of the allergy (input during initial allergy documentation; based on documented reaction)

Drug-drug interactions

Tiering:

  • Level 1 – unable to bypass (i.e., hard stop)

  • Level 2 – interruptive and required override reason

  • Level 3 – informational (i.e., not displayed to provider)

Tiering based on First Databank (e.g., very high, high)

Color differentiation used for tiers of alerts

Tailored based on multidisciplinary, expert opinion/published evidence/quality improvement efforts

Two alerts included that were not in the commercial system

‘Contraindicated’ medications able to bypassed

Geriatric

Presented to all healthcare providers

Majority presented as warning to only ordering provider

‘Contraindicated’ alerts only presented to all providers

Provided alternative medications and recommended doses

Provided alternative medications but no recommended dosing

Renal

Provided alternative medications and recommended doses (including degree of renal impairment)

Provided alternative medications but no recommended dosing


#

3.2 Evaluation of ADEs

A post hoc analysis evaluating for ADEs was performed for the overridden ‘contraindicated drug combination’ DDI alerts in the commercial system. This was completed as these alerts would have been analogous to Level 1 alerts in the legacy system and therefore, would not be able to be overridden. This evaluation required at least one administration of each medication in the DDI combination. Patient medical records were evaluated by one pharmacist reviewer for ADEs potentially resulting from the overridden alert (e.g., elevated creatinine kinase or documentation of myopathy for an interaction between selected macrolides and HMG-CoA reductase inhibitors).


#

3.3 Statistical Analysis

Descriptive statistics were used to summarize patient and alert characteristics. A Mann-Whitney U test was performed to compare the differences between groups for continuous variables (e.g., median number of alerts per patient). A chi-square test or Fisher’s exact test were used to compare the differences between groups for categorical variables (e.g., override rate for alert category), as appropriate. A p-value of < 0.05 was considered significant. Statistical analysis was completed using SAS 9.3 (SAS Institute Inc., Cary, NC, USA).


#
#

4. Results

A total of 1,030 and 5,535 overridden alerts were identified from the legacy and commercial systems, respectively. Characteristics of patients and the overridden alerts are described in ►[Table 2]. A further breakdown of the number of unique overridden alerts per patient (i.e., a patient with multiple alerts for a penicillin allergy with the same medication would only count as one alert), the number of unique patients per alert type, and the median number of overridden alerts per patient per category is described in ►[Table 3]. The most common medication(s) overridden per alert category are detailed in ►[Table 4].

Table 2

Characteristics of patients and overridden alerts.

Legacy

Commercial

p-value

Number of medication orders, n

42,097

36,800

N/A

Total # of unique patients admitted to ICU, n

684

783

N/A

Medical ICU, n (%)

247 (36.1)

274 (35.0)

0.655

Neurology ICU, n (%)

232 (33.9)

272 (34.7)

0.741

Surgical ICU, n (%)

205 (30.0)

237 (30.3)

0.901

Number of unique patients with alerts, n (%)

254 (37.1)

706 (90.2)

< 0.0001

Total # of overridden alerts, n

1,030

5,535

N/A

Allergy, n (%)

486 (47.2)

697 (12.6)

< 0.0001

DDI, n (%)

326 (31.7)

4,566 (82.5)

< 0.0001

Geriatric, n (%)

207 (20.1)

4 (0.1)

< 0.0001

Renal, n (%)

11 (1.0)

268 (4.8)

< 0.0001

DDI = drug-drug interaction; ICU = intensive care unit.
Table 3

Characteristics of overridden alerts per alert category.

Legacy

Commercial

p-value

Allergy

Number of unique patient-alerts[*]

143

221

N/A

Number of unique patients

105

139

N/A

Median # unique alerts per patient[]

1.0 (1.0, 1.0)

1.0 (1.0, 2.0)

0.992

Median # alerts per patient (IQR)

3.0 (2.0, 5.0)

3.0 (2.0, 6.0)

0.451

DDI

Number of unique patient-alerts[*]

148

1390

N/A

Number of unique patients

114

487

N/A

Median # unique alerts per patient[]

1.0 (1.0, 1.0)

4.0 (2.0, 6.0)

< 0.0001

Median # alerts per patient (IQR)[]

2.0 (1.0, 4.0)

4.0 (2.0, 10.0)

< 0.0001

Geriatric

Number of unique patient-alerts[*]

113

4

N/A

Number of unique patients

89

4

N/A

Median # unique alerts per patient

1.0 (1.0, 2.0)

1.0 (1.0, 1.0)

0.105

Median # alerts per patient (IQR)[]

2.0 (1.0, 3.0)

1.0 (1.0, 1.0)

< 0.0001

Renal

Number of unique patient-alerts[*]

7

180

N/A

Number of unique patients

7

68

N/A

Median # unique alerts per patient[]

1.0 (1.0, 1.0)

2.0 (1.0, 3.5)

< 0.0001

Median # alerts per patient (IQR)

1.0 (1.0, 2.0)

2.5 (1.0, 5.0)

< 0.0001

* Indicates the number of unique alerts per patient (i.e., a patient with multiple alerts for a penicillin allergy would only count as one). † Indicates p-value < 0.05. DDI = drug-drug interaction.
Table 4

Most common medication(s) overridden per alert category.

Alert Category

Legacy

Commercial

Allergy, n (%)

Penicillin; 105 (21.6)

Codeine; 150 (21.5)

Codeine; 94 (19.3)

Penicillins; 80 (11.5)

Oxycodone + Acetaminophen; 38 (7.8)

Oxycodone + Acetaminophen; 46 (6.6)

DDI, n (%)

Haloperidol-Ondansetron; 36 (11.0)

Haloperidol-QTc Prolonging Agent; 436 (9.5)

Citalopram-Omeprazole; 34 (10.4)

ACE Inhibitors; ARBs-Potassium Preparations; 395 (8.7)

Diltiazem-Simvastatin; 32 (9.8)

Beta-2 Agonists / Non-cardioselective beta-blockers; 366 (8.0)

Geriatric, n (%)

Bisacodyl; 142 (68.6)

Nifedipine; 3 (75.0)

Meperidine; 13 (6.3)

Indomethacin; 1 (25.0)

Clonazepam; 12 (5.8)

Renal, n (%)

Aspirin; 7 (63.6)

Potassium chloride; 127 (47.4)

Hydrochlorothiazide; 4 (36.4)

Magnesium sulfate; 52 (19.4)

Desmopressin; 19 (7.1)

ACE = angiotensin-converting enzyme; ARBs = angiotensin receptor blockers; DDI = drug-drug interaction.

4.1 Evaluation of Allergy Alerts

The override rates for allergy alerts for the legacy and commercial system were 90.7% and 93.4%, respectively (p<0.001). The commercial system included the functionality of denoting a reaction to a medication as an allergy or intolerance. However, only 23 (3.3%) alerts were associated with documentation of intolerance.


#

4.2 Evaluation of Drug-Drug Interaction Alerts

The override rate for DDIs for the legacy and commercial system were 72.9% and 97.0%, respectively (p<0.001). There were a significantly increased number of potential DDI alerts and provider exposure to alerts in the commercial system compared to the legacy system. A comparison of the override rates of three commonly overridden alerts in the commercial system with a related alert in the legacy system is detailed in ►[Table 5]. Most alerts that were overridden in the legacy system were included in the alerts from the commercial system. Exceptions were an alert for bupivicaine for epidural use and anticoagulants (n=26, 8.0%) (different alert type in commercial system) and an alert for the co-administration of ceftriaxone and intravenous calcium for neonates (n=21, 6.4%), which is not clinically relevant to the adult ICU population and was found to be an error in the system. A total of 6 alerts that were not available in the legacy system were available in the commercial system. The two most common examples were β-agonists and non-cardioselective β-blockers (n=404, 8.8%) and propofol and QT-prolonging agents (n=261, 5.7%). In comparing the level of severity for DDI alerts in each system, a total of two were a higher severity level in the legacy system, while 30 were a higher severity level in the commercial system. For example, an alert for a DDI involving neuromuscular blockers and polypeptide antibiotics (e.g., colistin) was a level 2 interaction in the legacy system but was a moderate alert in the commercial system, which would correlate to a level 3 DDI in the legacy system (e.g., informational). Therefore, the DDI severity was higher in the legacy system than the commercial system.

Table 5

Difference in override rates of three common drug-drug interactions in commercial system.

Legacy

Commercial

p-value

Amiodarone – QTc prolonging agents – # overridden/total # alerts (%)

20/30 (66.7)

138/141 (97.9)

<0.001

Ondansetron – QTc prolonging agents – # overridden/total # alerts (%)

46/68 (67.6)

357/385 (92.7)

<0.001

Haloperidol – QTc prolonging agents – # overridden/total # alerts (%)

60/82 (73.2)

436/453 (96.2)

<0.001

In evaluating the ‘contraindicated drug combination’ DDI alerts in the commercial system, a total of 73 were overridden (1.6% of all overridden alerts). This was accounted for by 37 unique patient-alerts and 36 unique patients. Of the 73 alerts, only 12 (16.4%) were level 1 alerts in our legacy system. Only 43 (56.6%) of the alerts had documented administration of both medications in the DDI combination to the patient. Two types of alerts accounted for the majority of these alerts: increased levels of HMG-CoA reductase inhibitors (n=20, 46.5%) and increased risk of neuroleptic malignant syndrome (n=18, 41.9%). We found no ADEs with the DDI combinations that were overridden.


#

4.3 Evaluation of Geriatric Alerts

The availability of geriatric alerts and medications to avoid in this population was a functionality that was significantly changed in the commercial system, with the only overridden alerts comprised of indomethacin and short-acting nifedipine (override rate 100%). They had changed to be presented to the ordering provider as a different warning earlier in the order entry process, which suggested substitute medications that could be chosen in place of the originally selected medication. However, no dose recommendations were offered if the ordering provider chose to accept this change. Additionally, these alerts were not presented to pharmacists or nurses in evaluating the order itself, and there was no history of an override, unlike the other alerts that we evaluated. It appeared that only medications that were “contraindicated” based on the commercial system came through as an alert. These ‘overrides’ are not documented as an overridden alert, as it is not defined as a medication CDS alert in our study.


#

4.4 Evaluation of Renal Alerts

Renal alerts differed significantly between systems in number and presentation. The override rate was 96.1% in the commercial system (unable to be obtained for legacy system). The difference in the number of renal alerts between the two systems is largely due to the inclusion of electrolyte CDS alerts in the commercial system. In this system, intravenous electrolyte orders accounted for the 10 most common overrides. The alerts in the commercial system still occurred despite the majority being written for a once order or as an electrolyte protocol per institutional guidelines. All other orders were for a set number of doses, appropriate for the serum electrolyte level. Desmopressin was the most common non-electrolyte with an overridden renal alert in the commercial system, which were all ordered as single doses. Our legacy system also calculated creatinine clearance and offered medication substitution recommendations for dose and frequency adjustments, CDS which is not currently provided with the commercial system. Similar to the geriatric alerts, renal alerts were presented differently to the ordering provider (►[Figure 1]). In the legacy system, the provider was provided with an informational notification that a change in the default dosage of a renally cleared medication in a patient with renal insufficiency had been made and the provider did not need to take further action. In the commercial system, an alert was presented stating that a change was required based on renal function, but no recommendations for new dosage were made by the system.

Zoom Image
Fig. 1 Comparison of an order for allopurinol in a patient with renal dysfunction between the two systems. Figure 1a and 1b are derived from the legacy system, while 1c is from the commercial system. The legacy system provided dose recommendations for an order for allopurinol 300 mg daily, while the commercial system identifies the presence of renal impairment but does not recommend a specific dose. (c) 2017 epic Systems Corporation. Used with permission.

#
#

5. Discussion

We evaluated the differences in the presentation and overrides of medication-related CDS between two CDS systems at our institution. There were significant differences in the studied number of alerts per patient and number of unique alerts per patient between the two difference systems, with the largest difference in the DDI alerts. Several differences in the presentation of medication CDS and overrides were apparent between the two systems with the most notable differences involving the DDI, geriatric and renal alerts. There is certainly room for improvement with the commercial system, especially since the tailoring that was performed with the legacy systems had largely been lost. In a small sample of ‘contraindicated’ DDI alerts which were overridden, we did not find evidence of harm related to the overrides.

The findings of this study add to the limited data comparing different CDS systems, especially in the ICU. Common CDS tools such as dosing support and drug-allergy interaction functionality have been identified to be common but how they are implemented has shown to have significant variation [[12], [13]]. Available literature is limited to general CDS and is not specific to the ICU. Evaluations of CDS in the ICU have mainly focused on internally developed systems and few have been done [[14]]. With the growing proportion of healthcare institutions adopting commercially available EHR systems, this study shows the importance of evaluating changes in CDS after the transition to ensure that quality improvement efforts are continued. Evaluation of overridden CDS by appropriateness of the override and also the harm associated with an override would be potential ways to evaluate the performance of these alerts [[14]]. A study evaluating the ICU patient population found a weighted appropriateness rate of 92.3%, which was heavily weighted by the number of allergy alerts. ADEs were found to be rare and with a higher incidence in inappropriately overridden alerts [[14]].

We believe that based on the small relative differences in the number of patients admitted to the included ICUs and the number of medication orders, the differences we found in alert and override rates appear to be related to the change in CDS. A large change in the presentation of the CDS was the time of presentation to the provider from upon ordering with the legacy system to upon signing in the commercial system. In speaking with providers at our institution, most believed that the time of alert presentation interfered with their workflow, especially when they had to re-enter all orders even if there was only one medication they wanted to change. Another explanation for the differences in override rates and alert frequency is due to the knowledge base used. Our institution had continuously tailored the medication-related CDS at our institution based on published data, end-user feedback and recommendations of a multidisciplinary, expert committee to refine the legacy system CDS. Finally, “alert fatigue” may be even more prominent with users of the commercial system and has been associated with overrides [[15]]. This is supported by the significant difference in override rates of the QTc-prolonging DDIs, which would be expected to be similar between the two systems.

5.1 Discussion of Specific Alert Categories

Evaluation of the allergy alerts indicated that there were a similar number of alerts per patient between the two systems. This is expected as the same knowledge base was used for these alerts. Override rates were statistically significantly different between the two systems but was extremely high regardless (legacy: 90.7%; commercial: 93.4%). This may indicate that refinement of allergy CDS is beneficial, regardless of the system used.

One of the greatest differences between the two systems was in the overridden DDI alerts (override rates: legacy: 72.9%; commercial: 97.0%). Differences in the tiering of DDI alerts are due to the lack of standardization given the large number of knowledge bases available [[16]]. Although only level 2 DDIs were evaluated in the legacy system, they accounted for most of the overridden ones in the commercial system (i.e., levels 1 and 3 DDIs were rare in the commercial system because they were either contraindicated medication combinations or were strictly informational). A part of the difference in the DDIs found between the two systems may be that level 1 and 3 DDIs were not included, but we would anticipate there to remain a large difference between the two systems, as level 2 DDIs account for the majority of all DDIs in the legacy system. There was no DDI alert for propofol and QTc-prolonging medications in the legacy database. There is conflicting data on the clinical significance of this potential interaction [[17], [18]]. Studies evaluating the incidence of clinically significant QTc prolongation (QTc > 500 ms or QTc increase from baseline > 60 ms) in the ICU have found rates up to 48%, depending on the ICU studied [[19], [20]]. CDS focused on QTc prolongation in general has shown to significantly reduce this risk in a pre-post intervention study by 35% [[21]]. In 2016 (after our study period), “medium” severity level DDIs were removed from provider alerting (i.e., filtered from the alert display and only visible upon opening ‘filtered messages’), accounting for approximately 50% of overridden DDI alerts in our cohort. This was in an effort to reduce the alert burden for providers, which is a reasonable option. However, there are some significant interactions that are common in the ICU patient population that were removed due to this action (e.g., quetiapine and QTc-prolonging agents). Evaluation of large changes in alert burden would be interesting to identify if there is a subsequent change in override rate or if the rate remains consistent, due to already existing alert fatigue. A better method would be the ability to “turn off” alerts at the individual drug-drug level, instead of a severity level [[22], [23]]. It would be interesting to evaluate the effect of such a large decrease in alert burden on alert overrides, as well as the potential harm that may have resulted from this elimination of alerts. Although it could be argued that these interactions may be well-known and the ordering provider has had experience with the DDI prior, new providers and those who have not been exposed to the alert prior may be at a disadvantage.

The evaluation of ADEs associated with the contraindicated DDIs did not find any events. This may largely be explained by the rarity of harm or difficulty of diagnosis (e.g., neuroleptic malignant syndrome and as needed medications) associated with the interaction, or the need for an extended duration of treatment for an ADE to occur (e.g., myopathy/rhabdomyolysis from statins) [[24]–[27]]. Given the importance of these interactions, it is still reasonable to tier these interactions as hard stops, as the package inserts of these medications state that these drug combinations are contraindicated.

There were significant differences between the systems in regards to the geriatric and renal alerts. The finding that only a small number of overrides were associated with geriatric alerts in the commercial system was surprising. In evaluating the non-overridden geriatric alerts, there were numerous medications that were identified as ‘precautions’ that were ‘cancelled.’ In evaluating the individual orders, these medications had actually not been cancelled and were actually ordered and administered to these patients, which may just be a limitation of our data source. The renal alerts in the new application were very non-specific, and electrolytes (magnesium and potassium) accounted for the majority of these overrides. Since many were ordered per our institution’s electrolyte protocol or as “once” orders, they were likely appropriately ordered. Even for the non-electrolyte overrides, most of the remaining medications that were overridden had low clinical importance (e.g., desmopressin ordered as once orders, intravenous voriconazole) [[28], [29]].


#

5.2 Recommendations for Improving Clinical Decision Support

Areas for improvement from our legacy system include the ability to denote the type of reaction for a patient allergy and the removal of alerts that should not have been included initially in our knowledge base (e.g., short-duration laxatives in geriatrics). However, the presentation of the allergy alerts to the provider appears to be similar and not tiered, despite the severity or type of documented allergy. Presentation of only severe alerts (including severe intolerance to a medication) would likely increase the specificity of these alerts. Areas for improvement in the commercial system include incorporation of hard stops for certain DDI combinations shown to be harmful to patients (i.e., contraindicated), inclusion of additional alerts for medications to avoid in geriatric patients and age-appropriate starting dosages for older patients, and improving the renal alerts so that a dosage specific to the patient’s characteristics is suggested [[1], [7]]. Improvements in CDS should be focused on a human factors approach and not just on training of end-users of this technology [[7], [30], [31]].

Given the increased override rates apparent with the allergy and DDI alerts in the commercial system, this might be remediated in part by changing the time of presentation of alerts to the time of ordering, instead of at signing, and also by adjusting the underlying database [[31]]. Requiring the provider to “double-back” mentally to rethink an order once it has already been placed is not an efficient use of time. Although there are recommended alternatives associated with the medications that are recommended to not be used in geriatric patients, they are only shown to the ordering provider [[32]]. Presentation to others involved in the medication process (e.g., pharmacists) could increase the likelihood of appropriate prescribing of these medications. Increased specificity of the renal alerts could also be accomplished by eliminating alerts for appropriately ordered electrolyte protocols per institution policy, once orders for electrolytes given that the patient’s serum level is above or below a certain threshold, and for a set number of doses, expected to be appropriate given the patient’s last serum level.

Our study has several limitations. First, this was a single-center study. The findings of this study may not be applicable to those with a legacy system or those with a tailored commercial system. However, given the recent increase in adoption of EHRs and CDS in the United States, our study has identified potential areas of improvement for other institution’s EHR systems. Additionally, given the market share of our EHR (11.8%) and knowledge base providers, our experiences are likely reflective of a significant portion of the market [[33]]. Second, there was a 4-year gap between our evaluation periods. There is the potential for secular changes in CDS alerts in both EHR systems but we believe this would not change the findings our study. Thirdly, we did not evaluate the appropriateness of the overrides. Appropriateness evaluation of overrides has shown to be variable, based on the setting and the type of alert [[12], [34]]. Evaluation of the commercial system would allow us to further identify areas of improvement. Finally, we only evaluated a short time period of overrides. A more recent period of evaluation for the legacy system or a more extended time period may have identified more areas for improvement or identification of actual patient harm from the contraindicated DDI overrides. This would potentially have given us more objective outcome data to fuel potential changes in our EHR.


#
#

6. Conclusions

The medication-related decision support performed better in the legacy system than in the commercial system, though the analysis illustrates many opportunities for improvement with both. We will be using these data to help iteratively refine the new system. These findings are likely also an issue in many other institutions which are now using commercial EHRs.


#

Multiple Choice Question

1. In implementing a new clinical decision support system, which of the following is an important consideration in regards to presentation of alerts to reduce the potential for patient harm?

  • Presentation of alerts only to the ordering provider

  • Presentation of all alerts in knowledge base to the provider

  • Presentation of all alerts at time of signing

  • Presentation of only limited alerts as an interruptive alert

The correct answer is D. Only a limited number of alerts should be interruptive to signify the importance of these alerts in regards to severity of patient harm. As mentioned in our study, most geriatric alerts are presented only to the ordering provider. Allowing for these alerts to be presented to other professions would allow for a double-check of the ordered medication (Answer A). Presentation of all alerts that are available would likely lead to alert fatigue, reducing the effectiveness of presented alerts given the sheer amount (Answer B). Similarly, presentation of all alerts at the same time may overload a provider and lead to alert fatigue (Answer C).


#
#

Conflict of Interest

This study was funded by grant #U19HS021094 from the Agency for Healthcare Research and Quality (AHRQ). Dr. Bates reported receiving equity from Intensix, which makes software to support clinical decision-making in intensive care; being named as coinventor on patent No. 6029138 held by Brigham and Women’s Hospital on the use of decision support software for medical management, licensed to the Medicalis Corporation, and holding a minority equity position in Medicalis, which develops web-based decision support for radiology test ordering; consulting for Early-Sense, which makes patient safety monitoring systems; receiving equity and cash compensation from QPID Inc, a company focused on intelligence systems for electronic health records; receiving cash compensation from CDI (Negev) Ltd, which is a not-for-profit incubator for health IT startups; receiving equity from Enelgy, which makes software to support evidence-based clinical decisions, from Ethosmart, which makes software to help patients with chronic diseases, and from MDClone, which takes clinical data and produces deidentified versions of it. The remaining authors have disclosed that they do not have any conflicts of interest.

Clinical Relevance Statement

In this comparison of a legacy EHR and commercial EHR system in regards to CDS, we found significant differences in alert characteristics and frequency. We discuss recommendations to improve both systems, which may be extrapolated to other institutions. Recommendations include the timing of presentation of alerts to time of order instead of signing, increasing specificity by including patient-specific factors, and suggested doses based on a patient’s presentation.


Human Subjects Protections

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the Partners Institutional Review Board.



Correspondence to:

David Bates
Division of General Internal Medicine and Primary
Care, Brigham and Women‘s Hospital
Boston/USA


Zoom Image
Fig. 1 Comparison of an order for allopurinol in a patient with renal dysfunction between the two systems. Figure 1a and 1b are derived from the legacy system, while 1c is from the commercial system. The legacy system provided dose recommendations for an order for allopurinol 300 mg daily, while the commercial system identifies the presence of renal impairment but does not recommend a specific dose. (c) 2017 epic Systems Corporation. Used with permission.