Keywords
medical order entry systems - drug interactions - alert fatigue - alert systems -
clinical decision support
Background and Significance
Background and Significance
Drug–drug interactions (DDIs) are an important cause of preventable morbidity and
mortality in the hospital setting.[1] A DDI occurs when medications that are taken concurrently interfere with the anticipated
effect of one another, potentially resulting in an adverse drug event.[2]
[3] To minimize the risk of DDI-related adverse drug events in hospital patients, DDI
alerts are embedded into computerized provider order entry (CPOE) systems.[4]
There is limited research demonstrating that DDI alerts are effective in reducing
DDIs and subsequent patient harm.[5] A persistent and widespread problem appears to be that DDI alerts are frequently
ignored and overridden by prescribers.[6]
[7]
[8] This is likely due to the large number of false-positive DDI alerts presented to
prescribers, resulting in alert fatigue. Research has shown that prescribers could
experience between 20 and 145 DDI alerts per 1,000 prescriptions,[9]
[10] and that even 20 DDI alerts per 1,000 medication orders can result in alert fatigue.[10] Thus, reducing the quantity of alerts alone is not sufficient to ensure alerts are
read and acted upon. The clinical relevance of alert information should also be evaluated
to help guide alert inclusion.
To determine which alerts to include and exclude in CPOE requires evaluation of their
clinical relevance. Typically, this involves a review of alerts which are frequently
overridden by users.[11] Alerts that rarely result in a medication order being changed are viewed to be of
limited value. However, this methodology requires DDI alerts to be operational in
a CPOE before being evaluated.
Interestingly, evaluation of DDI alerts is rarely done prior to implementation in
a system.[12]
[13] Instead, DDI alerts, often part of the “out-of-the-box” vendor functionality of
CPOE, are implemented to meet Meaningful use or accreditation requirements.[14]
[15] However, evaluation and the subsequent removal of low-quality alerts following implementation
of DDI alerts has been shown to be a complex and challenging task.[11]
[16]
[17] For example, in one study, frequently overridden DDI alerts were reviewed by an
expert panel, but the panel could not agree that any of the 86 DDI alerts identified
should be removed from the system.[11]
Two approaches available to organizations wishing to evaluate their alerts prior to
implementation in a CPOE are drug compendia and expert panel review. A common approach
involves comparing the agreement between multiple drug interaction compendia on the
severity of detected interactions.[18]
[19] High agreement between compendia, regarding the severity of DDI alerts, could indicate
that the alert is clinically relevant.[18]
[19]
In other studies, clinical experts have been tasked with reviewing alerts, with the
aim of excluding irrelevant alerts from the system.[20]
[21] However, this approach is sometimes problematic, as poor agreement between panelists
regarding the clinical relevance of alerts has been found.[11]
[17]
[22]
[23] This may be because panelists are often required to rate alerts independently, with
no opportunity to discuss their decisions, and alerts are often presented in the absence
of the clinical context in which they are triggered, possibly making it difficult
for panelists to conceptualize clinical relevance. In previous studies, panelists
have been required to indicate whether an alert is relevant or not, and rarely given
the opportunity to identify the contexts in which they believe a specific DDI alert
to be clinically relevant.[11] Only triggering DDI alerts in the presence of particular context factors has been
suggested as a method of reducing false-positive alerts.[24] For example, if a particular DDI is harmful in newborns but not adults, such as
the coadministration of ceftriaxone and calcium, then the patient's age is a context
factor that could be used to dictate whether to trigger the DDI alert.[24]
At our study hospital, a decision was made to evaluate DDI alerts prior to implementation
in a CPOE. An audit of “hypothetical” DDI alert numbers revealed that DDI alert rate
would be high (147 DDI alerts per 1,000 medication orders) and would likely contribute
to the development of alert fatigue. Thus, this provided a strong rationale to evaluate
DDI alert quality in terms of clinical relevance before implementation into a CPOE.
Objective
The objective of this study was to trial two commonly used methods (compendia review
and an expert panel) to assess clinical relevance of DDI alerts before implementation.
The results of the study would inform which DDI alerts to implement and under what
context factors they should trigger.
Methods
Study Setting
This study was conducted in a 379-bed public teaching hospital in Australia. The hospital
uses the CPOE MedChart (referred to herein as MedChart). MedChart allows electronic
prescribing, review, and administration of medications.[25] Several computerized alerts are operational in the system including allergy alerts,
therapeutic duplication alerts, and local messages (e.g., reminders about antibiotic
restrictions). At the time of the study, DDI alerts were not enabled.
This study was approved by the hospital's human research ethics committee.
Identification of DDI Alerts for Testing Clinical Relevance
To generate DDI alerts, medications for a sample of patients (n = 78) were entered into MedChart's training environment and all hypothetical DDI
alerts were noted. The sample consisted of all patients that were discharged over
two consecutive days. The patients were from seven specialties including: neurology,
gastroenterology, infectious diseases, geriatrics, oncology, cardiology, and cardiothoracic
surgery.
The DDI compendium utilized by MedChart is MIMS, and as the hospital planned to implement
only DDI alerts of the highest severity, only severe DDI alerts were subsequently
assessed for quality.[26] In MIMS, severe DDI alerts warn against interactions between medications that may
be life-threatening or cause permanent damage.[26]
Assessing Quality of DDI Alerts Using Compendia
The severe DDI alerts were entered into three other drug interaction compendia: Stockley's
Drug Interactions, Micromedex, and YouScript. These compendia were selected based
on high usage and reputation.[9]
[27]
[28]
[29]
The severity rating of each drug pair that triggered a DDI alert was reviewed in the
three compendia (see [Appendix A]). Agreement between the three compendia and MIMS was assessed using Krippendorff's
α. An α of 1 indicates perfect agreement between compendia, a value of 0 no agreement,
and a value of –1 indicates an inverse agreement between compendia.[30] IBM SPSS Statistics Version 23 was used for analysis.
Assessing Quality of DDI Alerts Using an Expert Panel
The panel consisted of five health care professionals: two clinical pharmacologists,
two senior clinical pharmacists, and one geriatrician. Six patient cases were randomly
selected and presented to the panel. These cases would have triggered 13 different
hypothetical severe DDI alerts if enabled in a CPOE. Short case presentations were
delivered to the panel, including information on medical history and progress during
admission.
Panel members were asked to review the patient's medication charts and were provided
with supporting information including: the relevant hypothetical DDI alert, how frequently
the alert would fire in our sample of 78 patients if enabled, the literature summaries
from the four compendia, as well as the alerts severity rankings from each compendium.
For each of the DDI alerts, panelists individually decided whether the alert should
be included in MedChart. Then, all panelists presented their independent view as a
prelude to a general open discussion. During the discussion, the panelists explained
how and why they determined whether an alert should or should not be included in a
CPOE and attempted to form a consensus. The panel was encouraged to discuss context
factors that related to the patient (i.e., age), the medications (i.e., dose and route
of administration), and the organization (i.e., whether the prescriber was a junior
medical officer). Consensus was defined as a minimum of four of the five panelists
reaching agreement, after open discussion. Panelists were informed that it was not
necessary for a consensus opinion to be reached.
Results
Alert Quality Using Compendia
A total of 147 DDI alerts were triggered by 45 unique DDI drug pairs in the 78 hospital
patients. These 45 drug pairs were entered into Stockley's Drug Interactions, Micromedex,
and YouScript. In total, 8 of the 45 unique drug pairs (18%) were ranked as severe
in all four compendia. These 8 drug pairs accounted for 15 of the 147 (10%) alerts.
Statistical analysis confirmed that there was poor agreement between drug compendia
on the severity classification of DDIs. The Krippendorff's α was 0.03 with a 95% confidence
interval of –0.07 to 0.14.
The six drug pairs that would have triggered the highest number of DDI alerts, using
the MIMS interaction module, are shown in [Table 1]. Oxycodone and oxycodone/naloxone was the drug pair that triggered the most alerts
in our sample, but this interaction was only classed as a severe DDI by MIMS.
Table 1
Top six drug pairs resulting in DDI alerts
|
Drug pair
|
Number of DDI alerts triggered (% of total)
|
Compendia that ranked DDI as severe
|
|
Oxycodone and naloxone/oxycodone
|
13 (9)
|
1/4
|
|
Amiodarone and furosemide
|
10 (7)
|
2/4
|
|
Amiodarone and tacrolimus
|
9 (6)
|
3/4
|
|
Morphine and naloxone/oxycodone
|
9 (6)
|
3/4
|
|
Itraconazole and tacrolimus
|
7 (5)
|
3/4
|
|
Amiodarone and warfarin
|
7 (5)
|
2/4
|
Abbreviation: DDI, drug–drug interaction.
Alert Quality Using an Expert Panel
The panelists reached consensus after open discussion on 12 of the 13 alerts triggered
in the patient cases ([Table 2]). Nine were recommended for inclusion in the system, but for four, it was suggested
that these should trigger only in certain clinical contexts. These context factors
are shown in [Table 2].
Table 2
Findings from the expert panel
|
Drug pair
|
Panel's response after open discussion
|
Key context factors identified
|
|
Furosemide and gentamicin
|
Include
|
|
|
Amiodarone and domperidone
|
Include
|
|
|
Amiodarone and warfarin
|
Include
|
|
|
Ondansetron and domperidone
|
Include
|
|
|
Amiodarone and ondansetron
|
Include
|
|
|
Heparin and salicylates
|
Include in certain clinical contexts
|
• IF patient is older than 66 y old THEN trigger
• IF renally impaired THEN trigger
• IF hyperkalemic THEN trigger
|
|
Methadone and ondansetron
|
Include in certain clinical contexts
|
• IF no ECG performed THEN trigger
• IF dose (ondansetron) > 16 mg THEN trigger
• IF dose (methadone) > 80 mg THEN trigger
• IF route of ondansetron is parenteral THEN trigger
|
|
Enoxaparin and warfarin
|
Include in certain clinical contexts
|
• IF INR is elevated THEN trigger
• IF junior medical officer prescribing THEN trigger
• IF renal impairment THEN trigger
|
|
Temazepam and olanzapine
|
Include in certain clinical contexts
|
• IF patient is older than 75 THEN trigger;
• IF more than 5 h between administrations THEN do not trigger
• IF STAT dose THEN do not trigger
|
|
Lorazepam and olanzapine
|
No consensus
|
|
|
Amiodarone and bisoprolol
|
Exclude
|
• The alert is warning for the desired effect (i.e., bradycardia)
|
|
Aspirin and selective serotonin receptor inhibitors
|
Exclude
|
• Well-known DDI with likely low significance in the average person
• Common combination that is taken without adverse drug events
|
|
Ramipril and spironolactone
|
Exclude
|
• Indications outweigh the risks of DDI
|
Abbreviations: DDI, drug–drug Interaction; ECG, electrocardiogram; INR, international
normalized ratio.
Discussion
This study used two common methods to evaluate the clinical relevance of DDI alerts
before implementation into a CPOE. In assessing the quality of alerts using compendia,
only a small number of hypothetical alerts (10%) were classed as severe by all four
compendia. The very poor agreement found between compendia with respect to classifications
of severity, brings into question the usefulness of compendia alone to determine which
alerts should be included or excluded in a CPOE. This poor agreement between compendia
is echoed in other studies[18]
[31]
[32] and is due to different, nontransparent DDI prediction models used to classify the
severity of interactions.[20] The poor agreement is concerning, as it is common for prescribers to only consult
one compendium when reviewing risk of adverse drug events.
When a subset of DDI alerts were presented to an expert panel in the context of individual
patient cases and panelists were given the opportunity to discuss their views, we
found high agreement between panelists with respect to whether alerts should be included
in a CPOE. Of the 13 alerts presented, panelists reached a consensus on 12 of the
alerts. Panelists agreed that three should be excluded and four included only in certain
contexts. Previous studies have reported moderate to poor agreement between panelists
when determining the importance of DDI alerts.[11]
[22]
[23] The high agreement observed in our study could be attributed to the methodology
adopted, which differed from the approach taken in other studies.[22]
[23] First, the panelists were encouraged to openly discuss the reasoning for their recommendations,
while other studies have utilized an independent and noncollaborative approach to
the assessment of alert relevance.[22]
[23] Second, alerts in this study were presented in the context of particular patient
cases. Consideration of the patient and their clinical context is a crucial element
in determining clinical relevance. The panel members were able to consider the context
factors impacting on the probability DDI resulting in an adverse effect.[24] For example, the panel agreed that the increased risk of bleeding due to the coadministration
of enoxaparin and warfarin was only clinically relevant when the patient's international
normalized ratio (INR) (a laboratory indicator for bleeding risk) was elevated. Incorporating
INR information into the alerting system would ensure that when triggered, the DDI
alert would be relevant to the prescriber's decision.
Research has shown that only triggering alerts in the presence of relevant context
factors could reduce overall alert burden significantly.[24] Our current evaluation demonstrates that presenting alerts in conjunction with patient
context and allowing for open discussion, not only facilitated agreement between panelists
on clinical relevance of DDI alerts but allowed the identification of context factors
for improving alert specificity.
This study had several limitations. The expert panel was limited by time and was only
able to review a sample of 13 alerts that triggered in 6 patient cases. Level of experience,
profession, and expertise of panel members may have impacted results; however, we
attempted to minimize these influences by including a range of different professionals
(i.e., pharmacists, clinical pharmacologists, and geriatricians) and providing each
panelist with an opportunity to share their view. Finally, we reassured the panelists
that reaching a consensus was not necessary.
Although utilizing an expert panel is more resource-intensive and time-consuming than
inputting medications into compendia and reviewing agreement, we suggest presenting
a large sample of alerts to the expert panel before implementation. This would ensure
that alerts likely to cause the highest burden to prescribers are clinically relevant.
An evaluation of alert burden and alert relevance postimplementation would reinforce
the effectiveness of this approach to minimize the risk of alert fatigue.
Conclusion
In this study, we assessed the clinical relevance of DDI alerts in terms of clinical
relevance prior to their implementation in a CPOE. We found drug compendia to be unreliable
in their classification of DDI alerts, but the expert panel was highly consistent
in their assessments of clinical relevance and in their identification of context
factors, most likely because alerts were presented in the context of specific patient
cases, and experts were permitted to share their clinical knowledge and discuss any
differences in opinion. Although more resource-intensive, we recommend expert panel
review as an effective approach for assessing clinical relevance of DDI alerts prior
to alert implementation, to minimize the risk of alert fatigue before prescribers
are exposed to alerts.
Clinical Relevance Statement
Clinical Relevance Statement
Hospitals across the world are utilizing CPOE systems and, commonly, enable extra
functionalities such as drug–drug interaction alerts. However, due to overexposure,
alert fatigue is a common problem. This article identifies a method that may reduce
alert rate by improving clinical relevance of alerts before they are implemented in
a CPOE.
Multiple Choice Question
What did this study do differently, that potentially resulted in panelists having
higher agreement on whether DDI alerts should be included or excluded from the CPOE?
-
Blinded the panelists to each other's answers to keep views unbiased.
-
Blinded the panelists to the clinical context of the patient to ensure that the results
were generalizable.
-
Allowed the panelists to discuss their reasoning openly with each other with respect
to the clinical relevance and important context factors with each DDI alert.
-
Used only panelists from one specialty so that they would reach consensus.
Correct Answer: The correct answer is option c. Allowing for open discussion gave way to fruitful
debate between the panelists and was successful at helping them come to agreement
on 12 of the 13 DDI alerts they reviewed.
Appendix A
Severity ratings from multiple drug compendia
|
Comparative ranking
|
MIMS drug interaction checker
|
Micromedex
|
YouScript
|
Stockley's Drug Interactions
|
|
SEVERE
|
Severe: The interaction between these medications may be life-threatening or may cause permanent
damage. These medications are not usually used concurrently; medical intervention
may be required
|
Contraindicated: Drug pairs are contraindicated for concurrent use
|
Contraindication: This drug has an interaction that is contraindicated in the product insert due to
the potential for a severe or life-threatening reaction. This combination should not
be administered together
|
Severe: Interactions that may totally incapacitate a patient or result in permanent detrimental
effect. Can be life-threatening
|
|
Major: Potentially life-threatening interactions and/or require medical intervention to
minimize serious adverse effects
|
Major Clinical Impact: This drug has an interaction that may result in severe clinical effects or large
changes in drug levels. The risks of the interaction generally outweigh the benefits
of prescribing the drug
|
|
MODERATE
|
Moderate: These medications may interact resulting in the potential deterioration of the patient's
condition. The patient should be monitored for the possible manifestations of the
interaction. Medical intervention or a change in therapy may be required
|
Moderate: The interaction may result in exacerbation of the patient's condition
and/or require an alteration in therapy
|
Moderate Clinical Impact: This drug has an interaction that may result in substantial clinical effects or
moderate changes in drug levels. Changes in therapy, such as making dose adjustments
or prescribing alternatives, may be warranted
|
Moderate: For interactions that could cause considerable distress or partial incapacitation
of patients. Unlikely to be life-threatening
|
|
MINOR
|
Minor: Clinical effects of the interaction are limited and may be bothersome but would not
usually require a major change to therapy. The patient should be monitored for the
possible manifestations of the interaction
|
Minor: The interaction would have limited clinical effects. Manifestations may include an
increase in the frequency or severity of the side effects but generally would not
require a major alteration in therapy
|
Minor Clinical Impact: This drug has an interaction that may result in minor clinical effects or small changes
in drug levels. The benefits of prescribing the drug generally outweigh the risks
of the interaction. Major changes in therapy are not expected, although minor dose
adjustments may be appropriate
|
Mild: For interactions that are unlikely to result in an effect or that if an effect was
to occur it would be mild and unlikely to incapacitate the majority of patients
|
|
Caution: The interaction may occur based on the mechanism of action of the coadministered
medicines. Be alert for increased or decreased effect, depending on the combination
of medicines
|
Minimal Clinical Impact: This drug may be associated with clinically insignificant and/or favorable interactions.
No change in therapy is necessary
|
Note: Compendia used nonstandardized severity terminology. This table shows how these
inconsistent labels were grouped for comparison across compendia.
