Keywords clinical decision support systems - heart failure - prescribing - PRISM - RE-AIM -
implementation science
Background and Significance
Background and Significance
Over the past two decades, there have been numerous advances in drug therapy for the
treatment of patients with heart failure and reduced ejection fraction (HFrEF). These
medications include β blockers (BB), which have demonstrated improvements in quality
of life, rehospitalization, and mortality.[1 ]
[2 ]
[3 ]
[4 ]
[5 ] As such, BBs are recommended for patients with HFrEF.[6 ]
[7 ]
[8 ]
[9 ] Exceptions to these recommendations are limited to patients with contraindications
or who are intolerant of BBs. Unfortunately, prescribing of BBs remains suboptimal,[10 ] stymying their impact on patient outcomes.
To increase uptake of these medications and their impact, many have tested a variety
of different approaches with varying success. These approaches include provider education,
direct to patient education and empowerment,[11 ] new care delivery models,[12 ] and national efforts such as Get With The Guidelines.[13 ] However, a rate-limiting step of many approaches is resource availability for initial
delivery and ongoing maintenance. In contrast to approaches that require additional
personnel to deliver and maintain, automated health IT solutions within the electronic
health record (EHR) are alternatives that are receiving increasing attention.[14 ]
[15 ]
[16 ]
As the technical capacity of EHRs advance and increasingly automate health care workflows,
health systems are learning how to effectively implement user-centered health IT solutions.
Such solutions include clinical decision support (CDS) tools, which when well designed,
can expedite the translation of evidence to practice by providing clinicians with
the right information at the right time.[17 ] Within this evolution of learning how to effectively leverage CDS, there are examples
of CDS implemented to improve guideline concordant prescribing for HFrEF. The earliest
examples of such CDS demonstrated limited or no change in prescribing behavior, likely
because they were constrained by the technical capacity of EHRs at that time. These
early examples were also designed with limited consideration of what are now considered
best practices in the design of CDS.[18 ]
[19 ] The best practices in CDS design includes a user-centered design process to bolster
contextual relevance and careful consideration of implementation issues.[20 ]
[21 ]
[22 ]
[23 ] However, the best practices provide limited direction on the implementation issues,
which can be addressed through the field of implementation science (IS).[24 ]
IS methods aim to improve the systematic uptake of research and evidence-based practices
into real-world settings.[25 ] IS prioritizes the use of pragmatic methods and study designs to develop feasible
and sustainable implementation strategies that maximize external relevance and the
use of iterative, multilevel partner engagement to promote representativeness and
address contextual issues.[25 ]
[26 ]
[27 ]
[28 ] Central to IS methods is use of a theory, model, or framework to guide systematic
assessment of contextual factors and alignment of the intervention with the context
to maximize uptake, reproducibility, and sustainability.[29 ]
[30 ]
[31 ] Many IS frameworks, including the Practical, Robust Implementation and Sustainability
Model (PRISM),[32 ]
[33 ] are broadly applicable across settings and clinical situations; thus, they are used
with other criteria or standards specific to the situation at hand. In this case of
implementing a CDS to improve HFrEF prescribing, an IS framework helps to integrate
the HFrEF clinical practice guidelines and established best practices in CDS design
to inform the design of the CDS.[34 ]
In a recent randomized controlled trial, we found that using the PRISM IS framework
to inform the design of an “enhanced” CDS resulted in higher rates of BB prescription
for patients with HFrEF compared with a traditional or “commercial” CDS (0 vs. 14%
led to prescription, p = 0.006).[35 ] Although these results suggest that a CDS informed by IS principles and methods
may be more effective at improving guideline-concordant prescribing, the study had
a relatively small sample size (87 patients, 118 clinicians) and analyses did not
account for clustering due to clinic or adjust for patient characteristics. The trial
was also not able to evaluate the sustained effect of the enhanced CDS over time.
Therefore, we conducted a natural experiment with a larger sample size to better assess
the differences between the two CDS tools and to examine the sustained effect of the
enhanced CDS. The purpose of this study is to evaluate the real-world effect of applying
an IS framework and methods that considers best practices in CDS design compared with
a commercially available CDS on the reach, adoption, and sustainment of clinician
prescribing for patients with HFrEF.
Methods
Study Design
We conducted a natural experiment that expands the follow-up of our prior cluster
randomized trial,[35 ] where all sites were transitioned to the enhanced CDS tool. Briefly, in the prior
trial, 28 primary care clinics were randomized to either a commercial or enhanced
CDS using block randomization to allocate each clinic to one of the two CDS. Both
CDS were interruptive and designed to alert during a primary care visit and recommend
clinicians initiate one of three evidence-based BBs (i.e., metoprolol succinate, carvedilol,
bisoprolol) for patients with HFrEF, if not already prescribed. The enhanced CDS was
developed using a multilevel user-centered design process informed by the best practices
in CDS design and the PRISM IS framework.[34 ] PRISM guides the systematic assessment and alignment of evidence-based interventions
with the multilevel contextual determinants that influence implementation success
and sustainability.[32 ]
[33 ] PRISM also includes the widely used Reach, Effectiveness, Adoption, Implementation,
and Maintenance (RE-AIM) evaluation outcomes to inform pragmatic outcome measures.[36 ]
[37 ] The commercial CDS was provided by the EHR vendor (Epic Systems, Verona, Wisconsin,
United States), thus to our knowledge was not informed by a user-centered design process
or PRISM. A summary of the key differences in design of the two CDS is provided in
[Table 1 ] and a detailed summary is described elsewhere.[35 ]
Table 1
Key differences between the commercial and enhanced clinical decision support tools
Design feature
Commercial CDS tool
Enhances CDS tool
Inclusion criteria
Adults with any ICD-9/10 diagnosis for HF and an ejection fraction </= 40%
No documented allergy to a BB or β-agonist (relied on vendor supplied knowledge content,
which erroneously included β-agonists with BB)
Adults with an ICD-9/10 diagnosis that explicitly states the EF is </= 40% or an ejection
fraction </= 40%
No documented BB allergy (was curated by clinician informaticists to ensure accuracy/completeness)
Format and timing
Interruptive pop-up at time of opening encounter
Response options (time period that a future alert is delayed for)
Open an order set with many options including order medications, laboratories, referrals,
echocardiogram
or
Indicate “contraindicated” (90 d), “cost concern” (90 d), “patient declines” (90 d)
or
Dismiss option
Pend an order for a BB at staring doses
or
Indicate “never appropriate” (>20 y), “remind me later (1 mo)” (28 d), “provide comment”
(28 d)
Informational content displayed
Most recent EF, blood pressure, heart rate
BB recommended
Rationale for recommendation with link to more information
Guidance of which BB are evidence-based and that asthma/chronic obstructive pulmonary
disease are not contraindications
Most recent EF
BB recommended
Abbreviations: BB, β blockers; CDS, clinical decision support; EF, ejection fraction;
HF, heart failure.
At the conclusion of the randomized controlled trial, the health system decided to
discontinue the commercial CDS and switch those 14 clinics to the customized CDS.
Details of the prior randomized trial, study population and both CDS interventions
are described elsewhere.[35 ] This study evaluates the real-world effect of the enhanced CDS when expanded to
all 28 clinics.
[Fig. 1 ] summarizes the different exposure groups over time.
Fig. 1 Overview of the exposure groups over time. CDS, clinical decision support.
Data Collection
We manually reviewed EHR records to collect data on patient characteristics, characteristics
of health care utilization, and identify potential reasons why a clinician may not
have prescribed a BB. We did not rely on audit logs to determine whether a medication
was prescribed. To identify potential reasons, unstructured clinical documentation
and clinician comments in response to the enhanced CDS alert were evaluated. Clinicians
were not permitted to provide comments in response to the commercial CDS. Following
a rapid qualitative analysis approach, three investigators (L.W., V.W., and K.E.T.)
iteratively and inductively discussed and defined themes of responses, which informed
the development of a structured template. This structured template was then used to
assignment each unstructured data element into a theme.[38 ] The manual review of unstructured documentation and comments was conducted by two
independent reviewers with clinical training (L.W. and V.W.). A second independent
investigator with advanced clinical pharmacy training (K.E.T.) validated a 20% sample
of the manually collected data and reviewed the assignment of every unstructured data
element into a theme. Data were also collected from automated audit logs that monitor
CDS activity, including clinician-stated actions in response to the CDS alerts (buttons
clicked). All data from the original randomized trial that were collected from the
health system's virtual data warehouse were recollected via manual chart review to
facilitate consistency of comparisons.
Outcome Measures
Outcomes were informed by PRISM's RE-AIM outcome measures.[36 ] RE-AIM outcomes of implementation and maintenance were outside the scope of this
study.
The primary effectiveness outcome measure was the proportion of alerts resulting in
prescription of an evidence-based BB when indicated. Possible reasons for not prescribing
a BB were identified by reviewing patient medical records and comments clinicians
made in response to the alerts. Secondary effectiveness outcomes included safety,
specifically instances of bradycardia (i.e., heart rate <50 bpm), hypotension (i.e.,
blood pressure < 90/60), acute heart failure (HF) exacerbation requiring hospitalization
of emergency department visits, and unintended consequences such as duplicate therapy
during the 1-month period after each alert.
Other secondary outcome measures included patient reach and clinician adoption. Reach
was measured as the number of alerts: (1) overall, (2) for unique patients, (3) for
unique visits, and (4) for unique clinicians. Adoption was measured as the proportion
of clinicians who responded to the alert. For the definition of adoption, clinician
response was defined as doing something other than outright dismissing the alert.
In the case of the enhanced CDS that did not have a “dismiss” option and would default
to pending a BB order if “accept” was selected, adoption included instances when a
clinician selected any of the response options or ordered an evidence-based medication.
When a clinician initially selected “accept” in response to the enhanced CDS alert
and then canceled the pended BB, this was considered nonadoption. More details and
rationale for how adoption was measured are described elsewhere.[35 ] Adoption was also summarized based on the number of unique patients and clinicians
for which the CDS alerted. Representativeness of reach, adoption, and effectiveness
was evaluated by descriptively comparing outcomes across patient gender, race, and
ethnicity. All outcomes were measured at the alert level.
Analytic Plan
Descriptive statistics were produced for each of the four exposure groups (A–D illustrated
in [Fig. 1 ]). The four exposure groups were then collapsed into two treatment groups: commercial
(exposure group A) or enhanced (exposure groups B–D). Demographic and baseline characteristics
were compared between treatment groups (enhanced vs. commercial) using the chi-square
test or Fisher's exact test for categorical variables, and Wilcoxon rank sum test
for continuous or discrete variables. A multivariable logistic regression model was
used to estimate the odds of adoption comparing enhanced versus commercial. A multivariable
linear model was used to estimate the difference in effectiveness between enhanced
and commercial alerts. Both models used a generalized estimating equation approach
with an exchangeable correlation structure to account for clustering of patients within
clinics and adjusted for repeated measures within clinics, time (pre vs. post), diagnosis
of chronic obstructive pulmonary disease, and clinician type. All analyses were conducted
using SAS software (version 9.4, SAS Institute Inc.).
Results
Patient Reach
The two CDS alerted 367 times for 183 unique patients and for 171 unique clinicians:
the commercial CDS alerted 59 times for 26 unique patients and 24 unique clinicians;
and the enhanced CDS alerted 308 times for 169 unique patients and 155 unique clinicians.
[Table 2 ] describes the baseline characteristics of the study population. Specific baseline
characteristics to highlight include: the mean age of patients exposed to one of the
two alerts was 73.9 (13) years, 34.4% were female, 13.7% were non-White and 81.4%
had Medicare insurance. When considering the distribution of treatment group, there
were no significant differences based on patient race, ethnicity, or gender. There
were significant differences in treatment group based on patient age (79.4 [15.1]
years assigned to commercial vs. 72.9 [12.5] years assigned to enhanced; p = 0.02), diagnosis of chronic obstructive pulmonary disease (26.9% commercial vs.
11.5% enhanced; p = 0.03), and type of clinician (96.2% commercial vs. 69.3% enhanced alerted for an
attending physician; p = 0.016). As described in [Table 3 ], there was a 34% increase in patient reach in the postperiod for clinics that had
the enhanced CDS in the preperiod and an 85% increase for clinics that transitioned
from commercial CDS to enhanced CDS.
Table 2
Baseline characteristics for unique patients exposed to the alerts
Characteristic
Commercial alert (n = 26)
Enhanced
alert (n = 169)
p- Value
Age (y), mean (SD)
79 (15)
73 (13)
0.02
Male, n (%)
19 (73)
123 (64)
0.38
White, n (%)
22 (85)
168 (88)
0.58
Hispanic, n (%)
2 (8)
12 (6)
0.79
Medicare, n (%)
23 (88)
156 (82)
0.39
Clinician type: attending physician, n (%)
25 (96)
131 (69)
0.02
Heart rate, mean (SD)
74 (15)
79 (14)
0.12
Heart rate < 50, n (%)
0 (0)
3 (2)
0.52
Last heart rate < 50, n (%)
1 (4)
7 (4)
0.96
Blood pressure < 90/60, n (%)
1 (4)
4 (2)
0.58
Last blood pressure < 90/60, n (%)
0 (0)
11 (6)
0.21
≥1 visit with cards[a ] in past 1 y, n (%)
19 (73)
120 (63)
0.31
≥1 visit with cards in past 2 y, n (%)
21 (81)
134 (70)
0.26
Past BB, ever, n (%)
18 (69)
145 (76)
0.46
BB allergy per chart review[b ], n (%)
0 (0)
4 (2)
0.46
Nonevidence-based BB prescribed, n (%)
4 (15)
12 (7)
0.24
Prescribed metoprolol tartrate, n (%)
8 (31)
51 (27)
0.66
Prescribed ACE, ARB, or ARNI, n (%)
15 (58)
88 (46)
0.27
Prescribed angiotensin receptor-neprilysin inhibitor, n (%)
0 (0)
4 (2)
0.46
Prescribed mineralocorticoid receptor antagonist, n (%)
5 (19)
36 (19)
0.96
Prescribed sodium-glucose cotransporter 2 inhibitor, n (%)
0 (0)
1 (1)
0.71
Prescribed nondihydropyridine calcium channel blocker, n (%)
1 (4)
10 (5)
0.76
Chronic obstructive pulmonary disease, n (%)
7 (27)
22 (12)
0.03
Asthma, n (%)
1 (4)
16 (8)
0.42
Coronary artery disease[c ], n (%)
15 (58)
108 (57)
0.91
Atrial fibrillation or atrial flutter, n (%)
16 (62)
89 (47)
0.15
Type 2 diabetes mellitus, n (%)
8 (31)
52 (27)
0.71
Abbreviations: ACE, angiotensin converting enzyme inhibitor; ARB, angiotensin receptor
blocker; ARNI, angiotensin receptor–neprilysin inhibitor; CDS, clinical decision support;
SD, standard deviation.
Notes: Of the 183 unique patients, 12 were exposed to both CDS. Significant differences
are bolded.
a Cards: outpatient cardiology provider.
b These patients were incorrectly not excluded from the alert.
c Coronary artery disease includes myocardial infarction, percutaneous coronary intervention,
bypass, coronary artery disease, and angioplasty.
Table 3
Patient reach, clinician adoption, and effectiveness of changing prescribing across
exposure groups
Outcome
Commercial CDS preperiod (Exposure A)
Enhanced CDS postperiod (Exposure C)
Enhanced CDS preperiod (Exposure B)
Enhanced CDS postperiod (Exposure D)
Patient reach
unique patients alerted, n
26
48
61
82
# of total alerts, n
59
65
106
137
Adoption
alerts not dismissed, n (%)
17 (29%)
61 (94%)
66 (62%)
124 (91%)
Effective
alerts resulting in BB prescription, n (%)
0 (0%)
5 (8%)
15 (14%)
13 (10%)
Abbreviations: BB, β blockers; CDS, clinical decision support.
Note: Effectiveness rates were calculated based on those alerts that were adopted.
Clinician Adoption
After adjusting for repeated measures within clinics, time (pre vs. post), diagnosis
of chronic obstructive pulmonary disease, and clinician type, the odds of adoption
were 4.17 (95% confidence interval [CI]: 1.96–8.85) times as high for the enhanced
CDS compared with the commercial CDS. Additionally, time (pre vs. post) was significantly
associated with the odds of adoption, with alerts occurring in the postperiod at more
than 7 times greater odds (odds ratio: 7.07; 95% CI: 4.07–12.26) of adoption compared
with alerts in the preperiod.
From [Table 3 ], the absolute increase in adoption for clinics that received the enhanced CDS in
the preperiod was 29%. Of the 13 patients for whom commercial alerts were not adopted,
3 patients were then exposed to the enhanced CDS and all 3 enhanced alerts (100%)
were adopted for these patients. Across all 38 patients exposed to one of the two
CDS that were not adopted in the preperiod, 26 were later exposed to the enhanced
CDS in the postperiod and 10 (38%) of these enhanced alerts were adopted.
Effectiveness of Changing Prescribing
After adjusting for repeated measures within clinics, time (pre vs. post), diagnosis
of chronic obstructive pulmonary disease, and clinician type, the increase in effectiveness
for the enhanced CDS was 26.1% (95% CI: 17.0–35.1%) higher compared with the commercial
CDS. Additionally, clinician type was significantly associated with effectiveness,
with physicians (attendings, resident, fellows) having a 10.7% increase in alerts
that were effective compared with advanced practice providers. Alerts in the postperiod
(exposure groups C and D) were associated with a decrease in effectiveness compared
with alerts in the preperiod (exposure groups A and B; −13.9% CI: −3.1 to −24.7%).
From [Table 3 ], there was an absolute decrease in effectiveness for clinics exposed to the enhanced
CDS in the preperiod (14–10%). Of the 26 patients exposed to the commercial CDS in
the preperiod (none were effective), there were 6 patients who were later exposed
to the enhanced CDS and none (0%) were prescribed a BB. Across all 38 patients exposed
to one of the two CDS that was not effective in the preperiod, 26 were later exposed
to the enhanced CDS in the postperiod and 1 patient (3.9%) was prescribed a BB with
the enhanced alert.
Reasons for Not Prescribing
Although the commercial alert did not allow clinicians to provide comments, clinicians
provided 68 comments after receiving the enhanced CDS. Based on the clinician comments
in response to the enhanced CDS and chart review of clinical documentation, the most
common reasons clinicians appear to have avoided prescription of a BB was a concern
of an adverse event or because the patient was either referred to or being managed
by a cardiologist. [Table 4 ] describes potential reasons clinicians may have avoided prescribing a BB.
Table 4
Summary of β blockers intolerance or reason β blockers was not appropriate per clinical
documentation or clinician comments in response to alerts, n (%)
Reason provided
“Exposure A” commercial CDS
n = 59 alerts
“Exposure B, C, or D” enhanced CDS
n = 275[a ] alerts that did not result in prescription of a BB
Documented reason in clinical notes or in comments in response to CDS (total across
alerts)
33 (55.9)
161 (58.5)
Comments provided in response to CDS (total across alerts)
N/A[a ]
68 (24.7)
Reasons based on comment or documented in clinical notes
Seeing cards or referred to cards
0 (0)
47 (17.1)
History of adverse drug event
7 (21.2)
35 (12.7)
Other reason
5 (15.15)
16 (5.8)
Ejection fraction >40%
2 (6.1)
15 (5.5)
Decompensated/HF exacerbation
5 (15.2)
7 (2.5)
Conduction disorder or device placement planned
4 (12.1)
6 (2.2)
Defer to primary care provider
0 (0)
8 (2.9)
On nonevidence-based BB
0 (0)
8 (2.9)
Concern for hypotension and BP ≥ 90/60
2 (6.1)
6 (2.2)
Concern for hypotension and BP < 90/60
1 (3)
6 (2.2)
Patient refuses
3 (9.1)
4 (1.5)
Concern for bradycardia and HR ≥ 50
4 (12.1)
2 (0.7)
Concern for bradycardia and HR < 50
0 (0)
1 (0.4)
Abbreviations: BB, β blockers; BP, blood pressure; CDS, clinical decision support;
HF, heart failure; HR, heart rate; N/A, not applicable.
Note: Table represents those CDS alerts that did not result in prescription of an
evidence-based BB.
a The commercial CDS was not enabled to allow clinicians to provide comments.
Representativeness of Outcomes
[Table 5 ] describes the representativeness of outcomes by gender, race, and ethnicity. The
CDS alerted for more patients that were male, White, and non-Hispanic. Clinician adoption
and effectiveness rates were incrementally higher for female patients. Although the
adoption rate was incrementally higher for non-White patients, effectiveness rates
were not higher for non-White patients.
Table 5
Representativeness of reach, adoption and effectiveness outcomes based on patient
gender, race, and ethnicity
Patient gender
Patient race
Patient ethnicity
Male
Female
White patient
Non-White patients
Hispanic
Non-Hispanic
Alerts that fired, n
(reach)
258
114
316
53
17
352
Alerts that were adopted, n (%)
182 (71)
91 (80)
229 (72)
41 (77)
13 (76)
257 (73)
Alerts that were effective, n (%)
18 (7)
15 (13)
29 (9)
3 (6)
2 (12)
30 (9)
Safety Outcomes
With respect to safety outcomes, there were no documented instances of bradycardia,
hypotension, or acute HF exacerbation requiring hospitalization or emergency department
visits in patients prescribed a BB. As previously reported,[35 ] there were two near misses in the preperiod, in which clinicians inadvertently prescribed
a BB in response to the enhanced CDS, but the medication never reached the patient.
Manual chart review did not reveal any new instances of near misses in the postperiod.
Discussion
This study provides further empirical support that application of IS principles and
methods with CDS design best practices can improve adoption and effectiveness of CDS
tools and also suggests this effect is sustained over time. In our study, the enhanced
CDS was informed by IS to be contextually relevant to the local health system and
significantly improved guideline-concordant prescribing of BBs for patients with HFrEF
compared with a commercially available CDS. The current study validates the findings
of our earlier randomized controlled trial[35 ] by addressing some of its potential limitations. Although the original study was
randomized, the sample size was relatively small, which limited the ability to adjust
for patient characteristics and within-clinic clustering in analyses. In the present
study, we extended the follow-up beyond the original randomized controlled trial to
conduct comparisons within the same study groups with increased sample size and longer
follow-up. Our use of a natural experimental design as a pragmatic follow-up method
is a strength and not only validates the findings of the original randomized controlled
trial, but also facilitates examination of the long-term sustained effects of the
enhanced CDS tool. Thoughtful natural experiments can facilitate a timely and low-cost
method to observe the effects of interventions in generalizable, real-world scenarios
without threatening internal validity.[39 ]
In the present study, we observed significantly higher rates of clinician adoption
with the enhanced CDS compared with the commercial CDS. In fact, the rate of adoption
exceeded 90% with the enhanced CDS groups during the postperiod. This finding is especially
notable when considering that the number of unique patients reached by an alert and
the total number of alerts fired significantly increased during the postperiod as
well. Despite receiving more frequent alerts, clinicians were more likely to adopt
the enhanced CDS. Some features of the enhanced CDS design that we believe contributed
to this high adoption rate include linking to specific medication orders rather than
a broad order set and defaulting the ordering of an evidence-based BB and starting
dose. We also made it more difficult to ignore the recommendation by removing the
“dismiss” button. To bypass the recommendation, the clinician would need to manually
deselect the medication order within the alert or deliberately cancel the pended order
after clicking “accept.” These features align with current best practices in CDS design
and nudge theory, which encourage leveraging small changes in alert presentation to
drive clinician behavior and minimize the effort and time required to perform the
recommended action.[17 ]
[34 ]
[40 ]
Historically, medication-related commercial CDS have been associated with high dismissal
rates (low adoption) up to 96%.[40 ]
[41 ] In our study, adoption rates of the enhanced CDS were substantially higher (62%
in the preperiod and >90% in the postperiod), which suggests a positive return on
investment when IS principles and methods are used with CDS design best practices
to customize CDS to the local context. Interestingly, the adoption rate of the enhanced
CDS increased from 62% in the preperiod to >90% in the postperiod despite no changes
to its design. Although the frequency of a given clinician seeing the enhanced CDS
more than once was low, one possible explanation may be related to shifts in clinician-perceived
usefulness of CDS overall, which may have resulted in clinicians being more likely
to adopt instead of outright dismiss the CDS; however, this effect has not yet been
observed in other CDS situations at our insitution. The finding of high adoption rates
is important, given it is a rate-limiting step for our primary outcome of effectiveness.
Unless clinicians pay attention to the CDS (adopt it), effectiveness is not possible.
Our application of IS principles and methods prioritized alignment of the enhanced
CDS design to meet the needs and preferences of clinicians, which may explain the
high rates of adoption that persisted over the study period.
Previous findings of effectiveness were also sustained over time. We found a 26.1%
increase in effectiveness with the enhanced CDS, which is consistent with the 24%
increase found in the 2021 study. Although the overall effectiveness of the enhanced
CDS decreased in the postperiod, this may be explained by saturation of patients eligible
for BB therapy for which the CDS recommendation was deemed relevant (e.g., due to
patient preference or other reasons). Among the 130 patients in the postperiod, 20%
(26 patients) were observed in both time periods and only 1 was prescribed a BB in
the postperiod, which may suggest that decisions to not prescribe an evidence-based
BB as a result of the CDS during the preperiod were maintained in the postperiod.
Given the increase in total number of alerts in the postperiod, we considered the
possibility that alert fatigue could explain the change in effectiveness; however,
this is contradicted by the fact that clinician adoption increased from the pre- to
post time period.
Over time, CDS design best practices, including user-centered approaches are increasingly
being employed when developing CDS,[16 ]
[19 ]
[42 ]
[43 ] but few have formally applied IS principles and methods.[34 ]
[35 ]
[44 ] The recent PRagmatic trial Of Messaging to Providers about Treatment of Heart Failure
(PROMPT-HF) study applied a user-centered design approach to create CDS for four categories
of guideline-recommended HFrEF medications to be used in both cardiology and primary
care practices.[16 ] PROMPT-HF found a significant increase of >40% in guideline-concordant prescribing
compared with no CDS. When specifically considering changes in BB prescribing, PROMPT-HF
found a 3% improvement with their CDS. In contrast, we found that 20% of patients
who received the enhanced alert were prescribed a BB. Differences in prescribing rates
could be explained by the differences in the designs of the CDS or our application
of IS principles and methods. Interestingly, in our mixed methods evaluation, clinicians
generally preferred focused recommendations over more comprehensive order sets that
were included in PROMPT-HF's CDS. However, this preference may have been driven by
other design features of the CDS that made the order set less appealing; designed
differently, the order set may have been more appealing in our study.
Our previous study and this continuation study present an emerging approach to combining
IS methods with established best practices in CDS design. This approach enhances the
traditional user-centered design process to include additional perspectives beyond
the intended end user and to systematically consider how these perspectives dynamically
interact with other contextual factors (e.g., guidelines, best practices, resources
available). While the initial application of IS methods and CDS best practices may
require additional resources compared with the implementation of commercially available
CDS tools, our findings support that resource utilization at the design or planning
stage can result in significant improvements and sustainment of clinician adoption
and guideline-concordant prescribing practices. Further, by applying IS, the reproducibility,
scalability, relevance, and representativeness of CDS implementation can be elevated.
An important aspect of PRISM and other IS frameworks is the consideration of the sustainability,
representativeness, and long-term population health impact of the intervention from
multiple perspectives.[26 ]
[27 ]
[28 ]
[45 ]
There are several limitations of this study. First, our study focused on clinics in
the primary care setting at a single health system for one clinical situation (HF
prescribing) with one commercial CDS and may not be generalizable to all health systems,
clinical situations, or all commercially available CDS. Although the natural experimental
design increases external validity, challenges to this design are the risk of selection
bias or selective exposure to the intervention and inability to control for temporal
changes.[46 ] To address these risks, we controlled for potential confounders that may influence
a patient's indication for BB therapy and a clinician's adoption of the CDS tool in
our analysis. Further, this study aimed to validate or refute the findings from the
original randomized controlled trial, which was designed to control for potential
confounders and issues related to study group imbalance. There were also outcomes
that we were not able to evaluate robustly such as equity and cost of implementation
of the enhanced CDS. Although we assessed the RE-AIM issues of representativeness
of reach, adoption, and effectiveness outcomes across different demographic groups,
the study was not powered to test for statistical significance and the CDS were not
designed to consider differences based on key social determinants. We did find a clinically
significant difference in the number of alerts based on sex, which is likely due to
a greater proportion of men with HFrEF, but more in-depth analysis of the implications
of this difference is warranted. Similarly, additional investigation of the reason
for and implications of the customized CDS alerting for more attending physicians
is warranted. Given limitations of EHR data, including lack of structured EHR data
to accurately identify patients without contraindications to BB prescription,[47 ]
[48 ] we were unable to accurately report the proportion of alerts that fired for patients
who are true candidates for BB therapy; thus, we instead described the reach outcome
of the PRISM/RE-AIM framework as the number of alerts overall.
Future studies should evaluate the effect of applying IS to promote evidence-based
care when designing CDS for other clinical situations and compared with other commercially
available CDS. Such studies should also proactively consider equity and unintended
consequences in the design and evaluation of CDS tools. In addition, the cost of using
this IS approach is an important measure of scalability to other CDS within and external
to our health system. In future work, we plan to evaluate the cost of implementing
CDS using this IS-based approach using a time-driven activity-based cost approach,[49 ] given the bulk of the cost is in personnel time. Another important area of future
research is exploring ways to increase the specificity of such alerts by leveraging
natural language processing systems to transform data from unstructured clinical narratives
into structured data elements, which can also be used to more robustly evaluate reach.[50 ]
Future CDS should also consider methods to address primary care hesitance to intensify
therapy when patients are comanaged by cardiology. In our study, comanagement with
or deferral to cardiology was identified as a common reason for primary care clinicians
opting to not follow the CDS recommendation to initiate an evidence-based BB. Given
how we identified comanagement as a reason, our estimates of its influence are likely
under representative of the true incidence. Others have also identified comanagement
as a deterrent to optimizing guideline-concordant prescribing.[51 ] CDS design features to address this deterrent could include functions that facilitate
ease of communication across specialties. Such features could support ease of e-consults
or include automated cross-specialty notifications regarding medication changes or
reasons a change was not made.
Conclusion
In this natural experiment, applying IS methods with CDS best practices to design
a customized CDS tool resulted in improved and sustained clinician adoption and effectiveness
compared with a commercially available CDS tool. Future research is needed to replicate
and test the generalizability of this approach and to evaluate the value of additional
resources needed to develop CDS tools that are aligned with the local context and
adhere to CDS design best practices.
Clinical Relevance Statement
Clinical Relevance Statement
Applying IS methods with best practices in CDS may result in improved and sustained
clinician adoption and effectiveness compared with commercially available CDS tools.
Others seeking to implement CDS solutions may benefit from customizing CDS tools to
meet the needs and preferences of the local context while also considering best practices
in CDS design versus relying on commercially available CDS tools that are designed
for the general or average needs of health systems.
Multiple-Choice Questions
Multiple-Choice Questions
Which one of the following is a potential outcome of applying IS principles and methods
to the design and implementation of CDS tools?
Increased likelihood of sustaining or maintaining the effect
Increased likelihood of developing a cost-effective tool
Decreased likelihood of scaling the tool to other settings
Decreased likelihood of gaining clinician buy-in
Correct Answer: The correct answer is option a.
Which of the following is true regarding the use of IS principles and methods for
CDS tools?
Does not provide guidance on outcome measures
Does not require additional resources
Does consider representativeness and equity
Does result in improved outcomes in all settings
Correct Answer: The correct answer is option c.
