Clinical Decision Support and Implications for the Clinician Burnout Crisis

Summary

Objectives: This survey aimed to review aspects of clinical decision support (CDS) that contribute to burnout and identify key themes for improving the acceptability of CDS to clinicians, with the goal of decreasing said burnout.

Methods: We performed a survey of relevant articles from 2018-2019 addressing CDS and aspects of clinician burnout from PubMed and Web of Science™. Themes were manually extracted from publications that met inclusion criteria.

Results: Eighty-nine articles met inclusion criteria, including 12 review articles. Review articles were either prescriptive, describing how CDS should work, or analytic, describing how current CDS tools are deployed. The non-review articles largely demonstrated poor relevance and acceptability of current tools, and few studies showed benefits in terms of efficiency or patient outcomes from implemented CDS. Encouragingly, multiple studies highlighted steps that succeeded in improving both acceptability and relevance of CDS.

Conclusions: CDS can contribute to clinician frustration and burnout. Using the techniques of improving relevance, soliciting feedback, customization, measurement of outcomes and metrics, and iteration, the effects of CDS on burnout can be ameliorated.

Keywords

Decision Support Systems - Clinical - burnout - professional - Electronic Health Records

1 Introduction

The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 resulted in the rapid incorporation of electronic health records (EHRs) into hospitals and clinics, with the promise of improving healthcare quality and efficiency [1]. Unintentionally, the EHR has created frustrating workflow interruptions, hours of documentation at home, clerical data entry for reimbursement and regulations, cumbersome security requirements, and an interrupted clinician-patient relationship [2]. In one study, 70% of physician EHR users reported stress related to healthcare information technology, almost two-thirds felt that EHRs added to their daily frustrations, and almost half disagreed with the sentiment that EHRs improved patient care [3]. Although superior in many ways to the paper charts they replaced, EHRs’ F-grade usability [1] has become a significant driver of burnout in clinicians [1] [2] [4] [5].

Burnout is a serious issue affecting healthcare professionals. It is characterized as a syndrome of emotional exhaustion, depersonalization, and a low sense of personal accomplishment. The causes of burnout are multifactorial, but include workload pressures, inefficiencies, and moral distress from ethically undesirable situations [2]. Currently, it is estimated that 35-60% of clinicians experience symptoms of burnout [2], though one large study of medical interns in the United States found rates of up to 79% in subscales of the Maslach Burnout Inventory [6]. These rates are higher than in other professions [7], suggesting factors intrinsic to medicine drive burnout. Importantly, burnout is a consequence of employment and not a mental health disorder [2].

Burnout affects not only the health of clinicians but also that of patients and the healthcare system overall. As studied in physicians, burnout is associated with increased risks of depression, substance abuse, occupational injury, and suicide [2] [7]. Patients also suffer when clinicians are burned out. Increases in medical errors, recovery times, and patient mortality, as well as decreased patient satisfaction, have been associated with healthcare team burnout [2] [7]. From the point of view of the healthcare system, burnout is costly. Physician burnout is associated with decreased productivity, absenteeism, intent to leave one’s practice or medicine altogether, and increased malpractice claims [2] [7]. Physician burnout is estimated to cost over $4 billion annually [2].

There is no single cause of burnout in healthcare providers. The National Academy of Medicine in 2019 released a 333-page report on clinician burnout [2] that identified contributors such as long work hours; low nurse-to-patient ratios; patient messaging portals; ethical tensions between serving individual patients, the patient population, and the healthcare system; excessive documentation; threats of harm; and the tendency to always place patient needs above one’s own. The report also identified administrative burdens, such as those related to EHRs, as “the most prominent current complaint by clinicians about their workplaces,” and specifically highlighted how poor usability of healthcare technology contributes to clinician burnout.

Clinical decision support (CDS) is an important aspect of modern EHRs, with the potential to limit patient care errors and improve adherence to evidence-based medicine. For example, CDS alerts can identify if a patient is allergic to a prescription or if they are due for a flu shot. However, although CDS systems can reduce some errors, they can also be the cause of others, such as treatment delays [8]. CDS has had several impacts on clinicians, including changes in workflow, more or new kinds of work, alert/alarm fatigue, spillover of work into personal time, loss of autonomy, and anxiety about the medico-legal ramifications of CDS suggestions [2] [7] [8] [9]. Concerns about these changes are not new [10], and many were highlighted in the Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-based Medicine a Reality, published in 2003 [11]. Nonetheless, it remains unclear whether factors that affect clinician burnout are being considered and used to inform CDS design and deployment in vivo. For this reason, we undertook a survey of two years of publications on CDS and clinician burnout, in order to identify current trends and synthesize evidence-based standards for consideration in any CDS implementation project.

2 Methods

Two databases, PubMed and Web of Science™ (WoS), were queried for relevant studies in English published between January 1^st, 2018 and December 31st, 2019, yielding 189 total results, which was narrowed to 150 articles when duplicates were excluded ([Figure 1]). To capture CDS, terms including “cognitive aid,” “user interface,” “expert system,” “decision support systems,” “cds,” “clinical reminder,” “best practice advisory,” “best practice alert,” “decision support,” “Decision Support Systems, Clinical” [MeSH], and “Decision Support Systems, Management” [MeSH] were included in the search terms. To capture burnout concepts, the terms “burnout,” “Stress, Psychological”[MeSH], “pain point,” “workaround,” “psychology,” “stress,” “stressful,” “fatigue,” “alarm fatigue,” “alert fatigue,” “provider resistance,” and “provider satisfaction” were used. The general terms “alarm,” “alert,” and “human factors” were not included as the search results were too broad.

Fig. 1 Flowchart describing the literature search from January 1st, 2018 to December 31st, 2019

Articles were excluded if their relevance could not be determined (e.g., abstract was not available), if the CDS was not related to technology (e.g., a paper worksheet), if the CDS was not aimed at clinicians, or if the tool was used exclusively for teaching. After applying the exclusion criteria, the remaining articles were evaluated for their themes and study design, whether the evaluation of factors that could contribute to burnout was actively incorporated into the design of the CDS tool, and which metrics, if any, were used to evaluate CDS tools. Study design was categorized from each paper’s methods section and adapted to research hierarchies [12] [13] [14]. Metrics and themes were considered relevant if they addressed aspects of usability, user preferences or frustrations, burden, user efficiency at tasks, or other concepts related to burnout and wellness as described in section 3.2, even if the term “burnout” was not used explicitly. If these metrics could not be ascertained from the abstract, the whole paper was reviewed, which was the case for the majority of papers.

3 Results

Of the 89 articles that met inclusion criteria, 12 were identified as review articles, of which seven were systematic reviews. Two were editorials. Five described randomized controlled trials (RCTs). Seventeen studies included a comparator group, such as pre-post or cohort designs. Five publications were pre-trial descriptions of intended studies. The remaining 48 publications were descriptive, including cross-sectional and case study designs ([Table 1]). Themes and features of review articles and non-review articles were considered separately.

Table 1 Categorization of the design of each publication from the results of the literature survey

Study type	Manuscripts
Systematic Review or Meta-Analysis Review Randomized Controlled Trial Controlled Studies Descriptive Studies Opinion or Editorial Pre-Trial Study Proposal	18,20,21,23-25,108 16,17,19,22,109 78,80,82,92,97 27,46,49,51,55,56,69,72,73,76,79,81,85,89,91,93,94 26,28-41,43-45,47,50,52,53,57-68,70,71, 74,75,77,83,84,87,90,95,96,98,99,104 102,110 42,48,54,86,88

3.1 The Big Picture – Review Articles

The 12 review articles that met inclusion criteria in this literature survey can be broadly categorized as either prescriptive or analytic. The prescriptive reviews offered a perspective on how CDS should work. For example, Marcial et al. proposed that CDS should support individual patients as well as their clinical care teams and emphasized the previously-established “CDS Five Rights” [15]: the right information at the right time, given to the right people, in the right format, via the right channel [16]. Other articles argued that nurse practitioners, as stakeholders in CDS, should be involved in design, implementation, and optimization [17] [18]. Wilbanks and McMullen reviewed cognitive workload associated with EHRs and generally found that it is hard to measure with validity. Specifically regarding CDS, they proposed that the related concepts of alarm fatigue and desensitization were the most pressing for patient safety and they suggested minimizing within-patient identical alerts to ameliorate the negative effects on cognitive load [19]. Finally, Tolley et al. systematically reviewed CDS literature from 2007-2014 to identify areas of improvement. They suggested that sensitivity and specificity of alerts should always be considered, with an emphasis on high specificity to decrease alert fatigue. They also proposed tiering of alerts to improve acceptance and consideration of human factors in CDS design [20]

The analytic reviews analyzed the features of previously implemented CDS tools. Several reviews examined alert use, generally finding acceptance rates to be low [21] and override rates to be high [22] [23] [24]. Hussain et al. delved into why alerts are so commonly overridden and found that although interruptive alerts are the most common design, they are also the least accepted. The authors also found that role-tailoring alerts to pharmacists instead of physicians had improved acceptance [21]. Carli et al. undertook a systematic review of positive predictive values (PPVs) of CDS alerts as a proxy for clinical relevance, finding massive PPV variations from 0% to 97%, with most in the 20%-40% range. PPVs were higher when the contextual information of the individual patient was considered in CDS, so the authors concluded that incorporating as much specific patient data as possible to improve PPV is more important than using a large database of knowledge that may generate many false positives and thus promote alert fatigue [24]. In a systematic review to understand CDS for drug allergies, Légat et al. found that significant inaccuracies in the EHR due to the difficulty of structured documentation of allergies, the lack of standard allergy terminology, and irregular updating of allergy databases resulted in poor specificity and overall performance of CDS allergy alerts, along with override rates of up to 90%. The authors emphasized that clinicians and programmers should review CDS at regular intervals post-implementation to ensure that the allergy rules are up-to-date and to minimize alert fatigue [23].

Finally, Powers et al. evaluated 32 articles on “hard stop” CDS in which the clinician could not proceed without third-party override, if at all, and the effects on healthcare delivery. The majority of studies showed improvement in process outcomes and four of eight that reported patient outcomes found improvements. Only two studies pre-specified patient health outcomes, with one finding no adverse effect and the other being stopped early due to treatment delays from the hard stop alert. In evaluating the user experience, the authors found that CDS for which the end users were involved in design and iteration were the most acceptable. Hard stops for documentation purposes were poorly received [25].

3.2 How and How Much – Methods and Metrics

The 75 non-review, non-opinion articles used a variety of metrics to evaluate the usability and acceptability of their CDS tools, with many using more than one metric. The most common evaluation was via interviews (n=29) [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54], followed by surveys or questionnaires (n=16) [27] [29] [50] [53] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66]. Other qualitative feedback responses were obtained by focus groups or workshops (n=5) [43] [57] [61] [67] [68]. The alert firing rate (n=12) [43] [55] [62] [69] [70] [71] [72] [73] [74] [75] [76] [77] and alert acceptance rate (n=15) [27] [29] [43] [58] [61] [62] [69] [74] [78] [79] [80] [81] [82] [83] [84] were also common metrics used to understand alarm fatigue and acceptability. Although metrics of relevance such as sensitivity, specificity, and positive predictive value (PPV) were referenced several times [77] [84] [85] [86] [87] [88], only one study evaluated these metrics as pre-specified outcomes [49]. Tools for capturing clinician burnout, stress, or satisfaction after using a CDS tool were used in nine studies [26] [44] [78] [80] [82] [89] [90] [91] [92]. Psychometric measures of the user interface, such as mouse clicks, eye movement tracking, time spent on the page, errors using the tool, or direct observation were captured in seven studies [29] [49] [54] [59] [82] [89] [93]. Efficiency metrics of task completion time or clinic appointment time were obtained in seven studies [29] [44] [60] [61] [89] [91] [94]. How often a non-mandatory CDS tool was used, a proxy of whether clinicians found the tool useful and acceptable, was studied in five publications [27] [53] [56] [93] [95]. Whether or not a CDS tool made a difference in patient outcomes, such as cardiovascular events [48], A1c [46], or mortality [80], was evaluated in 11 studies [28] [43] [47] [48] [77] [80] [81] [82] [90] [96] [97]. Only one study considered the economic cost of a CDS system [48].

3.3 Personalized Medicine (Alerts) –The Relevance of CDS Prompts

Similar to the review articles, the relevance of alerts to patients featured prominently in studies of CDS systems. As one article put it: “high PPV is critical for successful deployment of clinical decision support interventions” [49]. Irrelevant alerts fail to improve patient care, contribute to alarm fatigue, and interrupt workflow, all of which have negative effects on clinician wellbeing and that of patients. For example, one study found that their sepsis alert had a PPV of only 14.6% and was associated with a 66.6% override rate, which the authors felt negatively impacted the ability of the CDS to improve patient outcomes [49]. In another case, CDS for a precision-medicine genotype test to detect patients at risk of a drug toxicity yielded 71% normal status and no case of complete deficiency in tested patients. Dosing was adjusted in only three patients after 500 alerts [96]. Kizzier-Carnahan et al. point out, “the average clinician must deal with both the inappropriate presence as well as the inappropriate absence of alerts,” [98] and targeting alerts to those who stand most to improve would improve clinician trust in the CDS system [97].

Relevance could also be captured indirectly by measuring alert acceptance rates, with the caution that some studies found alerts were accepted just to silence an annoying alert [49], or that alarm fatigue could cause even appropriate alerts to be dismissed [69] [72]. Nonetheless, a cross-sectional study of medication CDS alerts examined the appropriateness of overrides, finding that overall 60% of overrides were appropriate and, in subgroup analysis, >95% of duplicate drug and patient allergy alerts were appropriately overridden [57]. In an example of successful CDS, Wasylewicz et al. describe how relevance was improved in patients with hypokalemia when pharmacists were only alerted if there was no potassium repletion order. Doctors were no longer called about lab results they had already acted upon and their acceptance rate of this targeted alert was 88% [81].

To better capture relevance of CDS to clinicians, one study parlayed “number needed to treat” into “number needed to remind,” a measure of the number of CDS alerts the clinician sees before the desired intervention is undertaken. For the intervention in question, the “number needed to remind” was approximately 83 [45]. Another way of understanding CDS relevance was by directly soliciting feedback from users, for example using a Likert scale. A study of an antibiotic susceptibility CDS found that clinicians using a Likert scale felt that fewer than 30% of alerts were relevant [84]. Improved CDS relevance to individual patients was also high on the wish-lists of clinicians giving feedback on CDS alerts [29] [97].

3.4 Drowning in Information – Alarm Fatigue

In addition to relevance, one of the major themes across studies was the burden of CDS tools. Even with perfect PPV, a thousand daily alerts would soon become overwhelming. There is currently no consensus on how to use CDS without causing alert fatigue [64]. Worries about alert fatigue and the burden of CDS were common in qualitative studies on CDS tools [29] [31] [34] [83] [93] [99], which is perhaps unsurprising given that one study estimated the average ICU clinician is subjected to over 900 active and passive alerts per day [98]. Alarm fatigue can lead to important alerts being overlooked [29]. When one hospital switched from a legacy EHR to a commercial one, the alert burden increased six-fold but the alert acceptance rate plummeted nearly twelve-fold, from 100% to 8.4% for high-severity alerts [69]. This drop suggests that poor relevance or simply alarm fatigue can have a disproportionate effect on CDS burden. One RCT determined that their CDS generated approximately 14,400 alerts during the study period, almost 95% of which were dismissed [80]. Another study described alert rates 51 per 100 orders at one hospital, indicating that an alert was generated for every other order. Over 90% of these alerts were dismissed [72].

3.5 Make it Easy – Workflow, Efficiency, and Integration

The burden of CDS tools is not only an excessive number of alerts, but also how the clinician interacts with CDS. Usability of the tool itself is one example. Encouragingly, several studies emphasized developing tools that incorporated user-centered design, such as clean, concise, and intuitive interfaces [26] [30] [50] [67]. However, several studies cited the need for an improved CDS user interface, such as limiting mouse-clicks or not having the pop-up window block access to the chart [35] [93] [99]. Usability was measured directly in several studies, including by observation or tracking of eye movements, click counts, and use errors. Interestingly, users sometimes had differing opinions on design elements [63] [64].

The ability to use CDS tools efficiently can limit excessive documentation burden and improve efficiency. Relatedly, how CDS incorporated into clinician workflow was commonly addressed. For example, one study found that a large number of antibiotic susceptibility alerts were already addressed by standard workflow (e.g., the clinician reviewing new culture results and adjusting empiric antibiotics appropriately), so early alerts just contributed to alert burden [84]. Another study noted the pharmacists using a tool for antibiotic stewardship were frustrated by automatic log-offs [33]. Additionally, a CDS tool for screening for pregnancy complications in Ghana was criticized by clinicians for adding up to 30 minutes to an antenatal visit [38]. Positively, some studies were specifically targeted at improving workflow, such as early work using natural language processing to pre-populate imaging orders based on clinicians’ notes [87].

One major theme in the usability of CDS addressed in the recent papers was integration. Prior studies have shown that CDS needs to be minimally-interruptive to workflow [100], and literature in this survey emphasizes that this goal is better accomplished by integrating it into the existing EHR. Needing to reference an external database is a deterrent to CDS use [27]. The inability of CDS to understand records from outside the hospital system, such as scanned lab results, can lead to misfiring of CDS alerts [96]. The lack of integration of CDS into EHRs leads to needing to “do work twice” – once to satisfy the CDS alert and then again, for example, to actually place the order in the EHR [99]. Better integration of CDS into the EHR was identified as a key change that would improve acceptance of decision support systems [34] [78] [83] [85]. For example, automatic documentation of patient vital signs that feed into CDS systems decreased both clinician workload and frustration [91].

3.6 Ends Justify the Means – Measuring Outcomes

Given the potential negative effects on workflow and the risks of alarm fatigue, with the associated burnout-related sequelae, it is important to establish that CDS is accomplishing its patient-centered goals. Prior meta-analyses evaluating RCTs of CDS found only weak evidence, if any, of improved patient outcomes [100] [101]. Only a minority of surveyed studies here explicitly evaluated the effectiveness of the CDS tools. Some studies identified the importance but did not undertake evaluation of effectiveness [36]. One study of a CDS tool for deprescribing found that only 1.2% of alerts resulted in medication discontinuation, which was overall branded ineffective [45]. Another study compared interruptive vs. non-interruptive alerts for ACE inhibitor prescription, finding that the non-interruptive alerts were less annoying but less seen [51]. Unintended consequences should be assessed in addition to expected outcomes [59]. In one case, CDS did not change pre-specified outcomes relating to clinician confidence but was preferred by the users anyway [92].

Studies evaluating the effects of CDS on clinically relevant patient outcomes were uncommon. A CDS tool to point clinicians away from prescribing fluoroquinolone antibiotics showed decreased fluoroquinolone prescription rates but did not examine if patients had fewer adverse events or if their infections resolved appropriately [76]. A hard-stop best practice advisory to promote intensification of blood pressure control in patients with diabetes was successful in increasing treatment, but control of hypertension in the population was not assessed [79]. An evaluation of patient outcomes based on whether the physician accepted or declined CDS found no difference in A1c but statistically significant improvements in blood pressure and low-density lipoprotein (LDL) [46]. As mentioned earlier, a sepsis alert with poor PPV did not change length of stay or mortality [49]. One RCT of CDS found no difference in length of stay or mortality, though a modest increase in alert resolution [80]. Another RCT of a CDS system to assist in the management of patients with cardiovascular risk showed a 2.24% improvement in the surrogate outcome of annual change in predicted 10-year cardiovascular risk [82]. Of the 11 studies that examined CDS effects on end-users as well as patient health outcomes, only two [46] [82] showed benefit in some domains.

3.7 I’m a Believer – Clinician Buy-in

Multiple authors posited that for a CDS tool to be accepted and used by healthcare professionals, they must be convinced that the tool solves a meaningful clinical problem [31] [47] [92] [102], building on previous work on the importance of clinician buy-in for CDS uptake [103]. An investigation into why clinician adoption of a CDS tool was only 4-14% identified lack of buy-in as one of the factors deterring use [93]. Some studies noted concerns that CDS threatened clinician autonomy [67] or that CDS use would be monitored for punitive reasons [33]. For example, users were offended by the CDS prompt to use basal-bolus insulin when only sliding scale insulin had been ordered [67], demonstrating that even when CDS is supporting the evidence-based regimen the prompt needs to be palatable to clinicians. Accordingly, another study proposed increasing buy-in by framing CDS positively instead of negatively and making users feel recognized for their efforts [28]. From the surveyed literature, for there to be optimal clinician buy-in, a CDS system must foster the clinician’s belief that a worthwhile problem is being addressed, that CDS can solve the problem (patient outcomes), that it is targeting the correct patients (sensitivity and specificity, trust), that the alerts are relevant (PPV and alert burden), and that workflow interruptions are worth the benefits, all while using language that lifts up instead of denigrating the clinician. The need for clinician buy-in for CDS success is so significant that Kawamanto et al. state that there should be “a requirement that proposed CDS is actually desired by intended recipients” [75].

3.8 Fix It – Improving CDS

The surveyed studies on CDS and clinician burnout explored how CDS tools could become more acceptable to users. As a participant in one study put it: “[Not having an] alert is better than a poorly designed alert” [65]. The most important theme in improving CDS was iteration, meaning evaluating a tool, revising it, and then evaluating the revised tool. Multiple studies used iterative designs to fine-tune their CDS [26] [34] [42] [68] [74]. Mann et al. provided a description of the development of a CDS tool for antibiotic prescriptions: (i) Pre-deployment, the CDS team collected qualitative feedback during “think aloud” sessions on the draft versions of the tool, (ii) Post-deployment, they solicited feedback in group interviews and measured utilization rates as well as antibiotic prescription rates. Their CDS tool was based on a previous version developed via user-centered iteration that had nearly two-thirds utilization; however, one year after implementation, the utilization of the new tool was 4-14% across sites. Their inability to replicate a previous tool’s utilization success demonstrates that even intensive feedback and iteration may not be able to overcome factors that dissuade clinicians from CDS use, such as the fact that alert fatigue for clinicians had increased since the earlier version [93].

In contrast to the intensive qualitative evaluations of user perceptions undertaken by Mann et al., Yoshida et al. described an automated approach of CDS monitoring. Using this approach, the CDS team monitored for changes in firing rates or patterns indefinitely post-implementation. In addition to automated monitoring, the team undertook targeted monitoring of silent alert firing patterns for two weeks prior to roll-out, including chart review to capture false positives and negatives, post-implementation monitoring and chart review, and ad hoc monitoring whenever users reported problems. Their automatic monitoring system identified 128 issues with the CDS system over two years, of which 24 were false positives [60].

In a story of successful CDS improvement, Kawamanto et al. described the establishment of a CDS governance that involved a CDS committee, CDS for desired outcomes only, data analytics to monitor alert frequency, a push to switch alerts to other areas of the EHR, and experimental designs to improve effectiveness, resulting in a greater than 50% reduction in alerts [75]. In another example of improvement of CDS, Bubp et al. described the multi-phased development of a drug-disease alert system. In the first phase, pharmacists and physicians scored alerts from a database based on clinical utility and the scores were validated by a survey. During the second phase, alerts fired silently, and alert burden was evaluated. Finally, the alert system was rolled out, and post-implementation analysis showed decreased alert burden and the second-highest acceptance rate of any alert type at nearly 22%, behind dose alerts [62].

Similar to Bubp et al., expert panels were used in several studies to improve the relevance of CDS [62] [71] [88]. Expert panels can be advantageous over stock alerts. For example, one study found only 18% concordance on high severity alerts amongst four drug interaction compendia, whereas an expert panel was able to reach consensus on the relevance of 12/13 alerts [104]. A downside of expert panels is that they do not scale well given time limitations [85] [104].

Customizability was another factor that could improve user experience with CDS. Customizability can be at the scale of the hospital or clinic [69] [73] or by the end-user altering, for example, which alerts are seen [44]. Additionally, role-tailoring, in which CDS is targeted to users by role such as nurse, pharmacist, physician, etc., was also seen as a way to reduce burnout and increase acceptability in CDS [21] [51]. In one study, non-physician care-coordinators were responsible for data entry to support CDS [46]. In another, nurses rooming patients triggered the CDS [82]. Alerts tailored by role could also be used in cases where the ownership of an alert would otherwise be ambiguous, such as results of a genomic test [37]. Separate from role-tailoring, CDS that “nudges” [26] could be used to combat concerns about loss of autonomy [67]. Relevance could also be improved by tiering of alerts [20] [22]. For example, when one hospital disabled the least severe drug interaction alerts, alert burden fell by over 50% [69]. Another hospital was able to decrease “major” alerts by 62% via customization of the stock alerts [73].

Finally, multiple publications discussed the importance of end-user feedback throughout the design, implementation, and modification process. Several publications noted that nurses and nurse practitioners are also stakeholders who should be involved in CDS design [17] [18] [27]. Feedback before iteration was emphasized in multiple studies [42] [47] [68] [74] [93] and “the importance of listening to user experience” was explicitly highlighted [36]. Also important was the idea that implementation does not end with roll-out, but that CDS is an ongoing process of monitoring, feedback, and iteration [52] [60] [62] [65] [72].

4 Discussion

The rapid and widespread implementation of EHRs has contributed to a crisis of clinician dissatisfaction [105]. Clinical decision support is an important aspect of EHRs that is “not merely the use of technology; it is using technology to find meaningful information to make clinical decisions and provide the best possible patient care” [18]. As described above, how CDS is designed and implemented can have significant impacts on clinician users. Alarm fatigue from both a high volume of alerts and alerts of poor relevance, such as low PPV, were commonly described problems, leading to high rates of alert overrides or avoidance of CDS. Although the ideal PPVs, number of alerts, or alert override rates have not been established, this review identifies that generally poor performance of many CDS tools has been reported. Overrides, avoidance, and workarounds decrease effectiveness of CDS [95] in implementing behavioral change, though few studies directly evaluated the effectiveness of CDS on changing health-related patient outcomes. Studies also identified that CDS tools need to have clinician buy-in, which is partly achieved by fostering trust that the CDS is relevant to their patients but also by utilizing user-centered interfaces with integration into the workflow and EHR.

There were many suggestions on how to improve CDS tools in the surveyed literature, however it is not clear that they are being consistently implemented, perhaps because of a lack of randomized trials to guide design [106] [107]. Aspects of CDS that were found to be most helpful and harmful to clinician burnout in the recent RCTs and systematic reviews are highlighted in [Table 2]. As the meta-analysis by West et al. demonstrated, structural and organizational changes can result in meaningful reductions in burnout [107], so organizations designing and implementing CDS could potentially play a significant role in the wellness of front-line clinicians.

Table 2 Factors that ameliorate (helpful) or contribute to (harmful) clinician burnout as extracted from the systematic reviews and randomized controlled trials in this literature survey

Based on our survey of the recent literature, we have consolidated recurrent ideas into key factors that should be considered when designing and implementing CDS in order to minimize the effects on clinician burnout:

1. Be relevant. CDS should solve problems that clinicians feel need to be solved. CDS alerts should incorporate as much patient-specific information as possible to maximize PPV and minimize the number needed to remind.

2. Solicit feedback. End-users should be involved in all aspects of design, pre-testing, and implementation.

3. Customize. Whether allowing expert panels to tier alerts or clinicians to choose how and when to see CDS tools, customization can minimize alert burden and improve relevance as well as clinician satisfaction.

4. Measure outcomes. The effects on alert burden, override rates, workflow, efficiency, burnout, satisfaction, patient outcomes, etc. must be evaluated. Tools should either improve efficiency, patient outcomes, or both. Tools that do neither should be abandoned, especially if they add to alert burden or burnout.

5. Iterate. CDS requires ongoing maintenance based on feedback and outcomes, as well as updates to clinical practice standards.

Using these principles, future CDS tools can minimize their impacts on the multifactorial problem that is clinician burnout.

A strength of this study is that it included a wide array of healthcare professionals’ experiences including physicians [37] [62], advanced practice providers [17] [18] [63], nurses [90] [92], pharmacists [29] [33], midwives [38], and paramedics [59]. We also surveyed a heterogenous mix of publication types including systematic reviews, randomized trials, quality improvement initiatives, and opinion pieces. A limitation is that manual review of articles was undertaken by one individual and some relevant articles, study designs, or themes may have been misclassified or overlooked.

5 Conclusion

Clinical decision support tools can contribute to clinician frustration and burnout. Using the techniques of improving relevance, soliciting feedback, customization, measurement of outcomes and metrics, and iteration, the effects of CDS on burnout can be ameliorated.

Acknowledgements

We would like to thank Stanford Information Services Librarian Christopher Stave for his assistance in formulating the database queries. This research was supported in part by the NIH Big Data 2 Knowledge initiative via the National Institute of Environmental Health Sciences under Award Number K01ES026837 as well as by the Diabetes, Endocrinology, and Metabolism Training Grant 5T32DK007217-44. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

• References

• 1 Melnick ER, Dyrbye LN, Sinsky CA, Trockel M, West CP, Nedelec L. et al. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin Proc [Internet]. 2019 Nov 14 [cited 2019 Nov 15]; Available from: http://www.sciencedirect.com/science/article/pii/S0025619619308365
• 2 National Academies of Sciences Engineering, and Medicine; National Academy of Medicine. Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being [Internet]. Washington, DC: The National Academies Press; 2019. Available from https://www.nap.edu/catalog/25521/taking-action-against-clinician-burnout-a-systems-approach-to-professional
• 3 Gardner RL, Cooper E, Haskell J, Harris DA, Poplau S, Kroth PJ. et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc 2019; 26 (02) 106-14
• 4 Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J. et al. Relationship Between Clerical Burden and Characteristics of the Electronic Environment With Physician Burnout and Professional Satisfaction. Mayo Clin Proc 2016; 91 (07) 836-48
• 5 Kroth PJ, Morioka-Douglas N, Veres S, Babbott S, Poplau S, Qeadan F. et al. Association of Electronic Health Record Design and Use Factors With Clinician Stress and Burnout. JAMA Netw Open 2019; 2 (08) e199609-e199609
• 6 Desai SV, Asch DA, Bellini LM, Chaiyachati KH, Liu M, Sternberg AL. et al. Education Outcomes in a Duty-Hour Flexibility Trial in Internal Medicine. N Engl J Med [Internet]. 2018 Mar 20 [cited 2019 Nov 15]; Available from: https://www-nejm-org.laneproxy.stanford.edu/doi/10.1056/NEJMoa1800965
• 7 West CP, Dyrbye LN, Shanafelt TD. Physician burnout: contributors, consequences and solutions. J Intern Med 2018; 283 (06) 516-29
• 8 Coiera E, Ash J, Berg M. The Unintended Consequences of Health Information Technology Revisited. Yearb Med Inform 2016; Aug; 25 (01) 163-9
• 9 Shanafelt TD, Schein E, Minor LB, Trockel M, Schein P, Kirch D. Healing the Professional Culture of Medicine. Mayo Clin Proc 2019; 94 (08) 1556-66
• 10 Weed LL. Medical Records That Guide and Teach. N Engl J Med 1968; 278 (12) 652-7
• 11 Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L. et al. Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-based Medicine a Reality. J Am Med Inform Assoc 2003; 10 (06) 523-30
• 12 Glogowski J. GSU Library Research Guides: Literature Reviews: Types of Clinical Study Designs [Internet]. [cited 2020 May 1]. Available from: https://research.library.gsu.edu/c.php?g=115595&p=755213
• 13 Daly J, Willis K, Small R, Green J, Welch N, Kealy M. et al. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol 2007; 60 (01) 43-9
• 14 Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs 2003; 12 (01) 77-84
• 15 Osheroff JA, Teich J, Levick D, Saldana L, Valesco F, Sittig D. et al. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. 2nd ed. Chicago, IL: HIMSS; 2012
• 16 Marcial LH, Richardson JE, Lasater B, Middleton B, Osheroff JA, Kawamoto K. et al. The Imperative for Patient-Centered Clinical Decision Support. EGEMS Wash DC 2018; 6 (01) 12
• 17 Fant C, Adelman D. Too many medication alerts: How alarm frequency affects providers. Nurse Pract 2018; 43 (11) 48-52
• 18 Borum C. Barriers for Hospital-Based Nurse Practitioners Utilizing Clinical Decision Support Systems: A Systematic Review. Comput Inform Nurs 2018; Apr; 36 (04) 177-82
• 19 Wilbanks BA, McMullan SP. A Review of Measuring the Cognitive Workload of Electronic Health Records. Comput Inform Nurs 2018; Dec; 36 (12) 579-88
• 20 Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving medication-related clinical decision support. Am J Health-Syst Pharm AJHP Off J Am Soc Health-Syst Pharm 2018; 75 (04) 239-46
• 21 Hussain MI, Reynolds TL, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc 2019; 26 (10) 1141-9
• 22 Carroll AE. Averting Alert Fatigue to Prevent Adverse Drug Reactions. JAMA 2019; 322 (07) 601
• 23 Legat L, Van Laere S, Nyssen M, Steurbaut S, Dupont AG, Cornu P. Clinical Decision Support Systems for Drug Allergy Checking: Systematic Review. J Med Internet Res 2018; 20 (09) e258
• 24 Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review. JMIR Med Inform 2018; 6 (01) e3
• 25 Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc 2018; 25 (11) 1556-66
• 26 Luo S, Botash AS. Designing and Developing a Mobile App for Clinical Decision Support: An Interprofessional Collaboration. Comput Inform Nurs 2018; Dec; 36 (10) 467-72
• 27 Mahabee-Gittens EM, Dexheimer JW, Tabangin M, Khoury JC, Merianos AL, Stone L. et al. An Electronic Health Record-Based Strategy to Address Child Tobacco Smoke Exposure. Am J Prev Med 2018; Jan; 54 (01) 64-71
• 28 Brown B, Balatsoukas P, Williams R, Sperrin M, Buchan I. Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety. J Biomed Inform 2018; 77: 62-80
• 29 Bagri H, Dahri K, Legal M. Hospital Pharmacists’ Perceptions and Decision-Making Related to Drug-Drug Interactions. Can J Hosp Pharm 2019 72. (4)
• 30 Poncette AS, Spies C, Mosch L, Schieler M, Weber-Carstens S, Krampe H. et al. Clinical Requirements of Future Patient Monitoring in the Intensive Care Unit: Qualitative Study. JMIR Med Inform 2019; 7 (02) e13064
• 31 Johansson-Pajala RM. Conditions for the Successful Implementation of Computer-Aided Drug Monitoring From Registered Nurses’ Perspective-A Case Site Analysis. Comput Inform Nurs 2018; Oct; 37 (04) 196-202
• 32 Nilsson L, Fagerstrom C. Decision-makers and mediators in a home healthcare digitisation process: nurses’ experiences of implementation and use of a decision support system. Contemp Nurse 2018; 54 (4–5): 511-21
• 33 Giuliano CA, Binienda J, Kale-Pradhan PB, Fakih MG. “I Never Would Have Caught That Before”: Pharmacist Perceptions of Using Clinical Decision Support for Antimicrobial Stewardship in the United States. Qual Health Res 2018; Apr; 28 (05) 745-55
• 34 Goodspeed A, Kostman N, Kriete TE, Longtine JW, Smith SM, Marshall P. et al Leveraging the utility of pharmacogenomics in psychiatry through clinical decision support: a focus group study. Ann Gen Psychiatry 2019; 18: 13
• 35 Asan O, Nattinger AB, Gurses AP, Tyszka JT, Yen TWF. Oncologists’ Views Regarding the Role of Electronic Health Records in Care Coordination. JCO Clin Cancer Inform 2018; 2: 1-12
• 36 Thoma-Lurken T, Lexis MAS, Bleijlevens MHC, Hamers JPH. Perceived added value of a decision support App for formal caregivers in community-based dementia care. J Clin Nurs 2019; 28 (1–2): 173-81
• 37 Pet DB, Holm IA, Williams JL, Myers MF, Novak LL, Brothers KB. et al. Physicians’ perspectives on receiving unsolicited genomic results. Genet Med Off J Am Coll Med Genet 2018; Jun; 21 (02) 311-8
• 38 Abejirinde IOO, Douwes R, Bardaji A, Abugnaba-Abanga R, Zweekhorst M, van Roosmalen J. et al. Pregnant women’s experiences with an integrated diagnostic and decision support device for antenatal care in Ghana. BMC Pregnancy Childbirth 2018; 18 (01) 209
• 39 Banks J, Farr M, Salisbury C, Bernard E, Northstone K, Edwards H. et al. Use of an electronic consultation system in primary care: a qualitative interview study. Br J Gen Pract J R Coll Gen Pract 2018; 68 (666): e1-8
• 40 Nguyen KA, Patel H, Haggstrom DA, Zillich AJ, Imperiale TF, Russ AL. Utilizing a user-centered approach to develop and assess pharmacogenomic clinical decision support for thiopurine methyltransferase. BMC Med Inform Decis Mak 2019; 19 (01) 194
• 41 Bjorkman A, Salzmann-Erikson M. When all other doors are closed: Telenurses’ experiences of encountering care seekers with mental illnesses. Int J Ment Health Nurs 2018; Oct; 27 (05) 1392-400
• 42 Chokshi SK, Belli HM, Troxel AB, Blecker S, Blaum C, Testa P. et al Designing for implementation: user-centered development and pilot testing of a behavioral economic-inspired electronic health record clinical decision support module. Pilot Feasibility Stud 2019; 5: 28
• 43 Dehlendorf C, Reed R, Fitzpatrick J, Kuppermann M, Steinauer J, Kimport K. A mixed-methods study of provider perspectives on My Birth Control: a contraceptive decision support tool designed to facilitate shared decision making. Contraception 2019 Aug 9
• 44 Pirnejad H, Amiri P, Niazkhani Z, Shiva A, Makhdoomi K, Abkhiz S. et al. Preventing potential drug-drug interactions through alerting decision support systems: A clinical context based methodology. Int J Med Inform 2019 Jul;127:18–26
• 45 Alagiakrishnan K, Ballermann M, Rolfson D, Mohindra K, Sadowski CA, Ausford A. et al. Utilization of computerized clinical decision support for potentially inappropriate medications. Clin Interv Aging 2019; 14: 753-62
• 46 Singh K, Johnson L, Devarajan R, Shivashankar R, Sharma P, Kondal D. et al. Acceptability of a decision-support electronic health record system and its impact on diabetes care goals in South Asia: a mixed-methods evaluation of the CARRS trial. Diabet Med J Br Diabet Assoc 2018; Dec; 35 (12) 1644-54
• 47 Vousden N, Lawley E, Nathan HL, Seed PT, Brown A, Muchengwa T. et al. Evaluation of a novel vital sign device to reduce maternal mortality and morbidity in low-resource settings: a mixed method feasibility study for the. BMC Pregnancy Childbirth 2018; 18 (01) 115
• 48 Cox JL, Parkash R, Abidi SS, Thabane L, Xie F, MacKillop J. et al. Optimizing primary care management of atrial fibrillation: The rationale and methods of the Integrated Management Program Advancing Community Treatment of Atrial Fibrillation (IMPACT-AF) study. Am Heart J 2018; 201: 149-57
• 49 Austrian JS, Jamin CT, Doty GR, Blecker S. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc 2017; 25 (05) 523-9
• 50 Schild S, Sedlmayr B, Schumacher AK, Sedlmayr M, Prokosch HU, St Pierre M. A Digital Cognitive Aid for Anesthesia to Support Intraoperative Crisis Management: Results of the User-Centered Design Process. JMIR MHealth UHealth 2019; 7 (04) e13226
• 51 Blecker S, Pandya R, Stork S, Mann D, Kuperman G, Shelley D. et al. Interruptive Versus Noninterruptive Clinical Decision Support: Usability Study. JMIR Hum Factors 2019; 6 (02) e12469
• 52 Thayer JG, Miller JM, Fiks AG, Tague L, Grundmeier RW. Assessing the Safety of Custom Web-Based Clinical Decision Support Systems in Electronic Health Records: A Case Study. Appl Clin Inform 2019; Mar; 10 (02) 237-46
• 53 Winthereik AK, Neergaard MA, Jensen AB, Vedsted P. Development, modelling, and pilot testing of a complex intervention to support end-of-life care provided by Danish general practitioners. BMC Fam Pract 2018; 19 (01) 91
• 54 Baysari MT, Zheng WY, Li L, Westbrook J, Day RO, Hilmer S. et al. Optimising computerised decision support to transform medication safety and reduce prescriber burden: study protocol for a mixed-methods evaluation of drug-drug interaction alerts. BMJ Open 2019; 9 (08) e026034
• 55 Muhlenkamp R, Ash N, Ziegenbusch K, Rampe N, Bishop B, Adane E. Effect of modifying dose alerts in an electronic health record on frequency of alerts. Am J Health Syst Pharm 2019; ;76(Supplement_1): S1-8
• 56 Grout RW, Cheng ER, Carroll AE, Bauer NS, Downs SM. A six-year repeated evaluation of computerized clinical decision support system user acceptability. Int J Med Inform 2018; 112: 74-81
• 57 Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL. et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25 (05) 476-81
• 58 Ollerenshaw A, Wong Shee A, Yates M. Towards good dementia care: Awareness and uptake of an online Dementia Pathways tool for rural and regional primary health practitioners. Aust J Rural Health 2018; Apr; 26 (02) 112-8
• 59 Porter A, Dale J, Foster T, Logan P, Wells B, Snooks H. Implementation and use of computerised clinical decision support (CCDS) in emergency pre-hospital care: a qualitative study of paramedic views and experience using Strong Structuration Theory. Implement Sci 2018; 13 (01) 91
• 60 Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The Value of Monitoring Clinical Decision Support Interventions. Appl Clin Inform 2018; Jan; 9 (01) 163-73
• 61 Wong JC, Izadi Z, Schroeder S, Nader M, Min J, Neinstein AB. et al. A Pilot Study of Use of a Software Platform for the Collection, Integration, and Visualization of Diabetes Device Data by Health Care Providers in a Multidisciplinary Pediatric Setting. Diabetes Technol Ther 2018; Dec; 20 (12) 806-16
• 62 Bubp JL, Park MA, Kapusnik-Uner J, Dang T, Matuszewski K, Ly D. et al. Successful deployment of drug-disease interaction clinical decision support across multiple Kaiser Permanente regions. J Am Med Inform Assoc 2019; 26 (10) 905-10
• 63 Saman DM, Walton KM, Harry ML, Asche SE, Truitt AR, Henzler-Buckingham HA. et al. Understanding primary care providers’ perceptions of cancer prevention and screening in a predominantly rural healthcare system in the upper Midwest. BMC Health Serv Res [Internet] 2019 Dec 30 [cited 2020 Apr 29];19. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6937782/
• 64 Long D, Capan M, Mascioli S, Weldon D, Arnold R, Miller K. Evaluation of User-Interface Alert Displays for Clinical Decision Support Systems for Sepsis. Crit Care Nurse 2018; Aug; 38 (04) 46-54
• 65 Page NJ, Baysari MT, Westbrook JI. Selection and use of decision support alerts in electronic medication management systems in Australian hospitals: a survey of implementers. J Pharm Pract Res 2019; 49 (02) 142-9
• 66 Tu MH, Chang P, Lee YL. Avoiding Obsolescence in Mobile Health: Experiences in Designing a Mobile Support System for Complicated Documentation at Long-term Care Facilities. Comput Inform Nurs 2018; Oct; 36 (10) 501-6
• 67 Helmle KE, Edwards AL, Kushniruk AW, Borycki EM. Qualitative Evaluation of the Barriers and Facilitators Influencing the Use of an Electronic Basal Bolus Insulin Therapy Protocol to Improve the Care of Adult Inpatients With Diabetes. Can J Diabetes 2018; 42 (05) 459-464e1
• 68 Salz T, Schnall RB, McCabe MS, Oeffinger KC, Corcoran S, Vickers AJ. et al. Incorporating Multiple Perspectives Into the Development of an Electronic Survivorship Platform for Head and Neck Cancer. JCO Clin Cancer Inform 2018; 2: 1-15
• 69 Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record. J Gen Intern Med 2018; 33 (11) 1868-76
• 70 Grabel MZ, Vaughan BL, Dexheimer JW, Kirkendall ES. Mathematical Model for Computer-Assisted Modification of Medication Dosing Rules. Biomed Inform Insights 2019; 11: 1178222619829079
• 71 Heringa M, van der Heide A, Floor-Schreudering A, De Smet PAGM, Bouvy ML. Better specification of triggers to reduce the number of drug interaction alerts in primary care. Int J Med Inform 2018; 109: 96-102
• 72 Saiyed SM, Davis KR, Kaelber DC. Differences, Opportunities, and Strategies in Drug Alert Optimization—Experiences of Two Different Integrated Health Care Systems. Appl Clin Inform 2019; 10 (05) 777-82
• 73 Horn J, Ueng S. The Effect of Patient-Specific Drug-Drug Interaction Alerting on the Frequency of Alerts: A Pilot Study. Ann Pharmacother 2019; 53 (11) 1087-92
• 74 Khan S, Richardson S, Liu A, Mechery V, McCullagh L, Schachter A. et al. Improving Provider Adoption With Adaptive Clinical Decision Support Surveillance: An Observational Study. JMIR Hum Factors 2019; 6 (01) e10245
• 75 Kawamanto K, Flynn MC, Kukhareva P, El Halta D, Hess R, Gregory T. et al. A Pragmatic Guide to Establishing Clinical Decision Support Governance and Addressing Decision Support Fatigue: a Case Study. AMIA Annu Symp Proc AMIA Symp 2018; 2018: 624-33
• 76 Gunn LR, Tunney R, Kelly K. Nonmodal Clinical Decision Support and Antimicrobial Restriction Effects on Rates of Fluoroquinolone Use in Uncomplicated Infections. Appl Clin Inform 2018; Jan; 9 (01) 149-55
• 77 Khairat S, Coleman C, Ottmar P, Bice T, Koppel R, Carson SS. Physicians’ gender and their use of electronic health records: findings from a mixed-methods usability study. J Am Med Inform Assoc 2019; 26 (12) 1505-14
• 78 Rapoport MJ, Zucchero Sarracini C, Kiss A, Lee L, Byszewski A, Seitz DP. et al. Computer-Based Driving in Dementia Decision Tool With Mail Support: Cluster Randomized Controlled Trial. J Med Internet Res 2018; 20 (05) e194
• 79 Ramirez M, Maranon R, Fu J, Chon JS, Chen K, Mangione CM. et al. Primary care provider adherence to an alert for intensification of diabetes blood pressure medications before and after the addition of a “chart closure” hard stop. J Am Med Inform Assoc 2018; 25 (09) 1167-74
• 80 Moja L, PoloFriz H, Capobussi M, Kwag K, Banzi R, Ruggiero F. et al. Effectiveness of a Hospital-Based Computerized Decision Support System on Clinician Recommendations and Patient Outcomes. JAMA Netw Open [Internet] 2019 Dec 11 [cited 2020 May 4];2(12). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6991299
• 81 Wasylewicz ATM, Korsten EHM, Egberts TCG, Grouls RJE. Clinical rule-guided pharmacists’ intervention in hospitalized patients with hypokalaemia: A time series analysis. J Clin Pharm Ther 2020; 45 (03) 520-9
• 82 Sperl-Hillen JM, Crain AL, Margolis KL, Ekstrom HL, Appana D, Amundson G. et al. Clinical decision support directed to primary care patients and providers reduces cardiovascular risk: a randomized trial. J Am Med Inform Assoc 2018; 25 (09) 1137-46
• 83 Cresswell K, Callaghan M, Mozaffar H, Sheikh A. NHS Scotland’s Decision Support Platform: a formative qualitative evaluation. BMJ Health Care Inform 2019 26. (1)
• 84 Canovas-Segura B, Morales A, Juarez JM, Campos M, Palacios F. Impact of expert knowledge on the detection of patients at risk of antimicrobial therapy failure by clinical decision support systems. J Biomed Inform 2019; 94: 103200
• 85 Divakaran S, Singh A, De Filippis EM, Churchill TW, Cuddy S, Ge Y. et al. Appropriateness of inpatient stress testing: Implications for development of clinical decision support mechanisms and future criteria. J Nucl Cardiol [Internet] 2019 Nov 18 [cited 2020 May 4]; Available from: https://doi.org/10.1007/s12350-019-01955-x
• 86 Wulff A, Montag S, Steiner B, Marschollek M, Beerbaum P, Karch A. et al. CADDIE2—evaluation of a clinical decision-support system for early detection of systemic inflammatory response syndrome in paediatric intensive care: study protocol for a diagnostic study. BMJ Open 2019; 9 (06) e028953
• 87 Rousseau JF, Ip IK, Raja AS, Valtchinov VI, Cochon L, Schuur JD. et al. Can Automated Retrieval of Data from Emergency Department Physician Notes Enhance the Imaging Order Entry Process?. Appl Clin Inform 2019; 10 (02) 189-98
• 88 Bakker T, Klopotowska JE, Eslami S, de Lange DW, van Marum R, van der Sijs H. et al. The effect of ICU-tailored drug-drug interaction alerts on medication prescribing and monitoring: protocol for a cluster randomized stepped-wedge trial. BMC Med Inform Decis Mak 2019; 19 (01) 159
• 89 Johansson-Pajala RM, Martin L, Jorsater Blomgren K. Registered nurses’ use of computerised decision support in medication reviews. Int J Health Care Qual Assur 2018; 31 (06) 531-44
• 90 Ankem K, Cho S, Simpson D. Nurses’ perceptions and problems in the usability of a medication safety app. Inform Health Soc Care 2019; Jan; 44 (01) 48-69
• 91 Skyttberg N, Chen R, Koch S. Man vs machine in emergency medicine - a study on the effects of manual and automatic vital sign documentation on data quality and perceived workload, using observational paired sample data and questionnaires. BMC Emerg Med 2018; 18 (01) 54
• 92 Thoma-Lurken T, Bleijlevens MHC, Lexis MAS, Hamers JPH. Evaluation of a decision support app for nurses and case managers to facilitate aging in place of people with dementia. A randomized controlled laboratory experiment. Geriatr Nurs N Y N 2018; Nov; 39 (06) 653-62
• 93 Mann D, Hess R, McGinn T, Mishuris R, Chokshi S, McCullagh L. et al Adaptive design of a clinical decision support tool: What the impact on utilization rates means for future CDS research. Digit Health 2019; 5: 2055207619827716
• 94 Fortin C, van Schaik P, Aubin-Fournier JF, Bettany-Saltikov J, Bernard JC, Ehrmann Feldman D. The acceptance of the clinical photographic posture assessment tool (CPPAT). BMC Musculoskelet Disord 2018; 19 (01) 366
• 95 Bertoni CB, Prusakov P, Merandi J, Bartman T. Clinical Decision Support to Improve Dosing Weight Use in Infants with Neonatal Abstinence Syndrome. Pediatr Qual Saf 2019; 4 (04) e184
• 96 Ubanyionwu S, Formea CM, Anderson B, Wix K, Dierkhising R, Caraballo PJ. Evaluation of prescriber responses to pharmacogenomics clinical decision support for thiopurine S-methyltransferase testing. Am J Health-Syst Pharm AJHP Off J Am Soc Health-Syst Pharm 2018; 75 (04) 191-8
• 97 Biswas A, Parikh CR, Feldman HI, Garg AX, Latham S, Lin H. et al. Identification of patients expected to benefit from electronic alerts for acute kidney injury. Clin J Am Soc Nephrol 2018; 13 (06) 842-9
• 98 Kizzier-Carnahan V, Artis KA, Mohan V, Gold JA. Frequency of Passive EHR Alerts in the ICU: Another Form of Alert Fatigue?. J Patient Saf 2019; Sep; 15 (03) 246-50
• 99 Arts DL, Medlock SK, van Weert HCPM, Wyatt JC, Abu-Hanna A. Acceptance and barriers pertaining to a general practice decision support system for multiple clinical conditions: A mixed methods evaluation. PloS One 2018; 13 (04) e0193187
• 100 Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM. et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ [Internet] 2013 Feb 14 [cited 2020 Apr 29];346. Available from: https://www.bmj.com/content/346/bmj.f657
• 101 Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V. et al. Effectiveness of Computerized Decision Support Systems Linked to Electronic Health Records: A Systematic Review and Meta-Analysis. Am J Public Health 2014; 104 (12) e12-22
• 102 Cimino JJ. Putting the “why” in “EHR”: capturing and coding clinical cognition. J Am Med Inform Asso 2019; 26 (11) 1379-84
• 103 Liberati EG, Ruggiero F, Galuppo L, Gorli M, González-Lorenzo M, Maraldi M. et al. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci 2017; 12 (01) 113
• 104 Meslin SMM, Zheng WY, Day RO, Tay EMY, Baysari MT. Evaluation of Clinical Relevance of Drug–Drug Interaction Alerts Prior to Implementation. Appl Clin Inform 2018; 9 (04) 849-55
• 105 Auerbach AD, Khanna R, Adler-Milstein J. Letting a Good Crisis Go to Waste. J Gen Intern Med [Internet] 2019 Nov 19 [cited 2019 Nov 21]; Available from: https://doi.org/10.1007/s11606-019-05552-z
• 106 Panagioti M, Panagopoulou E, Bower P, Lewith G, Kontopantelis E, Chew-Graham C. et al. Controlled Interventions to Reduce Burnout in Physicians: A Systematic Review and Meta-analysis. JAMA Intern Med 2017; 177 (02) 195-205
• 107 West CP, Dyrbye LN, Erwin PJ, Shanafelt TD. Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis. Lancet 2016; 388 10057 2272-81
• 108 Fitzmaurice MG, Wong A, Akerberg H, Avramovska S, Smithburger PL, Buckley MS. et al. Evaluation of Potential Drug–Drug Interactions in Adults in the Intensive Care Unit: A Systematic Review and Meta-Analysis. Drug Saf 2019; 1-10
• 109 Gesner E, Gazarian P, Dykes P. The Burden and Burnout in Documenting Patient Care: An Integrative Literature Review. Stud Health Technol Inform 2019; 264: 1194-8
• 110 Zeng-Treitler Q, Nelson SJ. Will Artificial Intelligence Translate Big Data Into Improved Medical Care or Be a Source of Confusing Intrusion? A Discussion Between a (Cautious) Physician Informatician and an (Optimistic) Medical Informatics Researcher. J Med Internet Res [Internet] 2019 Nov 27 [cited 2020 May 4];21(11). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6906615/