Evaluation of an Antimicrobial Stewardship Decision Support for Pediatric Infections

Abstract

Objectives Clinical decision support (CDS) has promise for the implementation of antimicrobial stewardship programs (ASPs) in the emergency department (ED). We sought to assess the usability of a newly developed automated CDS to improve guideline-adherent antibiotic prescribing for pediatric community-acquired pneumonia (CAP) and urinary tract infection (UTI).

Methods We conducted comparative usability testing between an automated, prototype CDS-enhanced discharge order set and standard order set, for pediatric CAP and UTI antibiotic prescribing. After an extensive user-centered design process, the prototype CDS was integrated into the electronic health record, used passive activation, and embedded locally adapted prescribing guidelines. Participants were randomized to interact with three simulated ED scenarios of children with CAP or UTI, across both systems. Measures included task completion, decision-making and usability errors, clinical actions (order set use and correct antibiotic selection), as well as objective measures of system usability, utility, and workload using the National Aeronautics and Space Administration Task Load Index (NASA-TLX). The prototype CDS was iteratively refined to optimize usability and workflow.

Results Usability testing in 21 ED clinical providers demonstrated that, compared to the standard order sets, providers preferred the prototype CDS, with improvements in domains such as explanations of suggested antibiotic choices (p < 0.001) and provision of additional resources on antibiotic prescription (p < 0.001). Simulated use of the CDS also led to overall improved guideline-adherent prescribing, with a 31% improvement for CAP. A trend was present toward absolute workload reduction. Using the NASA-TLX, workload scores for the current system were median 26, interquartile ranges (IQR): 11 to 41 versus median 25, and IQR: 10.5 to 39.5 for the CDS system (p = 0.117).

Conclusion Our CDS-enhanced discharge order set for ED antibiotic prescribing was strongly preferred by users, improved the accuracy of antibiotic prescribing, and trended toward reduced provider workload. The CDS was optimized for impact on guideline-adherent antibiotic prescribing from the ED and end-user acceptability to support future evaluative trials of ED ASPs.

Keywords

clinical decision support - user testing - antimicrobial stewardship

Background and Significance

Antimicrobial resistance is a major public health threat. Each year, antibiotic-resistant bacteria account for 2.8 million infections and 35,000 deaths in the United States.[1] A large contributor to the development of antibiotic resistance is poor adherence to guideline-recommended antibiotic prescribing. In the emergency department (ED), approximately 30 to 50% of the 10 million outpatient antibiotic prescriptions are inappropriate or unnecessary.[2] [3] Consequently, multiple national organizations have emphasized the need for antimicrobial stewardship programs (ASPs) in the ED.[4] [5] [6] ASPs are effective for inpatient settings, reducing costs and the impact of resistant bacteria through the promotion of narrow-spectrum and guideline-adherent antibiotic prescribing.[7] [8] [9] [10] However, there are barriers to ASP implementation in the ED, including erratic workflow, shift work by clinical providers, and the need for empiric therapy in the absence of confirmed diagnosis.[11] In addition, the failure to adapt national guidelines into ED local context has impaired uptake of antibiotic-prescribing guidelines.[12]

Our investigative team recently determined that automated clinical decision support (CDS) systems are strongly preferred by ED and hospital antimicrobial stewardship leaders.[13] Using a user-centered design approach, our team rigorously developed a prototype electronic health record (EHR)-based CDS to improve guideline-adherent prescribing for two high-priority pediatric infections, community-acquired pneumonia (CAP), and urinary tract infections (UTIs).[14] [15] The prototype antimicrobial stewardship CDS presents locally adapted prescribing recommendations and accounts for ED provider preferences and workflow.[15] The long-term goal is to create a platform-agnostic CDS that can be used interchangeably within EHRs and be widely implemented in EDs.

User-centered methods in the evaluation, development, and deployment of EHRs, and specifically CDS, are recommended to improve system usability. Formal usability testing is essential to evaluate the effectiveness of health information technology,[16] though underused even in initial development and configuration of the hospital-based EHR.[17] [18] Development of effective CDS with user-centered design methods has the potential to reduce cognitive burden, improve efficiency, and minimize EHR burnout in providers.[19] [20] [21] [22] Furthermore, CDS has the potential to drive clinical improvements including adherence to clinical guidelines and evidence-based care for common infectious diseases such as CAP.[23] [24] [25] Using a simulated ED environment, the objective of our pilot study was to test the usability and effectiveness of a new, prototype CDS developed through user-centered design for antibiotic prescribing for pediatric CAP and UTI.

Methods

Study Design

This was a user-testing study comparing a CDS-enhanced EHR order set (prototype CDS) for outpatient antibiotic prescribing for pediatric CAP and UTI, according to the current standard EHR order set used in our ED. The study was deemed exempt by our institutional review board.

Study Setting and Participants

This was a scenario-based laboratory study conducted among ED providers from a large, tertiary care, academic pediatric health system participating in the Pediatric Emergency Care Applied Research Network (PECARN). The EDs within the health system consist of a tertiary care referral ED, three community-based EDs, and two free-standing urgent care (UC) centers that receive approximately 170,000 visits annually. All sites operate on the same EHR (Epic Systems Inc., Verona, Wisconsin, United States); the appearance and function of the EHR are identical at all sites. The ED provider group consists of over 150 pediatric emergency medicine specialists, emergency general pediatricians, and advanced practice providers and trainees. For the study, we purposively sampled ED providers using e-mail invitations and targeted a variety of providers that clinically practice in the EDs and UCs. Verbal consent was obtained at the start of user testing sessions.

Description of the Prototype Clinical Decision Support for Community-Acquired Pneumonia and Urinary Tract Infection Antibiotic Prescribing

Using user-centered design principles, our study team developed a prototype CDS for ED antibiotic prescribing with input from three PECARN institutions to enhance the generalizability of the CDS content and activation. Our team formally adapted CAP and UTI CDS antibiotic recommendations into the local context to account for current institutional pathway recommendations, antibiotic resistance patterns, and pharmacy availability.[26] [27] We refined guideline-adherent recommendations using patient characteristics, such as age, presence of fever, penicillin allergy, and results of diagnostic testing; the team translated these adapted prescribing recommendations into algorithms which were included in the CDS prototype. Key stakeholder interviews and provider focus groups informed CDS format and strategies for integration into ED care, formal workflow observation and analysis informed EHR triggering and presentation of the CDS.[14] [15] Our overall approach used passive CDS techniques rather than hard stops. The CDS was designed in accordance with the five rights of CDS and underwent heuristic review by investigators and users to ensure functionality.[28] The prototype CDS consisted of orders embedded in the usual patient discharge workflow to guide end-users toward the correct antibiotic choice, paired with best practice advisories (BPAs) displays of informational resources regarding CAP and UTI antibiotic prescribing ([Fig. 1]). The CDS also included a notification banner alerting providers if a patient had a prior diagnosis of a UTI and directing users to review prior cultures/sensitivities.

Fig. 1 Explanatory text for CAP and UTI present in prototype CDS. CAP, community-acquired pneumonia; CDS, clinical decision support; PCN, penicillin; UTI, urinary tract infection;

Study Procedure and Usability Testing Sessions

We used a scenario-based approach to compare the prototype CDS to the current approach for antibiotic prescribing in the EHR.[29] [30] A simulated EHR test environment was created with realistic clinical scenarios for CAP and UTI ([Supplementary Appendix], available in the online version). In total, there were six simulated patients for CAP and eight for UTI. Simulated patients were designed to address and represent various factors present in the aforementioned antibiotic-prescribing algorithms. Each case scenario could be used to measure the current approach to antibiotic prescribing or to test the usability of the prototype CDS.

We conducted usability testing with three simulated patients for each participant. The first case presented to the participant tested the current ordering approach, followed by two cases using the prototype CDS. Simulated patients were selected at random for each user test to ensure that the cases were distributed among the participants in a balanced fashion.[31] The first case assigned to participants was randomized to be either a CAP or UTI patient, and the second and third cases included both a CAP and UTI patients presented in random order.

Participants were instructed to read each virtual case as presented on paper, and then review the patient chart in the EHR as desired, with the intent of ultimately discharging the patient from the ED. Each participant completed the task of discharging their first assigned patient using the current approach to outpatient prescribing in the existing (unenhanced) EHR workflow. The next two simulated patients presented were designed to trigger the appearance of the prototype CDS in the EHR discharge workflow. The appearance of the CDS was specifically triggered by the entry of one of a preselected set of discharge diagnoses for CAP or UTI. For the discharge diagnosis to activate the CDS, there had to be a recent order for a chest X-ray for patients presenting with CAP or a positive urinalysis result for patients who presented with symptoms of a UTI. These orders and results were built into the simulated patient scenarios prior to user testing in order to direct the end-user toward discharge diagnoses and antibiotic prescribing rather than diagnostic workup. Participants independently conducted all antibiotic ordering tasks; redirection was briefly provided by the study team observer only if necessary. Specific participant behaviors were documented on a standardized data collection form. Testing sessions were conducted by one of two observers (E.A.M. and R.D.M.), who instructed participants to follow “think-aloud” protocols throughout each case scenario, which called for participants to verbalize all thoughts as they interact with the EHR.[32] In addition to recording objective measures, observers took qualitative notes as participants made think-aloud statements. Each user testing session took approximately 25 to 30 minutes.

Study Phases

Our study was conducted in two phases. The intent of having two phases was to determine if major functional issues existed with the prototype CDS, or if minor adjustments were needed to improve appearance or usability. Phase 1 involved initial user testing. After nine user-test sessions in Phase 1, it was determined that only minor design and workflow changes were recommended for the prototype CDS, based on participant feedback. Subsequently, 12 user tests were conducted phase after prototype CDS revision. We present the majority of results as overall (Phase 1 and Phase 2 combined) as iterative changes were expected to be minor and not expected to substantially change our results. However, we analyzed and present our provider assessments by each phase of the study, as well.

Outcome Measures of User Testing

The study team identified seven functions of the CDS to assess provider satisfaction with current EHR ordering and the prototype CDS approach to antibiotic prescribing ([Table 1]). Each item was rated on a five-point Likert scale (1—very dissatisfied; 5—very satisfied). After participants worked through each clinical scenario, they completed assessments specific to the CAP and UTI cases and each diagnosis-specific CDS. Provider perceptions of the usability of the current and CDS enhanced order set were also assessed through participant surveys; domains assessed in these are summarized in [Table 2]. We utilized five-point Likert scales to assess agreement with the domains of the CDS (1—strongly disagree; 5—strongly agree).

Table 1 Domains assessed for provider satisfaction with the prototype CDS and disease-specific scenarios using five-point Likert scales

Domain	Assessment description
Antibiotic prescribing well structured	Assess whether the disease-specific CDS is well structured and well-integrated into the ED EHR workflow
System helped with patient management	Assess whether the disease-specific CDS improves outpatient antibiotic prescription and overall patient management in the ED EHR workflow
Preferred over prior methods	Assess whether the user prefers the disease-specific CDS over the current EHR workflow
Saved time	Assess whether the prototype CDS saves time compared with the current EHR workflow for the disease-specific antibiotic prescription process
Overall satisfaction	Assess overall satisfaction with how the CAP or UTI CDS is designed to aid providers in outpatient antibiotic prescribing

Abbreviations: CAP, community-acquired pneumonia; CDS, clinical decision support; ED, emergency department; EHR, electronic health record; UTI, urinary tract infection.

Table 2 Functions of the CDS used to assess provider perceptions of usability

Function	Function description
Patient identification	Identification of patients with evidence of CAP or UTI
Antibiotic selection	Ease of selecting appropriate antibiotics for treatment of CAP or UTI
Integration into workflow	Ease of outpatient antibiotic prescription in the ED EHR workflow
Suggestion of antibiotic alternatives	Provision of specific and appropriate alternative antibiotics when patients have antibiotic allergies
Explanation of suggested antibiotics	Provision of explanations for recommended antibiotic choices
Provision of prescribing resources	Provision of additional resources to help the user better understand suggested antibiotic choices
Overall satisfaction	Overall satisfaction with how the CDS is designed to aid providers in outpatient antibiotic prescribing

Abbreviations: CAP, community-acquired pneumonia; CDS, clinical decision support; ED, emergency department; EHR, electronic health record; UTI, urinary tract infection.

The study assessed the ordering systems using established usability testing and performance measures, including task completion and errors related to decision-making and system usability.[32] Decision-making outcomes included participant choices for appropriate CAP and UTI treatment actions, including the selection of the appropriate diagnosis to trigger the CDS order set and the choice of guideline-adherent antibiotic therapy for CAP and UTI. Usability outcomes included those related to appropriate participant actions used to engage with the ordering system, including interacting with best-practice alerts and completion of all prescribing tasks. For the purposes of the study, we calculated rates of errors in participant decision-making and CDS usability. “Decision-making errors” were defined as errors in clinical decisions and included failure to activate the CDS with an appropriate diagnosis and prescription errors: ordering a nonguideline-adherent antibiotic, not ordering an antibiotic when one was warranted or ordering an antibiotic when not recommended per local antibiotic-prescribing guidelines. “Usability errors” were related to system usability or errors made causing deviations from the most direct and correct pathway to patient discharge. Usability errors included failure to select the correct discharge order set without prompting or prescription of the incorrect antibiotic only if the user subsequently corrected the order based on CDS recommendations (correcting an otherwise decision-making error). The occurrence of decision-making and usability errors was compared between current prescribing workflow and the prototype CDS.

The study measured workload using the National Aeronautics and Space Administration Task Load Index (NASA-TLX).[33] [34] [35] The NASA-TLX is a multidimensional scale designed to obtain workload estimates from one or more operators, while they are performing a task or immediately afterward and has been used to evaluate EHR-based tools.[36] [37] Study observers administered the NASA-TLX to participants twice: once after the completion of the current antibiotic-prescribing method test case, and a second time after the completion of both scenarios using the prototype CDS. Participants completed a posttest survey after finishing all cases and NASA-TLX tools.

Data Collection and Analysis

The study team summarized all quantitative data using standard descriptive statistics. Planned comparisons included those between the current ordering process and the prototype CDS approach for all items, as well as between the two study phases to determine if iterative changes were influencing testing results. Proportions were compared using chi-square testing or Fisher's Exact test, where appropriate. For Likert scale items and NASA-TLX, we calculated both medians and interquartile ranges (IQRs). Though most data approximated normality, we used nonparametric statistics for comparison owing to the limited sample size. Therefore, we also reported means and standard deviations (SDs) for clinical relevance. Statistical comparisons were made using the Wilcoxon signed-rank test as these represent a pre–post-analysis within the same population. Analysis of the NASA-TLX raw scores was conducted as instructed by the developers, including paired tests using the Wilcoxon signed-rank test.[38] Results of hypothesis testing were reported with both the effect size r = Z / √N, where Z is Wilcoxon's test statistic and N is the sample size, as well as p-values, with a significance level of <0.05.[39] Magnitude of effect sizes were interpreted according to Cohen's criteria where r = 0.2 small, r = 0.5 medium, and r = 0.8 large.[40] All quantitative data were analyzed using IBM SPSS version 27.0 (IBM Co, Armonk, New York, United States). Qualitative comments were thematically summarized for presentation.

Results

Demographics of Study Participants

The study team conducted a total of 21 user tests. Nine providers participated in Phase 1 of the study, and 12 providers participated in Phase 2. All results presented are overall analyses with both phases combined (n = 21), unless otherwise stated. Participants included residents (n = 10; 47.6%), fellows and attendings in pediatric emergency medicine (n = 9; 42.9%), pediatricians (n = 1; 4.8%), and advanced practice providers (n = 1; 4.8%). Most participants worked as providers in the tertiary care referral ED (85.7%). The median number of years in the practice was 4.5 years (range of 1.5–20 years). Most providers (81.0%) indicated that they prescribed outpatient antibiotics on a weekly basis.

Overall Assessment of Current System and Prototype Clinical Decision Support

The prototype CDS exhibited moderate-to-large preferences in usability as compared to the current EHR order set in each domain ([Table 3]). Overall results demonstrated that the prototype CDS exhibited significant improvement in the identification of patients needing antibiotic treatment, antibiotic selection, suggestion of antibiotic alternatives, and overall satisfaction ([Table 3]), with a large effect on the explanation of suggested antibiotics (r = 0.773) and the provision of prescribing resources (r = 0.848). These findings were also reflected in the qualitative data provided by study participants. One participant commented “It's nice that the [link to the] clinical pathway is right here,”, while another noted that they “really like the explanations here [in the Best Practice Advisory box].” Compiled qualitative data are available in [Supplementary Table S1] ([Supplementary Material], available in the online version).

Table 3 Perceptions of system usability assessed by participants (n = 21) for current and CDS ordering systems (5-point Likert scale: 1 = very dissatisfied; 5 = very satisfied)

Function	Current system		Prototype CDS		Effect size (r)	p-Value[a]
	Median (IQR)	Mean ± SD	Median (IQR)	Mean ± SD
Patient identification	4 (3–5)	3.4 ± 0.8	4 (3–5)	4.2 ± 0.6	0.660	0.002
Antibiotic selection	4 (3.5–4.5)	3.8 ± 0.6	5 (4–5)	4.5 ± 0.8	0.612	0.005
Integration into workflow	4 (3–5)	3.8 ± 0.7	4 (3–5)	4.2 ± 0.8	0.42	0.053
Suggestion of antibiotic alternatives	3 (2–4)	3.2 ± 0.8	5 (4–5)	4.4 ± 0.8	0.62	0.004
Explanation of suggested antibiotics	3 (2–4)	2.6 ± 0.8	4 (3–5	4.0 ± 1.1	0.773	<0.001
Provision of prescribing resources	3 (2–4)	2.5 ± 0.6	5 (4–5)	4.4 ± 0.8	0.848	<0.001
Overall satisfaction	4 (3–5)	3.5 ± 0.6	5 (4–5)	4.4 ± 0.8	0.700	0.001

Abbreviations: CDS, clinical decision support, IQR, interquartile range, SD, standard deviation.

^a Statistical comparisons made using the Wilcoxon Signed-Rank test. Effect size reported using effect size r = Z/√n, where Z is the Wilcoxon test statistic and n is the sample size.

Assessment of Current System and Prototype Clinical Decision Support: Phase 1 to Phase 2

Qualitative comments from Phase 1 participants suggested minor design and workflow changes were warranted. Trends toward preferences for the prototype CDS were present in Phase 1 of the study, though only the provision of prescribing resources reached statistical significance ([Table 4]). Phase 1 participants provided feedback regarding the number of appropriate antibiotic choices offered for the treatment of CAP in penicillin-allergic patients. In Phase 1, the CDS listed three cephalosporins as equivalent antibiotic choices for penicillin-allergic patients. One participant commented, “There are too many [health system] sanctioned cephalosporin options.” Another participant commented that it would be nice to stratify these options by cost or to include a comment stating that they are all equivalent if this were truly the case. Phase-specific data are present in the [Supplementary tables S2] and [S3] ([Supplementary Material], available in the online version).

Table 4 Function satisfaction by phase as measured by 5-point Likert scale[a]

	Phase 1 (n = 9)				Phase 2 (n = 12)
Function	Current System Median (IQR)	Prototype CDS Median (IQR)	Effect Size (r)	p-Value	Current System Median (IQR)	Prototype CDS Median (IQR)	Effect Size (r)	p-Value
Patient identification	4 (3–5)	4 (3–5)	0.544	0.102	3.5 (2.5–4.5)	4 (3–5)	0.761	0.008
Antibiotic selection	4 (3–5)	4 (2–5)	0.360	0.279	4 (3.25–4.75)	5 (4.25–5)	0.833	0.004
Integration into workflow	4 (4–4)	4 (3–5)	0	1.000	4 (3–5)	5 (4–5)	0.709	0.014
Suggestion of antibiotic alternatives	3 (2–4)	4 (2–5)	0.393	0.238	3 (1.25–4.75)	5 (4–5)	0.763	0.008
Explanation of suggested antibiotic choices	3 (1.5–4.5)	4 (1.5–5)	0.602	0.071	2.5 (1.5–3.5)	4.5 (3.5–5)	0.879	0.002
Provision of prescribing resources	2 (1–3)	4 (2.5–5)	0.853	0.011	3 (2–4)	5 (4–5)	0.873	0.002
Overall satisfaction	4 (3–5)	4 (3–5)	0.471	0.157	4 (3–5)	5 (4.25–5)	0.833	0.004

Abbreviations: CDS, clinical decision support, IQR, interquartile range.

^a Statistical comparisons made using the Wilcoxon's signed-rank test. Effect size reported using effect size r = Z / √n, where Z is Wilcoxon's test statistic and n is the sample size.

In response to participant feedback in Phase 1, the study team made several refinements to the prototype CDS. After discussion with institutional experts, we narrowed the suggested antibiotic choice for penicillin-allergic children with CAP to a single cephalosporin option. Of note, all three previous cephalosporin options were guideline adherent; therefore, the limitation to a single option simplified but did not influence the proportion of “correct” selections. To enhance CDS activation for CAP, trigger criteria were expanded to include multiple variations of the chest X-ray order (1 view, 2 view, and 3 view). Finally, the explanatory text content in the informational BPA was revised using feedback in order to be more streamlined in presentation. The revisions also included the addition of information regarding antibiotic prescription suggestions for penicillin-allergic patients. The results from Phase 2 of this trial demonstrated statistically significant improvement in all ordering functions assessed using the prototype CDS as compared to the current system, with high-moderate-to-large effect sizes ([Table 4]).

Disease-Specific Results for Community-Acquired Pneumonia and Urinary Tract Infection

For pediatric CAP, the prototype CDS led to an improvement in participant selection of the correct antibiotic for the treatment of CAP as compared with the current order set (55 vs. 86%) ([Table 5]). There was a decrease in the number of decision-making errors made by participants with the CDS; 64% of participants made decision-making errors when using the current EHR order set and only 14.3% of participants made decision-making errors when using the prototype CDS order set to prescribe antibiotics for the treatment of CAP ([Table 5]). Results from the pediatric UTI user tests were similar to those from CAP ([Table 5]). The implementation of the new prototype CDS led to a decrease in decision-making errors made by participants, with 20% making decision-making errors with the current EHR order set and only 9.5% of participants making decision-making errors when using the new CDS. Usability errors were also less frequent with the new CDS (40 vs. 9.5%). Provider interaction with associated informational BPA was greater for UTI than CAP. Over half of participants either commented on or read the BPA content, though fewer than one-third attempted to access local and national guidelines via embedded weblinks. No participants interacted with or commented on the notification or banner indicating that the patient had a history of a prior UTI diagnosis.

Table 5 Tasks assessed for CDS usability, decision-making errors, and usability errors in simulated scenarios for CAP and UTI

	CAP		UTI
	Current order set (n = 11)	Prototype CDS (n = 21)	Current order set (n = 10)	Prototype CDS (n = 21)
Tasks
Decision-making outcomes
Diagnosis placed	10 (90.9%)	21 (100%)	10 (100%)	21 (100%)
Correct antibiotic Selected	6 (54.5%)	18 (85.7%)	9 (90.0%)	19 (90.5%)
Usability outcomes
Used order set	9 (81.8%)	21 (100%)	5 (50%)	21 (100%)
Signed order	10 (100%)	11 (100%)	10 (100%)	21 (100%)
Interacted with EHR elements	6 (54.5%)	12 (57.1%)	9 (90.0%)	18 (85.7%)
Commented on BPA	–	8 (38.1%)	–	12 (57.1%)
Content read in BPA	–	7 (33.1%)	–	9 (42.9%)
Clicked on BPA links	–	6 (28.6%)	–	6 (28.6%)
Completed prescribing tasks	11 (100%)	21 (100%)	10 (100%)	21 (100%)
Overall error rates
Decision-making errors	4 (63.6%)	3 (14.3%)	1 (10.0%)	2 (9.5%)
Usability errors	3 (27.3%)	3 (14.3%)	5 (50.0%)	2 (9.5%)

Abbreviations: BPA, best practice advisory; CAP, community-acquired pneumonia; CDS, clinical decision support; EHR, electronic health record; UTI, urinary tract infection.

Provider satisfaction with the prototype CDS was high for both CAP and UTI ([Table 6]). Ratings were particularly high for UTI as compared to CAP, though the CDS for both infections demonstrated preferences greater than four out of five for the majority of domains, including ease of use and preference over the existing ordered system of antibiotic prescribing.

Table 6 Provider satisfaction with the CAP and UTI prototype CDS an using five-point Likert scales

Domain	CAP(n = 21)		UTI(n = 21)
	Median (IQR)	Mean ± SD	Median (IQR)	Mean ± SD
Antibiotic prescribing well structured	4 (3–5)	4.00 ± 0.89	5 (3.5–5)	4.24 ± 0.52
System helped with patient management	4 (3–5)	4.24 ± 0.77	5 (4–5)	4.33 ± 0.94
Preferred over prior methods	4 (3–5)	4.14 ± 0.85	5 (4–5)	4.29 ± 0.90
Saved time	4 (3–5)	3.81 ± 1.12	5 (3–5)	4.10 ± 1.1
Overall satisfaction	4 (3–5)	4.24 ± 0.77	4 (3–4)	4.10 ± 0.94

Abbreviations: CAP, community-acquired pneumonia; CDS, clinical decision support; IQR, interquartile range; SD, standard deviation; UTI, urinary tract infection.

National Aeronautics and Space Administration Task Load Index

The study team analyzed NASA-TLX results by comparing overall results, results by phase, and results for CAP and UTI cases. The NASA-TLX index results demonstrated a trend toward lower average workload reported by participants for the current system (median: 26, IQR: 11–41; mean ± SD: 30.1 ± 15.1) versus the prototype CDS system (median: 25, IQR: 10.5–39.5; mean: 25.3 ± 1), though not statistically significant (p = 0.117) and with limited effect size (r = 0.342). NASA-TLX scores reflected a trend toward reduction in participant workload in phase 2 of this study (median: 25, IQR: 0–52; mean: 31.5 ± 18.9 reduced to median 21, IQR: 2–40; mean 23.8 ± 13.3; p = 0.107) after improvements were made in the CDS, compared with Phase 1 (median: 31, IQR: 18–43; mean 28.4 ± 8.6) reduced to median 25, IQR: 13–37, mean 27.3 ± 7.6; r = 0.104; p = 0.635).

For the CAP CDS, the mean NASA-TLX score reported by participants for the current order set was median: 35, IQR: 17 to 53; mean = ± 16.5 compared with median 23, IQR: 9 to 37; mean: 25.4 ± 11.8 reported for CAP cases completed using the prototype CDS which was significantly improved (r = 0.617; p = 0.041). The mean NASA-TLX score for UTI CDS when using the current order set was median 22.5, IQR: 6.5–38.5; mean ± SD: 24.6 ± 11.8 compared with a mean NASA-TLX score of median: 25.5, IQR: 8–43; 25.2 ± 10.9 reported for UTI cases completed using the prototype CDS (r = 0.071; p = 0.80).

Qualitative Comments from User Tests

Participant comments reflected strong preferences toward the prototype CDS and specific components of the prototype CDS ([Supplementary Table] [[Supplementary Material]], available in the online version). Participants' statements included “I like it! Nice and clean,” and favored “fewer clicks” required for the prototype CDS to complete an antibiotic prescription. Several participants commented that what they liked most about the prototype CDS was the new BPA, which included the suggestion about when antibiotics are specifically not recommended. When asked whether they would know how to navigate the prototype CDS within the EHR workflow without additional training, 16 of the 17 respondents agreed that they would and all 17 indicated that they would trust the CDS system to give accurate recommendations regarding antibiotic treatment choices.

Discussion

Our findings demonstrate that our prototype CDS for outpatient ED antibiotic prescribing was preferred by users and led to improved adherence to antimicrobial stewardship guidelines in simulation. The CDS, which was created through a user-centered design approach, demonstrated improvements in several aspects of usability and potential effectiveness. Through our simulated patient approach for pediatric CAP and UTI, the novel prototype CDS increased satisfaction and function for users while also reducing rates of decision-making and functional usability errors. Therefore, we were able to meet a critical goal of many ED providers, using health information technology to support a common aspect of clinical care, antibiotic prescribing, through interaction with the EHR. These results support the use of our prototype CDS for outpatient antibiotic-prescribing ED and the future implementation of EHR-embedded CDS approaches to provide antimicrobial stewardship.

Our study supports the potential of CDS to improve outpatient antibiotic-prescribing ED. However, our findings also support the importance of user-centered design for the creation of an acceptable and feasible CDS. We developed our CDS using contextual considerations for the ED setting to integrate adapted prescribing recommendations and workflow considerations, as well as end-user preferences to further refine the CDS design. Our findings build upon existing studies demonstrating that usability testing combined with user-centered methods, including iterative design, results in more usable CDS that has the potential to contribute to improved outcomes including adherence to clinical guidelines, patient safety, and reducing EHR burnout.[41] [42] [43] [44]

Among the design features in our prototype CDS is passive activation, which does not require clinicians to engage or “seek” the prescribing recommendations. When designing our CDS, passive activation was important among numerous stakeholders and ED clinicians.[14] [15] Automated triggering and CDS activation contributed to perceived improvements in ease of use and resultant frequency of guideline-adherent antibiotic prescribing. We believe that the passive approach to our CDS, as opposed to the current EHR approach which requires active seeking of recommendations and antibiotic choices, substantially improved usability. Furthermore, our CDS trended toward a perceived reduced workload, without additional steps or forced user functions. Therefore, input from end-users at the onset of design led to the development of a more effective, better accepted, and more accessible decision-making tool and suggest that such an iterative approach be employed in future CDS design.

We developed standard EHR functionality which supports the application of user-centered methods by hospitals in the configuration of the EHR.[17] [18] This includes the use of CDS tools such as discharge order sets, BPAs, weblinks to local and national guidelines, and institutional antibiograms. These EHR and CDS components were well received; perhaps even more critical to success was the placement of these outpatient prescribing tools at discharge, which is often most logical in the ED workflow and decision-making process for outpatient treatment of many infections. By integrating the CDS into a discharge “order set,” we were able to naturally place recommendations at the point of care while avoiding unnecessary presentation of recommendations when not indicated. Furthermore, we were able to prepopulate the antibiotic prescriptions such that the user only needed to sign the prescription, avoiding the possibility of calculation errors in dosing and nonguideline adherent prescription durations of treatment.

Implementation of CDS through user-centered design, and subsequent rigorous user-testing, resulted in a prototype CDS with great potential for ability to improve ED clinical care using the EHR, a ubiquitous health care tool. Our CDS addresses the end-user desire to have a lower workload at the time of prescribing (e.g., not needing to seek allergy information) and to minimize clicks within the EHR using automated prescribing. For quality improvement approaches to improve patient outcomes, such as antimicrobial stewardship, using a CDS in our method of design facilitates the management of data from multiple sources and optimizes ED care through the reduction of prescribing variability. As demonstrated in our quantitative assessments and user qualitative comments, a user-friendly CDS can substantially improve the provider experience while simultaneously increasing guideline-adherent care, with potential for application to other ED conditions.

Limitations

There are limitations to our findings. First, we conducted the user testing in a simulated patient environment using cases typical of presentations of CAP and UTI in healthy children; therefore, it is possible that the findings may not reflect the complexity of care in the clinical ED. We did attempt to create realistic clinical, patient-based scenarios, and the study appearance of the EHR was identical to that used in patient care. Though we organized our test case-scenarios randomly, we conducted user testing with our current system before the two cases testing our novel CDS. All participants in the study had been previously exposed to the current system in their everyday practice, and thus, the assumption was made that there would be less potential for learning from this environment than there would be through exposure to the new features of the CDS including general practice guidelines. This ordering of the systems allowed us to minimize learning effects and test the current system without influencing antibiotic prescription practices. Our study was a pilot study of 21 users, limiting the determination of effect size and significance of some of our measurements. Nonetheless, we were we able to use a repeated measures approach which provided important data demonstrating strong effects and significance with respect to usability and user preferences. We combined user test assessments of attending faculty and trainees, who may vary in pediatric experience and training. Lastly, we conducted our user tests in single health system, which may limit applicability to EDs that vary in EHR, clinical practice, and antibiotic availability.

Conclusion

This study employed usability testing methodology to analyze the integration of CDS for antibiotic prescription in the outpatient setting into the ED EHR. We found that an enhanced, passively activated CDS with best practice alerts was strongly preferred over usual methods of antibiotic prescribing and demonstrated the potential for a lower cognitive burden for ED clinicians. Future steps will include the conduct of CDS implementation trials across multiple settings to test effectiveness.

Clinical Relevance Statement

User-centered design and iterative user testing are important for the development of an acceptable and effective EHR-based CDS in the ED. The design of passive CDS using key stakeholder and end-user engagement has great potential to augment guideline-adherent antibiotic prescribing for common outpatient ED infections.

Multiple-Choice Questions

1. Which of the following are useful methods of obtaining data to inform the design of clinical decision support systems that are optimized for end-users?

   • NASA Task Load Index measurement

   • Electronic health record vendor input

   • Creation of best practice alert

   • Clinician focus groups

     Correct Answer: The correct answer is option d. Input from end-users is essential to design effective interventions using the electronic health record and clinical decision support. To maximize the appearance, activation, and usable information from the intervention, engagement of end-users through focus groups and/or semistructured interviews can glean critical data necessary for successful design. Data from workflow observations and analysis can be equally important to design. With respect to clinical decision support, user-centered design can assist in the effective creation and adherence to the “Five Rights of Clinical Decision Support,” delivering the right information, to the right person, in the right intervention format, through the right channel, at the right time in workflow.

2. Important benefits of user-centered design for clinical decision support may include which of the following?

   • Decreased workload

   • Increased patient length of stay

   • Integration in multiple EHR vendors

   • Increased insurer reimbursement

     Correct Answer: The correct answer is option a. Ultimately, user-centered design for clinical decision support is used to help deliver improved patient care, including quality measures such as timely care, shorter length of stay, and adherence to the standard of care. However, by using input from end-users and key-stakeholder who may be affected by the clinical decision support, this intervention can also be used to meet the needs of the care providers. A more seamless clinical decision support intervention embedded in the electronic health record can help deliver faster care and mitigate frustrations and cognitive delays and reduce the cognitive burden introduced in navigating the electronic health record system.

Conflict of Interest

None declared.

Acknowledgments

The authors would like to thank the PECARN Steering Committee and PECARN subcommittees (Grants and Publications, Protocol Review and Development, Quality and Safety, and Feasibility and Budget) for their thoughtful review and assistance in the development of this project.

Protection of Human and Animal Subjects

The current study was deemed exempt by our institutional review board.

• References

• 1 CDC. Antibiotic Resistance Threats in the United States. GA: U.S. Department of Health and Human Services, CDC; 2019
• 2 Fleming-Dutra KE, Hersh AL, Shapiro DJ. et al. Prevalence of inappropriate antibiotic prescriptions among us ambulatory care visits, 2010-2011. JAMA 2016; 315 (17) 1864-1873
• 3 Poole NM, Shapiro DJ, Fleming-Dutra KE, Hicks LA, Hersh AL, Kronman MP. Antibiotic prescribing for children in United States Emergency Departments: 2009-2014. Pediatrics 2019; 143 (02) e20181056
• 4 Sanchez GV, Fleming-Dutra KE, Roberts RM, Hicks LA. Core elements of outpatient antibiotic stewardship. MMWR Recomm Rep 2016; 65 (06) 1-12
• 5 Society for Healthcare Epidemiology of America, Infectious Diseases Society of America, Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). Infect Control Hosp Epidemiol 2012; 33 (04) 322-327
• 6 Spellberg B, Blaser M, Guidos RJ. et al; Infectious Diseases Society of America (IDSA). Combating antimicrobial resistance: policy recommendations to save lives. Clin Infect Dis 2011; 52 (Suppl. 05) S397-S428
• 7 Avdic E, Cushinotto LA, Hughes AH. et al. Impact of an antimicrobial stewardship intervention on shortening the duration of therapy for community-acquired pneumonia. Clin Infect Dis 2012; 54 (11) 1581-1587
• 8 Carling P, Fung T, Killion A, Terrin N, Barza M. Favorable impact of a multidisciplinary antibiotic management program conducted during 7 years. Infect Control Hosp Epidemiol 2003; 24 (09) 699-706
• 9 Ohl CA, Luther VP. Antimicrobial stewardship for inpatient facilities. J Hosp Med 2011; 6 (Suppl. 01) S4-S15
• 10 Fishman N. Antimicrobial stewardship. Am J Infect Control 2006;34(5, suppl 1):S55–S63, discussion S64–S73
• 11 May L, Cosgrove S, L'Archeveque M. et al. A call to action for antimicrobial stewardship in the emergency department: approaches and strategies. Ann Emerg Med 2013; 62 (01) 69-77.e2
• 12 Mistry RD, Dayan PS, Kuppermann N. The battle against antimicrobial resistance: time for the emergency department to join the fight. JAMA Pediatr 2015; 169 (05) 421-422
• 13 Mistry RD, Newland JG, Gerber JS. et al. Current state of antimicrobial stewardship in children's hospital emergency departments. Infect Control Hosp Epidemiol 2017; 38 (04) 469-475
• 14 Chung P, Scandlyn J, Dayan PS, Mistry RD. Working at the intersection of context, culture, and technology: provider perspectives on antimicrobial stewardship in the emergency department using electronic health record clinical decision support. Am J Infect Control 2017; 45 (11) 1198-1202
• 15 Ozkaynak M, Metcalf N, Cohen DM, May LS, Dayan PS, Mistry RD. Considerations for designing EHR-embedded clinical decision support systems for antimicrobial stewardship in pediatric emergency departments. Appl Clin Inform 2020; 11 (04) 589-597
• 16 Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Annu Fall Symp 1997; 218-222
• 17 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
• 18 Hettinger AZ, Melnick ER, Ratwani RM. Advancing electronic health record vendor usability maturity: progress and next steps. J Am Med Inform Assoc 2021; 28 (05) 1029-1031
• 19 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care—an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-549
• 20 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 21 Nguyen OT, Jenkins NJ, Khanna N. et al. A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being. J Am Med Inform Assoc 2021; 28 (05) 974-984
• 22 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 23 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 24 Kouri A, Yamada J, Lam Shin Cheung J, Van de Velde S, Gupta S. Do providers use computerized clinical decision support systems? A systematic review and meta-regression of clinical decision support uptake. Implement Sci 2022; 17 (01) 21
• 25 Jones BE, Collingridge DS, Vines CG. et al. CDS in a learning health care system: identifying physicians' reasons for rejection of best-practice recommendations in pneumonia through computerized clinical decision support. Appl Clin Inform 2019; 10 (01) 1-9
• 26 Bradley JS, Byington CL, Shah SS. et al; Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. The management of community-acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis 2011; 53 (07) e25-e76
• 27 Roberts KB. Subcommittee on Urinary Tract Infection, Steering Committee on Quality Improvement and Management. Urinary tract infection: clinical practice guideline for the diagnosis and management of the initial UTI in febrile infants and children 2 to 24 months. Pediatrics 2011; 128 (03) 595-610
• 28 Osheroff JA. Improving Medication Use and Outcomes with Clinical Decision Support: A Step by Step Guide. 1st ed. Chicago, IL: HIMSS Publishing; 2009
• 29 Rosson MB, Carroll JM. Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Burlington, MA: Morgan Kaufmann Publishers Inc.; 2001
• 30 Russ AL, Saleem JJ. Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inform 2018; 78: 123-133
• 31 Cochran WG. Cox, Gertrude M, Experimental Designs. Hoboken, NJ: John Wiley & Sons; 1957
• 32 Robert Schumacher, Svetlana Lowry. (NISTIR 7741) NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records. Published online November 29, 2010.
• 33 Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In: Hancock PA, Meshkati N. eds. Advances in Psychology. Vol 52. North-Holland; 1988. :139–183
• 34 Hart SG. NASA-task load index (NASA-TLX); 20 years later. Proc Hum Factors Ergon Soc Annu Meet 2006; 50 (09) 904-908
• 35 Said S, Gozdzik M, Roche TR. et al. Validation of the Raw National Aeronautics and Space Administration Task Load Index (NASA-TLX) questionnaire to assess perceived workload in patient monitoring tasks: pooled analysis study using mixed models. J Med Internet Res 2020; 22 (09) e19472
• 36 Karavite DJ, Miller MW, Ramos MJ. et al. User testing an information foraging tool for ambulatory surgical site infection surveillance. Appl Clin Inform 2018; 9 (04) 791-802
• 37 Pollack AH, Pratt W. Association of health record visualizations with physicians' cognitive load when prioritizing hospitalized patients. JAMA Netw Open 2020; 3 (01) e1919301
• 38 NASA. NASA TLX: Task Load Index. Published online 2006. Accessed December 2, 2022 at: http://humansystems.arc.nasa.gov/groups/TLX/
• 39 Lang TA, Secic M. How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers. ACP Press; 2006
• 40 Cohen J. Set correlation and contingency tables. Appl Psychol Meas 1988; 12 (04) 425-434
• 41 Li AC, Kannry JL, Kushniruk A. et al. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform 2012; 81 (11) 761-772
• 42 Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Human Factors 2015; 2 (02) e14
• 43 Richardson S, Mishuris R, O'Connell A. et al. “Think aloud” and “Near live” usability testing of two complex clinical decision support tools. Int J Med Inform 2017; 106: 1-8
• 44 Ward MJ, Chavis B, Banerjee R, Katz S, Anders S. User-centered design in pediatric acute care settings antimicrobial stewardship. Appl Clin Inform 2021; 12 (01) 34-40

Publication History

Received: 07 May 2022

Accepted: 16 November 2022

Article published online:
08 February 2023