Appl Clin Inform 2018; 09(02): 248-260
DOI: 10.1055/s-0038-1641594
Research Article
Schattauer GmbH Stuttgart

Examining Workflow in a Pediatric Emergency Department to Develop a Clinical Decision Support for an Antimicrobial Stewardship Program

Mustafa Ozkaynak
1   College of Nursing, University of Colorado, Anschutz Medical Campus, Aurora, Colorado, United States
,
Danny T. Y. Wu
2   Department of Biomedical Informatics and Pediatrics, University of Cincinnati, Cincinnati, Ohio, United States
,
Katia Hannah
3   Skaggs School of Pharmacy and Pharmaceutical Sciences, University of Colorado, Anschutz Medical Campus, Aurora, Colorado, United States
,
Peter S. Dayan
4   Division of Pediatric Emergency Medicine, Department of Pediatrics, Columbia University College of Physicians and Surgeons, New York, New York, United States
,
Rakesh D. Mistry
5   Section of Emergency Medicine, Department of Pediatrics and Emergency Medicine, School of Medicine, University of Colorado, Anschutz Medical Campus, Aurora, Colorado, United States
› Institutsangaben
Funding This study was funded by grants from Emergency Medicine Foundation, University of Colorado School of Medicine and Children's Hospital Colorado.
Weitere Informationen

Address for correspondence

Mustafa Ozkaynak, PhD
College of Nursing, University of Colorado, Anschutz Medical Campus
Campus Box 288-18, Education 2 North Building
13120 East 19th Avenue, Room 4121, Aurora, CO 80045
United States   

Publikationsverlauf

02. Oktober 2017

21. Februar 2018

Publikationsdatum:
11. April 2018 (online)

 

Abstract

Background Clinical decision support (CDS) embedded into the electronic health record (EHR), is a potentially powerful tool for institution of antimicrobial stewardship programs (ASPs) in emergency departments (EDs). However, design and implementation of CDS systems should be informed by the existing workflow to ensure its congruence with ED practice, which is characterized by erratic workflow, intermittent computer interactions, and variable timing of antibiotic prescription.

Objective This article aims to characterize ED workflow for four provider types, to guide future design and implementation of an ED-based ASP using the EHR.

Methods Workflow was systematically examined in a single, tertiary-care academic children's hospital ED. Clinicians with four roles (attending, nurse practitioner, physician assistant, resident) were observed over a 3-month period using a tablet computer-based data collection tool. Structural observations were recorded by investigators, and classified using a predetermined set of activities. Clinicians were queried regarding timing of diagnosis and disposition decision points.

Results A total of 23 providers were observed for 90 hours. Sixty-four different activities were captured for a total of 6,060 times. Among these activities, nine were conducted at different frequency or time allocation across four roles. Moreover, we identified differences in sequential patterns across roles. Decision points, whereby clinicians then proceeded with treatment, were identified 127 times. The most common decision points identified were: (1) after/during examining or talking to patient or relative; (2) after talking to a specialist; and (3) after diagnostic test/image was resulted and discussed with patient/family.

Conclusion The design and implementation of CDS for ASP should support clinicians in various provider roles, despite having different workflow patterns. The clinicians make their decisions about treatment at different points of overall care delivery practice; likewise, the CDS should also support decisions at different points of care.


#

Background and Significance

Clinical Decision Support in Emergency Departments and Clinical Workflow

Electronic health record (EHR)-based clinical decision support (CDS) systems have the potential to change clinical behavior in the emergency department (ED) setting. Decision support interventions tailored to context and end-user needs have been shown to be more likely to result in desired behavior change.[1] Although CDS has promise to impact prescribing practices, the ED setting presents unique challenges to CDS implementation and effectiveness.[2] [3] Clinicians in the ED are frequently forced to make rapid decisions in a chaotic setting. Decision making is often complicated by patient turnover, multiple interruptions, the need for empiric therapy, and limitations such as a lack of perceived follow-up and medicolegal fears.[4] [5] [6] [7] To be successful, current literature and CDS experts strongly recommend that developers assess and account for local ED processes and clinician perceptions during the creation and refinement of CDS, but prior to implementation.[8] [9] [10] [11]

Antimicrobial stewardship programs (ASPs) aim to optimize appropriate antibiotic prescribing in the clinical setting.[12] CDS can deliver antibiotic prescribing recommendations at the point-of-care, and overcome barriers unique to the ED setting.[13] [14] [15] Therefore, CDS is a potentially effective method to implement ED-based ASPs. However, an understanding of ED workflow prior to development of CDS for antibiotic prescribing is necessary to develop an effective and sustainable ASP intervention. Proper design and implementation of CDS can be integrated into the usual workflow without disrupting ongoing practice.[16] [17] The purpose of this study was to characterize workflow for four clinician roles in a pediatric ED, to inform the future development of EHR-based CDS, as the centerpiece for an ED-based ASP.


#

Potential Role for CDS in Antimicrobial Stewardship Programs

Antimicrobial resistance is among the greatest threats to health care quality and pediatric public health.[18] In outpatient settings such as the ED, it is estimated that 25 to 63% of patients are prescribed inappropriate or unnecessary antibiotics, contributing to the development of antimicrobial resistance.[19] [20] [21] [22] ASPs can improve guideline adherence for antibiotic prescribing and minimize the impact of antibiotic-resistant organisms.[23] However, ASPs are not sufficiently utilized in ED settings, owing to the erratic workflow and variety of providers unique to this setting.[24] Innovative methods are required to implement ASPs into the ED setting.

Use of health information technology is one possible method to potentially optimize ASPs in the ED.[13] [14] Incorporation of health information technology and use of EHR is viewed as an essential method for future CDS interventions to mitigate antibiotic prescribing.[14] [15] [25] Implementation of CDS using the EHR as a vehicle to deliver recommendations has positively impacted provider antibiotic prescribing in various clinical settings.[15] [25] [26] [27] [28] [29] [30] Trials of CDS have demonstrated success in improving antibiotic prescribing in inpatient and office settings, though ED-based studies are limited.[25] [27] [28] [31] [32] [33] [34] [35] [36] However, the success of these CDS interventions largely depends on proper integration into the clinical workflow.[26] [37] [38]


#
#

Methods

Setting

Workflow was systematically examined in a single, tertiary-care academic children's hospital ED that serves as regional referral center, receiving ∼73,000 visits annually. The ED consists of 40 acute care beds and is staffed by a variety of providers, including attending physicians (pediatric emergency medicine [PEM] specialists and general pediatricians), nurse practitioners (NPs), physician assistants (PAs), and resident physician trainees. At any given time, the ED is usually staffed by 2 to 3 PEM specialists, 5 advanced practice providers (NPs or PAs), and between 2 and 4 residents. The ED is grouped into three teams of providers: two teams are a combination of PEM specialists, residents, and advanced practice providers. A third team is composed of two advanced practice providers who evaluate patients of low acuity. In the system, all advanced practice providers act independently, without direct supervision. Residents present to PEM specialists only, and are fully supervised.


#

Sample

ED clinicians of four provider types were observed in the ED over a 5-month period (March–July 2016). Data were collected from a total of 23 participants (11 attendings, 3 residents, 5 NPs, and 4 PAs). [Table 1] shows the participating providers and observation characteristics. Participants volunteered from a pool of 40 attending physicians, 30 residents (on ED rotation), and 70 advanced practice providers. We focused recruitment on the day shift, to ensure homogeneity of the data.

Table 1

Study sample

Type of providers

Clinicians

(n)

Sessions

(n)

Time spent in the ED

(h)

Time captured with observation tool

(h)

Decision point questions

(n)

Attending

11

13

41

38

65

Resident

3

4

11

10

12

Nurse practitioner

5

8

24

22

32

Physician assistant

4

6

14

13

18

Total

23

31

90

83

127

Abbreviation: ED, emergency department.



#

Procedure

Data were collected by direct observations supported by probing questions. Workflow activities and information regarding decision points were recorded on a tablet computer, using a user-friendly software tool for collection of time and motion data, where the tablet automatically records the time each recording of an activity was entered ([Fig. 1]). The software tool[39] was initially developed for and validated in an ambulatory care setting. We modified the tool by adding the ED setting and observer names while keeping the task list and other settings. Coding of workflow activities is outlined in [Appendix A]. A total of 90 hours of observations were conducted.

Zoom Image
Fig. 1 Data collection tool.
Appendix A

CATG_ID

CATG_DESC

A1

Computer–Communicating

B1

Dictating

C1

Computer–Entering

C2

Computer–Login

C3

Computer–Logout

C4

Computer–Processing

C5

Computer–Printing

C6

Computer–Reading

D1

Paper–Copying/Faxing

D2

Paper–Reading reference

D3

Paper–Reading/Writing

D4

Paper–Retrieving/Accepting

D5

Paper–Sorting/Filing/Delivering

D6

Paper–Writing

E1

Phone–Answering

E2

Phone–Calling

E3

Phone–Transferring

F1

Talking–Coworker

F2

Talking–Patient

F3

Talking–With others

G1

Walking–Alone

G2

Walking–With coworker

G3

Walking–With patient

H1

Meeting

I1

Performing–Exam room preparation/Cleaning

I2

Performing–Hand sanitization

I3

Performing–Handling

I4

Performing–Measuring

I5

Performing–Medical procedure

I6

Performing–Physical exam

I7

Performing–Other

J1

Personal–Cell phone

J2

Personal–Computer

J3

Personal–Eating/drinking + Socializing/chatting

J4

Personal–Restroom

J5

Personal–Other

K1

Cell phone/iPad–Reading/typing

To better understand the critical decision-making moment, clinicians were queried in real time about the timing of diagnosis, and disposition decision points, for each patient that had a prescription or discharge instruction entered in the EHR. Decision points were based on the following question: “For [this specific patient] at what point during her/his visit did you decide how to proceed with the treatment?” A total of 127 decision points were identified for providers ([Tables 2] and [3]). We primarily targeted diagnosis point rather than disposition with this question. However, we acknowledge that diagnosis and disposition can be very close temporally for some health care encounters. We acknowledge that consideration of both decision points is very important. Regardless, from the ASP CDS design perspective the ambiguity does not matter because in both cases (diagnosis or disposition) the function of the CDS would be the same.

Table 2

Summary of decision points

Decisions (total frequency of responses for each decision)

Provider type

Frequency

1. After/during examining or talking to patient or relative (55)

Attending

27

NP

17

PA

6

Resident

5

2. After talking to a specialist (13)

Attending

8

NP

3

PA

1

Resident

1

3. After test/image is resulted and discussion with patient/family (13)

Attending

8

NP

3

PA

2

4. During/after discussion with a resident (5)

Attending

5

5. After administration of medication and reexamining (4)

Attending

1

NP

2

PA

1

6. During/after discussing with an attending (2)

Resident

2

7. After reviewing patient history and conducting exam (2)

NP

2

8. Reading through the patient chart (2)

Attending

2

9. After seeing the test results and conducting a second physical exam (2)

PA

1

Attending

1

10. After making calculations on a Web site (1)

Attending

1

11. Talking to a respiratory therapist (1)

Attending

1

12. After going through head injury protocol; and observation protocol (1)

NP

1

13. After asthma assessments (1)

Resident

1

14. After gathering her history, conducting a physical exam, and talking to an attending (1)

PA

1

15. After obtaining a second opinion (1)

NP

1

16. After obtaining medical history and final exam (1)

NP

1

17. After referring to TBI algorithm (1)

PA

1

18. After medical procedure (1)

Attending

1

19. When hearing other attending taking the transfer phone call (1)

Attending

1

20. Prior to the patient's arrival at the ED, as well as after speaking with the patient's doctor who referred the patient (1)

Attending

1

21. After reading triage notes (1)

Resident

1

Total

110

Abbreviations: ED, emergency department; NP, nurse practitioner; PA, physician assistant; TBI, traumatic brain injury.


Table 3

Summary of cases with no specific decision

Reasons for no decision

Provider type

Frequency

1. Complex patient so no decision during the observation period

Attending

5

NP

2

PA

1

Resident

2

2. Social worker consultation

Attending

1

3. Psychiatry patient

Attending

2

PA

3

4. Observer could not ask

Attending

1

Total

17

Abbreviations: NP, nurse practitioner; PA, physician assistant.


The observer recorded reflection statements after each of the 31 observation sessions. These statements were used to clarify both qualitative and quantitative data. Reflection statements are important to explain why there was a decision or if there is any additional important information (e.g., location) that might be important for analysis.


#

Decision Analysis

We used a qualitative approach to analyze decision points and a quantitative approach to analyze workflow data. Qualitative analysis of the 127 decision point responses was accomplished by summarizing the decision points within the overall care delivery process and highlighting the patterns by roles of the four types of providers. Responses by the participants were categorized using constant comparison. Key events associated with decision making were identified.


#

Sequential Pattern Analysis

The quantitative analysis of workflow data consisted of two parts. In the first part, the analysis was accomplished by sequential pattern mining,[40] [41] a subfield in data mining that focuses on identifying frequent patterns in sequence data. Prior to applying the sequential pattern mining methods, we manipulated the workflow activity data into an appropriate format for the algorithm input. Specifically, tasks observed in the care delivery process were categorized ([Appendix A]) and transformed into a string sequence for each observation. For example, if a clinician walks alone (G1), talks to a patient (F2), and then performs a physical exam during the observation (I6), a sequence of “G1-F2-I6” for this observation is generated. To make the patterns more interpretable for clinical experts, we constrained the sequential pattern mining method to identify only adjacent events with a focus on start and end points. We ignored frequent task patterns that did not occur in a consecutive manner. To accommodate this constraint, we added a pseudo start and end point, “S1” and “S9” correspondingly, to each observation sequence.

The manipulated workflow activity data were then fed into a computerized algorithm developed by the second author (D.T.W.) to identify frequent sequence patterns. This identification was guided by the a priori algorithm, which has two parameters. The algorithm constrained a minimum support of 20%, i.e., a pattern is considered frequent when one-fifth of the observations contained this pattern, and the minimum confidence of 80%, i.e., a task is added to the frequent pattern list if this task is observed 4 out of 5 times using conditional probability. Once the frequent task sequences were identified, we visualized these patterns in a directed network graph ([Fig. 2]) for each clinical role, with the nodes being the frequent categories and the edges being the connection in the patterns. Based on these parameters, the network graphs were laid out in a circle to facilitate interpretation. It is worth noting that the research team iteratively reviewed and interpreted the frequent patterns in the network graphs to adjust the parameters to include a reasonable amount of meaningful patterns, resulting in the threshold of minimum 20% support and the 80% confidence.

Zoom Image
Fig. 2 Workflow patterns for the four providers.

The second part of the quantitative analysis focused on determining whether the frequency of, and the time allocated in the tasks, were significantly different among the providers based on their clinical role. In particular, the frequency (count) and the time allocation (percentage) were aggregated based on the task categories per observation using the same mapping definition in [Appendix A]. Multitasking events were split into several single events and their elapsed time were evenly distributed to the single events. For example, if event A lasts 10 seconds, event B lasts 15 seconds, A and B are overlapped by 5 seconds, then the multitasking part is split equally between A and B so that A is 7.5 seconds and B is 12.5 seconds in a single tasking manner. The mean of the frequency of, and the time allocated to the categories by clinical roles, was compared using the Kruskal–Wallis test, assuming two independent samples with a nonnormal distribution. The significance level was set at 0.05.

The data were manipulated in Python 2.7 and stored in a SQLite database (https://www.sqlite.org/), a relational file-based database format. The sequential pattern mining procedure and the statistical test were also conducted using Python 2.7. Specifically, the string matching procedure was implemented using the generic Python regular expression library; the Kruskal–Wallis test was implemented using the Python Numpy and Scipy libraries. The network graphs were produced using the igraph library in R-package in the R-Studio environment, an open-source interactive code editor for R.

During the study, we collected qualitative and quantitative data simultaneously. Our analysis of the qualitative data showed saturation before data collection was complete. We argue that our sample is sufficiently representative because: (1) we did not pick the participants purposefully, it was based on volunteering; (2) observers did not note a significant event during the observations that may affect the usual workflow; and (3) the sample size is consistent with similar studies.


#
#

Results

The 90-hour observations resulted in 6,060 records with 64 distinct activities. Among these activities, Five were conducted at different frequency or time allocation across the four clinical roles, including computer entering (C1) and reading (C6), talking with coworkers (F1) and walking alone (G1), and personal activities (J3) as listed in the second part of [Fig. 2]. We also identified the frequency of decision points and compared the sequential patterns of the activities across the four roles.

Decision Points

During the observation period, of 127 decision point questions, in 17 questions (9 attending, 2 NPs, 4 PAs, and 2 residents), no specific decisions were made toward the treatment (see [Table 3]). A common reason for the lack of a decision point was among those are complex patients; still need further tests and consultations, and those coming to ED for mental health reasons; these patients were being screened in the ED for medical clearance. The 110 remaining decision points within the care delivery process are presented in [Table 2].

The most frequent decision point for each of the four providers was “after/during examining or talking to patient or relative” in 55 (50%) responses. Other frequent decision points were: “after talking to a specialist” and “after test/image result is resulted and discussion with patient/family” with 13 (12%) total responses for each. The second and third most frequent decision points for attending physicians and NPs were “after talking to a specialist” and “after test/image result is resulted and discussion with patient/family.” “After test/image result is up (and talking to patient/family)” was the second frequent decision point for PAs, while “during/after discussing with an attending” was the second most frequent decision point for residents. Out of 110 visits, antibiotics were prescribed in 20 cases. In 12 of those cases, the decision was made “after/during examining or talking to patient or relative.”

The reflection notes highlight how various activities were conducted in various places within the ED. For example, conversations among ED staff members took place at physician stations, hallways, or in front of electronic board. Similarly (face-to-face or phone), conversations between ED clinicians and specialists took place in various places within the ED. Although the majority of physical exams took place in patient rooms, at least in one case it took place in a triage room. Test results were accessed by clinicians through desktop computers a majority of the time; however, attendings could be aware of test results by residents in hallways.


#

Sequential Patterns

[Fig. 2] illustrates the most frequent sequential pattern in a network graph for each clinical role. Attending physicians demonstrated the most clear-cut pattern (fewer frequent patterns in total) compared with other providers. The attendings frequently started with talking to coworkers (F1), walking alone (G1), and personal activities such as socializing (J3). They were frequently involved in computer entering (C1) and reading (C6) toward the end of the observations. Other providers, on the other hand, involved more tasks that the attendings did not frequently do, such as answering phone calls (E1), hand sanitization (I2), and performing physical exams (I6). This pattern reasonably reflects the roles these clinicians play on a care team. It is worth noting that residents and PAs were more likely to talk to patients (F2) in the beginning of the observations. They also tended to talk to their coworkers at the end of the observations (F1–>S9), which likely involved reporting to attending physicians and seeking guidance and approval. Residents exhibited the most complicated and frequent task patterns compared with other providers, meaning that residents had higher numbers of frequent patterns than others, and the sequence of these patterns had more steps and more switches between task categories.


#

Frequency and Time Allocation

Significant differences (p < 0.05) were seen in the frequency and task categories among providers ([Table 4]). Attending physicians demonstrated a very different pattern compared with the other clinical roles. For example, they more frequently talked to coworkers (F1) and read electronic documents (C6) than other providers. Attending physicians talked to coworkers about 20 minutes longer than other providers. NPs, on the other hand, tended to allocate more time than attendings to entering tasks on the computer (22.29 vs. 8.27 seconds), but performed measures (1.0 vs. 2.0%) less frequently than attendings. Residents walked alone longer then NPs (4.68% vs. 1.86%).

Table 4

Activities that were conducted in different amounts by four roles

Measure

Category

Roles compared (median)[a]

Number of observations

Median difference

p-Value

Time allocation (s)

F1: Talking–Coworker

Attending (2926.87)

NP (1140.65)

13, 8

–1786.22

0.000

Attending (2926.87)

PA (893.22)

13, 6

–2033.65

0.002

Attending (2926.87)

Resident (1520.75)

13, 4

–1406.12

0.000

Time allocation (%)

C1: Computer–Entering

Attending (8.27)

NP (22.29)

13, 8

14.02

0.001

F1: Talking–Coworker

Attending (30.58)

NP (12.54)

13, 8

–18.04

0.000

Attending (30.58)

Resident (19.02)

13, 4

–11.56

0.001

G1: Walking–Alone

NP (1.86)

Resident (4.68)

8, 4

2.82

0.006

Frequency

C6: Computer–Reading

Attending (57.38)

NP (22.13)

13, 8

–35.25

0.000

Attending (57.38)

PA (26.67)

13, 6

–30.71

0.000

Attending (57.38)

Resident (30.25)

13, 4

–27.13

0.002

E1: Phone–Answering

Attending (9.92)

NP (1.8)

13, 5

–8.12

0.001

Attending (9.92)

PA (2.0)

13, 3

–7.92

0.001

Attending (9.92)

Resident (3.0)

13, 4

–6.92

0.006

E2: Phone–Calling

Attending (6.45)

PA (1.8)

11, 5

–4.65

0.009

F1: Talking–Coworker

Attending (50.92)

NP (26.0)

13, 8

–24.92

0.000

Attending (50.92)

PA (15.17)

13, 6

–35.75

0.001

Attending (50.92)

Resident (29.75)

13, 4

–21.17

0.003

I4: Performing–Measuring

Attending (2.0)

NP (1.0)

3, 2

–1

0.000

I5: Performing–Medical procedure

Attending (1.14)

NP (2.0)

7, 2

0.86

0.001

Time allocation (s)

C1: Computer–Entering

Attending (707.50)

NP (2186.75)

13, 8

1479.25

0.001

Attending (707.50)

PA (1914.00)

13, 6

1206.50

0.011

C6: Computer–Reading

Attending (1546.50)

NP (923.58)

13, 8

–622.92

0.010

E1: Phone–Answering

Attending (419.17)

NP (80.00)

13, 5

–339.17

0.006

F1: Talking–Coworker

Attending (2880.50)

NP (992.33)

13, 8

–1888.17

0.000

Attending (2880.50)

PA (677.42)

13, 6

–2203.08

0.002

Attending (2880.50)

Resident (1520.75)

13, 4

–1446.50

0.005

G1: Walking–Alone

NP (182.5)

Resident (303.0)

8, 4

120.50

0.009

Time allocation (%)

C1: Computer–Entering

Attending (7.72)

NP (22.89)

13, 8

15.17

0.001

Attending (7.72)

PA (23.64)

13, 6

15.92

0.005

Attending (7.72)

Resident (16.92)

13, 4

9.20

0.005

E1: Phone–Answering

Attending (4.43)

NP (0.86)

13, 5

–3.57

0.009

F1: Talking–Coworker

Attending (31.52)

NP (10.95)

13, 8

–20.57

0.000

Attending (31.52)

PA (7.43)

13, 6

–24.09

0.011

G1: Walking–Alone

Attending (2.80)

Resident (4.62)

13, 4

1.82

0.012

NP (1.86)

Resident (4.62)

8, 4

2.64

0.003

I6: Performing–Physical exam

Attending (3.84)

Resident (6.39)

13, 4

2.55

0.012

Frequency

C6: Computer–Reading

Attending (58)

NP (22)

13, 8

–36.00

0.000

Attending (58)

PA (24)

13, 6

–33.50

0.001

Attending (58)

Resident (29)

13, 4

–29.00

0.006

E1: Phone–Answering

Attending (9)

NP (1)

13, 5

–8.00

0.003

Attending (9)

PA (2)

13, 3

–7.00

0.011

E2: Phone–Calling

Attending (5)

PA (1)

11, 5

–4.00

0.008

F1: Talking–Coworker

Attending (48)

NP (22)

13, 8

–26.00

0.000

Attending (48)

PA (10)

13, 6

–38.00

0.001

Attending (48)

Resident (31)

13, 4

–17.00

0.003

Abbreviations: NP, nurse practitioner; PA, physician assistant.


a Nonnormal distribution, using Kruskal–Wallis test to compare the pre- and postmedian.



#
#

Discussion

We examined general workflow to provide context around a specific task, i.e., antibiotic prescription to inform the development of ASPs, in EDs. Although the parent project of this study was ASP implementation and we conducted the study within the ASP context, the main focus of this study is to highlight workflow variability across clinical roles. Because of higher relevancy, we identified decision points within the workflow. The most frequent decision point for each of the four providers was after/during examining or talking to patient or relative in 55 (50%) responses. Where an antibiotic is prescribed, in 60% of the cases, the decision is made at this same point. However, as we demonstrated in this study, decisions are made at different times and at different locations. The sequential pattern analysis also demonstrated that clinicians with different roles may have different and more frequent patterns in their activities. Our analysis showed that residents have the most complicated frequent patterns involving more diverse (e.g., talking and computer-related) and interconnected activities. On the other hand, while attendings have more frequent and longer conversations with coworkers overall, they tend not to end their processes with talking to coworkers but computer reading and entering. This is consistent with their role in demonstrating participation in decision making and “signing off” on the appropriateness of the plan of care. Therefore, the design of CDS should consider clinical roles, the accessibility at different points of care, and physical locations. CDS should provide best decision using up-to-date evidence-based clinical guidelines and most available data about the patient. These findings highlight the following three main principles regarding the design and implementation of the CDS for antibiotic prescribing that would support development of an ED-based ASP.

  1. CDS should support a variety of workflows: CDS should activate at the time when clinical decisions are made.[42] Our results indicate that providers with different backgrounds often take different steps and make decisions at different points within care delivery. This difference reflects their role and expertise in health care. CDS has the potential to understand a user's practice and provide decision support individualized to clinicians' roles. For ASP, the CDS could not only identify the choice of antibiotic and potential variability in prescribing practices, but consistency of practice with guidelines. CDS can benefit best when providing some standardization without harming flexibility that is due to the nature of the work.

  2. CDS should support clinicians within different physical locations in the ED: Clinicians are often at different locations when they make decisions regarding treatment plans. For example, talking to a patient or their families mostly occurs at the patient's bedside while talking to a specialist often occurs in the hallways or on the phone in front of the computer screen. CDS should ideally be activated the moment when the decision regarding antibiotic prescription is made at these diverse locations.

  3. CDS should support decisions at different points of care: Qualitative analysis ([Table 2]) shows that depending on the patient's symptoms and diagnosis, prescription decisions are made at different temporal points within the care. Quantitative analysis showed that overall workflow varies across roles. Decision support systems should be contextually sensitive in terms of understanding the clinician's overall workflow for a specific patient, including identifying the timing of care to ensure that decision support is provided at the right time. If CDS is not activated in time, the recommendation is more likely to be ignored.

If the CDS is embedded in the EHR, convenient access to EHR at all locations and situations should be provided to clinicians. Having computer access (e.g., strategically placed screens on walls informed by workflow) and providing easy logging in/out features can be solutions to support these principles. Another solution would be providing mobile access to clinicians. CDS that follows these principles will be more context-sensitive and will potentially provide better cognitive support to clinicians who then can make better prescription decisions. Although our focus was CDS for ASP, these principles apply to CDS for other purposes as well. CDS vendors could improve CDS performance by developing diverse interaction points that allow clinicians to receive needed decision supports at different physical locations as well as stage in process. An important bottleneck of CDSs is availability of needed data (e.g., laboratory and imaging results) in real time. For sooner access to needed data, vendors can also improve performance by focusing on better interoperability across systems that provide the needed data.

While writing an outpatient antibiotic prescription may be similar to other medication orders, the decision to prescribe is based on more complex data than other medications. With antibiotic prescriptions, data from patient exam, results of diagnostic testing, prior antibiotic use, etc., combined with local factors such as bacterial resistance for the area, all must be considered. This decision making is usually less straightforward than selecting a bronchodilator or analgesic. However, from a workflow perspective, antibiotic prescription can be considered a prescription task. This specific study is the first step in designing an ASP initiative. We intended to understand workflow so that the new ASP initiative is congruent with the current practice.

Two recent studies[42] [43] highlighted the necessity of examining workflow for the success of ASP implementation through EHR in EDs. As opposed to previous studies[44] [45] that focused on time distribution of tasks across various roles, this study also examines the differences in the sequence of activities by various roles in ED. Sequence is an important building block of workflow in EDs.[3] [46] Development of next-generation CDS for ASP which follows the three principles listed above may involve mobile applications and systems, which may significantly change existing workflow. Therefore, the new workflow should be consistent with the variability of care delivery work and should not hurt needed flexibility of clinicians' workflow to accomplish high-quality patient care and avoid unintended consequences.

In terms of methodological contribution, we recorded the clinical workflow activities using a tablet-based application and analyzed the data using a set of computational methods developed by the authors.[39] The successful completion of the data collection and analysis of this study demonstrates the potential generalizability of the application and the analysis methods, which were designed based on two previous publications about standardizing time motion data collection[47] and advancing time motion data analysis to uncover hidden patterns.[48] Another methodological contribution of this study is the combination of statistical and sequential pattern analysis on the workflow activity data. The former provides an overview of the frequency and time allocation of tasks in each clinical roles; the latter dives into the processes to provide a more granular view of activity sequences. This combined method enables both macro and micro view of workflow data, facilitating a more detailed understanding of the nuances of behavior patterns in clinical workflow.

There are limitations to our findings. This study was conducted in a single setting and the majority of observations were conducted by a single observer. However, pilot observations (observations before we started collecting data that was not included in the analysis) were conducted by two observers before data collection, and demonstrated excellent interobserver reliability. Another limitation is that the workflow activity data collected in this study, using a time and motion design, may not be comprehensive due to the medium sample size. However, we believe our samples are representative and we focused only on important tasks in the data collection process. This specific study is the first step in designing an ASP initiative. We intended to understand workflow so that the new ASP initiative is congruent with the current practice. We focus on day shift to ensure homogeneity of the data. However, we added this as a limitation.

Future studies will include leveraging EHR data to complement observation data with patient-specific and operational information such as diagnosis, demographics, and workload demands of the ED. Future studies will also include applying the data collection tool in other scenarios (such as hand-offs) to improve its ability to be customized for the setting and the generalizability of tool. This study examined general workflow and diagnostic decision in general. Our future efforts will also focus on cases in which antibiotics were prescribed and more detailed steps specific to antibiotic prescription and its unique characteristics.


#

Conclusion

The design and implementation of the CDS to implement an ED-based ASP should support all four ED provider roles, who often have different workflows. Clinicians make their decisions about treatment at different points of care delivery; a CDS for antibiotic prescribing also should support decisions at these different points of care.


#

Clinical Relevance Statement

CDS is the preferred method for implementation of ASP;[24] however, timing of CDS activation and presentation to clinicians is challenging in the ED setting. The ED is characterized by an erratic interrupted workflow, and a variety of providers and provider types that make decisions regarding antibiotic prescribing at different time points during the ED visit.[6] Integration into the clinical workflow is among the most important preferences of ED providers with respect to ED-based ASP implementation.[42] The results of our study demonstrate the complexity of the ED workflow, and the differences that exist among various ED providers. These workflow patterns will be essential to inform and develop our CDS for antibiotic prescribing, which will serve as the centerpiece for an ED-based ASP.


#

Multiple Choice Question

Which of the following statements is incorrect when designing and implementing clinical decision support systems in EDs?

  • CDS should support workflow for various type of clinical roles

  • Clinical decision support should support clinicians within different physical locations in the ED

  • CDS should support decisions at different points of care

  • The functionality of CDS should depend time of the day

Correct Answer: The correct answer is option d. As data analysis revealed in this study, CDS should support workflow for various clinical role. CDS also support clinicians within different physical locations and CDS should support decisions at different points of care. Although functionality of CDS can depend on various contextual factors such as workload in ED or individual characteristics of the clinicians, there is no evidence that the functionality of CDS should depend on time of the day.


#
#

Conflict of Interest

None.

Acknowledgment

We thank the faculty and staff in the Section of Emergency Medicine at Children's Hospital Colorado for their participation in our study. We thank Ms. Suzanne Lareau for editorial support. We thank Dr. Kai Zheng for his support with the use of the data collection tool. This study was funded by grants from Emergency Medicine Foundation, and the Clinical and Operational Effectiveness Safety Grant Program through the University of Colorado School of Medicine and Children's Hospital Colorado.

Protection of Human and Animal Subjects

This study was approved as a quality improvement activity by the Organizational Research Risk and Quality Improvement Review Panel, under agreement with the local Institutional Review Board.


  • References

  • 1 Shaw B, Cheater F, Baker R. , et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2005; (03) CD005470
  • 2 Ozkaynak M, Brennan PF. Characterizing patient care in hospital emergency departments. Health Syst (Basingstoke) 2012; 1 (02) 104-117
  • 3 Ozkaynak M, Dziadkowiec O, Mistry R. , et al. Characterizing workflow for pediatric asthma patients in emergency departments using electronic health records. J Biomed Inform 2015; 57: 386-398
  • 4 Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias. Acad Emerg Med 2002; 9 (11) 1184-1204
  • 5 Goldberg RM, Kuhn G, Andrew LB, Thomas Jr HA. Coping with medical mistakes and errors in judgment. Ann Emerg Med 2002; 39 (03) 287-292
  • 6 May L, Cosgrove S, L'Archeveque M. , et al. A call to action for antimicrobial stewardship in the emergency department: approaches and strategies. Ann Emerg Med 2013; 62 (01) 69-77
  • 7 Schenkel S. Promoting patient safety and preventing medical error in emergency departments. Acad Emerg Med 2000; 7 (11) 1204-1222
  • 8 Shiffman RN, Michel G, Essaihi A, Thornquist E. Bridging the guideline implementation gap: a systematic, document-centered approach to guideline implementation. J Am Med Inform Assoc 2004; 11 (05) 418-426
  • 9 Sim I, Gorman P, Greenes RA. , et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001; 8 (06) 527-534
  • 10 Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag 2009; 23 (04) 38-45
  • 11 Waitman LR, Miller RA. Pragmatics of implementing guidelines on the front lines. J Am Med Inform Assoc 2004; 11 (05) 436-438
  • 12 Barlam TF, Cosgrove SE, Abbo LM. , et al. Executive summary: implementing an antibiotic stewardship program: guidelines by the Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America. Clin Infect Dis 2016; 62 (10) 1197-1202
  • 13 Kullar R, Goff DA. Transformation of antimicrobial stewardship programs through technology and informatics. Infect Dis Clin North Am 2014; 28 (02) 291-300
  • 14 Kullar R, Goff DA, Schulz LT, Fox BC, Rose WE. The “epic” challenge of optimizing antimicrobial stewardship: the role of electronic medical records and technology. Clin Infect Dis 2013; 57 (07) 1005-1013
  • 15 Sintchenko V, Coiera E, Gilbert GL. Decision support systems for antibiotic prescribing. Curr Opin Infect Dis 2008; 21 (06) 573-579
  • 16 Moja L, Liberati EG, Galuppo L. , et al. Barriers and facilitators to the uptake of computerized clinical decision support systems in specialty hospitals: protocol for a qualitative cross-sectional study. Implement Sci 2014; 9: 105
  • 17 Wright A, Sittig DF, Ash JS. , et al. Lessons learned from implementing service-oriented clinical decision support at four sites: a qualitative study. Int J Med Inform 2015; 84 (11) 901-911
  • 18 Spellberg B, Blaser M, Guidos RJ. , et al; Infectious Diseases Society of America (IDSA). Combating antimicrobial resistance: policy recommendations to save lives. Clin Infect Dis 2011; 52 (Suppl. 05) S397-S428
  • 19 Donnelly JP, Baddley JW, Wang HE. Antibiotic utilization for acute respiratory tract infections in U.S. emergency departments. Antimicrob Agents Chemother 2014; 58 (03) 1451-1457
  • 20 Gonzales R, Camargo Jr CA, MacKenzie T. , et al; IMPAACT Trial Investigators. Antibiotic treatment of acute respiratory infections in acute care settings. Acad Emerg Med 2006; 13 (03) 288-294
  • 21 Kroening-Roche JC, Soroudi A, Castillo EM, Vilke GM. Antibiotic and bronchodilator prescribing for acute bronchitis in the emergency department. J Emerg Med 2012; 43 (02) 221-227
  • 22 Paul IM, Maselli JH, Hersh AL, Boushey HA, Nielson DW, Cabana MD. Antibiotic prescribing during pediatric ambulatory care visits for asthma. Pediatrics 2011; 127 (06) 1014-1021
  • 23 Society for Healthcare Epidemiology of America; Infectious Diseases Society of America; Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). Infect Control Hosp Epidemiol 2012; 33 (04) 322-327
  • 24 Mistry RD, Newland JG, Gerber JS. , et al. Current state of antimicrobial stewardship in children's hospital emergency departments. Infect Control Hosp Epidemiol 2017; 38 (04) 469-475
  • 25 Shebl NA, Franklin BD, Barber N. Clinical decision support systems and antibiotic use. Pharm World Sci 2007; 29 (04) 342-349
  • 26 Bates DW, Kuperman GJ, Wang S. , et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
  • 27 Christakis DA, Zimmerman FJ, Wright JA, Garrison MM, Rivara FP, Davis RL. A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics 2001; 107 (02) E15
  • 28 Gerber JS, Prasad PA, Fiks AG. , et al. Effect of an outpatient antimicrobial stewardship intervention on broad-spectrum antibiotic prescribing by primary care pediatricians: a randomized trial. JAMA 2013; 309 (22) 2345-2352
  • 29 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346
  • 30 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
  • 31 Ansari F, Gray K, Nathwani D. , et al. Outcomes of an intervention to improve hospital antibiotic prescribing: interrupted time series with segmented regression analysis. J Antimicrob Chemother 2003; 52 (05) 842-848
  • 32 Carling P, Fung T, Killion A, Terrin N, Barza M. Favorable impact of a multidisciplinary antibiotic management program conducted during 7 years. Infect Control Hosp Epidemiol 2003; 24 (09) 699-706
  • 33 Jenkins TC, Knepper BC, Sabel AL. , et al. Decreased antibiotic utilization after implementation of a guideline for inpatient cellulitis and cutaneous abscess. Arch Intern Med 2011; 171 (12) 1072-1079
  • 34 Linder J, Schnipper JL, Volk LA. , et al. Clinical decision support to improve antibiotic prescribing for acute respiratory infections: results of a pilot study. AMIA Annu Symp Proc 2007; 11: 468-472
  • 35 Metlay JP, Camargo Jr CA, MacKenzie T. , et al; IMPAACT Investigators. Cluster-randomized trial to improve antibiotic use for adults with acute respiratory infections treated in emergency departments. Ann Emerg Med 2007; 50 (03) 221-230
  • 36 Parnes B, Fernald D, Coombs L. , et al. Improving the management of skin and soft tissue infections in primary care: a report from State Networks of Colorado Ambulatory Practices and Partners (SNOCAP-USA) and the Distributed Ambulatory Research in Therapeutics Network (DARTNet). J Am Board Fam Med 2011; 24 (05) 534-542
  • 37 Moxey A, Robertson J, Newby D, Hains I, Williamson M, Pearson SA. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc 2010; 17 (01) 25-33
  • 38 Sittig DF, Wright A, Osheroff JA. , et al. Grand challenges in clinical decision support. J Biomed Inform 2008; 41 (02) 387-392
  • 39 Zheng K, Ciemins EL, Lanham HJ, Lindberg C. Examining the Relationship Between Health IT and Ambulatory Care Workflow Redesign (Prepared by Billings Clinic under Contract No. 290–2010–0019I–1). Rockville, MD; 2015
  • 40 Agrawal R, Srikant R. . Fast Algorithms for Mining Association Rules in Large Databases, in Proceedings of the 20th International Conference on Very Large Data Bases; 1994, Morgan Kaufmann Publishers Inc.:487–499
  • 41 Han J, Cheng H, Xin D, Yan X. Frequent pattern mining: current status and future directions. Data Min Knowl Discov 2007; 15 (01) 55-86
  • 42 Chung P, Scandlyn J, Dayan PS, Mistry RD. Working at the intersection of context, culture, and technology: provider perspectives on antimicrobial stewardship in the emergency department using electronic health record clinical decision support. Am J Infect Control 2017; 45 (11) 1198-1202
  • 43 Baysari MT, Del Gigante J, Moran M. , et al. Redesign of computerized decision support to improve antimicrobial prescribing. A controlled before-and-after study. Appl Clin Inform 2017; 8 (03) 949-963
  • 44 Hollingsworth JC, Chisholm CD, Giles BK, Cordell WH, Nelson DR. How do physicians and nurses spend their time in the emergency department?. Ann Emerg Med 1998; 31 (01) 87-91
  • 45 Yen K, Shane EL, Pawar SS, Schwendel ND, Zimmanck RJ, Gorelick MH. Time motion study in a pediatric emergency department before and after computer physician order entry. Ann Emerg Med 2009; 53 (04) 462-468
  • 46 Ozkaynak M, Unertl KM, Johnson SA, Brixey JJ, Haque SN. Clinical workflow analysis, process redesign, and quality improvement. In Finnel J, Dixon B. , eds. Clinical Informatics Study Guide. New York: Springer International Publishing; 2016: 135-161
  • 47 Zheng K, Guo MH, Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc 2011; 18 (05) 704-710
  • 48 Zheng K, Haftel HM, Hirschl RB, O'Reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc 2010; 17 (04) 454-461

Address for correspondence

Mustafa Ozkaynak, PhD
College of Nursing, University of Colorado, Anschutz Medical Campus
Campus Box 288-18, Education 2 North Building
13120 East 19th Avenue, Room 4121, Aurora, CO 80045
United States   

  • References

  • 1 Shaw B, Cheater F, Baker R. , et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2005; (03) CD005470
  • 2 Ozkaynak M, Brennan PF. Characterizing patient care in hospital emergency departments. Health Syst (Basingstoke) 2012; 1 (02) 104-117
  • 3 Ozkaynak M, Dziadkowiec O, Mistry R. , et al. Characterizing workflow for pediatric asthma patients in emergency departments using electronic health records. J Biomed Inform 2015; 57: 386-398
  • 4 Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias. Acad Emerg Med 2002; 9 (11) 1184-1204
  • 5 Goldberg RM, Kuhn G, Andrew LB, Thomas Jr HA. Coping with medical mistakes and errors in judgment. Ann Emerg Med 2002; 39 (03) 287-292
  • 6 May L, Cosgrove S, L'Archeveque M. , et al. A call to action for antimicrobial stewardship in the emergency department: approaches and strategies. Ann Emerg Med 2013; 62 (01) 69-77
  • 7 Schenkel S. Promoting patient safety and preventing medical error in emergency departments. Acad Emerg Med 2000; 7 (11) 1204-1222
  • 8 Shiffman RN, Michel G, Essaihi A, Thornquist E. Bridging the guideline implementation gap: a systematic, document-centered approach to guideline implementation. J Am Med Inform Assoc 2004; 11 (05) 418-426
  • 9 Sim I, Gorman P, Greenes RA. , et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001; 8 (06) 527-534
  • 10 Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag 2009; 23 (04) 38-45
  • 11 Waitman LR, Miller RA. Pragmatics of implementing guidelines on the front lines. J Am Med Inform Assoc 2004; 11 (05) 436-438
  • 12 Barlam TF, Cosgrove SE, Abbo LM. , et al. Executive summary: implementing an antibiotic stewardship program: guidelines by the Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America. Clin Infect Dis 2016; 62 (10) 1197-1202
  • 13 Kullar R, Goff DA. Transformation of antimicrobial stewardship programs through technology and informatics. Infect Dis Clin North Am 2014; 28 (02) 291-300
  • 14 Kullar R, Goff DA, Schulz LT, Fox BC, Rose WE. The “epic” challenge of optimizing antimicrobial stewardship: the role of electronic medical records and technology. Clin Infect Dis 2013; 57 (07) 1005-1013
  • 15 Sintchenko V, Coiera E, Gilbert GL. Decision support systems for antibiotic prescribing. Curr Opin Infect Dis 2008; 21 (06) 573-579
  • 16 Moja L, Liberati EG, Galuppo L. , et al. Barriers and facilitators to the uptake of computerized clinical decision support systems in specialty hospitals: protocol for a qualitative cross-sectional study. Implement Sci 2014; 9: 105
  • 17 Wright A, Sittig DF, Ash JS. , et al. Lessons learned from implementing service-oriented clinical decision support at four sites: a qualitative study. Int J Med Inform 2015; 84 (11) 901-911
  • 18 Spellberg B, Blaser M, Guidos RJ. , et al; Infectious Diseases Society of America (IDSA). Combating antimicrobial resistance: policy recommendations to save lives. Clin Infect Dis 2011; 52 (Suppl. 05) S397-S428
  • 19 Donnelly JP, Baddley JW, Wang HE. Antibiotic utilization for acute respiratory tract infections in U.S. emergency departments. Antimicrob Agents Chemother 2014; 58 (03) 1451-1457
  • 20 Gonzales R, Camargo Jr CA, MacKenzie T. , et al; IMPAACT Trial Investigators. Antibiotic treatment of acute respiratory infections in acute care settings. Acad Emerg Med 2006; 13 (03) 288-294
  • 21 Kroening-Roche JC, Soroudi A, Castillo EM, Vilke GM. Antibiotic and bronchodilator prescribing for acute bronchitis in the emergency department. J Emerg Med 2012; 43 (02) 221-227
  • 22 Paul IM, Maselli JH, Hersh AL, Boushey HA, Nielson DW, Cabana MD. Antibiotic prescribing during pediatric ambulatory care visits for asthma. Pediatrics 2011; 127 (06) 1014-1021
  • 23 Society for Healthcare Epidemiology of America; Infectious Diseases Society of America; Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). Infect Control Hosp Epidemiol 2012; 33 (04) 322-327
  • 24 Mistry RD, Newland JG, Gerber JS. , et al. Current state of antimicrobial stewardship in children's hospital emergency departments. Infect Control Hosp Epidemiol 2017; 38 (04) 469-475
  • 25 Shebl NA, Franklin BD, Barber N. Clinical decision support systems and antibiotic use. Pharm World Sci 2007; 29 (04) 342-349
  • 26 Bates DW, Kuperman GJ, Wang S. , et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
  • 27 Christakis DA, Zimmerman FJ, Wright JA, Garrison MM, Rivara FP, Davis RL. A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics 2001; 107 (02) E15
  • 28 Gerber JS, Prasad PA, Fiks AG. , et al. Effect of an outpatient antimicrobial stewardship intervention on broad-spectrum antibiotic prescribing by primary care pediatricians: a randomized trial. JAMA 2013; 309 (22) 2345-2352
  • 29 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346
  • 30 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
  • 31 Ansari F, Gray K, Nathwani D. , et al. Outcomes of an intervention to improve hospital antibiotic prescribing: interrupted time series with segmented regression analysis. J Antimicrob Chemother 2003; 52 (05) 842-848
  • 32 Carling P, Fung T, Killion A, Terrin N, Barza M. Favorable impact of a multidisciplinary antibiotic management program conducted during 7 years. Infect Control Hosp Epidemiol 2003; 24 (09) 699-706
  • 33 Jenkins TC, Knepper BC, Sabel AL. , et al. Decreased antibiotic utilization after implementation of a guideline for inpatient cellulitis and cutaneous abscess. Arch Intern Med 2011; 171 (12) 1072-1079
  • 34 Linder J, Schnipper JL, Volk LA. , et al. Clinical decision support to improve antibiotic prescribing for acute respiratory infections: results of a pilot study. AMIA Annu Symp Proc 2007; 11: 468-472
  • 35 Metlay JP, Camargo Jr CA, MacKenzie T. , et al; IMPAACT Investigators. Cluster-randomized trial to improve antibiotic use for adults with acute respiratory infections treated in emergency departments. Ann Emerg Med 2007; 50 (03) 221-230
  • 36 Parnes B, Fernald D, Coombs L. , et al. Improving the management of skin and soft tissue infections in primary care: a report from State Networks of Colorado Ambulatory Practices and Partners (SNOCAP-USA) and the Distributed Ambulatory Research in Therapeutics Network (DARTNet). J Am Board Fam Med 2011; 24 (05) 534-542
  • 37 Moxey A, Robertson J, Newby D, Hains I, Williamson M, Pearson SA. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc 2010; 17 (01) 25-33
  • 38 Sittig DF, Wright A, Osheroff JA. , et al. Grand challenges in clinical decision support. J Biomed Inform 2008; 41 (02) 387-392
  • 39 Zheng K, Ciemins EL, Lanham HJ, Lindberg C. Examining the Relationship Between Health IT and Ambulatory Care Workflow Redesign (Prepared by Billings Clinic under Contract No. 290–2010–0019I–1). Rockville, MD; 2015
  • 40 Agrawal R, Srikant R. . Fast Algorithms for Mining Association Rules in Large Databases, in Proceedings of the 20th International Conference on Very Large Data Bases; 1994, Morgan Kaufmann Publishers Inc.:487–499
  • 41 Han J, Cheng H, Xin D, Yan X. Frequent pattern mining: current status and future directions. Data Min Knowl Discov 2007; 15 (01) 55-86
  • 42 Chung P, Scandlyn J, Dayan PS, Mistry RD. Working at the intersection of context, culture, and technology: provider perspectives on antimicrobial stewardship in the emergency department using electronic health record clinical decision support. Am J Infect Control 2017; 45 (11) 1198-1202
  • 43 Baysari MT, Del Gigante J, Moran M. , et al. Redesign of computerized decision support to improve antimicrobial prescribing. A controlled before-and-after study. Appl Clin Inform 2017; 8 (03) 949-963
  • 44 Hollingsworth JC, Chisholm CD, Giles BK, Cordell WH, Nelson DR. How do physicians and nurses spend their time in the emergency department?. Ann Emerg Med 1998; 31 (01) 87-91
  • 45 Yen K, Shane EL, Pawar SS, Schwendel ND, Zimmanck RJ, Gorelick MH. Time motion study in a pediatric emergency department before and after computer physician order entry. Ann Emerg Med 2009; 53 (04) 462-468
  • 46 Ozkaynak M, Unertl KM, Johnson SA, Brixey JJ, Haque SN. Clinical workflow analysis, process redesign, and quality improvement. In Finnel J, Dixon B. , eds. Clinical Informatics Study Guide. New York: Springer International Publishing; 2016: 135-161
  • 47 Zheng K, Guo MH, Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc 2011; 18 (05) 704-710
  • 48 Zheng K, Haftel HM, Hirschl RB, O'Reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc 2010; 17 (04) 454-461

Zoom Image
Fig. 1 Data collection tool.
Zoom Image
Fig. 2 Workflow patterns for the four providers.