Principles for Designing and Developing a Workflow Monitoring Tool to Enable and Enhance Clinical Workflow Automation

Abstract

Background Automation of health care workflows has recently become a priority. This can be enabled and enhanced by a workflow monitoring tool (WMOT).

Objectives We shared our experience in clinical workflow analysis via three cases studies in health care and summarized principles to design and develop such a WMOT.

Methods The case studies were conducted in different clinical settings with distinct goals. Each study used at least two types of workflow data to create a more comprehensive picture of work processes and identify bottlenecks, as well as quantify them. The case studies were synthesized using a data science process model with focuses on data input, analysis methods, and findings.

Results Three case studies were presented and synthesized to generate a system structure of a WMOT. When developing a WMOT, one needs to consider the following four aspects: (1) goal orientation, (2) comprehensive and resilient data collection, (3) integrated and extensible analysis, and (4) domain experts.

Discussion We encourage researchers to investigate the design and implementation of WMOTs and use the tools to create best practices to enable workflow automation and improve workflow efficiency and care quality.

Keywords

workflow (L01.906.893) - data collection (E05.318.308) - data analysis (H01.548.338) - expert systems (L01.224.050.375.190)

Background and Significance

National efforts have recently prioritized the automation of health care workflows through modern computing techniques.[1] Workflow automation in health care through the support of information technology and machine intelligence can improve efficiency of care delivery, leading to reduced clinician burnout, higher patient satisfaction, and safer health care.[2] [3] [4] [5] Workflow automation and its broader discipline and business process management have been extensively studied in the field of information systems (IS) in the past two decades. To fulfill changing needs in a rapidly evolving business environment, IS researchers have proposed the use of agent-based technologies to develop a workflow management system in a flexible and efficient manner.[6] [7] Other methodologies include visual simulation, conversational digital assistance, and robotics.[8] [9] [10]

While these workflow automation methodologies and technologies are applicable to many industries and fields, health care has unique characteristics introducing challenges to workflow automation. One such characteristic is the dynamic nature of health care, making it a complex sociotechnical system where the system itself keeps changing and the multiple users (clinicians) interact with each other to provide care.[11] Carayon et al suggested using sociotechnical system analysis (STSA) to address care delivery and quality issues and proposed seven STSA research challenges, one of which being cognitive design of health information technology (HIT).[12] The cognitive design highlighted the importance to understand human behaviors and the human-automation interaction, the relationship between the human operators, the automated systems, and their interfaces.

In this vein, understanding clinicians' workflows would be a critical first step before developing any workflow automation solutions. If clinician workflow patterns are not carefully captured and translated into automation and optimization requirements, workflow automation will have low reliability and user adoption and can introduce unintended negative consequences and even patient harms.[13] [14] [15] [16] We adopted this viewpoint and argue that health care workflow automation should be enabled and continuously enhanced by a workflow monitoring tool (WMOT) which explores and identifies the needs and opportunities for workflow automation. Monitoring of clinical workflow involves gathering and analyzing relevant workflow data and providing real-time feedback and guidance. A pioneering study by Vankipuram et al in 2011 proposed a system to automatically analyze and visualize clinical workflow in critical care environments.[17] Another study was conducted by Zhu et al to develop a real-time radiology workflow-based monitoring dashboard.[18] Recently, analyzing clinical workflow has been a popular research topic in various clinical settings, such as emergency medicine, primary care, nursing, and medical scribes.[19] [20] [21] [22] While these workflow analysis studies demonstrated success, they required extensive resources for planning and conduction. Originated but expanded from clinical workflow analysis, a WMOT can reduce resource burdens and standardize the data collection and analysis process. To date, no framework exists to guide the development of WMOT.

Therefore, we aim to generate principles for designing and developing such a WMOT to enable and enhance workflow automation in health care. It is worth noting that a WMOT is separated from electronic health record (EHR) systems but interfaces with them; a WMOT is served as a diagnosis tool to help identify workflow patterns, bottlenecks, and context of use to guide workflow automation. While a WMOT is not used in clinical care, its results can help improve care routines and enhance local EHR systems. In the present study, we shared our experience in clinical workflow analysis through three distinct case studies. We summarized and synthesized our case studies, generate the design principles, and discuss our future plan to develop a WMOT.

Methods

Workflow Analysis Tools and Cases

We present three workflow analysis case studies in various clinical settings. All case studies used at least one of the two tools we developed to support workflow analysis. The two tools together were considered as a preliminary version of a WMOT. One tool is called Time Motion Data Collector (TMDC) which supports direct observations on clinical workflow. TMDC was designed and developed following the Suggested Time And Motion Procedures (STAMP) checklist to standardize time motion studies and make results compatible and comparable.[23] The other tool is called Clinical Workflow Analysis Tool (CWAT, pronounced as “see-what”) which analyzes tasks, locations, and timestamps in a workflow dataset. The CWAT used data-mining techniques with interactive visualization to help researchers identify and interpret workflow patterns. TMDC and CWAT were developed based on the first (D.T.Y.W.) and the senior author (K.Z.)'s previous researches.[24] Both TMDC and CWAT are capable of handling multitasking and interruptions which are unique capabilities comparing to existing workflow data collection and analysis tools.[25] All three case studies were focused on identifying the workflow issues and seeking for automation opportunities. Follow-up studies will be or are being conducted to redesign and automate the clinical workflow.

Case Analyses

The three cases were synthesized in their similarities and differences with a focus on workflow analysis methods and findings to form the foundation of design principles of a WMOT since the idea of clinical workflow monitoring builds upon clinical workflow analysis. Specifically, we utilized a data science process model to guide the comparison and contrast of workflow analysis methods which include raw data collection, data processing, data cleansing, exploratory data analysis, modeling and algorithms, visualization and communication, and production.[26] The case analyses also focused on the challenges faced and the lessons learned.

Results

Case Study 1: Antimicrobial Stewardship in a Pediatric Emergency Department

This study was conducted in a pediatric emergency department (PED).[27] The objective was to develop an EHR-embedded clinical decision support system (CDSS) for the antimicrobial stewardship programs (ASP) which aims to optimize appropriate antibiotic prescribing. To design and develop such a CDSS, we characterized the workflow of a sample of 23 clinicians in four clinical roles (attending physician, nurse practitioner, physician assistant, and resident). The workflow data were collected through TMDC for a total of 90 hours of direct observations and real-time queries for timing of diagnosis and disposition decision points. The observational data were analyzed using CWAT. The observers' reflections after each session were also recorded and analyzed to clarify the questions from the observational data. The TMDC and CWAT were used as standalone tools with no connections to other systems in this case. The results revealed 127 decision points in three themes (after/during examining or talking to patient or relative, after talking to a specialist, and after diagnostic test/image) with distinct workflow patterns exhibited by each of the clinical roles. The findings highlighted three principles in developing a CDSS to achieve optimal workflow in ASP, including supporting a variety of workflow, physicians in different locations in the ED, and decisions at different points of care. This study was limited in its single-site setting and medium sample size of the time and motion design.

Case Study 2: Bedside Nurse Documentation in Inpatient Wards

This study was conducted in a surgical ward and a medical ward with a total of 40 nurses and 23 nurse assistants. The study aimed to evaluate the impact of a mobile application on bedside documentation.[24] [25] The mobile application was developed to address the issue of burdensome EHR documentation and fragmented documentation processes. The application was used as a HIT-based intervention in this study while TMDC and CWAT were used as standalone tools to collect workflow data and examine workflow patterns. The workflow data collected in the intervention stage were compared with those in the baseline stage where documentation was done using paper (later transcribed) and computers on wheels. Duration and location of all documentation activities were observed and recorded through the TMDC in all shifts between 7 a.m. and 5 p.m. with a pre- and postdesign. In addition, EHR and application event logs were collected and analyzed as part of the workflow data. All timestamp analyses were supported by the CWAT. The results showed that the beside mobile application successfully streamlined the workflow and reduced overall EHR documentation time, allowing nurses to spend more time with their patients with less interruptions, potentially leading to higher quality of care and improved patient satisfaction. The analysis also showed some variability between the two wards. This study has a few challenges and limitations, including the high cost in planning and executing the direct observations, the statistical insignificance due to small samples, the various application utilization due to time pressure, workload, and learning curve, and the shorter study duration. After the research project, the application was implemented in several wards, until a recent major revision of the EHR architecture. The application is currently undergoing major revision due to this new architecture, and will include novel functionalities (such as alerts and photo uploads).

Case Study 3: Sedation in a Pediatric Emergency Department

This study was conducted in a PED with two objectives.[28] First, it aimed to compare the workflow of normal versus prolonged cases to identify potential factors contributing to delays in starting a sedation or in prolonged durations. Prolonged sedation cases were defined as cases with more than a 180-minute duration from patient arrival to ketamine given. Second, the study aimed to augment traditional quality improvement (QI) strategies (e.g., process maps) through objectively recorded time-based data. The study deployed a survey to physicians and nurses of the sedation team to assess their perceptions of the drivers that were most responsible for sedation delay. Eighty-eight out of the 215 clinicians responded to the survey (41% response rate). In addition, the study collected timestamps from the EHR and the real-time locating system (RTLS). The RTLS captured patient, staff, and device movement in the form of event logs. The survey data were summarized statistically and the EHR and RTLS timestamps were merged and analyzed using the CWAT. Results from the survey analysis showed that physicians and nurses considered each other's availability and simultaneous readiness as the key factors contributing to the sedation delay which is inconclusive and even conflicting. On the other hand, the objective timestamp data generated five measurable workflows for patients, sedation and orthopaedic physicians, residents, and medications. Through comparing and contrasting, the timestamp data on 54 sedation cases, 33 of which were considered prolonged, the analysis of the timestamp data was able to identify workflow patterns and significant bottlenecks. In fact, both physicians and nurses contributed to the delay at different points of the sedation process. Furthermore, the timestamp analysis identified patient arrival to being placed in a room as another key driver of the delay which was not captured in the survey. The workflow analysis resolved the discrepancy found in the self-reported survey data and inferred potential root causes. The study highlighted the challenges in combining EHR and RTLS timestamps and suggested close collaboration with domain experts to determine the best “source of truth.” Moreover, the study advocated for incorporating objective workflow data into traditional QI methodologies to develop deliberate QI strategies and effective interventions.

Lessons Learned from the Case Studies

We identified three lessons learned from the cases described in the previous sections. First, workflow data can be messy and incomplete. For example, time and motion data only covered certain work hours during a day and a portion of the clinicians. EHR or RTLS data were usually recorded based on predefined categories and granularity. Moreover, qualitative data can enable the understanding of the workflow context while quantitative data can show workflow patterns and trends in a larger scale. It is unlikely to have one source of data to reflect the full spectrum of clinical workflow which emerge within a complex sociotechnical system. Often times it requires triangulation and synthesis of multiple sources of data. Therefore, determining the source of truth and have proper research questions and a scope are important to guide the workflow analysis.

The second lesson was the limitation of TMDC and CWAT, both of which were initially developed to analyze clinical workflow in rural outpatient clinics.[24] Although TMDC and CWAT can support workflow data management and exploratory analysis, in each of the three case studies, we created specific features and functions to clean, manipulate, analyze, and visualize the workflow data. We realized that some of the features and functions, even advanced algorithms, were already implemented in other tools. However, the data structure of these tools can be so different that it may take more time to import our data into another tool than just implementing the specific features and functions on our own to answer our research questions. Meanwhile, the workflow measures of other tools may be differently defined, which makes the analysis results incomparable and hinders knowledge accumulation in this field. These can be addressed by creating a common data model, a set of standardized workflow measures, the ability to combine multiple tools, and the best practice analysis procedures.

The last lesson was getting the buy-in from the clinicians being studied. A workflow study requires input from these clinicians at all stages. The clinicians can (and may have to be) an integral part of a workflow study providing the context of the current workflow issues, supporting participant recruitment, refining study protocol to improve the feasibility and practicability, interpreting analysis results, and evaluating the facial validity of a solution. All of our case studies cannot be done without a close collaboration with the clinicians. However, to support and participant a workflow study, clinicians have to take additional tasks on top of their busy routines. It is therefore important to transform workflow analysis to workflow monitoring to remove the burden from clinicians while systematically and meaningfully addressing clinical workflow issues through informatics and domain expert engagement.

Discussion

We described and synthesized three case studies and propose four design principles for developing a WMOT in health care. The key components of a WMOT are discussed in the following sections with [Fig. 1] describing the system structure. It is worth noting that the four principles are not made for workflow automation and optimization. They are used to develop a WMOT that can inform workflow redesign and generate a better automation and optimization plan.

Fig. 1 The system structure of a workflow monitoring tool (WMOT). It engages domain experts and supports their goals (principles 1 and 4), and creates a feedback loop to interactively improve workflow. It includes multiple data sources with both qualitative and quantitative data (principle 2). All of the data are imported to a common data model and analyzed by a dedicated engine with interoperability to other workflow analysis algorithms and tools (principle 3). EHR event logs are a specific type of EHR timestamps, which record clinicians' clicks and system output. EHR, electronic health record; REST, representational state transfer; RTLS, real-time locating system; TMS, time and motion study.

Design Principles

Goal Orientation

Our case studies showed that clinical workflow analysis should start with a clear primary goal. Otherwise, the analysis can be difficult to carry over with massive and noisy workflow data and would not generate meaningful results. Scope creep can also occur. Each of our three case studies had distinct goals: understanding the workflow patterns and the context of decisions to design a CDSS, evaluating the impact of a bedside mobile application on clinical documentation, and comparing normal and prolonged sedation cases to identify hidden factors contributing to delays. Complex workflow projects may naturally yield multiple potential goals, yet explicit prioritization and definition of a primary goal will prevent complicated analyses and results that are difficult to interpret or functionally insignificant. Therefore, a WMOT should provide goal-specific recommendations. Moreover, its data collection and analysis procedures should be adjusted accordingly.

Comprehensive and Resilient Data Collection

Workflow data can be collected from multiple sources and different clinical roles. Oftentimes, combining datasets to create a more comprehensive understanding of the workflow and identify bottlenecks is helpful. In each of the three cases studies, we used at least two data sources (e.g., interview and time and motion study (TMS) data, TMS and EHR event logs, and survey and EHR/RTLS timestamps) to achieve our study objectives. We also focused on the secondary use of EHR and RTLS data for clinical workflow analysis which has been a major research topic in clinical informatics in recent years.[29] [30] [31] When collecting workflow data, it is important to keep in mind the key elements, that is, actors, actions, artifacts, and outcomes,[32] to make sure the data collection is comprehensive. This is particularly important because some data collection methods are time- and resource-consuming (e.g., interviews and TMS). On the other hand, creating flexible and resilient procedures to collect workflow data are important to accommodate clinicians' busy schedules. For example, while our TMDC can support TMS adequately and collect interruption and multitasking behaviors, it was not designed to rapidly respond to the changes in the observation schedules (e.g., merging observation sessions, changing an observee, and reassigning an observer). Therefore, a WMOT should have the ability to incorporate multiple sources and types of workflow data, integrate data collection tools, as well as create procedures to accommodate dynamics in data collection processes.

Integrated and Extensible Analysis

This principle suggests integrating multiple clinical workflow analysis methods and allowing for easy extension to new methods and techniques. The use of both qualitative and quantitative analysis is critical, either using one to shape the analysis of the other, or triangulating the results, to focus on key workflow patterns and to address the corresponding workflow issues. For workflow data collected from interviews with clinicians, the data should be analyzed both qualitatively (e.g., thematic analysis) and quantitatively (e.g., topic modeling and visualization).[33] [34] The qualitative interview analysis results will be included manually in a table format while the quantitative text analysis results will be included in WMOT automatically.

In addition to traditional-grounded theory-based qualitative methods, several quantitative measures and methods have been proposed and used in workflow timestamp data. Since improving workflow efficiency is a common goal, measures such as duration, frequency, and time allocation (in percentage) are frequently used. Zheng et al proposed a novel workflow fragmentation measure with the assumption that the higher the frequency of task switches, the more cognitive overload it could cause.[35] Recently, Sinsky et al proposed metrics to assess physician activities using EHR event logs.[36] On the other hand, Vankipuram et al proposed a method to transform, analyze, and visualize location-tracking data to monitor clinical workflow.[37] Their method used advanced techniques such as temporal sequence extraction and probabilistic modeling to analyze work processes as whole rather than discrete tasks. Moreover, the authors suggest structuring the analysis techniques as representational state transfer (REST) services to improve the generalizability of the method. Based on the lessons learned from the three cases, a common data model plus a REST service should be a core of a WMOT to allow for easy extension to future analysis methods. We also suggest that WMOT should incorporate additional types of timestamp data (e.g., pager and safety incident timestamps).

Domain Experts

Domain experts should be engaged in the development of WMOT. The domain experts refer various roles, including but not limited to physician champions, QI specialists, administrators, and informaticians. Domain experts play a vital role in the development and use of a WMOT because they are responsible for interpreting the information generated by the system to achieve workflow analysis goals. As described in the user-centered design,[38] a WMOT should consider the whole user experience and specify the context of use to support the identification of workflow bottlenecks. A WMOT should be designed in a way to allow domain experts to shape the data collection and analysis processes. A WMOT should also be constantly evaluated in a user-centered manner and coevolve with domain experts since the analysis needs may change in a dynamic health care environment.

Limitations

This study has at least two limitations. First, each case study was conducted in one institution and the sample size was limited with the specific goals of each study which was not to achieve statistical significance. The case studies used mixed methods to understand workflow patterns, bottlenecks, and context of use to inform future workflow redesign and automation. The focus of the present study is to synthesize the experience of all cases to generate the design principles of WMOT. Second, we did not focus on solutions to address the quality of the workflow data and the potentially low accuracy, reliability, and completeness of the data. We separated the data cleansing and quality issue from the design of a WMOT to focus on the system architecture. The performance of WMOT will depend on the quality of workflow data but this issue is out of the scope of the present study.

Contributions

Our study contributes to the workflow automation literature by highlighting the importance of understanding clinician workflows and context of use and proposing design principles to develop a monitoring tool (i.e., WMOT) to achieve better understanding. More importantly, workflow monitoring should be an integrated part of clinical workflow automation to inform automation requirements and evaluate automation outcomes. In addition, our study expends the current literature in clinical workflow analysis by integrating various workflow analysis techniques (e.g., time and motion studies, log analysis, and qualitative interviews) to create a more comprehensive picture of workflow patterns and bottlenecks. While studies have been using clinical workflow analysis to support workflow automation and provide evidence-based planning such as[39] our study takes the workflow analysis into the next level and advocates for continuous monitoring, detailed understanding, and user-centered feedback loops.

Conclusion

A successful workflow automation and optimization requires workflow monitoring. In this study, we proposed the idea of a WMOT with four design principles. Such a WMOT can demonstrate its efficacy through enhancing situational awareness, enabling clinical predictions,[40] and improving medical and administrative decision making. Our study serves as a starting point to develop WMOTs to support workflow automation and optimization in variety of clinical settings. We encourage researchers to continue investigating the design and implementation, such as WMOT, to create best practices to improve workflow efficiency, care quality, and patient safety.[41]

Clinical Relevance Statement

Many quality improvement projects focus on checklists (e.g., whether an action is completed or not), and maintaining new actions over time is often difficult. Clinical workflow analysis focuses on the processes, exploring ways to ensure that required actions are included in patient care, which can help maintain changes over time. Clinical workflow analyses should be largely included in quality improvement projects to identify solutions that are more closely adapted to clinicians' needs: these may be the timing of new actions in a given sequence, or providing more specific information needed to improve the clinical documentation, or simply better efficiency without decreasing quality or safety of care. The proposed WMOT takes clinical workflow analyses into the next level as a monitoring and diagnosis tool to enable and enhance workflow automation.

Multiple Choice Questions

1. Which of the following may not be a good consideration to develop a WMOT?

   • Goal orientation

   • Sole focus on quantitative measures and methods

   • Domain experts

   • Comprehensive data collection

   Correct Answer: The correct answer is option b. We think the use of both qualitative and quantitative analysis is critical, either using one to shape the analysis of the other, or triangulating the results, to focus on key workflow patterns and address the corresponding workflow issues. All other options (a, c, and d) are major principles of a WMOT.

2. Which of the following statements is true?

   • Collecting workflow data are easy so careful planning is not needed

   • There is no benefit to use multiple types of data for workflow analysis

   • Clinical workflow analysis needs to have a clear primary goal

   • WMOT users just read the information from the system and do not need to provide feedback

   Correct Answer: The correct answer is option c. This first option (a) is not correct because collecting workflow data (e.g., TMS and interviews) can be very time- and resource-consuming. The second option (b) is not correct because oftentimes two or more types of data are used for workflow analysis to construct a more comprehensive picture of workflows. The fourth option (d) is not correct because users (domain experts) play a vital role in WMOT and should be involved in the analysis process and provide feedback.

Conflict of Interest

None declared.

Acknowledgment

We thank Mr. Ruthvik Abbu at the University of Cincinnati, College of Medicine, for his effort on proofreading the manuscript.

Protection of Human and Animal Subjects

No human and/or animal subjects were included.

Author Contributions

The first author drafted the manuscript. All coauthors helped improve the clarity and value of the manuscript by reviewing and revising the manuscript and contributed significantly to the synthesis of the major takeaways, that is, the four principles to developing a WMOT.

• References

• 1 The Office of the National Coordinator for Health Information Technology (ONC). Health information technology workflow automation policy development. Accessed May 25, 2021: https://www.healthit.gov/topic/scientific-initiatives/health-information-technology-workflow-automation-policy-development
• 2 Zayas-Cabán T, Haque SN, Kemper N. Identifying opportunities for workflow automation in health care: lessons learned from other industries. Appl Clin Inform 2021; 12 (03) 686-697
• 3 Otokiti AU, Craven CK, Shetreat-Klein A, Cohen S, Darrow B. Beyond getting rid of stupid stuff in the electronic health record (Beyond-GROSS): protocol for a user-centered, mixed-method intervention to improve the electronic health record system. JMIR Res Protoc 2021; 10 (03) e25148
• 4 Alam S, Osama M, Iqbal F, Sawar I. Reducing pharmacy patient waiting time. Int J Health Care Qual Assur 2018; 31 (07) 834-844
• 5 Tsai P, Liu C, Kahler DL, Li JG, Lu B, Yan G. A self-checking treatment couch coordinate calculation system in radiotherapy. J Appl Clin Med Phys 2020; 21 (01) 43-52
• 6 Wang M, Wang H, Xu D. The design of intelligent workflow monitoring with agent technology. Knowl Base Syst 2005; 18 (06) 257-266
• 7 Ehrler L, Fleurke M, Purvis M, Savarimuthu BTR. Agent-based workflow management systems (WfMSs): JBees: a distributed and adaptive WfMS with monitoring and controlling capabilities. Inf Syst E-Bus Manag 2006; 4 (01) 5-23
• 8 Holzmüller-Laue S, Schubert P, Göde B, Thurow K. Visual simulation for the bpm-based process automation. In: Kobyliński A, Sobczak A. eds. Perspectives in Business Informatics Research. Vol 158. Lecture Notes in Business Information Processing (LNBIP, volume 158). Berlin, Germany: Springer; 2013: 48-62
• 9 Rizk Y, Isahagian V, Boag S. et al. A conversational digital assistant for intelligent process automation. In: Asatiani A, García JM, Helander N. et al., eds. Business Process Management: Blockchain and Robotic Process Automation Forum. Vol 393. Lecture Notes in Business Information Processing (LNBIP, volume 393). Switzerland: Springer International Publishing; 2020: 85-100
• 10 Romao M, Costa J, Costa CJ. Robotic Process Automation: A Case Study in the Banking Industry. In: 2019 14th Iberian Conference on Information Systems and Technologies (CISTI). IEEE; 2019: 1-6 Coimbra, Portugal:
• 11 Effken JA. Different lenses, improved outcomes: a new approach to the analysis and design of healthcare information systems. Int J Med Inform 2002; 65 (01) 59-74
• 12 Carayon P, Bass E, Bellandi T, Gurses A, Hallbeck S, Mollo V. Socio-technical systems analysis in health care: a research agenda. IIE Trans Healthc Syst Eng 2011; 1 (01) 145-160
• 13 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care–an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-549
• 14 Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc 2011; 18 (01) 82-90
• 15 Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA 2017; 318 (06) 517-518
• 16 Ogundaini O, de la Harpe R, McLean N. Unintended consequences of technology-enabled work activities experienced by healthcare professionals in tertiary hospitals of sub-Saharan Africa. Afr J Sci Technol Innov Dev 2021; 1-10
• 17 Vankipuram M, Kahol K, Cohen T, Patel VL. Toward automated workflow analysis and visualization in clinical environments. J Biomed Inform 2011; 44 (03) 432-440
• 18 Zhu Q, Nie H, Lu X, Duan H. Radiology workflow-based monitoring dashboard in a heterogeneous environment. In: 2010 3rd International Conference on Biomedical Engineering and Informatics. IEEE; 2010: 2494-2498 Yantai, China:
• 19 Patel VL, Denton CA, Soni HC, Kannampallil TG, Traub SJ, Shapiro JS. Physician workflow in two distinctive emergency departments: an observational study. Appl Clin Inform 2021; 12 (01) 141-152
• 20 Sieja A, Kim E, Holmstrom H. et al. Multidisciplinary sprint program achieved specialty-specific EHR optimization in 20 clinics. Appl Clin Inform 2021; 12 (02) 329-339
• 21 Yu D, Obuseh M, DeLaurentis P. Quantifying the impact of infusion alerts and alarms on nursing workflows: a retrospective analysis. Appl Clin Inform 2021; 12 (03) 528-538
• 22 Gold JA, Becton J, Ash JS, Corby S, Mohan V. Do you know what your scribe did last spring? The impact of COVID-19 on medical scribe workflow. Appl Clin Inform 2020; 11 (05) 807-811
• 23 Zheng K, Guo MH, Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc 2011; 18 (05) 704-710
• 24 Zheng K, Ciemins E, Lanham H, Lindberg C. Examining the relationship between health IT and ambulatory care workflow redesign. . Accessed December 10, 2021: https://digital.ahrq.gov/sites/default/files/docs/citation/examining-the-relationship-between-health-it-and-ambulatory-care-workflow-redesign-final-report.pdf
• 25 Zheng K, Westbrook J, Kannampallil TG, Patel VL. Cognitive Informatics: Reengineering Clinical Workflow for Safer and More Efficient Care. Switzerland: Springer Nature; 2019
• 26 O'Neil C, Schutt R. Doing Data Science. 1st ed.. Newton MA: O'Reilly Media Inc.; 2013
• 27 Ozkaynak M, Wu DTY, Hannah K, Dayan PS, Mistry RD. Examining workflow in a pediatric emergency department to develop a clinical decision support for an antimicrobial stewardship program. Appl Clin Inform 2018; 9 (02) 248-260
• 28 Barrick L, Wu DTY, Frey T. et al. Improving care delivery: location timestamps to enhance process measurement of a clinical workflow. Pediatr Qual Saf 2021; 6 (05) e475
• 29 Hribar MR, Read-Brown S, Goldstein IH. et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
• 30 Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020; 27 (04) 531-538
• 31 Overmann KM, Wu DTY, Xu CT, Bindhu SS, Barrick L. Real-time locating systems to improve healthcare delivery: a systematic review. J Am Med Inform Assoc 2021; 28 (06) 1308-1317
• 32 Unertl KM, Novak LL, Johnson KB, Lorenzi NM. Traversing the many paths of workflow research: developing a conceptual framework of workflow terminology through a systematic literature review. J Am Med Inform Assoc 2010; 17 (03) 265-273
• 33 Maguire M, Delahunt B. Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars. All Irel J High Educ 2017;9(03):
• 34 Mei Q, Cai D, Zhang D, Zhai C. Topic modeling with network regularization. In: Proceeding of the 17th International Conference on World Wide Web - WWW '08. ACM Press; 2008: 101
• 35 Zheng K, Haftel HM, Hirschl RB, O'Reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc 2010; 17 (04) 454-461
• 36 Sinsky CA, Rule A, Cohen G. et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (04) 639-643
• 37 Vankipuram A, Traub S, Patel VL. A method for the analysis and visualization of clinical workflow in dynamic environments. J Biomed Inform 2018; 79: 20-31
• 38 usability.gov. User-centered design basics. . Accessed December 10, 2021: https://www.usability.gov/what-and-why/user-centered-design.html
• 39 Staras S, Tauscher JS, Rich N. et al. Using a clinical workflow analysis to enhance eHealth implementation planning: tutorial and case study. JMIR Mhealth Uhealth 2021; 9 (03) e18534
• 40 Sills MR, Ozkaynak M, Jang H. Predicting hospitalization of pediatric asthma patients in emergency departments using machine learning. Int J Med Inform 2021; 151: 104468
• 41 Tanzini M, Westbrook JI, Guidi S, Sunderland N, Prgomet M. Measuring Clinical Workflow to Improve Quality and Safety. In: Donaldson L, Ricciardi W, Sheridan S, Tartaglia R. eds. Textbook of Patient Safety and Clinical Risk Management. Switzerland: Springer International Publishing; 2021: 393-402

Publication History

Received: 01 June 2021

Accepted: 22 November 2021

Article published online:
19 January 2022