Reporting Outcomes of Pediatric Intensive Care Unit Patients to Referring Physicians via an Electronic Health Record-Based Feedback System

Abstract

Background Many critically ill children are initially evaluated in front-line settings by clinicians with variable pediatric training before they are transferred to a pediatric intensive care unit (PICU). Because clinicians learn from past performance, communicating outcomes of patients back to front-line clinicians who provide pediatric emergency care could be valuable; however, referring clinicians do not consistently receive this important feedback.

Objectives Our aim was to determine the feasibility, usability, and clinical relevance of a semiautomated electronic health record (EHR)-supported system developed at a single institution to deliver timely and relevant PICU patient outcome feedback to referring emergency department (ED) physicians.

Methods Guided by the Health Information Technology Safety Framework, we iteratively designed, implemented, and evaluated a semiautomated electronic feedback system leveraging the EHR in one institution. After conducting interviews and focus groups with stakeholders to understand the PICU-ED health care work system, we designed the EHR-supported feedback system by translating stakeholder, organizational, and usability objectives into feedback process and report requirements. Over 6 months, we completed three cycles of implementation and evaluation, wherein we analyzed EHR access logs, reviewed feedback reports sent, performed usability testing, and conducted physician interviews to determine the system's feasibility, usability, and clinical relevance.

Results The EHR-supported feedback process is feasible with timely delivery and receipt of feedback reports. Usability testing revealed excellent Systems Usability Scale scores. According to physicians, the process was well-integrated into their clinical workflows and conferred minimal additional workload. Physicians also indicated that delivering and receiving consistent feedback was relevant to their clinical practice.

Conclusion An EHR-supported system to deliver timely and relevant PICU patient outcome feedback to referring ED physicians was feasible, usable, and important to physicians. Future work is needed to evaluate impact on clinical practice and patient outcomes and to investigate applicability to other clinical settings involved in similar care transitions.

Keywords

audit and feedback - electronic health records and systems - care transition - provider-provider communication - pediatrics

Background and Significance

Clinicians develop expertise by learning from past performance.[1] Clinicians who receive information about their performance can become better calibrated and thus make better diagnosis and treatment decisions.[2] Calibration is the alignment between clinicians' confidence in their accuracy and their actual accuracy[3] and can be achieved by receiving consistent feedback about one's patient outcomes.[3] [4] [5] [6] [7]

A total of 40% critically ill children with acute illness and injury are initially diagnosed and treated at community hospitals.[8] [9] Front-line clinicians have variable training in pediatrics and may infrequently encounter very ill children which can result in suboptimal care, most commonly undertreatment.[10] Timely interventions based on accurate assessments by community physicians have been shown to improve outcomes in critically ill children[11] [12]; thus, it is essential for referring clinicians to learn about their transferred patients' outcomes to improve future performance. However, in our highly compartmentalized model of health care, referring clinicians do not receive consistent feedback on individual patient outcomes to continuously improve their practice. In a statewide survey, we conducted in Iowa, referring clinicians received patient outcome feedback on only 40% of patients they transferred to pediatric intensive care units (PICUs) over a year. Among clinicians who obtained feedback,13% received information they did not expect; 40% of these clinicians indicated that this experience changed their practice.[13]

Objectives

Given this significant gap in feedback and its potential to improve the emergency care of critically ill children, our objective was to develop, implement, and evaluate a semiautomated electronic health record (EHR)-supported system at a single institution to deliver timely and relevant PICU patient outcome feedback to referring emergency department (ED) physicians. In this case study, we report the feasibility, usability, and relevance of the feedback system to both ED and PICU physicians.

Methods

Using a human factors engineering approach informed by the Health Information Technology (IT) Safety Framework,[14] we iteratively designed, implemented, and evaluated a semiautomated electronic feedback system leveraging the Epic EHR at a single institution. This project was reviewed and determined to be exempt from human subjects research oversight by the University of Iowa Institutional Review Board. In the next sections, we describe the clinical settings for the project and how we determined the need for feedback, designed the feedback system/report, and evaluated its performance.

Clinical Settings

The EHR-supported feedback system was developed for implementation at the University of Iowa Hospitals and Clinics (UIHC) ED and the UI Stead Family Children's Hospital PICU. Both serve mostly rural populations within a large catchment area. The UIHC ED is a tertiary referral academic ED with 46 beds accommodating approximately 15,000 pediatric visits per year and is staffed by 30 emergency medicine (EM) attending physicians (six pediatric EM trained). The UI PICU is the only academic tertiary referral PICU in Iowa, admitting approximately 1,200 patients per year and is staffed by 11 attending intensivists.

Determining the Need for Patient Outcome Feedback

We conducted two focus groups of ED physicians (six UI ED physicians and six non-UI-affiliated referring physicians) to substantiate our earlier survey indicating the gap in feedback[13] and explore physicians' perceptions of the relevance of receiving consistent patient outcome information. Referring physicians noted the various unsystematic ways in which they find out what happened to their patients. This included their own efforts to reaccess patients' medical records or call the PICU to obtain an update on patients. Sometimes, they received patient outcome information through third parties, such as colleagues, patients' family members, or via social media. Referring physicians underscored that knowing their patients' outcomes can help them evaluate and improve their practice. They also indicated that open communication between medical professionals who share patients across practice settings can improve transitions of care for critically ill children.

Development and Evaluation of the Feedback System Using a Human Factors Engineering Approach

Our team used the UIHC's existing Epic EHR platform to develop an electronic feedback system. Commonly used EHR platforms, such as Epic, have existing functionality, such as the ability to insert patient data into templated letters[15] [16] that can automate steps in the feedback process and can be readily adapted for implementation.[17]

We used a human factors engineering approach as applied to biomedical informatics projects to develop and test the EHR-supported feedback process. This approach uses a functional model of a human factors engineering lifecycle adapted for health care work situations and incorporates usability engineering methods to evaluate health care products and workflows.[18] Throughout development, we were guided by the Health Information Technology Safety Framework[14] [17] which provides a multidimensional sociotechnical approach for evaluating and ensuring safety of health IT in complex adaptive health care settings. This approach accounts for important interactions among clinicians, tasks/workflows, and technology that may present as facilitators or barriers to implementation. [Fig. 1] shows each phase of the project with its corresponding goals and methods.

Fig. 1 Overview of development and evaluation of an EHR-supported patient outcome feedback system using a human factors engineering approach. ED, emergency department; EHR, electronic health record; IT, information technology, PICU, pediatric intensive care unit; UI, University of Iowa.

Phase I: Analysis of the Sociotechnical System

To develop a deep understanding of the health care work system where the feedback process will be integrated, we performed semistructured interviews and focus groups with various stakeholders. During the same focus groups, we conducted with referring physicians above, in addition to determining the need for feedback, we also gathered information on physicians' clinical workflows and environments and elicited their perspectives on how an electronic feedback system may be best implemented. We also interviewed support staff (e.g., UI telephone operators), health IT/EHR platform specialists, and administrators (e.g., information officers and legal counsel), who we anticipated will play a role in the feedback process or have knowledge of resources relevant to delivering feedback. Finally, we reviewed organizational and state/federal data transfer regulations to be considered.

Phase II: Cooperative Design

We collaboratively designed the EHR-supported feedback system by translating stakeholder, organizational, and usability objectives (from Phase I) into feedback process and feedback report requirements. We designed a workflow for how feedback reports could be generated, delivered, and received. For each step, we identified tasks that need to be performed, who/what will perform the tasks (staff/clinicians versus EHR), and when/where tasks will be performed ([Fig. 2]). Stakeholders also designed the feedback report by considering clinical data relevant to ED clinicians and the EHR's capabilities to pull/push and summarize clinical data ([Fig. 3]).

Fig. 2 Feedback process systems design showing (A) overall feedback process requirements and (B) feedback workflow with functions of each step and information technology/human requirements. Epic applications and functions are shown since this was the specific EHR platform used for the project. ATC, admission and transfer center; ED, emergency department; EHR, electronic health record; IT, information technology; PICU, pediatric intensive care unit; UI, University of Iowa; UIHC, UI Hospitals and Clinics.

Fig. 3 Feedback report design showing (A) report requirements and (B) a prototype report generated in the EHR and sources of the report content. Epic was the specific EHR platform used for the project. EHR, electronic health record; PICU, pediatric intensive care unit.

Phase III: Implementation and Iterative Evaluation

Over 6 months, we completed three cycles of implementation, evaluation, and redesign. We generated feedback reports for all patients who were transferred to the PICU within 24 hours of a UIHC ED visit. As illustrated in [Fig. 2], patients eligible for feedback and their referring physicians were identified on the third PICU day and the initial feedback reports were manually generated by our team. Reports were then routed to PICU physicians who reviewed, edited, and sent them to ED physicians.

We also developed a set of evaluation metrics that would best reflect the feedback system's performance with regard to stakeholder goals while also considering the feasibility of data collection. These metrics and data sources are summarized in [Table 1]. We evaluated the feedback system's feasibility and timeliness of feedback delivery and receipt by reviewing EHR access logs to determine feedback report throughput. We analyzed feedback reports sent to determine the sources of feedback report content. We conducted usability testing[19] of the EHR interface with six PICU physicians, asking them to review, edit, and send actual feedback reports while thinking aloud.[19] [20] We audio-recorded physicians' comments and two research team members directly observed how they interacted with the electronic interface, paying particular attention to the functionality of relevant EHR applications (Epic In Basket and Epic Quick Letter) and problems in human–computer interaction. We also asked each physician to assess the process using the System Usability Scale (SUS), a validated 10-item survey to evaluate an individual's assessment of a system's usability. SUS scoring yields a composite measure between 0 to 100 which represents overall usability. Products with SUS scores >85 are generally considered highly usable.[21] Finally, we conducted semistructured interviews with PICU physicians and ED-referring physicians to determine how well the feedback process is integrated into physician workflows and to gather physicians' perceptions of the relevance of feedback reports to their clinical practice. Descriptive statistics were used to analyze and report quantitative data. Qualitative data from interviews were transcribed, coded, and synthesized into common themes.

Table 1 Evaluation metrics and data sources

Evaluation metrics	Data sources
Feasibility, timeliness, and impact on workload
Feedback report throughput[a]	EHR reports and access logs
Sources of feedback report content	Feedback report
Impact on workload	Direct observation, physician interviews
Usability of electronic interface
Time for PICU physician to complete editing and sending report	Direct observation
System Usability Scale score[b]	Rating by PICU physician
Navigation problems in electronic interface	Direct observation, physician interviews
Actions taken to confirm information automatically populated by the EHR or to seek additional information	Direct observation, physician interviews
Changes made to information automatically populated by the EHR	Direct observation, physician interviews
Relevance to clinical practice	Physician interviews

Abbreviations: EHR, electronic health record; PICU, pediatric intensive care unit.

^a Feedback report throughput includes the number of reports generated, routed to PICU physicians, sent to emergency department (ED) physicians, and reviewed by ED physicians and the length of time between these events.

^b The System Usability Scale (SUS) is a validated 10-item survey used to evaluate an individual's assessment of a system's usability. Scoring of the SUS yields a composite measure between 0 and 100 that represents overall usability. Products with SUS scores >85 are generally considered highly usable.[21]

Results

Feasibility and Timeliness of Feedback Delivery

Over 6 months, 119 feedback reports were generated and routed to PICU physicians. PICU physicians edited and sent 98 (82%) feedback reports. Referring ED clinicians received and reviewed 86 (72%) reports. Overall, feedback reports were sent by 10 unique PICU physicians and reviewed by 25 unique ED clinicians.

During the first two cycles of implementation, 21 (18%) feedback reports were inappropriately deleted before PICU physicians were able to review them. Our IT team identified that this was due to a preexisting custom Epic communication management workflow, wherein pended feedback reports were deleted when other users created and sent new unrelated reports using the same application (Epic Quick Letter). This was addressed by revising the programmed Epic workflow, preventing further loss of reports.

Otherwise, we found that the EHR-supported feedback process performed technically well. Majority of the feedback report content (86%) was automatically pulled in from the EHR, with only 12% of the content typed in by PICU physicians (most commonly to clarify/add diagnoses and include information on clinical course and status). Feedback reports were generated and routed to PICU physicians a mean of 3.4 days from PICU admission. PICU physicians completed and sent the reports a mean of 0.7 days after they were routed to them. Reports were reviewed by ED clinicians a mean of 1.8 days after being sent and a mean of 6.8 days from their encounter with the patient in the ED ([Table 2]).

Table 2 Technical characteristics of EHR-enabled patient outcome feedback system and update report

Technical characteristics	Mean or proportion
Update report content, total word count, mean (SD)	277 (59)
Information automatically pulled from EHR, word count, mean (SD, % of total)	239 (47, 86%)
Information selected by clinician from dropdown choices, word count, mean (SD, % of total)	7 (5, 3%)
Information manually added by clinician, word count, mean (SD, % of total)	31 (26, 12%)
Feedback process delivery metrics[a]
Time to routing of initiated update report to PICU physician from PICU admission, days, mean (SD)	3.4 (0.7)
Time to completion of report once received by PICU physician, days, mean (SD), days, mean (SD)	0.7 (1.9)
Time to report viewed by ED clinician once sent by PICU clinician, days, mean (SD)	1.8 (3.8)
Time to report viewed by ED clinician from ED visit, days, mean (SD)	6.8 (4.3)
Update report received by the correct ED clinician,  n (%)	90 (92)

Abbreviations: ED, emergency department; EHR, electronic health record; PICU, pediatric intensive care unit; SD, standard deviation.

^a Of 98 letters sent to ED clinicians by PICU physicians.

Usability

Direct observation of six PICU physicians (two physicians per iterative cycle), four of which were novice users (less than 10 feedback reports previously sent), showed an overall mean of 4.7 minutes to review, edit, and send feedback reports. However, nonnovice users performed faster (mean of 2.5 minutes). Usability testing with the same six physicians yielded a mean SUS score of 88.3, suggesting an “excellent” usability.[22] Key observations included difficulties in navigation and physicians confirming information by reviewing other sections of the EHR (both unique to novice users). Half of physicians did not edit diagnoses automatically pulled from EHR problem lists, while all physicians typed in additional information regarding the patient's clinical course and intermediate outcomes ([Table 3]).

Table 3 Usability characteristics of EHR-enabled patient outcome feedback system and update report

Usability characteristics[a]
Time for PICU physician to complete reviewing, editing, and sending update report (min), minutes, mean (SD)	4.7 (2.9)
System Usability Scale score,[b] mean (SD)	88.3 (7.9)
Key observations from usability testing with PICU physicians, n	Frequency (n=6)
Had difficulties with navigating from EHR messaging application to letter editing/sending application (only observed among novice users)	3
Needed to confirm information by reviewing other sections of EHR	3
Did not edit diagnoses automatically pulled from EHR problem list (i.e., did not include a narrative summary of diagnoses)	3
Added more information describing the patient's clinical course and outcomes	6

Abbreviations: EHR, electronic health record; PICU, pediatric intensive care unit; SD, standard deviation.

^a Usability testing was performed with six PICU physicians with varying levels of experience in sending feedback reports prior to testing.

^b The System Usability Scale (SUS) is a validated 10-item survey to evaluate an individual's assessment of a system's usability. SUS scoring yields a composite measure between 0 and 100, which represents overall usability. Products with SUS scores >85 are generally considered highly usable.[21]

Impact on Workload and Relevance of Feedback Reports to Clinical Practice

Interviews with six PICU physicians and four ED physicians revealed that the feedback process was well-integrated into their respective clinical workflows. Because reports were routed or sent to physicians' EHR in-baskets, sending and receiving reports closely aligned with the physicians' existing identical workflows for signing clinical notes. ED physicians also appreciated that receiving feedback reports within the EHR provided them easy access to additional information in the patient's chart if needed. Both PICU and ED physicians indicated that they sent/reviewed feedback reports at the same time that they signed their notes and that the overall time burden for completing/reviewing reports was acceptable. ED physicians noted that they received reports within an acceptable timeframe as well.

PICU physicians indicated that sending patient outcome feedback to their ED colleagues is a worthwhile endeavor and will overall help improve the emergency care of children prior to PICU transfer. ED physicians indicated that receiving patient outcome feedback is relevant to their clinical practice, since it helps them identify whether they have appropriately triaged patients (i.e., to the PICU vs. general wards) and provides them with a venue to reflect on potential improvements in their diagnostic reasoning and clinical management.

Iterative Changes

After the first cycle, we disseminated a navigation guide to PICU physicians, this supported the learnability of the process and served as a quick reference while the task was performed. We also sent e-mails to PICU physicians at the beginning of each clinical service week to remind them to review and send reports. This was phased out after 5 weeks when the process was well-integrated into the workflow. After the second cycle, we responded to questions from PICU physicians regarding specific clinical information that ED physicians found helpful (additional diagnoses since the ED visit, evolution of the clinical course, and patient disposition) by reporting back results of our interviews with ED physicians. We also revised the EHR access log reports we created to track feedback report throughput, so we can more clearly delineate time intervals between tasks. Our IT team also identified and addressed the software error which caused feedback reports to be inappropriately deleted, as described earlier. After the third cycle, we added ED resident physicians as feedback recipients at the request of the pediatric ED medical director. We also noted that significant time and effort was needed to manually identify eligible patients and their respective referring physicians and PICU physicians to ensure correct routing and delivery of feedback reports, suggesting the need for more automation in this process.

Discussion

In this case study, we showed that a semiautomated EHR-supported system to deliver timely and relevant PICU patient outcome feedback to referring physicians is technically feasible, usable, and relevant to both ED and PICU physicians at a single institution, sharing a common EHR platform. Although our system is distinct from prior published work, deploying electronic interventions to improve subspecialty referrals have been used in the past with varying success.[23] [24]

The main challenges we encountered included the need for PICU physician review of feedback reports (due to limitations in the EHR's ability to accurately summarize clinical data[25]), PICU physicians' clinical workload affecting timely completion of feedback reports, and the significant effort in manually identifying eligible patients and their corresponding ED and PICU physicians for the purpose of generating feedback reports for routing to PICU physicians. Expanding the electronic process to deliver feedback beyond our institution has also been challenging due to the limited interoperability of EHR platforms, making it difficult to send feedback reports between institutions with different EHR systems and even between those with the same but differently configured EHR system. Pending progress on this issue, delivering feedback outside of our health care organization will necessitate a hybrid electronic-paper (faxed reports) feedback process.

Limitations and Strengths

Our case study has limitations. The feedback system was tested in only one institution and only between two groups of physicians, although efforts are underway in our institution to expand the process outside of our health care organization in different clinical settings. We also only determined the technical performance and relevance of the feedback system and did not measure impact on clinical practice and patient outcomes which will be essential next steps.

Despite these limitations, we expect that the system we have developed can be adapted to other institutions and/or other clinical settings across the continuum of patient care. Although we used the Epic EHR platform to develop and test the feedback process, we anticipate that other EHR platforms have similar functions that can be used to build a similar feedback system. We recommend that institutions follow a similar step-wise human factors approach to development to ensure that the system is feasible in their setting and fulfills their specific feedback goals. The following are the key lessons we learned:

• Create a sense of serious urgency (a “burning platform”) to highlight the importance and need for feedback between clinicians in the target settings.

• Involve all stakeholders from the beginning, especially referring clinicians, to achieve consensus and clarity around feedback goals and the processes to be implemented to attain these goals.

• Understand the context of the health care system and the environment where the feedback intervention will be embedded in both clinical settings sending and receiving feedback, not just the IT or EHR platform to be used.

• Collaboratively design the feedback process involving all stakeholders.

• Identify important and feasible metrics to evaluate the performance and impact of the feedback process.

Conclusion

In conclusion, an EHR-supported system to deliver timely and relevant PICU patient outcome feedback to referring ED physicians was technically feasible, usable, and important to physicians. Next steps include wider implementation to referring clinicians from other institutions, evaluation of the feedback system's impact on clinical practice and patient outcomes, and investigation of potential applicability to other clinical settings involved in similar care transitions.

Clinical Relevance Statement

Patient outcome feedback is an important way by which clinicians can improve their performance and the care they provide to patients. Leveraging the EHR to support an electronic semiautomated feedback system that can deliver timely and relevant patient outcome information to clinicians is a feasible way to operationalize consistent clinician feedback. The feedback system we developed can potentially be adapted for implementation in other clinical settings involved in similar care transitions.

Multiple Choice Questions

1. Why is it important to conduct stakeholder interviews and focus groups before implementing a new information technology-supported process?

   • To determine perceived facilitators and barriers to the process to inform development

   • To inform stakeholders of the new workflow and specific tasks that they will now need to do

   • To convince stakeholders that implementation of the new process will be of benefit to them

   • To emphasize the importance of adhering to the new process

   Correct Answer: The correct answer is option a. It is important to determine stakeholders' perspectives regarding how proposed new processes can be facilitated or hindered within their specific health care environments to proactively leverage facilitators and develop solutions to minimize barriers for more successful implementation.

2. What data can usability testing provide that are useful to inform the development and evaluation of a new electronic process?

   • Functionality of electronic applications

   • Problems in the human–computer interface

   • Impact on clinicians' workload

   • All of the above

   Correct Answer: The correct answer is option d. Usability testing is important to (1) determine how well electronic programs perform their specified functions, (2) identify problems in the program interface that may interfere with human operation, and (3) measure the workload conferred by the new process (e.g., time to completion of tasks, effort required to complete tasks, and others).

Conflict of Interest

None declared.

Acknowledgments

The authors thank Elizabeth Beaman, BS, Raquel E. Gomez, BA, and Sonali Ramesh, MD, for their assistance in coordinating and collecting data from focus groups, interviews, and usability testing.

Protection of Human and Animal Subjects

This project was performed in compliance with the World Medical Association Declaration of Helsinki on ethical principles for medical research involving human subjects. It was reviewed and determined to be exempt from human subjects research oversight by the University of Iowa Institutional Review Board.

• References

• 1 Duffy FD, Holmboe ES. Self-assessment in lifelong learning and improving performance in practice: physician know thyself. JAMA 2006; 296 (09) 1137-1139
• 2 Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med 2008; 121 (5, suppl): S2-S23
• 3 Meyer AND, Singh H. The path to diagnostic excellence includes feedback to calibrate how clinicians think. JAMA 2019; 321 (08) 737-738
• 4 Croskerry P. The feedback sanction. Acad Emerg Med 2000; 7 (11) 1232-1238
• 5 Schiff GD. Minimizing diagnostic error: the importance of follow-up and feedback. Am J Med 2008; 121 (5, suppl): S38-S42
• 6 Graff LG, Chern CH, Radford M. Emergency physicians' acute coronary syndrome testing threshold and diagnostic performance: acute coronary syndrome critical pathway with return visit feedback. Crit Pathw Cardiol 2014; 13 (03) 99-103
• 7 Chern CH, How CK, Wang LM, Lee CH, Graff L. Decreasing clinically significant adverse events using feedback to emergency physicians of telephone follow-up outcomes. Ann Emerg Med 2005; 45 (01) 15-23
• 8 Gregory CJ, Nasrollahzadeh F, Dharmar M, Parsapour K, Marcin JP. Comparison of critically ill and injured children transferred from referring hospitals versus in-house admissions. Pediatrics 2008; 121 (04) e906-e911
• 9 Odetola FO, Clark SJ, Gurney JG, Dechert RE, Shanley TP, Freed GL. Effect of interhospital transfer on resource utilization and outcomes at a tertiary pediatric intensive care unit. J Crit Care 2009; 24 (03) 379-386
• 10 Walls TA, Chamberlain JM, Strohm-Farber J, Klein BL. Improving pretransport care of pediatric emergency patients: an assessment of referring hospital care. Pediatr Emerg Care 2010; 26 (08) 567-570
• 11 Han YY, Carcillo JA, Dragotta MA. et al. Early reversal of pediatric-neonatal septic shock by community physicians is associated with improved outcome. Pediatrics 2003; 112 (04) 793-799
• 12 Akcan Arikan A, Williams EA, Graf JM, Kennedy CE, Patel B, Cruz AT. Resuscitation bundle in pediatric shock decreases acute kidney injury and improves outcomes. J Pediatr 2015; 167 (06) 1301-5.e1
• 13 Cifra CL, Tigges CR, Miller SL, Herwaldt LA, Singh H. Updates to referring clinicians regarding critically ill children admitted to the pediatric intensive care unit: a state-wide survey. Diagnosis (Berl) 2020; 7 (02) 123-128
• 14 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: The Health IT Safety Framework. BMJ Qual Saf 2016; 25 (04) 226-232
• 15 Klappe ES, de Keizer NF, Cornet R. Factors influencing problem list use in electronic health records-application of the unified theory of acceptance and use of technology. Appl Clin Inform 2020; 11 (03) 415-426
• 16 Cheng DR, Katz ML, South M. Integrated electronic discharge summaries-experience of a tertiary pediatric institution. Appl Clin Inform 2018; 9 (03) 734-742
• 17 Cifra CL, Sittig DF, Singh H. Bridging the feedback gap: a sociotechnical approach to informing clinicians of patients' subsequent clinical course and outcomes. BMJ Qual Saf 2021; 30 (07) 591-597
• 18 Beuscart-Zéphir MC, Elkin P, Pelayo S, Beuscart R. The human factors engineering approach to biomedical informatics projects: state of the art, results, benefits and challenges. Yearb Med Inform 2007; 109-127
• 19 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76
• 20 Savoy A, Militello LG, Patel H. et al. A cognitive systems engineering design approach to improve the usability of electronic order forms for medical consultation. J Biomed Inform 2018; 85: 138-148
• 21 Brooke J. SUS: a retrospective. J Usability Stud 2013; 8 (02) 29-40
• 22 Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4: 114-123
• 23 Nakayama M, Inoue R, Miyata S, Shimizu H. Health information exchange between specialists and general practitioners benefits rural patients. Appl Clin Inform 2021; 12 (03) 564-572
• 24 Poku MK, Behkami NA, Bates DW. Leveraging the electronic health record to get value from referrals. Appl Clin Inform 2017; 8 (04) 1054-1056
• 25 Feblowitz JC, Wright A, Singh H, Samal L, Sittig DF. Summarization of clinical information: a conceptual model. J Biomed Inform 2011; 44 (04) 688-699

Publication History

Received: 20 December 2021

Accepted: 14 March 2022

Article published online:
11 May 2022