Ascertaining Design Requirements for Postoperative Care Transition Interventions

Abstract

Background Handoffs or care transitions from the operating room (OR) to intensive care unit (ICU) are fragmented and vulnerable to communication errors. Although protocols and checklists for standardization help reduce errors, such interventions suffer from limited sustainability. An unexplored aspect is the potential role of developing personalized postoperative transition interventions using artificial intelligence (AI)-generated risks.

Objectives This study was aimed to (1) identify factors affecting sustainability of handoff standardization, (2) utilize a human-centered approach to develop design ideas and prototyping requirements for a sustainable handoff intervention, and (3) explore the potential role for AI risk assessment during handoffs.

Methods We conducted four design workshops with 24 participants representing OR and ICU teams at a large medical academic center. Data collection phases were (1) open-ended questions, (2) closed card sorting of handoff information elements, and (3) scenario-based design ideation and prototyping for a handoff intervention. Data were analyzed using thematic analysis. Card sorts were further tallied to characterize handoff information elements as core, flexible, or unnecessary.

Results Limited protocol awareness among clinicians and lack of an interdisciplinary electronic health record (EHR)-integrated handoff intervention prevented long-term sustainability of handoff standardization. Clinicians argued for a handoff intervention comprised of core elements (included for all patients) and flexible elements (tailored by patient condition and risks). They also identified unnecessary elements that could be omitted during handoffs. Similarities and differences in handoff intervention requirements among physicians and nurses were noted; in particular, clinicians expressed divergent views on the role of AI-generated postoperative risks.

Conclusion Current postoperative handoff interventions focus largely on standardization of information transfer and handoff processes. Our design approach allowed us to visualize accurate models of user expectations for effective interdisciplinary communication. Insights from this study point toward EHR-integrated, “flexibly standardized” care transition interventions that can automatically generate a patient-centered summary and risk-based report.

Keyword

continuity of care - care transition - requirements analysis and design - handoffs - surgery - anesthesia - intensive and critical care - machine learning

Background and Significance

Handoffs, defined as the transfer of patient information, responsibility, and accountability from one clinician to another, are essential for ensuring care continuity during transitions of care.[1] Postoperative handoffs from operating room (OR) to intensive care unit (ICU) settings require an orchestrated coordination of both physical patient transfer in conjunction with transfer of information, responsibility, and accountability between interdisciplinary teams representing anesthesiology, surgery, and critical care.[2] However, they can be vulnerable to communication breakdowns, technical errors, and environmental distractions,[3] [4] [5] leading to process failures.[6] [7] [8]

Standardization using process-based protocols[9] and structured information transfer checklists[10] are implemented to mitigate these care transition failures. Initial evaluations suggest that these standardized strategies were successful in reducing information loss, technical errors, and process defects while increasing clinician satisfaction and teamwork.[9] [11] However, based on our recent systematic review, we identified inconsistent evidence on effectiveness of current handoff tools has been inconsistent and mixed[12] [13] coupled with limited intervention sustainability over time,[14] which can be partially attributed to current tool limitations. Primarily, postoperative handoff tools were (1) lacking support for interdisciplinary teamwork and anticipatory guidance during handoffs, (2) paper based[15] [16] with few exceptions,[17] [18] [19] [20] [21] and (3) focused on improving standardization of process-driven protocols[18] with limited support for supporting communication interactions and coordination needs of receiving teams.[22]

To address these limitations, we conducted a user-centered study[23] to explore design requirements for an electronic health record (EHR)-integrated intervention to support effective, efficient, and interactive handoffs supporting interdisciplinary team workflows. We also examined the potential roles and integration of artificial intelligence (AI) and machine learning (ML) via the EHR[24] to augment handoffs that can foster anticipatory management by summarizing scattered EHR elements into concrete risks for the patient.

Objectives

Our three-fold study objectives were: (1) to identify factors affecting sustainability of handoff standardization, (2) to utilize a human-centered approach to develop design ideas and prototyping requirements for a sustainable handoff intervention, and (3) to explore the potential role for AI risk assessment during handoffs.

Methods

Study Setting and Participants

The study was conducted at a large academic medical center with 1,249 staffed beds. Among the 2019 to 2020 discharges, 14,488 surgical patients were transferred from OR to ICU. On-site hospital units included the OR, cardiothoracic ICU (CTICU), surgical ICU (SICU), and neurology–neurosurgical ICU (NNICU). Patients admitted to these ICUs are also remotely monitored by an electronic ICU (eICU), a telemedicine center staffed by ICU clinicians for additional surveillance, and 24/7 support. Participants were recruited with the support of residency and nursing coordinators using a convenience sampling approach.

Existing Postoperative Handoff Protocol

A standardized process-based protocol supported postoperative bedside ICU handoffs.[19] The protocol included (1) process steps to be followed during handoffs and (2) an information transfer report template ([Fig. 1]). Although laminated protocol copies were available at bedside for reference, there were no formal handoff documentation tools. The eICU team observed all handoffs.

Fig. 1 Institutional OR–ICU standardized handoff protocol. ICU, intensive care unit; OR, operating room.

Data Collection

We conducted design workshops in three phases to (1) obtain clinician insights on the current handoff protocol, (2) identify requirements for a handoff intervention with support for communication and documentation, and (3) explore AI integration into our risk assessments to augment postoperative handoff communication ([Fig. 2]). Workshops were audio-recorded and led by C.R.K. (clinician) and J.A. (qualitative expert).

Fig. 2 Phases of design workshops (supported by focus groups).

Phase 1: we used a semistructured guide to discuss the group's perceptions about the current handoff process and gather perspectives on effective OR–ICU handoffs. Participants were oriented to the goal of an integrated handoff intervention and asked to consider how it might fit into their workflow.

Phase 2: we elicited information requirements for an EHR-integrated handoff intervention with closed card sorting. Card sorting[25] was used to explore users' preferences on functionality, overall structure, navigation, and labeling.[26] Participants were given a list of content elements based on prior studies.[1] [19] Labeled sticky notes were sequentially placed on a board visible to all participants. Through group brainstorming, participants discussed and modified their ranking decisions, adding additional elements through nomination.

Phase 3: we adopted a scenario-based design ideation approach to gather intervention design ideas and elicit feedback on low-fidelity intervention sketches (printed sheets).[27] Handoff scenarios were drawn from our retrospective database.[28] [29] Additionally, to examine the potential utility of AI- and ML-generated risk assessment during handoffs, we supplied cross-validated risks for adverse events[28] [29] [30] (acute kidney injury [AKI], arrhythmia, pneumonia, acute heart failure, delirium, reintubation, unplanned ICU admission, wound infection, and venous thromboembolism). Three scenarios representing a diversity of adverse event risks were selected ([Fig. 3]).

Fig. 3 Scenarios used for design ideation and prototyping (phase 3). AKI, acute kidney injury; CAD, coronary artery disease; EBL, estimated blood loss; VTE, venous thromboembolism. *Substantially elevated risk for adverse events using published methods.[28]

Clinician participants were given scenario narratives, printed deidentified assessments, and anesthesia records. They were also shown the ML-risk predictions used to screen cases in a variety of formats (tabular, graphical, decomposed in force plots,[31] relative and absolute scales, and various reference points). “Important variables” for the prediction were identified using Shapely values[31] and permutation-based importance. Similar questions from phases 1 and 2 were used to gather their perspectives within the context of these scenarios in phase 3.

Data Coding and Analysis

After reading focus group transcripts multiple times, we assigned each statement with data-driven (or open) codes.[32] Similarities and areas of overlap between codes based on relationships were identified to synthesize unifying codes into subthemes. Finally, these subthemes were compared against each other based on similarities to generate higher level themes within and across transcripts. This involved multiple rounds of review and refinement based on theme relevance to our study objectives (see [Supplementary Appendix A] for coding example [available in the online version]). All transcripts were independently coded by authors (J.A. and A.M.), and all discrepancies were discussed to achieve team consensus. Information elements from card sorting were tallied based on both frequency of clinician selection and ranking of importance.

Results

A total of 24 participants (5 clinical anesthesiology fellows, 9 ICU registered nurses, and 10 anesthesiology/critical care residents) participated across four design workshops were conducted. Each workshop lasted an hour on average. We report on major themes identified: factors affecting handoff protocol use and sustainability, intervention components and role of ML-generated risks in handoff intervention, and intervention design requirements and implementation features. Representative quotations and additional data are provided in the [Supplementary Appendix B] (available in the online version).

Factors Affecting Postoperative Handoff Protocol Use and Sustainability

Two major factors impacted handoff protocol use and sustainability. First, over half of the clinicians believed there was a lack of awareness about the current standardized protocol. Without ongoing training, compliance with the standardized protocol was perceived to be limited. Residents especially reported that although some of them knew about the protocol, no one adhered to it: “I've never used it, nobody goes by (the protocol) ….” (Res-4)

Second, the EHR failed to adequately support effective information transfer during handoffs. There was limited awareness on “how to” access and interpret the pre- and intraoperative information found in the anesthesia record by ICU clinicians, “where to” document handoff information by OR clinicians, and “what” information in the record was critical and essential for maintaining care continuity versus irrelevant.

“If you want to look at the actual intra-op and look at what was going on in a concise, easy format, you have to actually look at the intra-procedure tab on the anesthesia thing. Otherwise it's kind of confusing and disorienting.” (Res-1)

Almost all clinicians agreed that anesthesia details were often hard to find but important to include in a handoff intervention.

Handoff Intervention Elements

Core Elements

The importance of elements was determined through card sorting and open-ended interview questions. [Fig. 4] shows how frequently participants chose each element. Elements viewed as necessary by more than half of our clinicians were considered core elements. A full list of participant-selected elements is presented in the [Supplementary Appendix C] (available in the online version).

Fig. 4 Frequency of element selection. Red text indicates free-text suggestions from participants to include in our intervention. BMI, body mass index; OSA, obstructive sleep apnea; EBL, estimated blood loss;

Among participants, residents tended to mark more elements as important to include in the intervention, while fellows and nurses tended to ignore items such as “age” and “preoperative diagnosis.” [Table 1] shows the frequency with which elements were viewed as important by participants. Additional elements included diuretics given (once), paralytic reversal given (four times), and endotrachial tube size and position (once).

Table 1 Prioritization of content elements

Element	No. of participant votes who viewed these elements as one of the top 10 most important elements to include
Age, EBL, blood products given, difficult intubation	10+ participants
Preoperative diagnosis, scheduled procedure, duration of anesthesia, crystalloid given, colloid given, antibiotics given (and time), neuromuscular blockade given (twitches and TOF), opioid analgesics given, arterial line present, central line present, preinduction vital signs	6–9 participants
Allergy list, regional block present, high-risk OSA, height/weight/BMI, average vital signs last 15 minutes, β-blockers given/continued, premedication given, insulin given	5 or fewer participants

Abbreviations: BMI, body mass index; EBL, estimated blood loss; TOF, train of four; OSA, obstructive sleep apnea.

Flexible Elements and Machine Learning–Generated Risk Predictions

Items that could be included depending on patient pre- and intraoperative management and postoperative risks, such as risk of VTE, intraoperative abnormalities, risk of pneumonia, 30-day mortality, and risk of AKI, were regarded as “flexible elements.” Most residents and nurses believed lines, average vital signs within the last 15 minutes, and insulin given were important to include, while fellows did not. In contrast, many critical care fellows found that anesthesia providers tended to focus on intraoperative management details of little meaning to them, stating they only wanted to know intraoperative information if it directly affected their ICU care.

“Why do we care about the dose of fentanyl? Why do we care about the opioid dose? (The anesthesiology team is) like, ‘We gave 600 of fentanyl.’ I'm like, ‘I don't care.’ Why do I care how much you gave intra-op? I don't know. I don't know how that affects me.” (Fellow-2)

More than two-thirds of our clinicians were only interested in actionable or modifiable risks. Information that would not affect care was irrelevant to them, and some clinicians worried that receiving reports on nonimmediate concerns would only increase their workload.

“We get a lot of the global (concerns) from teleICU too. It would just be double the work.” (RN-7)

There was, however, significant interest in the comparison of case patients to the average patient pre-, intra-, and postoperatively. Both nurses and residents stressed that understanding baselines were crucial in interpreting intraoperative data.

“I think having their pre-op info is good because then we know what their baseline is. As far as the averages for everyone else, it gives us an idea of where they should be as opposed to where they are, which is useful, and then what they actually are. So that way we know before surgery their baseline function was this. After surgery it should be this, but theirs is actually this one so we know if something's going well or something didn't go so good.” (RN-4)

Those responsible for interpreting risk information and adjusting patient care plans accordingly only reported interest in significantly elevated risks and use of various thresholds to distinguish risk severity. There was mixed feedback on how awareness of risks would affect patient management over the course of patients' ICU stays. While residents said they could use the risks to develop patient-centered care plans, nurses and fellows believed this risk information would not affect their patient care.

Unnecessary Elements

Elements including allergies, transfusions, and most intraoperative medications were considered unnecessary to include within an EHR-integrated handoff intervention. Nurses believed antibiotics can also be excluded. High-risk obstructive sleep apnea (OSA), height, weight, and body mass index (BMI) were frequently noted by receiving clinicians as never to be included in the intervention but garnered some interest from residents on the sending team.

Most clinicians reported that unnecessary information would lead to information overload and clutter. Fellows were particularly vocal in their beliefs that only crucial or hard to find information should be included in the intervention.

Clinicians had mixed feelings on whether information that was verbally communicated should also be documented on an intervention. Half of our clinicians believed verbal report information should be included while the other half believed it should not.

“I feel like those things … could be verbally communicated to the team that's receiving the patient, and then at 2 hours later, nobody cares. So the fact that this is going to stay in the patient's chart for a week while they're in ICU—I don't care anymore. I feel like a lot of that will clutter up the sheet and make it much harder to get the couple big things you want to see out of it.” (Fellow-2)

Nurses further explained that they heard three different accounts of a patient's status from three different types of clinicians, and felt that confirming information accuracy (and getting the facts straight) was difficult when accounts were not necessarily reliable. Therefore, verbal report information should also be included in an EHR intervention so they could verify the documented information during handoff communication.

Intervention Design Requirements and Implementation Features

Structural Presentation Format and Visualization Considerations

Clinicians suggested that a yes/no format could be implemented to present certain core elements (e.g., airway) across the header of the intervention, similar to how the EHR interface provided information in the past (A.1.10).

“And [information access to certain patient information] it actually used to be easy. And now, since they reformatted, I think it's hard. Because it used to be when it was in the—they had the header at the top, if you clicked on where it said Difficult Airway ‘Yes/No,’ it would actually bring up their most recent intubation document.” (Fellow-1)

When asked about the format for presenting risks, clinicians unanimously preferred absolute risk statements over percentiles and effect sizes.

“I mean in certain things, there's certain criteria that are gonna be elevated in different patients that's gonna make [patients susceptible to] VTE likely. But looking at the specifics on that particular patient that put them over the top that may be helpful.” (RN-2)

All nurses and some residents strongly preferred qualitative risk descriptions over absolute numbers. Residents particularly preferred graphs to visualize risks.

“For me, I would just like the graphs. Everything else would be too much data. But I'm not (other resident's name), so ….” (Res-9)

Irrespective of presentation choice, all clinicians strongly desired explanations of risks (to understand features contributing to predictions) in addition to the absolute score/qualitative narrative. These clinicians felt that knowing which pre- and intraoperative features explained the elevated risks would provide insights for postoperative management.

Intervention Modality and Access Considerations

Clinicians believed that a handoff intervention integrated into the EHR would be more useful than paper. Additionally, residents and fellows thought they would prefer to access an EHR-integrated intervention directly over a phone or computer, as they tended to be more mobile; however, the only concern was the lack of computer access in certain instances. Nurses stated that having a snapshot of the patient handoff within the EHR would “help (them) take care of the patient and anticipate needs …” (RN-1). However, they preferred to print the intervention form for personal use and control (i.e., editing, perusing information) at bedside.

“We know the patient's coming and we can go into documents and their chart and just automatically print it before they come. We would like if we have control of (printing) it.” (RN-4)

Nevertheless, regardless of the modality, all clinicians preferred to access the intervention before bedside handoff to better prepare and used time during the verbal report to ask appropriate questions.

Discussion

As recommended by the Joint Commission,[33] several U.S. hospitals have implemented handoff tools that adhere to structured information transfer and standardized handoff processes to improve safety during care transitions.[11] While these tools improve rate of information transfer, reports suggest limited sustainability in certain process and clinical outcomes over time.[9] [34] Furthermore, operative details are often prioritized over anticipatory guidance.[22] [35] As suggested in this study, this might be due to the inclusion and prioritization of some elements in standardized interventions or patient information irrelevant to specific postoperative care. Ascertaining which data elements are relevant to the receiving care team is crucial in preventing information overload and reducing the risk for care transition failures.[36]

Furthermore, in a study conducted on individual clinician performance, standardized lists of risks were seen to drive action in only a few clinicians.[37] Hence, a balance between standardization and adaptive flexibility is necessary to ensure timely and seamless patient care.[38] This is consistent with our findings that point toward communicating individualized, situational topics, such as postoperative risks, that are critical for implicit handoff functions (e.g., anticipatory guidance and contingency planning). These points of communication prepare the receiving team to better manage postoperative complications and anticipate related resource needs.[6]

Adaptive and patient-centered handoff interventions can potentially mitigate some of the standardized protocol compliance issues along with interdisciplinary teamwork gaps. In developing these interventions, we can streamline the handoff process, support transfer of core elements (pre- and intraoperative), highlight flexible elements including ML-generated patient-specific risks, promote a shared understanding about expectations (or “common ground”) among interdisciplinary teams, and, lastly, require minimal clinician effort for handoff preparation with EHR integration.[10]

Furthermore, we emphasize that any adaptive handoff tool is meant to augment rather than replace verbal handoff communication. For example, electronic tools cannot include information which has not been charted (e.g., subjective assessments and rationales). Our study pointed to important pieces of information that may not be documented before handoffs (e.g., extubation details, sedation for transport, or rescue medications immediately before or after extubation). Like any other form of missing data, incomplete charting/documentation can reduce accuracy of risk predictions. However, EHRs include time of handoff documentation, and deep learning techniques can both impute missing data and recognize missing data patterns.[39] [40] [41] Automatic identification of “probably missing” data in a handoff tool can potentially remind the sending team to fill-in potential documentation gaps during the verbal exchange.

Data visualizations are commonly used to communicate risks; however, design and presentation of these risks are crucial in influencing risk perception and decision-making.[42] Risk perception and accuracy of participant inference was the greatest in prior studies when icon arrays were present.[43] Additionally, the ability to see disease or risk progression was crucial when considering treatment options during development of clinical risk report intervention prototypes, simple graphs and designs were preferred and complex visualizations were rarely utilized to their full potential.[44] These preferences aligned with our findings, where many clinicians stated that they preferred simple, color-coded graphs for visualizing trends.

Limitations

Our study comes with limitations. First, given the exploratory nature of this work, the study used a small sample size of participants from a single site, with an uneven distribution of clinicians. This mixed cohort and distribution may skew which elements were prioritized. However, the main intent of our user-centered design (UCD), the study was to focus on exploring innovative design ideas and conducting low-fidelity prototype evaluations to ensure the development and implementation of a user friendly intervention that can be easily integrated within the clinical workflow and the EHR system. Prior work on similar UCD methods found that the number of stakeholders typically involved is low, commonly between 6 and 12 users per focus group and 10 to 20 users involved in card sorting.[45] Second, the group dynamics underlying the design focus group workshops varied. While we facilitated discussion among participants, we did observe participants who dominated certain conversations and at times, swayed the opinions of others. We attempted to mitigate this effect through a multimethod approach to ensure individual opinions were collected without discussion with other participants. Third, during our design workshops, physician participants provided their perspectives, both as a sender and a receiver given their clinical practice and experience in both roles. We were hence unable to make concrete distinctions between element preferences in our analysis. To address these limitations, we are recruiting additional participants for a more balanced distribution of both sending and receiving teams including surgery and certified registered nurse anesthetists. Lastly, we acknowledge that clinician overreliance on adaptive postoperative handoff interventions can be prone to information omissions. However, we believe that such interventions should serve as cognitive aids supporting handoffs, similar to clinical decision support systems.

Conclusion

Current postoperative handoff tools focus largely on standardization of information transfer and care transition processes. Our human-centered methodology allowed us to glean clinician perspectives about OR–ICU handoffs along with an accurate model for individual and shared team expectations for effective and efficient interdisciplinary care transitions. Insights from this study point toward an EHR-integrated “flexibly-standardized” or adaptive care transition intervention with an AI-generated, tailored handoff patient summary and risk-based intervention that is easily accessible to interdisciplinary sending and receiving teams.

Clinical Relevance Statement

Our study highlights addressable barriers to handoff standardization and also characterizes handoff information elements that are critical for designing postoperative care transition tools and can address these barriers. Furthermore, our findings suggest that EHR-integrated “flexibly standardized” tools with artificial intelligence can potentially augment effective and efficient interdisciplinary communication of postoperative care management goals essential for anticipatory management and contingency planning, especially for high-risk patients.

Multiple Choice Questions

1. What kind of intervention is acceptable to all clinicians throughout the postoperative handoff process?

   • paper only

   • EHR-integrated only

   • EHR-integrated and printable

   • verbal only

   Correct Answer: The correct answer is option c.

2. What kind of handoff intervention might satisfy the need for standardized information without including details unnecessary to patient-specific cases?

   • physical step-based handoff protocols

   • multiple simultaneous mini-handoffs

   • handoffs over the phone

   • flexibly standardized information checklists

   Correct Answer: The correct answer is option d.

Appendix A

Example of thematic analysis

Clinician	Interview transcript text	Open code	Subthemes	Theme
Fellow	“Well, that just in the ICU, paper is not useful. In my opinion, if it's going to be a piece of a document of some sort, it would be more useful if it's in the computer.”	Paper versus electronic report Intervention usefulness	Clinicians preferred risk reports in an electronic format for convenience and easy access in the future	Intervention format
Registered nurse	“Where it's something in epic you can click on, choose to read it. Not necessarily print because once you print it you have HIPAA, confidentiality, you have people's stuff laying around the unit. Families walk by, ‘Oh look, there's my dad's …’ the more you print, the more there it's out there to be seen, so I would just leave it.”	Ease of access Patient risks Confidentiality

Abbreviation: ICU, intensive care unit.

Appendix B

Themes and related participant quotations

Clinician content requirements	Participant quotations
Details of the anesthesia that are harder to find are more important to put on the risk report	“I didn't circle age, allergy list, BMI because that stuff is super easy to find in epic. I put more of the harder to find details of the anesthesia. I put the stuff that I always want to hear on handoff, but that if you weren't there for handoff or it wasn't documented well, it disappears. Or you have to go through the anesthesia record to find.” (Fellow 2) “Usually what vasopressors they're on (is difficult to find), too. Antibiotic dosing sometimes. That's still difficult to find.” (CA1 9) “I'd like to see more surgical … “What are their restrictions, what can we give them out of bed? Do they need to be on bed rest, do they need a brace, do they need to be weight bearing, do they need to be not weight bearing? Do their brains need. What drain is this???” Can we feed them? Can we not feed them?” (RN 3) “Transfusions. Access, tubes, direct … yeah. Yeah, that's difficult to see or figure out. Sometimes they forget to tell you how much opioid they use.” (CA1 6)
I care about intra-operative abnormalities and reports on dosing/intraoperative care only if it affects their ICU care	“I mean, I might care if they're extubated and then they're becoming somnolent. If they're intubated, usually we don't care.” (Fellow 1) “They (nurses) love hearing about access. They're very upset if there's not enough access.” (CA1 3)
The airway is important to put on the report	“And I feel like, in a difficult intubation, data that gets printed out—I feel like a yes/no is useful, but I also feel like the method that was used intraoperative is useful too because if it was not a difficult intubation, but it was video laryngoscopy with a bougie, that is still is not an easy airway.” (Fellow 2) “I think the nice thing about having a difficult airway always included in there is because it will prompt you to ask, like you said, because sometimes an airway might not be difficult for you, but for one of the surgery fellows who's never intubated before, might have difficulty with it.” (Fellow 3)
Unnecessary information
Unnecessary information clutters the report and makes me lose interest	“I agree that I don't care about its effect, but I feel like I want a piece of paper saying 6 hours or 13 hours.” (Fellow 2)
I think easy things like arterial/central line or duration of case are not important to put on the report	“I think it would be very cluttered … You can make the same argument that I don't really care how much fluids and EBL they have on that form. I can just look it up.” (Fellow 1) “I also don't care about [presence of a] central line. I can also look at them and then know that. Or duration of anesthesia. I mean, we kind of know how long a case is.” (Fellow 2)
Stuff that is communicated verbally during the handoff or on the way up does not need to be in the report	“Yeah, I feel like a lot of that will just clutter up the sheet and make it much harder to get the couple big things you want to see out of it.” (Fellow 2) “I can just look it up. I don't care what medications they've been given or (sedating) premedication they've been given.” (Fellow 1)
Risk specifications
Clinicians are interested in risks	“I think we wanna know why so we can prevent it early. I think that would be helpful.” (RN 1) “The top 5 reasons.” (RN 3) “I agree that I don't care about its effect, but I feel like I want a piece of paper saying 6 hours or 13 hours.” (Fellow 2)
Clinicians are especially interested in relative risks, comparison of current patient to the average patient preoperatively, intraoperatively, and now postoperatively	“I think that is a useful comparison and maybe the biggest reason or reasons why. Yeah, if the patient had persistent hypotension interop, and now they're at a much higher risk of AKI and a much higher risk of mortality, that's fairly useful.” (Fellow 1) “I think having their preop info is good because then we know what their baseline is. As far as the averages for everyone else, it gives us an idea of where they should be as opposed to where they are, which is useful, and then what they actually are. So that way we know before surgery their baseline function was this. After surgery it should be this, but theirs is actually this one so we know if something's going well or something didn't go so good.” (RN 4) “What the patient's baseline is, Like so, they know what to expect if the patient is totally normal before they went and now they're not oriented at all versus “Oh I picked them up in preoperative and they couldn't say their name. That's good to know so they have a baseline expectation.” (CA1 2)
Clinicians are interested only in actionable or modifiable risks	“I don't really care if they have a 1% increased chance of mortality. But if it's for a reason that might change something you do?” (Fellow 2) “… (Before) the surgery, I wanna know what the 30 day mortality will be to determine whether they need the surgery. But after the fact, it's like ‘Oh, there's an ICU handoff with thirty day mortality is.’ I don't care. I'm going to do whatever I can to keep them alive right now, I don't need to know what percentage chance of living the next 30 days is.” (CA1 1) “I'd wish for the longer term stuff, to be able to find it, but I find that when I take a new patient, I really don't care about their anticipated length of stay. However, I'd take both general and narrow risk reports.” (CA1 7)
Clinicians are interested in elevated risks	“I think only ones elevated because I can't see a reason if we don't include them, what is the consequence of not knowing.” (Fellow 1) “Yeah. I feel like for all these, I really only care if they're at significantly higher risk and then maybe if they're at significantly lower risk of afib in cardiac surgery patients. Other than that, I don't care if it's about average. I guess if they're at significantly elevated risk above the average for the procedure or if they're just at significantly elevated risk, period.” (Fellow 2)
Clinicians are interested in risk thresholds to distinguish risk severity	“Probably depends on the individual complication. And maybe it should be some amount higher than just the general risk for all people.” (Fellow 1) “30%. Above 30%.”—Fellow 2 (a popular sentiment among many clinicians was the 30% risk threshold)
Clinicians preferred absolute risks over percentiles and effect sizes	“I don't want to see the relationship that the computer imputes between the time and the risk. I don't want to see the effect size. They do absolute numbers but also is showing you—I thought they sometimes show you where you are in the population as well as red, yellow, green, or something like I think both of those are valuable. A lot of times, especially if you're having a discussion with family about risks of stuff surgeries, further surgeries or a further course of action, having the absolute values could be useful.” (Fellow 1)
Risk report format
Clinicians preferred absolute numbers instead of qualitative risk explanations (explanation)	“I mean, my thought is that I just can't work out—it effects it a lot. I don't necessarily need the number.” (Fellow 1) “But I do still like seeing the absolute numbers. Yeah, especially, like you said, for the ones that are low. I can see that, whatever it is, death, is really small. And so I can just look over and like, ‘Oh, it's 2.3%.’”—Fellow 2 (Interviewer asked if uncertainties mattered) “Not really. But I just mean, maybe we can cap it at 50 or 40%, depending on how high the risk predictors go in general.”—Fellow 2 (Interviewer: But stick with absolute instead of relative and keep them all on the same scale?) “Yeah.” (Fellow 2) “I don't really care for the picture, but different people are different. The absolute numbers are great.” (Fellow 3)
Clinicians had mixed feeling about numbers versus graphs for visualizing risks (visualization)	“I don't. I don't really care for the picture, but different people are different. The absolute numbers are great. [The diagram] takes too long to figure out.” (Fellow 3) “Graph. Just give me a bunch of bars and make them green and red.” (CA1 9) “Yeah. I like that format. Could we just have that? (National Surgical Quality Improvement Program format) Yes. It's really easy to look at. It gives you the risk, a visualization of how low and how high and where it is, relative to the norm. Yeah. But if nothing goes to 60% or above or something, we don't need that percentage on the sheet ever.” (Fellow 2) “For me, I would just like the graphs. Everything else would be too much data. But I'm not (other resident's name), so.” (CA1 9)
Clinicians are interested in explanations of risks	“Yeah. In that case, I would care more about why. But you would want to know that the elevated risk of mortality, even though all patients of that surgery have an elevated risk.” (Fellow 1)
Effect of the intervention
Clinicians argued that awareness of risks can impact their care management (impact)	“Almost 99% of the time I make changes to the default.” (CA1 2) “It would emphasize this perceived risk more. I would want them to start thinking of specific diagnoses to tell the PACU nurse or attending at the end of the case to say “Hey, consider these things that we had already thought of before” so they could have that heuristic and say hey, these are those things.” (CA1 7) “I would say people with lung disease, I order less opioids. If they need them, ask for them, have someone come reassess them. It does change those orders. Or if I think they were really sensitive to [medication], I'll mention that.” (CA1 4) There's not much on here that would change what I do. Because if they don't have AKI now. And we renally dose it anyways.” (Fellow 2) “I don't honestly know. We're still gonna do everything we can to prevent pneumonia, to prevent wounds. It's not like we're gonna try harder on this patient because the number says they're at a higher risk. We try all of them pretty much the same. They're all considered ‘high-risk.’” (RN 1)
Access to intervention
The report would be passed on, not just used in the moment	“I think it would still be a cool thing to have on their profile just so that when you have time to flip through and see it. Because it's a snapshot, isn't it? It would be a snapshot at the moment right before they're gonna change hands. This helps in predicting a statistic problem.” (RN 1) “I think it's something like that would stay. Cause it's a good prompt.” (RN 3)
Clinicians preferred risk report in an electronic format	“Well, that just in the ICU, paper is not useful. In my opinion, if it's going to be a piece of a document of some sort, it would be more useful if it's in the computer. Because if it's a piece of paper that you put in the thing here, that should ultimately get in the EMR as well.” (Fellow 1) “So like a mini H&P. That sounds like a nice–And then we may or may not want to print it if we have access to it then we may not need to print it and if this is just the OR, then.” (RN 1) “One of those stickies on the upper left-hand side of epic, you could like click on it and it would open up a new window. Paper would get lost.” (CA1 7)
Clinicians preferred to access the risk report before the handoff as opposed to during the handoff	“We'd like to be prepared.” (RN 3) We know the patient's coming and we can go into documents and their chart and just automatically print it before they come. Like we had control over how to get it. We would like if we have control of (printing) it.” (RN 4)
Current factors affecting handoff tool use
The current protocol is not followed	“Yes [we follow the protocol and listen to each other] I've never used it, nobody goes by. They use those sheets a lot during rounds but not so much during ICU handoffs.” (CA1 4) “Sometimes that's our attending wanting to get out of there before we even start report and so they're kind of instigating getting that information. I think [the protocol is] useful when they use it.” (RN 3) “… No (they [surgeons] don't really listen). A lot of times, whoever does the surgery isn't the person taking care of them ... So, whether that information gets transferred to the person is … doubtful.” (Res-1)
ICU clinicians use the presented information and their own review to understand patient care trajectory	“I think both. We usually do a bit of review and see what they did to compare and see if they match up.” (CA1 2) “I think most of the time they're gonna send new labs but it's not because they're not integrating the information I'm giving, it's just like, seems like it's the standard.” (CA1 9)
The eICU is currently not proactive and it would be helpful to utilize them further	“I feel like having an automated acuity of something we could look at of like ‘This is why they are sick, these are the criteria that make them sick.’ I think for the charge nurse it would be helpful to place the patient and prepare for the patient.” (RN 3) “EICU generates. They send us reports. We tend to throw them in the trash unfortunately but they can send you the report with risk of like. “These would be people that would be your best moves.” (RN 4)
Details that should go into the handoff report or be documented within the EHR get lost along the way if the providers are too busy, and communication amongst clinicians is tricky	“I've also seen too the surgery attendings sometimes are not. They don't come up for handoff so there's a breakdown between surgery attendings or fellows and that doesn't get relayed and suddenly a panic strikes at the 12-hour mark when we haven't got some tests. So you're right, it should be an order, but if no one puts it in, it doesn't exist.” (RN 2) “They're often checking their phones for … something I don't know.” (CA1 8)

Abbreviations: AKI, acute kidney injury; BMI, body mass index; EBL, estimated blood loss; HER, electronic health record; eICU, electronic intensive care unit; EMR, electronic medical record; OR, operating room.

Appendix C

Element preferences according to clinicians

Core elements
• Age • True procedure • Recent intubation/difficult airway • Anesthesia details (duration of anesthesia, EBL, blood products, and regional block) • OSA risk • Reactions to medication intra-operative • History of paralytics' issues • Malignant hyperthermia • Postoperative care plan/discharge plan • Surgeon notes (expectations for next dose and serious complications) • Positioning restrictions • Postoperative imaging • Fluids • Antibiotics • Restrictions (bed rest and diet) • Short-term concerns • Drain orders • Clinical overview (baseline and perception of patient status in relation to baseline) • Extubation plan • Risk of atrial fibrillation • Risk of stroke • Volume status • Tubes
Flexible elements
• Intraoperative abnormalities • Opioids • 30-day mortality • Method of intubation used intra-operatively • Risk of AKI • Risk of VTE • Risk of pneumonia • Length of ICU stay • Dressing changes • Urine output • Repeated surgery • Repeated debridement • Difficulty of closing • Reaction to anesthesia/medications • Interventions for the floor • Desaturation events • Vasopressors • Cardiology consult
Elements to never include
• Allergy list • Medications given • Duration of anesthesia • Central or arterial line • Intraoperative medications • Transfusions • Antibiotics • Drain line

Abbreviations: AKI, acute kidney injury;, EBL, estimated blood loss; ICU, intensive care unit; OSA, obstructive sleep apnea; VTE, venous thromboembolism.

Conflict of Interest

None declared.

Acknowledgments

We would like to thank our clinicians for their participation. Work included in this document was produced by machine learning risk predictions at operating room–intensive care unit handoff. This work was produced with the support of the Big Ideas Program, a BJC HealthCare and Washington University School of Medicine internal grant program, hosted by the Healthcare Innovation Laboratory and the Institute for Informatics.

Protection of Human and Animal Subjects

The study design was approved by the Washington University School of Medicine Institutional Review Board and consents were obtained from all participants.

• References

• 1 Segall N, Bonifacio AS, Schroeder RA. et al; Durham VA Patient Safety Center of Inquiry. Can we make postoperative patient handovers safer? A systematic review of the literature. Anesth Analg 2012; 115 (01) 102-115
• 2 Nagpal K, Arora S, Abboudi M. et al. Postoperative handover: problems, pitfalls, and prevention of error. Ann Surg 2010; 252 (01) 171-176
• 3 Catchpole KR, de Leval MR, McEwan A. et al. Patient handover from surgery to intensive care: using Formula 1 pit-stop and aviation models to improve safety and quality. Paediatr Anaesth 2007; 17 (05) 470-478
• 4 Agarwal HS, Saville BR, Slayton JM. et al. Standardized postoperative handover process improves outcomes in the intensive care unit: a model for operational sustainability and improved team performance*. Crit Care Med 2012; 40 (07) 2109-2115
• 5 Petrovic MA, Aboumatar H, Baumgartner WA. et al. Pilot implementation of a perioperative protocol to guide operating room-to-intensive care unit patient handoffs. J Cardiothorac Vasc Anesth 2012; 26 (01) 11-16
• 6 McElroy LM, Collins KM, Koller FL. et al. Operating room to intensive care unit handoffs and the risks of patient harm. Surgery 2015; 158 (03) 588-594
• 7 France DJ, Slagle J, Schremp E. et al. Impact of patient handover structure on neonatal perioperative safety. J Perinatol 2019; 39 (03) 453-467
• 8 France DJ, Slagle J, Schremp E. et al. Defining the epidemiology of safety risks in neonatal intensive care unit patients requiring surgery. J Patient Saf 2020
• 9 Northway T, Krahn G, Thibault K. et al. Surgical suite to pediatric intensive care unit handover protocol: implementation process and long-term sustainability. J Nurs Care Qual 2015; 30 (02) 113-120
• 10 Shah ACO, Oh DC, Xue AH, Lang JD, Nair BG. An electronic handoff tool to facilitate transfer of care from anesthesia to nursing in intensive care units. Health Informatics J 2019; 25 (01) 3-16
• 11 Yang J-G, Zhang J. Improving the postoperative handover process in the intensive care unit of a tertiary teaching hospital. J Clin Nurs 2016; 25 (7,8): 1062-1072
• 12 Gleicher Y, Mosko JD, McGhee I. Improving cardiac operating room to intensive care unit handover using a standardised handover process. BMJ Open Qual 2017; 6 (02) e000076
• 13 Lane-Fall MB, Pascual JL, Peifer HG. et al. A partially structured postoperative handoff protocol improves communication in 2 mixed surgical intensive care units: findings from the Handoffs and Transitions in Critical Care (HATRICC) prospective cohort study. Ann Surg 2018
• 14 Kamath SS, Helmers L, Otto A, Kirk D, Erdahl J, Wayling B. Operating room to pediatric intensive care unit handoff: improving communication and team relations while driving process improvement. . Accessed November 25, 2020 at: https://pdfs.semanticscholar.org/9af8/79d9c2679f30cff587e365d3dae03fcbc6fc.pdf?_ga=2.147604151.1480360050.1606310602-492149727.1565926388
• 15 Faiz T, Saeed B, Ali S, Abbas Q, Malik M. OR to ICU handoff: theory of change model for sustainable change in behavior. Asian Cardiovasc Thorac Ann 2019; 27 (06) 452-458
• 16 Gleich SJ, Nemergut ME, Stans AA. et al. Improvement in patient transfer process from the operating room to the PICU using a lean and six sigma-based quality improvement project. Hosp Pediatr 2016; 6 (08) 483-489
• 17 Chenault K, Moga M-A, Shin M. et al. Sustainability of protocolized handover of pediatric cardiac surgery patients to the intensive care unit. Paediatr Anaesth 2016; 26 (05) 488-494
• 18 Joy BF, Elliott E, Hardy C, Sullivan C, Backer CL, Kane JM. Standardized multidisciplinary protocol improves handover of cardiac surgery patients to the intensive care unit. Pediatr Crit Care Med 2011; 12 (03) 304-308
• 19 Krimminger D, Sona C, Thomas-Horton E, Schallom M. A multidisciplinary QI initiative to improve OR-ICU handovers. Am J Nurs 2018; 118 (02) 48-59
• 20 Mukhopadhyay D, Wiggins-Dohlvik KC, MrDutt MM. et al. Implementation of a standardized handoff protocol for post-operative admissions to the surgical intensive care unit. Am J Surg 2018; 215 (01) 28-36
• 21 Tun KS, Wai KS, Yin Y, Thein MK. Postoperative handover among nurses in an orthopedic surgical setting in Myanmar: a best practice implementation project. JBI Database Syst Rev Implement Reports 2019; 17 (11) 2401-2414
• 22 Karakaya A, Moerman AT, Peperstraete H, François K, Wouters PF, de Hert SG. Implementation of a structured information transfer checklist improves postoperative data transfer after congenital cardiac surgery. Eur J Anaesthesiol 2013; 30 (12) 764-769
• 23 Friesen MA, Herbst A, Turner JW, Speroni KG, Robinson J. Developing a patient-centered ISHAPED handoff with patient/family and parent advisory councils. J Nurs Care Qual 2013; 28 (03) 208-216
• 24 Maddox TM, Rumsfeld JS, Payne PRO. Questions for artificial intelligence in health care. JAMA 2019; 321 (01) 31-32
• 25 Spencer D, Warfel T. Card sorting: a definitive guide. . Accessed November 25, 2020 at: https://boxesandarrows.com/card-sorting-a-definitive-guide/
• 26 Kunjan K, Doebbeling B, Toscos T. Dashboards to support operational decision making in health centers: a case for role-specific design. Int J Hum Comput Interact 2019; 35 (09) 742-750
• 27 Cooper JB, Singer SJ, Hayes J. et al. Design and evaluation of simulation scenarios for a program introducing patient safety, teamwork, safety leadership, and simulation to healthcare leaders and managers. Simul Healthc 2011; 6 (04) 231-238
• 28 Fritz BA, Cui Z, Zhang M. et al. Deep-learning model for predicting 30-day postoperative mortality. Br J Anaesth 2019; 123 (05) 688-695
• 29 King CR, Fritz BA, Escallier K. et al. Association between preoperative obstructive sleep apnea and preoperative positive airway pressure with postoperative intensive care unit delirium. JAMA Netw Open 2020; 3 (04) e203125-e203125
• 30 Cui Z, Fritz BA, King CR, Avidan MS, Chen Y. A factored generalized additive model for clinical decision support in the operating room. AMIA Annu Symp Proc 2019; 2019: 343-352
• 31 Lundberg SM, Nair B, Vavilala MS. et al. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nat Biomed Eng 2018; 2 (10) 749-760
• 32 Crowe M, Inder M, Porter R. Conducting qualitative research in mental health: Thematic and content analyses. Aust N Z J Psychiatry 2015; 49 (07) 616-623
• 33 National Patient Safety Goals. . Accessed November 25, 2020 at: https://www.jointcommission.org/standards/national-patient-safety-goals/
• 34 Breuer RK, Taicher B, Turner DA, Cheifetz IM, Rehder KJ. Standardizing postoperative PICU handovers improves handover metrics and patient outcomes. Pediatr Crit Care Med 2015; 16 (03) 256-263
• 35 Anwari JS. Quality of handover to the postanaesthesia care unit nurse. Anaesthesia 2002; 57 (05) 488-493
• 36 Herasevich V, Ellsworth MA, Hebl JR, Brown MJ, Pickering BW. Information needs for the OR and PACU electronic medical record. Appl Clin Inform 2014; 5 (03) 630-641
• 37 Andreae MH, Maman SR, Behnam AJ. An electronic medical record-derived individualized performance metric to measure risk-adjusted adherence with perioperative prophylactic bundles for health care disparity research and implementation science. Appl Clin Inform 2020; 11 (03) 497-514
• 38 Abraham J, Kannampallil TG, Patel VL. Bridging gaps in handoffs: a continuity of care based approach. J Biomed Inform 2012; 45 (02) 240-254
• 39 Mehdipour Ghazi M, Nielsen M, Pai A. et al; Alzheimer's Disease Neuroimaging Initiative. Training recurrent neural networks robust to incomplete data: Application to Alzheimer's disease progression modeling. Med Image Anal 2019; 53: 39-46
• 40 Ghorbani A, Zou JY. Embedding for informative missingness: deep learning with incomplete data. . Accessed November 25, 2020 at: https://proceedings.allerton.csl.illinois.edu/2018/media/files/0202.pdf
• 41 Jerez JM, Molina I, García-Laencina PJ. et al. Missing data imputation using statistical and machine learning methods in a real breast cancer problem. Artif Intell Med 2010; 50 (02) 105-115
• 42 Roy G, Stewart M. Let's gamble: uncovering the impact of visualization on risk perception and decision-making. . Accessed November 25, 2020 at: http://visualdata.wustl.edu/files/lets-gamble.pdf
• 43 Ottley A, Metevier B, Han PKJ, Chang R. Visually communicating Bayesian statistics to laypersons. . Accessed November 25, 2020 at: https://www.eecs.tufts.edu/~alvittao/files/techReport.pdf
• 44 Hakone A, Harrison L, Ottley A. et al. PROACT: Iterative design of a patient-centered visualization for effective prostate cancer health risk communication. IEEE Trans Vis Comput Graph 2017; 23 (01) 601-610
• 45 Dwivedi S, Upadhyay S, Tripathi A. A working framework for the user-centered design approach and a survey of available methods. International Journal of Scientific and Research Publications. 2012; 2 (04) 12-19

Publikationsverlauf

Eingereicht: 20. August 2020

Angenommen: 10. November 2020

Artikel online veröffentlicht:
24. Februar 2021