A Qualitative Exploration of Nurses’ Information-Gathering Behaviors Prior to Decision Support Tool Design

Summary

Background: Large and readily-available clinical datasets combined with improved computational resources have permitted the exploration of many new research and clinical questions. Predictive analytics, especially for adverse events, has surfaced as one promising application of big data, and although statistical results can be highly accurate, little is known about how nurses perceive this new information and how they might act upon it. Objectives: Within the context of recognizing patients at risk for cardiopulmonary arrest, this study explored the possibility of incorporating predictive analytics into clinical workflows by identifying nurses’ current information gathering activities and perceptions of probability-related terms. Methods: We used a qualitative description approach for data collection and analysis in order to understand participants’ information gathering behaviors and term perceptions in their own words. We conducted one-on-one interviews and a focus group with a total of 10 direct care bedside nurses and 8 charge nurses. Results: Participants collected information from many sources that we categorized as: Patient, Other People, and Technology. The process by which they gathered information was conducted in an inconsistent order and differed by role. Major themes comprised: (a) attempts to find information from additional sources during uncertainty, (b) always being prepared for the worst-case scenario, and (c) the desire to review more detailed predictions. Use of the words probability, risk, and uncertainty were inconsistent. Conclusions: In an effort to successfully incorporate predictive analytics into clinical workflows, we have described nurses’ perceived work practices for gathering information related to clinical deterioration and nurses’ beliefs related to probability-based information. Findings from our study could guide design and implementation efforts of predictive analytics in the clinical arena.

Jeffery AD, Kennedy B, Dietrich MS, Mion LC, Novak LL. A Qualitative Exploration of Nurses’ Information-Gathering Behaviors Prior to Decision Support Tool Design. Appl Clin Inform 2017; 8: 763–778 https://doi.org/10.4338/ACI-2017-02-RA-0033

Keywords

Qualitative Research - Decision-Making - Computer-Assisted - Nursing

1. Background and Significance

With the last decade of widespread implementation of electronic health records (EHR), the rapid analysis of large datasets from a variety of sources using complex computational methods has opened the door to exploration of many new research and clinical questions. Although determining the most important questions to ask of this big data is a frequent topic of discussion [[1], [2]], predicting outcomes, such as high-cost patients, readmissions, triage, acute decompensation, adverse events, and treatment optimization, through the application of predictive analytics has been gaining popularity [[3]].

The statistical outputs of these predictive models can be highly accurate, but little is known about how nurses perceive this type of information and how they might act upon it. Nurses are experienced using information about a patient’s history and current condition (e.g., physical assessments, laboratory values), but information predicting a future outcome for a specific patient is a novel addition to his/her toolkit. Assuming we can leverage the big data at our disposal, the simple addition of another technological tool may or may not improve care delivery. While many technologies are developed to aid nurses in identifying important changes in patient conditions (i.e., enhance the signal), the wide variety of simultaneously available user interfaces and poor interoperability impedes nurses’ ability to cognitively process and appropriately use the data in caring for patients (i.e., too much noise) [[4]]. In contrast to the study of physicians and their workflows incorporating decision support, little attention has been placed on the nursing population.

Because the potential impact of predictive analytics on nurses’ clinical decision-making is a newer concept to consider, it is proposed that the initial step in investigating the usefulness of those analytics would include the study of a clinical event in which ideally a probability score could be presented close to the time when the actual event might occur. This approach facilitates real-time feedback and minimizes the influence of additional variables (e.g., other clinicians’ actions or non-hospital factors) on the connection between the probability of events and the actual occurrence of events. Using available data to predict events likely to occur within the subsequent 24–48 hours would be ideal for the in-hospital nurse. The risk of cardiopulmonary arrest (CPA) meets these criteria and served as the context for this study. With over 200,000 people treated for in-hospital cardiac arrest every year [[5]] and survival rates ranging from 23 to 37% [[6]], this is a clinically important adverse event that might be preventable [[7]].

2. Objectives

This study began the work of incorporating predictive analytics into clinical workflows by exploring nurses’ current activities involved in problem recognition, information gathering, and problem solving. Given the novelty of producing risk data for nurses, it was important to not only understand their use of probability-based information but also their perceptions and use of the terms probability, risk, and uncertainty. The aim of this study was to identify key concepts in those three areas to determine where probability-based tools might fit within the “orienting frames” nurses use for organization and conduct of their work.[[8]] Beginning with a qualitative approach was thought to provide the most suitable insight into how we can successfully deploy big data applications such as predictive analytics into the clinical environment.

3. Methods

3.1. Design

We used a qualitative description approach [[9], [10]] for data collection and analysis to understand participants’ information gathering behaviors and term perceptions. Naturalistic inquiry incorporating design research concepts yielded rich description of participants’ experiences gathering information and interacting with probability-based data. The naturalistic inquiry paradigm influenced our attempts to understand participants’ experiences while minimizing speculation, and describing the context and nature of nurses’ work is consistent with design research. Design Research methods [[11], [12]] can answer questions such as:

• What non-patient data are available within the healthcare system?

• How/when do nurses think and act on the topic of interest?

• What do different nursing roles need? and

• How do we incorporate our results into the workflow?

3.2. Setting and Participants

The study took place at a single academic medical center in an urban city in the mid-South region of the U.S. Participant recruitment involved e-mail and printed flyers along with face-to-face discussions with nurse leaders. Inclusion criteria comprised:

• bedside nurses who cared for a patient requiring activation of a rapid response team or CPA resuscitation efforts within the last 6 months for a duration of at least 2 hours before the event, and

• charge nurses who assigned nurses to patients at least twice per week over the last 6 months.

Our purposive sample consisted of 18 nurses. One-on-one interviews were conducted with ten direct care bedside nurses and five charge nurses. Bedside (i.e., direct care) nurses were included because they are the clinicians who spend the most time with patients and thus were appropriate to evaluate predicted versus actual CPAs. Charge nurses, who are responsible for determining nurse-patient assignments, were included because decision-making with predictive analytics might also serve organizational leaders who manage resources. One focus group of three charge nurses (rather than individual interviews) was also conducted because we assumed task recall for the complex activity of assignment making would be easier by hearing cues from others performing similar work.

3.3. Data Collection Procedures

3.3.1. Variables of Interest

Explorations of interdisciplinary (including non-healthcare) information behaviors highlight the importance of other people and the environment in gathering information for situation awareness [[13], [14]]. Studies of nurses’ general information behaviors, which primarily focus on critical care nurses, add patient records and monitoring technology as additional information sources while emphasizing the role of tool design [[15]–[17]]. To explore how predictive analytics tools might support nurses’ situation awareness with patients at risk for deterioration, data collection focused on three major areas of interest:

1. information gathering behaviors of participants related to deteriorating patients,

2. potential benefit of clinical prediction models for deteriorating patients, and

3. definitions of risk, uncertainty, and probability.

4. These focus areas guided the development of a semi-structured interview guide (see Supplemental Material), which was pilot-tested with practicing nurses and graduate nursing students before using.

Regarding information gathering behaviors, we used questions focused on the materials accessed (e.g., electronic health record), activities performed (e.g., obtaining vital signs, performing physical assessments), and people contacted (e.g., other clinicians) during participants’ efforts to gather information about a deteriorating patient. To prevent speculative findings, participants need to have experienced the phenomenon of interest. If the study setting does not allow participants to be observed experiencing the phenomenon, their input in interviews must be based on an actual experience. Therefore, to determine how prediction models might be perceived within clinical environments, we used weather scenarios as a proxy. Given that participants had prior exposure to probability-based weather information (but not clinical prediction tools), weather scenarios served as a surrogate to understand how they might use and interpret probability-based decision support tools. Scenarios and images were designed to elicit responses from participants that were similar to those we expected nurses might use when inferring a patient’s likelihood of experiencing clinical deterioration from a predictive tool. Scenarios included weather forecasting examples of daily summaries, hourly trends, images, numbers, and one scenario of a discrepancy in predictions between two different forecasting websites for the same geographic location (►[Figure 1] and ►[Figure 2]). To obtain perceptions of the terms probability, risk, and uncertainty, we simply asked participants to provide us with a definition and example.

Example of weather scenario provided to participants. This scenario depicts the chance of precipitation for the next 30 hours from a general overview and hourly probabilities [[33]].

Another example of weather scenario provided to participants. This depicts a scenario where two different weather sources forecast discrepant precipitation probabilities for the same location [[33], [34]].

3.3.2. Interview and Focus Group Methods

The Institutional Review Board approved the study, and participants provided informed consent. We asked charge nurse participants to write responses to guided questions on a worksheet before attending the interviews as a memory aid to minimize recall bias [[18]] for the complex task of assignment-making (see Supplemental Material). These questions were intended to elicit thoughts while making assignments on a clinical shift, and we discussed these written responses during the interviews and focus group.

Following researcher introductions, we used the pilot-tested, semi-structured interview guide for data collection with all study participants. Individual interviews lasted approximately 60 minutes, and the focus group lasted 120 minutes concluding when participants had no further information to add in response to questions and probes. All participant discussions were audio recorded and occurred in a private room at the medical center where participants were employed. Bedside nurses received a $30 gift card for participation, and charge nurses, because of the pre-interview worksheet activity requirement, received a $50 gift card for participation. Researchers collected field notes during the discussions with participants and during debriefing discussions with each other.

3.3.3. Moderators

The principal investigator (AJ), a male registered nurse and doctoral candidate with some qualitative research experience, was present for all interviews and the focus group. A co-investigator (LN), a female doctoral-prepared organizational anthropologist with expertise in qualitative research methodology, was present for three individual interviews and the focus group. LN’s presence for individual interviews focused on evaluating AJ’s moderating skills and immersing herself in a sample of the data while her presence in the focus group permitted the two-moderator approach. Consistent with a two-moderator approach to focus groups, the principal investigator (AJ) was able to be the subject matter expert while the co-investigator (LN) served as the focus group conduct specialist [[19]]. The moderators had no supervisory relationship with the participants.

3.4. Data Analysis

Thematic data analysis involved coding all transcribed discussions and written statements by two coders (AJ and LN). To develop the codebook, coders jointly applied codes to two interviews, discussing differences until consensus was reached on the set of codes and their definitions. The online qualitative analysis software Dedoose [[20]] was used to apply codes, group themes, look for patterns, and compare findings with existing literature. A Key Concepts analytic framework guided identification of factors most important to the study’s objectives.[[19]] Methodologists have recommended that this surface-level interpretation results in findings “closer to the data as given” (p. 78) [[21]].

4. Results

Eighteen participants (►[Table I]) from 15 interviews and 1 focus group produced 14.5 hours of recorded audio, 525 pages of transcripts, and many hand-written notes. Findings are presented in alignment with the three areas of interest: Information Gathering Behaviors, Potential Benefit of Clinical Prediction Models, and Perceptions of Probability, Risk, and Uncertainty.

Table 1

Descriptive statistics of participants (n=18).

Demographic Variable		n
Education[*]	Associate	2
	Bachelor	10
	Master	3
Age (Years)	Under 25	4
	26–35	8
	36–45	2
	46–55	3
	56–65	1
Nursing Experience (Years)	< 2	3
	2–5	4
	6–10	5
	11–15	2
	> 15	4
Patient Population	Adult	9
	Pediatric/Neonatal	9
Care Setting	Emergency Department	4
	Intensive Care Unit	7
	Ward/Floor	7

*Education variable not available for focus group participants

4.1. Information Gathering Behaviors

Participants reported collecting information from many sources when evaluating whether a patient was at risk for CPA. We categorize these sources as: Patient (e.g., physical assessment, past medical history), Other People (e.g., family members, physicians), and Technology (e.g., electronic health record, vital sign monitor). The process by which they gathered information differed by role (i.e., bedside nurse or charge nurse).

4.1.1. Bedside Nurses

Bedside nurses working in the emergency department reported gathering initial information from the patient while nurses in the inpatient setting gathered initial information from the previous shift’s nurse. Initial patient information quickly resulted in mental classification of the patient’s clinical status such as “previously healthy” versus “multiple comorbidities”, “high-risk” versus “low-risk” surgery, or “stable” versus “unstable” during previous shift. One participant noted:

I call it a drive-by assessment. It’s when you like, if you were to come in the ER and if you walk in…,that gives me a few minutes to go in there and see, because you’re probably stable because you’re walking into the room. –Bedside Nurse #9

The initial mental classifications appeared to result in a baseline assessment against which future information was integrated. A different participant commented on this phenomenon by stating:

When you have a patient, either whether it’s multiple days or there are multiple admissions… it’s always beneficial to the nurse, I believe, because you kind of see them through their, at their baseline, you see what’s going on with them from a day to day basis and you can evaluate kind of what’s going on with that patient based on what you’ve seen previously. –Bedside Nurse #3

After compiling the initial information, bedside-nurse participants communicated with the patient (and/or family or caregivers), conducted physical assessments, obtained vital signs, and reviewed the patient’s history, all in no particular or consistent order. When participants discovered discrepancies among information sources, for example a physical assessment finding inconsistent with the patient’s history, they sought additional information or communicated those inconsistencies with others, such as the charge nurse, physician, or rapid response team. Bedside nurses provided insight on what, where, and when nurses were charting in the EHR. For example, most participants noted that if the change in patient status was capable of harming the patient, documentation of the nurses’ findings did not occur in real-time but rather after the decisions and actions to safely manage the patients were finished. One participant noted:

… when he was probably decompensated I probably didn’t chart as often because I was doing… procedures with the patient I guess…if there’s a procedure at the bedside, we actually like paper chart. –Bedside Nurse #6

In addition to identifying which elements were not charted in real-time, participants were helpful in identifying additional variables for prediction model development versus those that are not beneficial for real-time decision support algorithms. For example:

I usually document any notifications, any provider notifications, and …who we spoke with, what the concern was, and what the result of the notification was…I’m sure early on I had documented some of my concerns [in real time], but then probably not in real time after that. –Bedside Nurse #7

4.1.2. Charge Nurses

When working with individual patients, charge nurses (who had previously worked as, or were currently working in the capacity of, a bedside nurse) expressed the same information gathering behaviors as bedside nurses. However, in their role of making the patient care assignments for the unit, charge nurses gathered information differently. Charge nurses reported needing less individual patient detail than bedside nurses because they had many patients’ needs to consider, balanced with available nursing personnel. Themes surrounding the number of available clinicians (including nurses, nursing assistants, and physicians, both on their own unit and on other units) surfaced frequently. Charge nurses tended to gather information from the bedside nurse more frequently than any other source, even though charge nurses did report talking to physicians during patient changes and reviewing the patient’s medical record upon admission. Charge nurses in the focus group summarized by stating:

I go a lot with the intuition of the bedside staff. Most of the time, the attending physician, I don’t trust most of the residents, but our attending physicians, our fellows, and the experienced nurses, especially the ones that I’ve personally worked with for a long time and trust, I would trump a nurse’s intuition over every other kind of objective data. –Charge Nurse #1

I agree with that totally. –Charge Nurse #2

Charge nurses also strongly considered individual staff members’ experience, expertise, and preferences along with individual patients’ needs and their relationship with nurses.

I knew that I needed, and this is our expression, I needed a strong nurse for this assignment because it could either end up in…, withdrawal, you know bereavement or escalating care…, even more so than what we already had. –Charge Nurse #7

4.2. Potential Benefit of Clinical Prediction Models

Three major themes and four minor themes emerged in participant responses to the weather scenarios (►[Table 2]). Major themes included attempts to find information from additional sources during uncertainty (i.e., triangulation), always being prepared for the worst-case scenario regardless of predictions, and the desire to review more detailed projections (e.g., viewing hourly predicted values rather than a daily summary and knowing the source of the information). Regarding this latter theme, participants also noted that consistency (i.e., hour-to-hour stability) and extreme values (i.e., 0% or 100%) provide a sense of confidence or certainty in the outcome. In contrast, certainty appeared to diminish with temporally distal predictions such as those more than 24 hours from the current time. Minor themes that surfaced among some participants included: compromising when faced with discrepant values, a deference to personal preference to simplify decision making, building a story to accompany the empirical data, and the ability of prior beliefs to supersede new evidence.

Table 2

Major and minor themes resulting from weather scenarios.

Major Themes & Description	Example Quotes
#1: Triangulation When faced with uncertainty, participants sought information from different sources (e.g., reviewing a weather map in addition to probabilities), facilitating discovery of a ‘true’ state.	“I would take a look at a map…because then you can see actual precipitation, you know what’s coming in, if there’s…a front or something like that, or if it’s just cloud like cover cause then you know that’s the one you’d lean towards.” –Bedside Nurse #10
#2: Always be Prepared Some participants never trusted the predictions because they have found them to be wrong too many times. Others seemed slightly risk averse. In both cases, they preferred to bring an umbrella or raincoat in the event of an undesirable outcome.	“Bring an umbrella, park close…be prepared” –Bedside Nurse #5
	“Depending on where I was planning on being or what I was planning on doing, I would just prepare for the worst.” –Charge Nurse #3
#3: Desire for Detail Almost all participants preferred an hourly forecast view. This level of detail provided insight into which activities can be performed by the participant and provided a trend by which one can make more-informed predictions.	“I’ll look at the hourly because there’s certain times when I’m going to be coming in from my car and going back to my car.” –Bedside Nurse #1
#3a: Consistency and Extreme Values Consistency and extreme values (e.g., repeated hours of 0% probability) promoted comfort, certainty, and confidence.	“R: There’s theoretically no chance of rain tomorrow and it’s down-trending.
	I: So, both the absolute 0% here and the trend moving down?
	R: Right.” –Bedside Nurse #7
#3b: The only certainty is uncertainty Predicting soon is, at best, uncertain – predicting the far future, one might as well not look, that is unknown.	“I mean just more uncertain because it’s so far out and so, obviously, our ability to predict tomorrow is much better than our ability to predict that day…and the days leading up to it.” –Bedside Nurse #7
	“I: What about 3 weeks out? Does that mean anything? R: It means that the weather man is a poor betting man.” –Bedside Nurse #5
Minor Themes
#1 Compromising When faced with the discrepant values, participants assumed the true value was somewhere in the middle.	“No wonder why we don’t trust the weatherman. [laughing] I’d be thinking it’s somewhere in-between that then.” –Bedside Nurse #6
#2 Deference to Preference Similar to the cognitive heuristic of simplifying decision making, participants noted that when faced with discrepancies, they would defer either to the outcome they prefer (i.e., no rain) or to the outcome suggested by their most trusted source.	“It matters what my day was like…I would like to choose the one that would fit into my day better.” –Bedside Nurse #9
	“I would probably choose to go with whichever you know website or app that I typically look at or you know whichever one I felt gave the most accurate information.” –Charge Nurse #8
#3 Story Building Giving a story or personality to the data (i.e., devising a rationale for why something might be displayed as it is).	“I: What are your thoughts on the discrepancy here? R: Well, it’s almost like this one just doesn’t want to commit.” –Bedside Nurse #10
#4 Prior beliefs can supersede new evidence Belief that context (e.g., geographic location) was so important that no new evidence would make someone come up with a different conclusion.	“If you’re in the tropics it’s going to rain whether you think it’s going to or not…if it’s Florida or California it’ll probably rain for 2 hours in the afternoon and you’ll be good for the rest of the day. And if it’s Jamaica and the Dominican, it always says it’s probably going to rain and it never does ‘cause it’s gorgeous, so it really just depends on where we are.” –Bedside Nurse #5

Of note, there were a few divergent views among our participants. With respect to Major Theme #3 regarding a desire for detail, two participants noted that too much detail (i.e., hour-by-hour predictions) showed the model was “too confident” and therefore could not be trusted. One of those participants noted:

You’ve got to apply it across the whole day because you can’t say that the wind is not gonna pick up at that moment and bring this 55% worth of showers into my next hour. –Bedside Nurse #7

This opinion was in contrast to that of another participant who felt some hourly variation increased her belief in the model. One of the focus group participants noted that abrupt changes made him disbelieve a prediction.

With the respect to Minor Theme #4 regarding prior beliefs, one participant (Bedside Nurse #5) expressed that context (e.g., geographic location) was so important that no new evidence would make her come up with a different conclusion.

• Interviewer [I]: …What would something like this [see [Figure 1]] mean to you if you were planning something, this is today, this is tomorrow, if you were planning something tomorrow outside, would you feel comfortable?

• Participant [P]: Nashville, Florida or the tropics, give me a location.

• I: Why does location matter?

• P: Because if you’re in the tropics it’s going to rain whether you think it’s going to or not… And if it’s Florida or California it’ll probably rain for 2 hours in the afternoon and you’ll be good for the rest of the day. And if it’s Jamaica and the Dominican it always says it’s probably gonna rain and it never does ‘cause it’s gorgeous, so it really just depends on where we are…

• I: So there’s a lot of context or prior knowledge that …

• P: Yeah. Yeah. I need more information.

Another participant made a similar comment that because she grew up in the area, her opinions on weather patterns were more trustworthy than the meteorologist’s predictions. Similarly, one participant stated she would not look for more information from different sources and did not care about the reputation of a source – only which sources appear more accurate based on her past experience.

As it related to the clinical intent of these questions, some participants expressed reservations about the ability of a prediction model to accurately predict cardiopulmonary arrest. One participant noted:

An arrest is so multifactorial and I think that, I don’t know. I would have a hard time seeing; could be if any, something that could pre-, predict it with the kind of accuracy beyond just this is a higher-risk patient because they’ve had a more complex surgery. –Bedside Nurse #7

Another participant stated:

How… are you collecting your data? How are you presenting it? Pretty and green doesn’t make a difference if you’re wrong, if you’re guessing, but neither does basic and blue, so it depends. At some point in time I just need a yes or no. [laughs] –Bedside Nurse #5

4.3. Perceptions of Probability, Risk, and Uncertainty

Regarding perceptions of the words probability, risk, and uncertainty, participants were inconsistent in their use of these words and frequently used the words interchangeably. Those words did not appear to have a consistent, intrinsic meaning between or within participants (►[Table 3]). For example, one participant noted:

Table 3

Themes related to perceptions of risk, uncertainty, and probability.

Theme	Description
#1: Probability = ‘probably’	Participants used ‘probability’ to imply that an event/outcome will ‘probably’ occur (i.e., more likely to occur than not to occur).
#2: Risk is related to a patient’s history and might assist with prioritizing.	Participants were able to mention some types or classifications of patients that could automatically be labeled as ‘high risk,’ which implied greater attention should be focused on that patient.
#3: Risk is related to harm.	Risk was the term most related to the concept of patient harm.
#4: Uncertainty = ‘unexpected, unpredictable, unknown.’	Participants noted that one can never fully expect or predict what will happen with a hospitalized patient.

I mean, I guess you could group them all kind of under the same classification if you’re like, there’s a probability of risk and then you can have a probability of uncertainty. And I think risk and uncertainty are pretty similar because when you’re uncertain there’s always a risk. –Bedside Nurse #3

Another participant (Bedside Nurse #1) noted, “…the risk of going to the unit is 100%, the probability of going to the intensive care is 100%, and the, I mean, you hear what I’m saying?”

4.3.1. Probability

Several participants related the term “probability” to mathematics; however, participants’ examples did not adhere to traditional rules of mathematics. The majority held a view summarized by a participant who stated:

I really don’t think in terms of like probability that something bad is going to happen…Probability is a more statistical term that I tend to stay away from. –Bedside Nurse #5

4.3.2. Risk

Participants related risk to a patient’s medical history and associated it strongly with harm. For example, one participant noted:

The risk of hypotension in a heart failure patient is high because…we give them very strong diuretics. –Bedside Nurse #1

Other participants indicated that risk stratification helped them prioritize the needs of their patients, both for the purpose of care planning and interpersonal communication with other clinicians.

4.3.3. Uncertainty

Regarding uncertainty, a few participants implied that clinicians can never fully expect or predict what will happen with a hospitalized patient. One participant (Charge Nurse #5) described uncertainty as “…an over-looming thing with everybody.” Another participant commented on the nurses’ role in managing the tension between statistical data and the lived experience of patients and families, noting:

as far as statistics, there’s some legitimacy to them, but they’re never, they should never dictate your care that you provide or the information you give a family. You can talk statistics with them, but you know what, allow them to keep their hope and say, “Look statistically speaking these are the things that happen.” You could have them mentally prepare themselves for those things, but also let them know, you know what? Things can get better too … things can be better than what statistically speaking and you know, you can have your hope and you can continue to push. –Bedside Nurse #2

Finally, a few participants noted that uncertainty was present when one does not know why something is occurring (e.g., a deteriorating patient with no rationale for that state). One participant described uncertainty as “insecurity” or a lack of confidence in the sense that a novice nurse might not know if she should share concerns with a physician.

5. Discussion

We have described nurses’ perceived workflows for gathering information related to clinical deterioration and nurses’ beliefs related to probability-based information. An initial qualitative approach provided a rich foundation for understanding behavior change (i.e. work processes) in response to predictive analytics to support design and evaluation. Although the content of information gathered by nurses was similar, a consistent temporal pattern in their information gathering was lacking. The lack of a predictable temporal pattern for information gathering has important implications for the design of decision support tools, similar to the non-linear workflows described in bar code medication administration [[22]]. Given that nurses gather a significant amount of information outside of the electronic health record (EHR) and that EHR documentation of clinical deterioration tends to occur after problem recognition and management, a predictive tool for this outcome likely belongs outside the EHR. Furthermore, brief and informal patient assessments (referred to by the participants as a “drive-by”) are not routinely considered important enough to document in the EHR, even though they contain information that influences a nurse’s anticipated trajectory of patient status.

Our finding that information gathering behaviors appeared to differ between bedside nurses and charge nurses suggests recommendations concerning information provision and decision support may not be transferable to all nurses but rather might require unique designs based on roles. To our knowledge, no studies have explored clinical decision support tool specifications that differ by nursing role. Our findings suggest that charge nurses might be key players in effectively incorporating decision support systems predicting negative patient outcomes because of their expressed preference for high-level overviews of patient status. Probability-based decisions support tools provide numerical outputs that aggregate information from multiple sources. Therefore, if summarized probability information could be provided for a charge nurse’s patients in one medium (e.g., ranked in order), such a display could provide assistance to charge nurses’ decisions in patient care assignments. When comparing responses based on other participant characteristics (e.g., experienced versus inexperienced, adult versus pediatric care settings), we did not identify distinct differences.

Discrepancies, whether in patient assessments or weather scenarios, were important in initiating the process of gathering additional information. This finding aligns with decision theory principles that one option for responding to uncertainty is the gathering of additional information in an effort to reduce uncertainty [[23]]. Participants expressed concerns surrounding a prediction model displaying results contradicting their clinical gestalt, but our findings suggest those discrepancies might simply result in seeking additional information.

Few studies have evaluated the direct impact of predictive analytics on the decisions nurses make [[24]], yet several researchers have explored nurses’ cognitive work in the context of the recognition and response to a deteriorating patient. For example, Braaten [[25]] conducted a cognitive work analysis with 12 nurses and reported, similar to us, that nurses gather information from a variety of sources and that they preferred the collection of objective clinical criteria to justify rapid response team activation. Additionally, as part of the systematic review of deterioration detection among ward patients conducted by Odell, Victor, and Oliver [[26]], the roles of intuition via pattern recognition, patients and families expressing concerns, and detecting abnormalities during routine care were identified as information gathering behaviors. Finally, similar findings by Cioffi [[27]] suggested that baseline mental stratification of patients and prior beliefs superseded new evidence.

Clinical prediction models provide outputs in the form of statistical probabilities that require analytical decision-making skills for interpretation. Parker [[28]] found that nurses who use analytical decision-making skills call rapid response teams more frequently than nurses who primarily use intuitive decision-making. However, if nurses tend to use intuitive decision-making skills frequently [[29]], the design and implementation of probability-based CDS tools should appreciate nurses’ propensity for intuitive decision-making while facilitating the analytical counterpart when prompting action. Given that several of our participants did not prefer statistical probabilities for decision-making, providing predictive model information will be an important consideration in future work.

The weak agreement of definitions for risk, uncertainty, and probability likely resulted from scarce decision theory knowledge among participants. This topic is not routinely covered in nursing curricula. Furthermore, many possible actions exist for nurses during a changing patient condition in which risk or uncertainty are present. The unique arrangements of actions and goals chosen by individual nurses might result in unique definitions. A few participants’ association of these terms with statistical probabilities and their statements regarding infrequent use is consistent with Kahneman’s [[30]] view that humans do not think probabilistically.

Several participants mentioned the role of context as influential in the extent to which nurses would accept the validity of a prediction model. This can have important implications for CDS tools (►[Table 4]) because the use of predictive analytics without an awareness and appreciation for the context of its application could lead to both:

• identification of correlation without an understanding of causation that could thwart future work and

• unsuccessful implementation of clinical tools developed from analytical methods.

Table 4

Implications of findings for probability-based decision support tools.

Finding	Implication for Decision Support Tools
Desire for detail	Provide trends. Identify sources of data.
Discrepancies promote information-gathering	Even if the tool does not align with clinical gestalt, it might still be helpful in determining if a patient has a problem.
Deference to trusted source	If decision support tools provide discrepant findings, they are less likely to prompt behavior because participants noted they are more likely to go to their trusted source when faced with uncertainty.

We propose that engaging nurses in the design of analytics solutions is one of the best solutions for these challenges. Specifically, Design Research methods can be both a starting point and an implementation aid for big data applications.

Strengths of our study included the variety of both nurses and settings, as well as the use of a proxy situation to elicit responses that were more likely to represent actual behaviors and decrease speculation. Limitations of our study included the single research site, small focus group size, and dependence on participant recall and perceptions. Ongoing and last-minute schedule conflicts contributed to the focus group’s small size and necessitated individual interviews to collect data from charge nurses; larger incentives might be needed in future recruitment efforts of this specific population. There is a need in future studies for documentation of what actually occurs rather than asking participants to recall what they did, similar to a recent report of critical care information gathering behaviors [[31]]. Additional nursing roles, such as advanced practice nurses, also need exploring, similar to the work of Weber [[32]].

Finally, we recognize nurses’ responses to a limited number of weather forecast scenarios are not likely an identical match to their responses to patient-focused predictive analytics tools. Our choice of weather scenarios, the common occurrence of inaccurate weather predictions by meteorologists, and the formal interview environment could have influenced our findings and resulted in the wide variety of observed responses. Because of this, we plan to examine nurses’ use of prediction models in simulated and real-world settings in the near future, and findings from our study will guide the design and implementation of big data applications into the clinical arena. Prior to embarking on large-scale experimental studies however, further qualitative investigations of technology’s influence on clinical decision making and predictive modeling’s impact on information technologies are needed.

6. Conclusions

Healthcare predictive analytics offers the potential for improving care delivery, but it provides only a portion of the total information nurses use to make decisions. As an initial step in identifying potentially beneficial roles of predictive analytics tools, this study explored nurses’ perceptions of their current work practices and the concepts of risk within the context of cardiopulmonary arrest. An increased understanding of nurses’ information behaviors will enhance the design, development, and implementation of effective probability-based clinical decision support tools while informing important expansions of nursing educational curricula. Our findings are consistent with others’ work related to situation awareness within and outside the nursing profession, and we have added to this work by focusing on the addition of prediction-based decision support tools as an information source. While these recommendations may seem intuitive to clinicians, these processes are not as well known to informaticians creating decision support systems intended for the clinical arena. Documentation of clinicians’ work practices will enable more opportunities for implementing decision support systems into their cognitive and physical workflows.

7. Clinical Relevance Statement

Big data applications (and particularly predictive analytics) are surfacing throughout the healthcare arena, but little is known about how nurses perceive this new information and how they might act upon it. This study described nurses’ current information-gathering practices related to patient deterioration and then explored how predictive analytics might fit within that work. Identifying how technology can complement current work will help to minimize interruptions and increase the likelihood of its adoption.

Multiple Choice Question

Which of the following rationales describes why qualitative research methods should be used when implementing new technologies in clinical environments?

• To facilitate adoption by understanding current work practices (correct answer)

• To assess for statistically significant effects of the intervention

• To convince clinicians to use the technology in their practice

• To persuade organizational leaders that front-line users want the technology

Rationale: Regardless of the phenomenon of interest, qualitative research methods focus on describing and understanding a lived experience. Within the realm of informatics, qualitative research methods can be helpful in understanding phenomena such as users’ experiences with technology, information needs that could be addressed by new or modified tools, and the current context in which informatics tools are implemented. This latter use was the focus of our study. Using qualitative methods to understand the context of work can provide a framework to guide design, development, and implementation of informatics tools that make success of the tools more likely.

Conflict of Interest

The authors declare that they have no conflicts of interest in the research.

Human and Animal Subjects Protections

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by Vanderbilt University Institutional Review Board. Animal subjects were not included in the project.

• References

• 1 Fallik D. For big data, big questions remain. Health Aff (Millwood) 2014; 33 (07) 1111-4.
• 2 Murdoch TB, Detsky AS. The inevitable application of big data to health care. JAMA 2013; 309 (13) 1351-2.
• 3 Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014; 33 (07) 1123-31.
• 4 Dekker S. Complexity, signal detection, and the application of ergonomics: Reflections on a healthcare case study. Appl Ergon 2012; 43 (03) 468-72.
• 5 Merchant RM, Yang L, Becker LB, Berg RA, Nadkarni V, Nichol G, Carr BG, Mitra N, Bradley SM, Abella BS, Groeneveld PW. American Heart Association Get With The Guidelines-Resuscitation I. Incidence of treated cardiac arrest in hospitalized patients in the United States. Crit Care Med 2011; 39 (11) 2401-6.
• 6 Go AS, Mozaffarian D, Roger VL, Benjamin EJ, Berry JD, Blaha MJ, Dai S, Ford ES, Fox CS, Franco S, Fullerton HJ, Gillespie C, Hailpern SM, Heit JA, Howard VJ, Huffman MD, Judd SE, Kissela BM, Kittner SJ, Lackland DT, Lichtman JH, Lisabeth LD, Mackey RH, Magid DJ, Marcus GM, Marelli A, Matchar DB, McGuire DK, Mohler ER, 3rd, Moy CS, Mussolino ME, Neumar RW, Nichol G, Pandey DK, Paynter NP, Reeves MJ, Sorlie PD, Stein J, Towfighi A, Turan TN, Virani SS, Wong ND, Woo D, Turner MB, American Heart Association Statistics C, Stroke Statistics S. Heart disease and stroke statistics--2014 update: a report from the American Heart Association. Circulation 2014; 129 (03) e28-e292.
• 7 Hodgetts TJ, Kenward G, Vlackonikolis I, Payne S, Castle N, Crouch R, Ineson N, Shaikh L. Incidence, location and reasons for avoidable in-hospital cardiac arrest in a district general hospital. Resuscitation. 2002; 54 (02) 115-23.
• 8 Hazlehurst B, McMullen C. Orienting frames and private routines: the role of cultural process in critical care safety. Int J Med Inform 2007; 76 01 S129-35.
• 9 Sandelowski M. Whatever happened to qualitative description?. Res Nurs Health 2000; Aug 23 (04) 334-40.
• 10 Kim H, Sefcik JS, Bradway C. Characteristics of Qualitative Descriptive Studies: A Systematic Review. Res Nurs Health 2017; 40 (01) 23-42.
• 11 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-53.
• 12 Reeder B, Hills RA, Turner AM, Demiris G. Participatory design of an integrated information system design to support public health nurses and nurse managers. Public Health Nurs 2014; 31 (02) 183-92.
• 13 Sonnenwald DH, Pierce LG. Information behavior in dynamic group work contexts: interwoven situational awareness, dense social networks and contested collaboration in command and control. Information Processing & Management 2000; 36 (03) 461-79.
• 14 Endsley MR. Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 1995; 37 (01) 32-64.
• 15 Reddy M, Dourish P. A finger on the pulse: temporal rhythms and information seeking in medical work. Proceedings of the 2002 ACM conference on Computer supported cooperative work; 2002: ACM; 2002: 344-53.
• 16 Koch SH, Weir C, Haar M, Staggers N, Agutter J, Gorges M, Westenskow D. Intensive care unit nurses‘ information needs and recommendations for integrated displays to improve nurses‘ situation awareness. J Am Med Inform Assoc 2012; 19 (04) 583-90.
• 17 Gorman P, Ash J, Lavelle M, Lyman J, Delcambre L, Maier D, Weaver M, Bowers S. Bundles in the wild: Managing information to solve problems and maintain situation awareness. Library Trends 2000; 49 (02) 266-89.
• 18 Schulz KF, Grimes DA. Case-control studies: research in reverse. Lancet 2002; 359 9304 431-4.
• 19 Krueger RA, Casey MA. Focus Groups: A Practical Guide for Applied Research. Los Angeles, CA: SAGE; 2009
• 20 SocioCultural Research Consultants, LLC.. Dedoose Version 7.0.23, web application for managing, analyzing, and presenting qualitative and mixed method research data. 2016 [cited October 4, 2016]; Available from: http://www.dedoose.com
• 21 Sandelowski M. What‘s in a name? Qualitative description revisited. Res Nurs Health 2010; 33 (01) 77-84.
• 22 Novak LL, Holden RJ, Anders SH, Hong JY, Karsh BT. Using a sociotechnical framework to understand adaptations in health IT implementation. Int J Med Inform 2013; 82 (12) e331-44.
• 23 Lipshitz R, Strauss O. Coping with uncertainty: A naturalistic decision-making analysis. Organ Behav Hum Decis Process 1997; 69 (02) 149-63.
• 24 Jeffery AD. Methodological Challenges in Examining the Impact of Healthcare Predictive Analytics on Nursing-Sensitive Patient Outcomes. Comput Inform Nurs 2015; 33 (06) 258-64.
• 25 Braaten JS. CE: Original research: hospital system barriers to rapid response team activation: a cognitive work analysis. Am J Nurs 2015; 115 (02) 22-32.
• 26 Odell M, Victor C, Oliver D. Nurses‘ role in detecting deterioration in ward patients: systematic literature review. J Adv Nurs 2009; 65 (10) 1992-2006.
• 27 Cioffi J. Nurses‘ experiences of making decisions to call emergency assistance to their patients. J Adv Nurs 2000; 32 (01) 108-14.
• 28 Parker CG. Decision-making models used by medical-surgical nurses to activate rapid response teams. Medsurg Nurs 2014; 23 (03) 159-64.
• 29 Tanner CA. Thinking like a nurse: A research-based model of clinical judgment in nursing. J Nurs Educ 2006; 45 (06) 204-11.
• 30 Kahneman D. Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux; 2013
• 31 Wright MC, Dunbar S, Macpherson BC, Moretti EW, Del Fiol G, Bolte J, Taekman JM, Segall N. Toward Designing Information Display to Support Critical Care. A Qualitative Contextual Evaluation and Visioning Effort. Appl Clin Inform 2016; 7 (04) 912-29.
• 32 Weber S. A qualitative analysis of how advanced practice nurses use clinical decision support systems. J Am Acad Nurse Pract 2007; 19 (12) 652-67.
• 33 The Weather Channel. Hourly Weather Forecast. 2014 [cited 2014]; Available from: http://www.weather.com/
• 34 AccuWeather. Hourly Weather Forecast. 2014 [cited 2014]; Available from: http://www.accuweather.com