Keywords
contextual inquiry - fault tree analysis - abnormal laboratory follow-up - human factors
- patient safety
Background and Significance
Background and Significance
Lack of timely follow-up and missed follow-up of abnormal test results (henceforth,
“missed results”) is a recognized patient safety concern.[1]
[2] In ambulatory settings, the incidence of missed results can be up to 65% and can
lead to delayed diagnosis or treatment.[2]
[3] Knowledge on individual-level (vs. system-level) factors contributing to missed
results is evolving. For example, it is imperative to understand individual's intent
at the point of care and decision making that contribute to missed results especially
within the complex sociotechnical context of electronic health records (EHRs)-enabled
health systems. Human factors methods could advance understanding of physician decision-making
processes and uncover factors related to individual decision-making processes.
Prior research on understanding causes of missed results is limited and has used retrospective
chart reviews,[2] EHR activity logs,[4]
[5] focus groups,[6]
[7] cognitive task analyses,[8] aggregated root cause analyses data,[9] and safety event reporting.[10] Our objective was to use human factors methods to further illustrate workflow and
process issues related to missed results and focus on contributing factors. By using
the human factors methods described in this study, we gathered additional detail about
physician's decision-making process, illustrating factors that contribute to missed
results.
We applied three human factors methods, including critical decision method (CDM)-based
interviews, contextual inquiry (CI)-based analysis methodology, and fault tree analysis
(FTA), to understand factors contributing to missed results. Missed follow-up of abnormal
laboratory result involves complex human–system interaction factors; to uncover the
related decision-making processes, we chose CDM-based interviews. Building on the
decision-making process, we applied CI and FTA to illustrate contributing factors
and interactions among the factors leading to missed follow-up. Such human factors
methods can enhance understanding of where to focus strategies to reduce or mitigate
negative outcomes.
Critical Decision Method–Based Interviews
CDM is a cognitive task analysis technique used to describe naturalistic decision
making, and improves understanding of situational awareness, mental models, and decision
points in particular situations.[11]
[12]
[13] CDM involves gathering information about a personally experienced incident via focused
interviews with task experts and identifying timelines, key decision points, and factors
influencing decision making, such as clinical decision making in critical care.[14]
[15]
[16] One of the limitations of the CDM method is the delay between the incident and the
interview.[17] By using near-real time detection of incidents, we reduce the problem of memory
decay.
Contextual Inquiry–Based Analysis
The CI methodology helps in understanding the context of actions, such as physicians'
responses to abnormal test results. CI is a structured methodology for modeling work
domains and identifying user needs, guiding both interviews, and analysis. It focuses
on four principles as follows: (1) identifying context of participants' work, (2)
partnering with participants to observe and discuss work, (3) interpreting insights
and relaying them back to the participant, and (4) using the research question to
guide the interactions. CI-based analysis helps generate models to represent different
aspects of how work functions: communication flow and coordination, culture, task
sequences, physical environment, and artifacts. CI has been used in developing tools
and implementing new workflows in health care.[18]
[19]
[20]
Fault Tree Analysis
FTA[21] is a form of root cause analysis used to illustrate and analyze complex interacting
pathways leading to process failures[21] and is used for developing error prevention, monitoring, and intervention strategies.[22]
[23]
[24] FTA models an outcome as a hierarchy of interacting contributing factors[21]
[25] using Boolean logic operators (“AND” and “OR”). Construction of fault trees requires
describing top-level outcomes and resolving them into basic (primary initiating events)
and intermediate events (immediate causes for basic events).[26] This method enables visual analytics and probabilistic modeling of factors contributing
to an outcome and has been applied in clinical use cases for studying factors related
to adverse events.[27]
[28]
[29]
A combination of these human factors methods could allow in-depth identification of
causes for missed results and inform targeted solutions to improve decision-making
processes.
Case Report
Setting
This study was performed at three large primary care clinics in Texas after Institutional
Review Board approval. Each clinic used EHRs and included trainees.
Case Selection
We queried the clinical data repository at each site from January 1, 2015 to September
30, 2015 to identify abnormal imaging and laboratory results ([Table 1]). A reviewer (V.B.) manually reviewed records to identify missed results, defined
as lack of documented action (repeat or subsequent testing, referral placement, medication
change, or patient notification) within 14 days. We then invited 30 physicians who
ordered the respective tests for interviews. We used maximum variation sampling techniques
to maximize heterogeneity in clinic site and test types with each subsequent interview.
Table 1
List of abnormal laboratory result cases not followed-up
|
Real
|
Vignette
|
Total
|
|
Chest X-ray
|
1
|
|
1
|
|
EKG
|
1
|
3
|
4
|
|
Hemoglobin
|
1
|
|
1
|
|
TSH
|
|
10
|
|
Urine albumin
|
2
|
|
2
|
|
Urine culture
|
2
|
|
2
|
|
Urine micro
|
8
|
2
|
10
|
|
Total
|
15
|
30
|
Abbreviations: EKG, electrocardiogram; TSH, thyroid stimulating hormone.
Interview
We created a CDM-based interview guide ([Appendix A]) to understand follow-up in the context of a physician's own missed result cases
including reasons for the miss. Questions identified factors delaying follow-up, not
necessarily in the same case. Questions also both elicited specific factors contributing
to missed results and identified relationships between work system and individual
decision-making factors. Interviews were audio recorded and transcribed.
Three investigators (M.W.S., D.F.S., and D. Roosan) performed semistructured interviews
with physicians using the CDM-based interview guide, and data were analyzed using
the four CI principles. For the first 15 cases, interviewees were aware that they
missed the follow-up (delay case interviews). However, this contributed to some reluctance
in responding to questions about causes of missed results. To ensure responses were
not constrained by recognition of their own potential oversight and enable more open
discussion, we modified the method, so the remaining 15 cases were not traditional
CDM interviews about the participants' own incidents, but instead vignettes similar
to that experienced by their patient (vignette case interviews). The vignettes were
generated by removing identifying information about the patient, treating physician,
and clinic.
Data Analysis
Two other independent reviewers (D. Rogith and T.S.) with human factors expertise
analyzed data using CI-based and FTA methods. We adapted the CI to include only the
flow diagram analysis. We adapted the FTA to consider all logical operators as “AND”
operators.[26] The sociotechnical model[30] guided identification of factors contributing to missed results.
Reviewers first analyzed transcripts to identify underlying factors contributing to
missed results. Because we aimed to identify information flow breakdowns related to
missed results, reviewers used the CI-based analysis methodology to develop flow models
of communication and coordination in result management decision making. Thus, the
reviewers independently reviewed transcripts and identified discussion about information
flow and workflow decisions related to managing the test result. These were represented
in the model in terms of information flow for both people (e.g., physicians and patients)
and data sources (e.g., laboratory results and EHR systems). Each reviewer then independently
combined their 30 flow models into a single model before collaboratively reconciling
into one final model ([Fig. 1]).
Fig. 1 Contextual inquiry flow model of follow-up of abnormal laboratory results. EHR, electronic
health record.
Reviewers then performed FTAs to identify events leading to inaction for each case
and used deductive reasoning to identify basic events from the interviews. To generate
fault trees based on actual events, we chose only the 15 cases where interviewees
were aware that they had missed the follow-up, and both reviewers (D. Rogith and T.S.)
independently conducted FTAs for each case to identify basic events. Basic events
were then grouped into intermediate events based on the flow model described above.
An interdisciplinary team discussed findings and consolidated intermediate events
to generate a cumulative fault tree ([Fig. 2]). Using this process, the basic and intermediate events were grouped into four categories:
patient, clinical condition, physician, and EHR.
Fig. 2 Fault tree analysis of events leading to no follow-up of abnormal laboratory results.
EHR, electronic health record.
Results
Contributory causes identified from interview data are listed in [Table 2]. During delay case interviews, workflow issues were predominant (e.g., forgetting
to notify patients about therapy changes based on results, diffusion of responsibility
between referring physicians and residents for results follow-up, and language barriers).
However, in vignette case interviews, EHR issues were more prominent. For example,
limited patient portal usage led physicians to not send messages about results.
Table 2
List of reasons for not following-up an abnormal laboratory result
|
Categories
|
Delay case interviews
|
Vignette case interviews
|
|
Test results
|
• Similar abnormal laboratory results in past
• Results deemed abnormal but not clinically serious
|
• Results arrive after patient's visit
• Laboratory results are scanned or faxed so not available in structured data tabs
• Noise in color coding of abnormal laboratory results
|
|
Physician actions
|
• Specialist expected to follow-up
• Follow up deemed to be resident physician's responsibility
• Communication breakdown during delegating follow-up action to be relayed by staff
• No feedback from staff that abnormal results and follow-up actions communicated
to patient
|
• Forgetfulness
• No dedicated staff for follow-up
• Follow-up deemed an unbillable activity
• Communication breakdown during delegating follow-up action to be relayed by staff
• No feedback from staff that abnormal results and follow-up actions communicated
to patient
|
|
Clinical actions
|
• Action taken in form of adding a clinical note or updating prescription without
communication to patient
• No treatment modifications necessary
|
• Need to explain results in detail
|
|
EHR system
|
• Patient does not use portal
• Unable to confirm whether patient accessed result via portal
|
• Patient does not use portal
• Unable to confirm whether patient accessed result via portal
• Multiple clinics, multiple EHR systems—forgets to act on time
• Time limit for auto notification is deemed to be too short (10 business days)
• Not sure of difference between “communication” vs “release of abnormal laboratory
result EHR features”.
|
|
Other communications
|
• Inability to reach patient via phone
• Physician prefers not to call patients
• Language barriers
• Mail—no feedback on status of mailing
|
• Inability to reach patient via phone. No patient contact information available
• Prefer direct communication (via SMS-like technology)
• Mail—no feedback on status of mailing
|
|
Patient factors
|
• Patient deemed responsible for follow-up
• Patient has another appointment within 2 weeks, so follow-up delayed
|
• Patient has another appointment within 2 weeks, so follow-up delayed
|
Abbreviation: EHR, electronic health record.
Note: Factors common in identified cases and vignette cases are in bold.
We developed a CI flow model describing physicians' processes for managing abnormal
results ([Fig. 1]). The flow model shows four different paths in physicians' action after test results
as follows: (1) Identifying abnormal results, (2) tracking follow-up, (3) delegating
follow-up, and (4) conducting follow-up. For each path, we identified barriers in
the follow-up process. This displays how physicians interact with abnormal results,
their expectations for managing these results, and user requirements for completing
the follow-up tasks. Key findings from the flow model included physicians' lack of
methods to track follow-up and mismatch in communication channels, timeframes, and
expectations between patients and physicians.
Several physicians described unwillingness to sending notifications to the EHR portals
to communicate results because they felt patients may not use the portal. Some physicians
reported that if the result was not acted on by a physician within a specific timeframe
(e.g., 10 days), the EHR automatically released results without a physician interpretation.
This removed the item from the physician's to-do list, limiting prompts to act. Furthermore,
some physicians preferred only in-person communication of abnormal results at patients'
next appointments, which may occur beyond the autorelease timeframe.
[Fig. 2] displays the FTA-based hierarchical model of factors contributing to missed test
results, which displays the frequency of each occurrence among the 15 delay case interviews.
The most common factors were physicians' assumption that ordering physicians are responsible
for follow-up (5 of 15 cases). While most institutions designate responsibility for
result follow-up to the ordering physician, physicians reported not being notified
when resident-ordered tests returned, adding delays to follow-up. In specialty referrals,
some physicians reportedly assumed that referred specialist physicians will manage
the results of the tests ordered by the referring physician because results would
arrive at the time of the specialists' appointments. The FTA hierarchical model ([Fig. 2]) also confirms mismatch in communication channels for follow-up (5 of 15 cases).
We found that some physicians shifted responsibility for follow-up to patients, such
as by releasing results electronically only when patients signed up to the organization's
patient portal and/or expecting patients to schedule follow-up visits to discuss results.
These intermediate events provide a high-level cause for the individual basic events.
Some intermediate events highlight professionalism issues, such as reliance on patient
appointments and poor communication etiquette. One physician asserted that patients
are responsible for scheduling follow-up visits within 2 weeks. In other cases, physicians
preferred to wait on nonurgent results for patients' next visit if scheduled within
2 weeks. However, several physicians felt that follow-up discussion was often forgotten
if patients missed or canceled visits.
Discussion
Using three complementary human factors methods, we identified causes for lack of
abnormal laboratory results follow-up. Reasons identified included physicians' expectations
that patients are responsible for scheduling follow-up, mismatch in physician–patient
communication preferences, and difficulties with managing abnormal results in EHR
systems. These first-person accounts using human factors methods differ from prior
studies in that individual decisionmaking, and workflow- and technology-related findings
were prominent, allowing identification of contributing factors and barriers to action.
Combining CI-based and FTA analysis methods with CDM-based interviews permitted uncovering
of contributing factors for missed results. For example, CDM-based interviews allow
identification of multiple causes for inaction on abnormal results, while adding CI-based
analysis enabled uncovering of contextual information such as information exchange
between physician and staff ([Fig. 1]). This helped categorize causes for missed results ([Table 2]). Application of FTA then yielded a hierarchical model of missed results. Identifying
basic contributory causes can assist in designing systems to manage abnormal test
results, implementing results follow-up policies, and training clinicians to reduce
breakdowns. Interestingly, we obtained richer detail with vignettes than delay cases,
suggesting that physicians remain hesitant to discuss care breakdowns that they are
involved in and providing guidance for future work in this area.
Limitations
Several limitations should be noted. First, our findings may be limited by socially
desirability bias given the potentially sensitive topic of missed results. In vignette
cases, physicians were unable to refer to their own experiences with the cases as
they would in traditional CDM interviews. Additionally, findings from these three
sites may not be generalizable to different practice settings and EHRs. Second, the
CI-based analysis was performed by reviewers who did not perform the interviews; however,
this offers a more independent assessment of findings that might not be apparent during
interviews. The initial interview was collected to understand the decision making
using CDM, and so for the secondary analysis using CI- and FTA-based methods, we used
independent reviews. Third, CI methodology involves both observations and analysis.
However, in clinical practice, it is impractical to directly observe rare events such
as those under study in this care report. Nevertheless, CI-based analysis allowed
useful information to be gleaned from postevent interview. Finally, we did not aim
to identify specific actions to improve efficiency of test result management; however,
our findings help inform future work to identify and test solutions.
Conclusion
We illustrate our application of diverse human factors methods, – CDM, CI, and FTA,
to understand factors in abnormal test result follow-up. Our methods identified multiple
factors contributing to missed follow-up, such as provider–patient communication channel
mismatch and diffusion of responsibility. We focused on identifying barriers to successful
follow-up and pathways leading to inaction. Future directions include expanding these
methods to facilitate design information systems and implementation of preventive
strategies to reduce missed test results.
Clinical Relevance Statement
Clinical Relevance Statement
Adverse events and care delays can occur when physicians miss taking actions on abnormal
test results. However, individual decision-making factors surrounding such events
are less understood. Using a combination of human factors methods described herein
can identify key contributory factors that guide development of preventive interventions.
Multiple Choice Questions
Multiple Choice Questions
1. Which method is useful in understanding information flow in decision making process?
-
Critical decision method
-
Contextual inquiry
-
Process mining
Correct Answer: The correct answer is option b.
2. Which method is useful in understanding sequence of events leading to an adverse
outcome?
-
Fault tree analysis
-
Process mining
-
Critical decision method
Correct Answer: The correct answer is option a.
Critical Decision Method Interview Protocol
Critical Decision Method Interview Protocol
Explanation of Study and Description of Interview
We are interested in how system factors affect the process of follow-up to abnormal
test results.
We've talked with leadership, IT, laboratory, and to some providers to get a high-level
overview of the general process. But because so much of what happens depends on specific
details, to fully understand all the factors, we also need to get concrete, and look
in depth at a sample of cases.
We are looking at a quasirandom sample of recent abnormal test results, sampling a
range of follow-up patterns.
Patient ___ and test results ____ on ____ is one we want to explore. We are interested
in the various system and other factors that played a role in the follow-up of this
case. We'd like to speak to you because you are most knowledgeable about this case
and what factors played a role.
I'd like to first walk through the case and get your description of things. Then I'll
quickly review it with you to make sure I've got the picture. Then I'll ask some more
questions to get a richer picture of what elements played a role in this particular
case.
Do you remember the case?
Do you want to pull up the chart?
Provider Account
Can you walk me through it? Your initial impressions during the appointment, what
your concerns were and what decisions you made ….
Starting at the visit where you ordered the test or before that if you want.
Timeline
Draw timeline
Include
Review with provider
Deepening
In our sampling we've seen similar results that had a more involved follow-up, and
also ones with a less involved follow-up. We've see similar results that had faster
resolution and ones that took more time to be resolved. Do you think there were any
factors that led to this test result being addressed in the way it was? Why wasn't
it slower or faster, or a more involved or less involved follow-up?
Probes for factors:
-
Busy day, staffing level
-
CPOE for this test, this patient, via this exam room/office
-
Patient engagement
-
Patient access to laboratory, radiology
-
Reviewing results this test, this patient, via this exam room/office
-
Confidence versus Uncertainty about patient trajectory
Probe questions:
-
What led you to order test?
-
Training regarding ordering this test via EHR?
-
What were your expectations? What led you to expect that?
-
In what context did you access the results—morning at work, evening at home …
-
What did the results mean to you?
-
What other clinical information did you use to make that assessment?
-
What was your main concern at that point?
-
Did you consider different options? How did you decide to take the next step?
-
What were your expectations about the patient's response? About how the next step
would occur? About the clinical trajectory of the patient?
