Semiautomated System for Nonurgent, Clinically Significant Pathology Results

Abstract

Background Failure of timely test result follow-up has consequences including delayed diagnosis and treatment, added costs, and potential patient harm. Closed-loop communication is key to ensure clinically significant test results (CSTRs) are acknowledged and acted upon appropriately. A previous implementation of the Alert Notification of Critical Results (ANCR) system to facilitate closed-loop communication of imaging CSTRs yielded improved communication of critical radiology results and enhanced adherence to institutional CSTR policies.

Objective This article extends the ANCR application to pathology and evaluates its impact on closed-loop communication of new malignancies, a common and important type of pathology CSTR.

Materials and Methods This Institutional Review Board-approved study was performed at a 150-bed community, academically affiliated hospital. ANCR was adapted for pathology CSTRs. Natural language processing was used on 30,774 pathology reports 13 months pre- and 13 months postintervention, identifying 5,595 reports with malignancies. Electronic health records were reviewed for documented acknowledgment for a random sample of reports. Percent of reports with documented acknowledgment within 15 days assessed institutional policy adherence. Time to acknowledgment was compared pre- versus postintervention and postintervention with and without ANCR alerts. Pathologists were surveyed regarding ANCR use and satisfaction.

Results Acknowledgment within 15 days was documented for 98 of 107 (91.6%) pre- and 89 of 103 (86.4%) postintervention reports (p = 0.2294). Median time to acknowledgment was 7 days (interquartile range [IQR], 3, 11) preintervention and 6 days (IQR, 2, 10) postintervention (p = 0.5083). Postintervention, median time to acknowledgment was 2 days (IQR, 1, 6) for reports with ANCR alerts versus 6 days (IQR, 2.75, 9) for reports without alerts (p = 0.0351). ANCR alerts were sent on 15 of 103 (15%) postintervention reports. All pathologists reported that the ANCR system positively impacted their workflow; 75% (three-fourths) felt that the ANCR system improved efficiency of communicating CSTRs.

Conclusion ANCR expansion to facilitate closed-loop communication of pathology CSTRs was favorably perceived and associated with significant improved time to documented acknowledgment for new malignancies. The rate of adherence to institutional policy did not improve.

Keywords

critical test results - pathology information systems - hospital communication systems - automated electronic alert notification - clinically significant test results

Background and Significance

Failure to follow-up test results has well-documented consequences, including delayed diagnosis and treatment, malpractice litigation, added health care costs, and potential patient harm.[1] [2] [3] [4] [5] In ambulatory settings, examples of test results that typically “fall through the cracks” include nonurgent, clinically significant results of radiology studies (e.g., incidental pulmonary nodules), pathology specimens, microbiology cultures, chemistry and hematology tests, and cardiology studies (e.g., echocardiograms).

Effectively communicating clinically significant test results (CSTRs) is a Joint Commission's National Patient Safety Goal.[6] CSTRs have been defined as “any result that requires further clinical action to avoid morbidity or mortality, regardless of the urgency of that action.”[7] Closed-loop communication is a key process to ensure that CSTRs are acknowledged and acted upon appropriately. Breakdowns in the closed-loop communication process are often due to failure to acknowledge and follow-up test results.[8] [9] [10] [11] [12] [13] [14] Automated systems that alert providers have been shown to improve communication and time to appropriate management of CSTRs.[15]

We previously implemented the Alert Notification of Critical Results (ANCR) system to facilitate closed-loop communication of radiology CSTRs. Alerts are transmitted through either synchronous (i.e., pager) or asynchronous (i.e., secure e-mail) mechanisms, both of which require acknowledgment to close the communication loop.[16] [17] This Web-based software has been optimized for mobile access and allows for a one-click acknowledgment process that leverages our secure institutional single sign-on process. Asynchronous, noninterruptive communication is particularly helpful for clinically significant but nonurgent findings (e.g., pulmonary nodules). Implementing ANCR has been shown to improve communication of critical radiology results and enhance adherence to institutional policies regarding CSTRs, with 95% of reports with critical results adherent to policy and sustained improvement over the 4 years after implementation.[17] [18] ANCR was widely adopted by radiologists—it was used for 81% of all communicated reports by 18 months after implementation; usage persisted for the remainder of the study and afterwards.[17]

With the successful adoption and associated improvement in CSTR management for radiology,[16] [17] [19] there was much interest in expanding the ANCR to other results-generating areas (RGAs) at our institution that struggled with closed-loop communication of CSTRs (e.g., RGAs that have had malpractice claims associated with failure to follow-up high-risk results such as new malignancies), and whose complexity require physician assessment of the clinical situation to determine if a result is clinically significant. For example, pathology results, such as reports of biopsy specimens that demonstrate new malignancies, may not be immediately life-threatening but do require subsequent and timely action (e.g., follow-up with a specialist, surgical intervention, or chemotherapy) to prevent morbidity.

Objective

The aims of this study were to: (1) implement ANCR in a pathology RGA, and (2) evaluate the impact of ANCR on closed-loop communication of a specific type of CSTRs, i.e., new malignancies. We hypothesized that ANCR, if properly integrated into the workflow, would be used to facilitate more timely, closed-loop communication of pathology CSTRs.

Materials and Methods

Study Site and Cohort

This Institutional Review Board-approved study was performed at a 150-bed community hospital affiliated with an academic medical center in a department of 4 pathologists performing over 14,500 cytopathology and surgical pathology exams annually. A total of 30,774 cytopathology or surgical pathology specimens were submitted for 23,948 adult patients during a 13-month preintervention and 13-month postintervention period from January 2012 through April 2014 ([Fig. 1]).

Fig. 1 Study cohort. NLP, natural language processing.

As part of an ongoing institution-wide initiative to standardize test results across our health care network, CSTRs were classified as Level 1, 2, or 3 based on predetermined criteria ([Table 1]).[7] [20] Pathologists could use these criteria to determine an appropriate level of urgency at the time of specimen interpretation. Because the majority of “Level 3” pathology results were new malignancies, we focused our evaluation on new malignancies. Malignancies were considered new if not previously sampled. For example, a malignancy in a biopsy of a breast lesion would be considered new whereas malignancy in the subsequent lumpectomy or mastectomy specimen would not be considered new. We excluded “Level 1” and “Level 2” CSTRs as these were exceedingly rare in our pathology RGA.

Table 1 CSTR criteria and acknowledgment time frames

Level 1: Critical, for example, fat in endometrial curettage or endoscopic polypectomy; acknowledgment of alert required by referring/ordering provider within 1 hour; reminders sent hourly until acknowledged

Level 2: Urgent, for example, leukocytoclastic vasculitis; acknowledgment of alert required by referring/ordering provider within 24 hours; reminders sent daily until acknowledged

Level 3: Nonurgent but clinically significant, for example, new malignancy; acknowledgment of alert required by referring/ordering provider within 15 days; reminders sent weekly until due then daily until acknowledged

Abbreviation: CSTR, clinically significant test result.

Study cohort selection is summarized in [Fig. 1]. To identify pathology reports with malignancies, we developed a natural language processing (NLP) algorithm using a publicly available document retrieval toolkit, Information from Searching Content with an Ontology-Utilizing Toolkit (iSCOUT).[21] We previously used iSCOUT to retrieve radiology reports containing critical results.[22] [23] [24] For this study, we used iSCOUT to identify reports with malignancies and exclude reports that negated the concept, using phrases such as “no carcinoma” or “carcinoma was not seen.” See Appendix A for a complete list of terms. We validated the NLP algorithm by manually reviewing 1,000 randomly selected reports to determine the presence of malignancy: 169 true-positive, 15 false-positive, 816 true-negative, and zero false-negative pathology reports documenting malignancies were identified, yielding a sensitivity of 100%, specificity of 98%, and precision of 92%.

Usual Care

Prior to intervention, to facilitate closed-loop communication, pathologists would open a separate application either to page or e-mail the referring provider with critical results, and then document acknowledgment in the pathology report once the referring provider responded. However, they would still rely on manual reminders to follow-up with those referring providers who did not respond and acknowledge the CSTR. Furthermore, if acknowledgment was delayed, the pathologist would need to addend the pathology report to document the provider's response. This manual workflow likely contributed to missed CSTRs at this pathology RGA.

Intervention

ANCR is a public domain, Web-based application developed to facilitate closed-loop communication of critical radiology test results (including nonurgent, CSTRs) and has been previously described.[16] We adapted ANCR for use within a single pathology department. In a series of meetings, we engaged administrative leadership and all practicing pathologists to understand their clinical workflow, identify potential access points, review policies, and set expectations for use according to our institutional policy.

The new system allowed pathologists to launch ANCR in the context of the patient's specimen being reported ([Fig. 2]) by clicking a button within their current laboratory information system (CoPath, Cerner, North Kansas City, Missouri, United States). The pathologist identified a CSTR according to the institutional policy (see [Table 1]), and alerted the referring provider via pager for Level 1 and Level 2 results, and via pager or secure e-mail for Level 3 results. The phrase “This report was entered into the Alert Notification of Critical Results (ANCR) System at the time of signout” was automatically documented in the pathology report, indicating use of the system (see [Appendix B]). The referring provider could then call the pathologist or follow a link in a secure e-mail directing him/her to the ANCR application to acknowledge the CSTR. Importantly, using ANCR, the pathologist did not have to rely on a manual process for remembering to follow-up with referring providers who did not acknowledge a critical result—the system sent automatic reminders (e.g., weekly until due then daily for Level 3 alerts, see [Table 1]) to both the pathologist and referring provider (e.g., surgeon, oncologist, gynecologist who ordered or performed the biopsy) until the alert was acknowledged. Furthermore, a clinical administrator for pathology monitored a worklist of unacknowledged alerts to ensure all alerts were closed.

Fig. 2 Laboratory information systems integration. The highlighted button launches Alert Notification of Critical Results (ANCR) in context of the patient and exam being reported.

Main Outcomes and Data Collection

The primary outcome was documented acknowledgment within 15 days for new malignancies (adherence to our institutional policy for Level 3 pathology CSTRs).[7] Because no preintervention and few postintervention results were acknowledged in ANCR, chart review was used as the gold standard to identify evidence of documented acknowledgment. Any documentation in clinical notes, result review notes, telephone encounters, pathology reports, and/or procedure notes that identified or addressed the key finding was considered acknowledgment (i.e., stating the key finding, notifying, and/or communicating with the patient or another health care provider, establishing or changing a diagnosis, recommending, ordering, or cancelling a test or study, and/or referring the patient to another provider). The secondary outcome was time to acknowledgment (measured in days), defined as the interval between submission of the specimen and documentation by a health care provider.

A random sample of 400 pathology reports from pre- and postintervention periods that were NLP positive for malignancy was manually reviewed. The electronic health record (EHR) was reviewed to determine if the malignancies were new and identify when new malignancies were first acknowledged. The first 10% were independently reviewed by two physician investigators and agreement in documented alert acknowledgment was calculated using the kappa statistic. All discrepancies were resolved; the final criteria were used by a single physician investigator to review the remaining cases.

We also assessed user reports of ANCR adoption. Pathologists were surveyed about use of ANCR 4 months after implementation. The survey ([Appendix B]) addressed usage patterns, including the types of findings and perceived proportion of reports that triggered use, as well as attitudes toward using ANCR. Responses were categorical, with the option of free-text response. We assessed satisfaction by asking whether ANCR had a positive impact on workflow. Finally, to quantify use of ANCR according to the institutional policy, we measured the proportion of new and nonnew malignancies in our sample for which an ANCR alert was sent.

Power Sample and Statistical Analysis

Based on prior studies, we expected to observe electronic documentation of acknowledgment in 40% of patients with actionable test results when a provider was aware of the result.[13] [25] [26] [27] We previously showed that e-mail notification of actionable test results doubles awareness at 72 hours after becoming available.[25] [26] Therefore, we anticipated that ANCR would result in an increase of documented acknowledgment from 20 to 40%. To detect this change with 80% power and an α of 0.05, we estimated that we would need to review 180 actionable pathology results (90 pre- and 90 postintervention). We reviewed 50 randomly sampled reports that were NLP-positive for malignancy of which 30 (60%) were considered new (and thus CSTRs). Therefore, we would need to review a total of 300 randomly sampled reports that are NLP-positive for malignancy to identify those with new malignancies. To be conservative, we sampled 400 NLP-positive reports: 200 pre- and 200 postintervention.

The proportion of reports with documented acknowledgment was compared in the pre- and postintervention periods using Fisher's exact test. Wilcoxon rank sum was used to compare times to acknowledgment for submitted specimens in the pre- and postintervention periods as well as between ANCR-acknowledged and EHR-acknowledged alerts in the postintervention period. The proportion of postintervention reports of NLP positive for malignancy for which an ANCR was sent was compared among those with new versus previously identified malignancy using Fisher's exact test. A two-sided p-value of < 0.05 was used to determine statistical significance.

Results

Study Cohort and Sample

Of 30,774 pathology reports (23,948 patients), 5,595 (18%; 4,099 patients) were NLP positive for malignancy ([Fig. 1]). Patient characteristics for the study cohort and sample are presented in [Table 2]. In general, patient characteristics in the study cohort and sample were similar in the pre- and postintervention periods. ANCR alerts were generated on a similar proportion of postintervention patients in the study and sample cohort. Of the 2,802 postintervention reports that were NLP positive for malignancy, ANCR alerts were sent on 213 (7.6%). Of the 200 randomly sampled NLP malignancy-positive postintervention reports, new malignancies were identified in 103 (51.5%) cases based on chart review; of these 103, ANCR alerts were generated on 15 (14.6%), a greater proportion than for all 200 NLP-positive reports (7.6%, p = 0.010). Although pathologists had the option of sending Level 3 ANCR alerts by pager or secure e-mail, all alerts were sent via e-mail.

Table 2 Demographics of patients' pathology reports

	NLP+ for malignancy (Pre) n = 2,793	NLP+ for malignancy (Post) n = 2,802	Preintervention sample, n = 200	Postintervention sample, n = 200
No. of unique patients	2,124	2,126	196	195
Age (y)	61 ± 14	62 ± 14	61 ± 14	61 ± 15
Female	75% (2,093)	72% (2,004)	76% (152)	75% (149)
Race
White	81% (2,264)	80% (2,238)	80% (160)	79% (158)
Black	5.6% (156)	5.3% (148)	3.0% (6)	7.5% (15)
Hispanic	2.5% (71)	2.7% (77)	5.5% (11)	3.5% (7)
Asian	1.9% (53)	1.5% (41)	3.0% (6)	1.0% (2)
Other	8.9% (249)	11% (298)	8.5% (17)	9.0% (18)
Insurance
Government	36% (1,004)	37% (1,036)	31% (62)	36% (71)
Private	64% (1,774)	63% (1,756)	69% (137)	64% (128)
Other	0.5% (15)	0.4% (10)	0.5% (1)	0.5% (1)
# of reports with new malignancies validated by chart review	107 (3.8%)	103 (3.7%)	107 (54%)	103 (52%)
# of reports on which an ANCR alert was generated	N/A	213 (7.6%)	N/A	15 (7.5%)[a]

Abbreviations: ANCR, Alert Notification of Critical Results; NLP, natural language processing.

Note: Multiple specimens resulting in one pathology report were treated as single specimens.

^a In the postintervention sample, one additional ANCR alert was generated on a false-positive NLP⁺ pathology report (i.e., without a new malignancy) based on independent chart review; hence, there were 16 (8.0%) NLP⁺ pathology reports on which an ANCR alert was generated in the postintervention sample.

Rate of Documented Acknowledgment

Rates of acknowledgment of pathology reports with new malignancies as documented in the EHR were similar in the pre- and postintervention periods ([Table 3]). For a random subsample of 40 reports, reviewers agreed that a new malignancy was present in 35 (Kappa = 0.73, 95% confidence interval, 0.51–0.94).

Table 3 Estimates of reports with documented acknowledgment

	Preintervention (n = 200)	Postintervention (n = 200)	p-Value
New malignancies with acknowledgment (% [95% CI])	106/107 (99 [95–100]%)	100/103 (97 [92–99]%)	0.3620
New malignancies with acknowledgment within 15 d (% [95% CI])	98/107 (92 [86–97]%)	89/103 (86 [80–93]%)	0.2294

Abbreviation: CI, confidence interval.

Time to Documented Acknowledgment

The median time to acknowledgment for all new malignancies was 7 days (interquartile range [IQR], 3, 11) preintervention and 6 days (IQR, 2, 10) postintervention (p = 0.5083). In the postintervention period, when ANCR was used to communicate and acknowledgment was documented in the EHR, median time to acknowledgment was significantly shorter (2 vs. 7 days, p = 0.0351) than when an ANCR alert was not generated ([Table 4]).

Table 4 Postintervention: differential impact of ANCR on time to acknowledgment

	ANCR alert (n = 15)	No ANCR alert (n = 85)	p-Value
New malignancies, days until acknowledgment (median, [IQR])	2 [1, 6]	7 [2.5, 10.5]	0.0351

Abbreviations: ANCR, Alert Notification of Critical Results; IQR, interquartile range.

In the postintervention sample, ANCR alerts were more often generated on reports with new malignancies compared with reports with malignancies that were not new (14.6% vs. 1.0%, p < 0.01); however, ANCR alerts were not sent on a large proportion of reports with new malignancies (85.4%, n = 88) that should have been designated as clinically significant according to our institutional criteria.

ANCR Pathology Adoption and Use

The four pathologists who comprised the pathology department during the study period (50% female, 6–27 years of experience) were surveyed and all completed the questionnaire. All generated ANCR alerts on less than 25% of reports. Findings that prompted ANCR use included malignancies (4 pathologists), corrected or addendum reports (3 pathologists), and discrepancies between findings and diagnoses documented in preliminary and finalized reports (3 pathologists). All pathologists felt that the ANCR system positively impacted their workflow: of the three pathologists who provided free-text clarifications, all cited increased efficiency in communicating CSTRs and one specifically stated that ANCR was more efficient than paging or e-mail.

Discussion

We observed that electronic documentation of acknowledgment for a specific type of Level 3 CSTR was sustained at a high rate for a pathology RGA pre- and post-ANCR implementation. We observed a significant decrease in time to documented acknowledgment associated with use of ANCR. Although pathologists uniformly had a favorable impression of ANCR with regard to improving workflow for communication of CSTRs, they used ANCR to facilitate closed-loop communication for only a small proportion of CSTRs in our cohort, defined as Level 3 according to the institutional criteria.

The rate of documented acknowledgment within 15 days was unexpectedly high. Prior studies reported rates between 3 and 50%.[13] [25] [26] [27] There are several explanations for this finding: (1) users may have a lower threshold for documenting communication about new malignancies on pathology specimens as compared with other types of results (e.g., microbiology, laboratory tests); (2) new malignancies tend to require complex follow-up actions which are typically documented in the EHR (e.g., consults, additional surgeries, chemotherapy, radiation); (3) nearly half of the malignancies in our cohort were breast cancer, a condition with a well-established workflow for follow-up at this institution; and (4) medicolegal concerns.

We did not observe an improvement in our primary outcome after implementation of ANCR. This may be explained partly by ceiling effects; however, lack of improvement could also be explained by a low rate (∼15%) of ANCR use to facilitate closed-loop communication for the new malignancies sampled in our cohort. Although we held preintervention meetings to review workflow and policy, we did not conduct a formal assessment to understand how ANCR would be used and identify potential usage barriers. Also, in contrast to our other deployments, we did not implement performance reporting to encourage pathologists to use ANCR to facilitate closed-loop communication of new malignancies. Routine performance reporting is an important feedback mechanism that was associated with improved rates of adherence to institutional policy in our prior efforts.[18]

The shorter time to acknowledge CSTRs communicated via ANCR compared with those communicated via routine pathology reports can be attributed to ANCR's Web-based and mobile workflow, and one-click acknowledgment process. Because responsible providers are actively notified of CSTRs via a “push” (e.g., e-mail notification external to the EHR), they are not required to open the EHR to search for new results. Active notification is important for critical results because earlier acknowledgment should lead to more timely action, especially when required follow-up is more urgent.[28] [29] When follow-up action is not urgent, as is often the case for Level 3 CSTRs, acknowledgment may not be documented until the action is actually performed (which may be days after the result is finalized). Thus, in theory, the decreased time to acknowledgment that we observed for ANCR may improve time to action for this subset of test results.

Overall, pathologists had a favorable impression of ANCR: strong support from administrative leadership, clear expectations for use through institutional policy, and tight integration into clinical workflow were key factors identified from our qualitative assessment. Nonetheless, despite uniform adoption of ANCR by all pathologists in our study, ANCR was not uniformly used to facilitate closed-loop communication of CSTRs in accordance with institutional policy. The etiology of the discrepancy was likely multifactorial, most probably due to differences in interpretation of policies at the local and enterprise level regarding types of results considered critical. Specifically, per local departmental policy, only an unexpected malignancy should trigger an alert, whereas per enterprise policy, any new malignancy should trigger an alert. Thus, a departmental pathologist may decide to send an alert just for the subset of new malignancies that were truly not anticipated (i.e., clinical suspicion for cancer was not the indication for biopsy). Alternatively, the pathologist may not remember the policy or decide to notify in another way (manual call/page) without using ANCR. In addition to implementing performance reporting as described above, additional educational training, further harmonization of policies at the local RGA and enterprise levels, and mechanisms to ensure accountability regarding best practices for using closed-loop communication technologies are often necessary to encourage adoption.

Test result management in EHRs remains a source of dissatisfaction[30] and poses a significant safety concern.[31] Further, many providers create potentially unsafe workarounds to fill gaps in functionality.[32] ANCR minimizes this by providing a modern and more familiar Web-based and mobile user experience to those individuals receiving alerts. Indeed, referring providers are much more likely to acknowledge radiology result alerts via ANCR than the EHR when alerted for the same result using both mechanisms and when acknowledgment via one mechanism closed the alert in both systems.[20] There is no reason to suspect that referring providers would find ANCR alerts unacceptable for pathology test results as we have described. Furthermore, ANCR worklists serve as an important memory aid: pathologists can track all alerts, including attempts at escalating those that remain unacknowledged, and referring providers can track both unacknowledged and acknowledged alerts that require follow-up. Also, ANCR improves efficiency of identifying critical pathology reports for referring providers by automated flagging through integration with the EHR. Finally, by ensuring receipt of alerts, ANCR fulfills a Joint Commission requirement for closed-loop communication of critical results.[6]

Our study has several limitations. First, because it was performed at a single institution, our findings may not be generalizable to other institutions with different workflows and policies. Second, although our sampling strategy was adequate, the low rate of ANCR alerts generated for new malignancies may have hindered our ability to detect an impact on our primary outcome. Third, we did not evaluate use of ANCR by referring providers to acknowledge critical findings. Fourth, in our sample size determination, we underestimated the rate of documentation of acknowledgment for new malignancies and unexpectedly encountered ceiling effects during our study. Nonetheless, we still were able to observe improvement in time to documented acknowledgment. Fifth, this study focused on evaluating the effect of a specific technology on closed-loop communication pre- and postimplementation—we did not measure other factors that could have impacted acknowledgment of new malignancies. Finally, our definition of acknowledgment included acting on a critical result; thus, it was not possible to analyze acknowledgment and action separately.

Conclusion

In summary, we observed improvement in time to documented acknowledgment after extending ANCR to facilitate closed-loop communication of nonurgent pathology CSTRs. We believe that routine use of performance reporting and educational/feedback initiatives during implementation could have encouraged more consistent use of ANCR according to institutional CSTR policy. Nonetheless, understanding how closed-loop communication technologies such as ANCR are used by health care providers for different result types and in different health care settings will help shape future interventions to improve follow-up of CSTRs, while addressing the heterogeneity in test result follow-up practices. Future studies should focus on evaluating impact on other types of pathology CSTRs, and expand use of ANCR or similar tools to other RGAs (e.g., echocardiology) and care settings (e.g., care transitions). Importantly, formal usability assessments should be conducted to thoroughly understand implementation barriers and facilitators. Clarifying institutional policy, harmonizing expectations among RGAs, and improving communication is paramount to ensure successful implementation of technologies to facilitate closed-loop communication of test results.[33]

Clinical Relevance Statement

A Web-based software tool developed to facilitate communication of radiology CSTRs can be successfully expanded to pathology, garnering a favorable impression from pathologists, reducing the time to CSTR acknowledgment, and setting the stage for expansion to other areas as an enterprise-wide solution. Despite a positive reception, the software was only used for a minority of new malignancies, all of which would be considered CSTRs by enterprise policy. This highlights the need for CSTR communication policy harmonization between local results generating areas and the enterprise, establishment of performance reviews, creation of education initiatives, and understanding the heterogeneity of test result follow-up practices.

Multiple Choice Questions

1. Which of the following is true regarding the Joint Commission's National Patient and Safety Goal for communication of critical results (NPSG.02.03.01)?

   • It leaves the definition of critical results up to local organizational leadership.

   • It specifies the acceptable length of time between the availability and reporting of critical results.

   • It applies to tests and diagnostic procedures in the inpatient and emergency care settings, but not the outpatient setting.

   • It provides a list of acceptable provider roles which can receive and acknowledge critical results.

   Correct Answer: The correct answer is option a. The Joint Commission's National Patient and Safety Goal for communication of critical results includes collaboration with organization leaders to define critical results, determine who gives and receives critical results communication, and set the time by which communication must occur. It applies in all patient care settings.

2. A Web-based application was developed to facilitate closed-loop communication of clinically significant test results was associated with:

   • High rates of use of the system by pathologists to report new malignancies.

   • Improved rates of acknowledgment by ambulatory clinicians of pathology reports with new malignancies.

   • No change in reported workflow efficiency by pathologists.

   • A reduction in median time to acknowledgment of a pathology report of new malignancy.

   Correct Answer: The correct answer is option d. When the system was used, there was a significant reduction in median time to acknowledgment and all pathologists thought the system improved their efficiency. However, the system was not highly used and there was no significant change in the high rates of acknowledgment before and after implementation.

Conflict of Interest

None.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the Brigham and Women's Institutional Review Board.

• References

• 1 Wahls T, Haugen T, Cram P. The continuing problem of missed test results in an integrated health system with an advanced electronic medical record. Jt Comm J Qual Patient Saf 2007; 33 (08) 485-492
• 2 Wahls TL, Cram PM. The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract 2007; 8: 32
• 3 Gandhi TK, Kachalia A, Thomas EJ. , et al. Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med 2006; 145 (07) 488-496
• 4 Callen J, Georgiou A, Li J, Westbrook JI. The safety implications of missed test results for hospitalised patients: a systematic review. BMJ Qual Saf 2011; 20 (02) 194-199
• 5 Singh H, Thomas EJ, Mani S. , et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential?. Arch Intern Med 2009; 169 (17) 1578-1586
• 6 National Patient Safety Goals | Joint Commission. Available at: http://www.jointcommission.org/standards_information/npsgs.aspx . Accessed April 22, 2016
• 7 Roy CL, Rothschild JM, Dighe AS. , et al. An initiative to improve the management of clinically significant test results in a large health care network. Jt Comm J Qual Patient Saf 2013; 39 (11) 517-527
• 8 Singh H, Arora HS, Vij MS, Rao R, Khan MM, Petersen LA. Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007; 14 (04) 459-466
• 9 Howanitz PJ, Steindel SJ, Heard NV. Laboratory critical values policies and procedures: a college of American Pathologists Q-Probes Study in 623 institutions. Arch Pathol Lab Med 2002; 126 (06) 663-669
• 10 Singh H, Thomas EJ, Petersen LA, Studdert DM. Medical errors involving trainees: a study of closed malpractice claims from 5 insurers. Arch Intern Med 2007; 167 (19) 2030-2036
• 11 Greenberg CC, Regenbogen SE, Studdert DM. , et al. Patterns of communication breakdowns resulting in injury to surgical patients. J Am Coll Surg 2007; 204 (04) 533-540
• 12 Poon EG, Haas JS, Louise Puopolo A. , et al. Communication factors in the follow-up of abnormal mammograms. J Gen Intern Med 2004; 19 (04) 316-323
• 13 Roy CL, Poon EG, Karson AS. , et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med 2005; 143 (02) 121-128
• 14 Gandhi TK. Fumbled handoffs: one dropped ball after another. Ann Intern Med 2005; 142 (05) 352-358
• 15 Parl FF, O'Leary MF, Kaiser AB, Paulett JM, Statnikova K, Shultz EK. Implementation of a closed-loop reporting system for critical values and clinical communication in compliance with goals of the joint commission. Clin Chem 2010; 56 (03) 417-423
• 16 Lacson R, O'Connor SD, Andriole KP, Prevedello LM, Khorasani R. Automated critical test result notification system: architecture, design, and assessment of provider satisfaction. Am J Roentgenol 2014; 203 (05) 491-496
• 17 Lacson R, Prevedello LM, Andriole KP. , et al. Four-year impact of an alert notification system on closed-loop communication of critical test results. Am J Roentgenol 2014; 203 (05) 933-938
• 18 Anthony SG, Prevedello LM, Damiano MM. , et al. Impact of a 4-year quality improvement initiative to improve communication of critical imaging test results. Radiology 2011; 259 (03) 802-807
• 19 Lacson R, O'Connor SD, Sahni VA. , et al. Impact of an electronic alert notification system embedded in radiologists' workflow on closed-loop communication of critical results: a time series analysis. BMJ Qual Saf 2016; 25 (07) 518-524
• 20 O'Connor SD, Dalal AK, Sahni VA, Lacson R, Khorasani R. Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up?. J Am Med Inform Assoc 2016; 23 (02) 333-338
• 21 Lacson R, Andriole KP, Prevedello LM, Khorasani R. Information from Searching Content with an Ontology-Utilizing Toolkit (iSCOUT). J Digit Imaging 2012; 25 (04) 512-519
• 22 Lacson R, Sugarbaker N, Prevedello LM. , et al. Retrieval of radiology reports citing critical findings with disease-specific customization. Open Med Inform J 2012; 6: 28-35
• 23 Gershanik EF, Lacson R, Khorasani R. Critical finding capture in the impression section of radiology reports. AMIA Annu Symp Proc 2011; 2011: 465-469
• 24 Warden GI, Lacson R, Khorasani R. Leveraging terminologies for retrieval of radiology reports with critical imaging findings. AMIA Annu Symp Proc 2011; 2011: 1481-1488
• 25 El-Kareh R, Roy C, Williams DH, Poon EG. Impact of automated alerts on follow-up of post-discharge microbiology results: a cluster randomized controlled trial. J Gen Intern Med 2012; 27 (10) 1243-1250
• 26 Dalal AK, Roy CL, Poon EG. , et al. Impact of an automated email notification system for results of tests pending at discharge: a cluster-randomized controlled trial. J Am Med Inform Assoc 2014; 21 (03) 473-480
• 27 Harrison AM, Thongprayoon C, Aakre CA. , et al. Comparison of methods of alert acknowledgement by critical care clinicians in the ICU setting. PeerJ 2017; 5: e3083
• 28 Kuperman GJ, Teich JM, Tanasijevic MJ. , et al. Improving response to critical laboratory results with automation: results of a randomized controlled trial. J Am Med Inform Assoc 1999; 6 (06) 512-522
• 29 Dalal AK, Schaffer A, Gershanik EF. , et al. The impact of automated notification on follow-up of actionable tests pending at discharge: a cluster-randomized controlled trial. J Gen Intern Med 2018; DOI: 10.1007/s11606-018-4393-y.
• 30 Poon EG, Gandhi TK, Sequist TD, Murff HJ, Karson AS, Bates DW. “I wish I had seen this test result earlier!”: dissatisfaction with test result management systems in primary care. Arch Intern Med 2004; 164 (20) 2223-2228
• 31 Kwan JL, Cram P. Do not assume that no news is good news: test result management and communication in primary care. BMJ Qual Saf 2015; 24 (11) 664-666
• 32 Menon S, Murphy DR, Singh H, Meyer AND, Sittig DF. Workarounds and test results follow-up in electronic health record-based primary care. Appl Clin Inform 2016; 7 (02) 543-559
• 33 Renshaw A, Gould EW. Quality assurance measures for critical diagnoses in anatomic pathology. Am J Clin Pathol 2012; 137 (03) 466-469