A Quality Improvement Initiative to Decrease Platelet Ordering Errors and a Proposed Model for Evaluating Clinical Decision Support Effectiveness

Abstract

Background and Objectives Clinical decision support (CDS) and computerized provider order entry have been shown to improve health care quality and safety, but may also generate previously unanticipated errors. We identified multiple CDS tools for platelet transfusion orders. In this study, we sought to evaluate and improve the effectiveness of those CDS tools while creating and testing a framework for future evaluation of other CDS tools.

Methods Using a query of an enterprise data warehouse at a tertiary care pediatric hospital, we conducted a retrospective analysis to assess baseline use and performance of existing CDS for platelet transfusion orders. Our outcome measure was the percentage of platelet undertransfusion ordering errors. Errors were defined as platelet transfusion volumes ordered which were less than the amount recommended by the order set used. We then redesigned our CDS and measured the impact of our intervention prospectively using statistical process control methodology.

Results We identified that 62% of all platelet transfusion orders were placed with one of two order sets (Inpatient Service 1 and Inpatient Service 2). The Inpatient Service 1 order set had a significantly higher occurrence of ordering errors (3.10% compared with 1.20%). After our interventions, platelet transfusion order error occurrence on Inpatient Service 1 decreased from 3.10 to 0.33%.

Conclusion We successfully reduced platelet transfusion ordering errors by redesigning our CDS tools. We suggest that the use of collections of clinical data may help identify patterns in erroneous ordering, which could otherwise go undetected. We have created a framework which can be used to evaluate the effectiveness of other similar CDS tools.

Keywords

patient safety - quality improvement - platelet transfusion - clinical decision support systems - human–computer interaction

Background and Significance

As increasing attention is directed toward preventing medical errors, new emphasis has been placed on the use of clinical information systems, including clinical decision support (CDS) and computerized provider order entry (CPOE), to improve health care quality and patient safety.[1] [2] The expansion of CDS and CPOE was accelerated by the Health Information Technology for Economic and Clinical Health Act, which created incentives for the “meaningful use” of the electronic health record (EHR) to improve health care quality.[3] CDS systems can provide evidence-based recommendations integrated into a provider's clinical workflow and may include order sets, alerts, reminders, and drug dosing recommendations.[4] [5] [6] [7] [8] CDS and CPOE have frequently been shown to improve guideline adherence,[5] [9] [10] [11] [12] [13] [14] [15] prevent medical errors,[12] [16] [17] [18] [19] and improve health care quality.[5] [20] [21] [22] [23] [24] Unfortunately, CPOE and CDS have also been shown to generate unintended consequences, including the creation of previously unanticipated errors.[25] [26] [27] [28] [29] [30] Many of these unintended consequences can lead to adverse events which go undetected.[29] [31] An example of this new kind of error includes a “juxtaposition” error, where a user unknowingly selects the wrong option between choices that were adjacent to one another in a selectable list.[26] [27] Guidelines exist for the development, governance, and monitoring of CPOE and CDS systems; however, they were not available when our institution implemented our EHR.[32]

Prophylactic platelet transfusion is widely recommended to prevent spontaneous hemorrhage in severely thrombocytopenic patients.[33] Pediatric platelet transfusion volume is typically weight based.[34] Transfusing less than the recommended volume could lead to increased risk of severe or life-threatening hemorrhage.[35] Additionally, repeated exposure to platelet products puts patients at increased risk for transfusion-related adverse events like infections, transfusion-related acute lung injury, and anaphylaxis.[35] [36] [37] At our hospital, development of order sets at different times for different services led to the existence of similar order sets that provide recommendations for weight-based pediatric platelet transfusion volume. Prior studies have shown that CDS can reduce ordering errors for blood products, but that physician noncompliance with recommendations remained high.[38] [39]

Objectives

The aim of this study was to (1) determine the frequency of platelet transfusion ordering errors using commonly used platelet transfusion order sets. Because we hypothesized that the error rates would differ between different order sets, we also aimed to (2) reduce the number of platelet transfusion ordering errors by 50% using the order set identified to have the highest percentage of ordering errors; and (3) to create a generalizable model for the postimplementation monitoring of commonly utilized CDS tools at our institution.

Methods

Setting

The study was conducted at Boston Children's Hospital in Boston, Massachusetts, United States, which is a 404-bed tertiary pediatric care hospital. This hospital trains fellows, residents, and medical students.

Our hospital utilizes a commercial EHR with customized order sets and other CDS. During the initial implementation of CPOE and CDS, the clinical informatics team designed and implemented order sets based on the clinical recommendations from subject matter experts. After the initial implementation of order sets, development of subsequent order sets was decentralized, leading to variation in order set design. Some years later, a Clinical Orders Oversight Committee was developed to provide governance over maintenance of existing order sets and creation of new ones. The Clinical Orders Oversight Committee includes physician and nurse informaticists, information technology staff, as well as staff from most functional departments in the hospital (i.e., laboratory staff, radiology staff, among others).

For this quality improvement (QI) initiative, we identified all order sets that provide recommendations for weight-based pediatric platelet transfusion volume. We focused on the two most commonly used order sets and orders placed without the use of an order set. The first order set is intended for use in Inpatient Service 1 and was created during the initial CPOE implementation. The other, intended for use in Inpatient Service 2, was developed several years later. One service is a pediatric intensive care unit (ICU), and the other is an inpatient pediatric hematology/oncology service. Although the patient populations on these units are different, those receiving platelet transfusions are often the same, and the transfusion volume recommendations are the same.

Design

In the initial planning phase for the study, we reviewed all existing platelet transfusion ordering CDS at a tertiary care pediatric hospital. Next, a retrospective cohort study was conducted to compare order error rates for platelet transfusions placed using the two mostly commonly used order sets, the Inpatient Service 1 and Inpatient Service 2 order sets, as well as orders placed without the use of any order set. Based on preliminary data review, we focused specifically on the order set with the highest order error percentage, Inpatient Service 1. The results of this study and our critical evaluation of the platelet transfusion order sets were reviewed with our Clinical Orders Oversight Committee.

Intervention

Based on preliminary data review, the display of the Inpatient Service 1 order set was modified. The original Inpatient Service 1 order set display had check boxes flanked by two different weight parameters, and required users to select a box for the appropriate weight category to place the order ([Fig. 1]). Each weight category has a recommended number of units of platelets for transfusion, with the volume of each unit approximately 20 to 40 mL. The display was updated using user-centered design principles to improve usability and reduce undertransfusion ordering errors ([Fig. 1]). After these modifications, the revised Inpatient Service 1 order set was reviewed by key stakeholders in the Inpatient Service 1 department including multiple nurse practitioners, attending providers, and the resident rotation director. The order set was additionally reviewed by the Clinical Orders Oversight Committee prior to implementation. After this updated order set was implemented, a quasi-experimental time-series design was used to track undertransfusion ordering errors prospectively.[40]

Fig. 1 Order sets.

All platelet transfusion orders placed over the previous year were collected and reviewed. The results of this review and the above changes were presented at an Inpatient Service 1 Mortality and Morbidity Conference.

Data Collection

We queried our enterprise data warehouse to identify all platelet transfusion orders that were placed from January 1, 2017 through December 12, 2017 (baseline period). We excluded platelet transfusion orders that were placed in preparation for surgery or other invasive procedures as we were specifically interested in prophylactic transfusion in the setting of thrombocytopenia, which was the clinical indication for use of the existing order sets. Additionally, we excluded orders placed in the cardiac ICU as those patients often had both different indications and thresholds for platelet transfusion as well as a clinical indication for transfusing lower platelet volumes due to the extreme fluid sensitivity of the patient population. Subsequently, we prospectively followed all platelet transfusion orders placed using the two most commonly used order sets (Inpatient Service 1 and Inpatient Service 2 order sets), as well as orders placed without the use of any order set, from December 13, 2017 through June 30, 2018 (intervention period). The same exclusion criteria as above were applied. All suspected errors identified were confirmed with manual chart review by which included a review of provider documentation for clinical indications which would justify purposefully limiting transfusion volume. The manual chart review was performed by two physicians who independently reviewed the charts of all suspected errors.

Measures

The primary outcome measure was the percentage of platelet transfusion undertransfusion ordering errors. Errors were defined as platelet transfusion volumes ordered which were less than the amount recommended by the order set used. We did not characterize orders placed for greater than the recommended volume of platelets as errors as there are frequently clinical and patient-specific indications for such orders.

Analysis

For our primary outcome measure, we compared the percentage of ordering errors when using the Inpatient Service 1 or Inpatient Service 2 order set, or the use of no order set, during the baseline and intervention periods. A chi-squared test was used to compare proportions. p-Values of < 0.05 were considered significant. Statistical process control charts were used to track the outcome measure over time and to assess for special cause variation. The centerline is the average platelet transfusion order error percentage in the baseline period. Control charts were created using CHARTrunner version 3.0 (PQ Systems, Dayton, Ohio, United States). Special cause variation was defined as a run of 6 or more consecutive points above or below the centerline, signaling a shift.[41] The interrater agreement of the manual chart review was measured using a kappa statistic.

An additional subgroup analysis was done to exclude errors when the ordering weight was within 1 kg of the category below as this could have been ordered at the clinician's discretion, given the proximity to the lower weight category. For example, if the patient weighed 12.7 kg, and was ordered for the platelet transfusion volume recommended for the less than 12 kg category, this would not count as an error in this subgroup analysis.

Results

We identified six different ways in which platelet transfusions were ordered, including five unique order sets and placing an order without the use of any order set. Sixty-two percent of all platelet transfusion orders were placed using either the Inpatient Service 1 or Inpatient Service 2 platelet transfusion order set, while 35% were placed without the use of any order set, and approximately 4% used another platelet transfusion order set. We did not analyze the order sets used for this 4% of orders given the low usage rate. About 1% of orders were placed using one of two remaining order sets, which were not analyzed given the low rate of usage. We found that orders placed without the use of any order set increased from 33% of all orders in the preintervention period to 37% of all orders placed in the postintervention period (p-value < 0.05).

We identified 4,144 platelet transfusion orders that were placed using either the Inpatient Service 1 or Inpatient Service 2 order set. We additionally identified 2,111 orders that were placed without the use of any order set. Of the orders placed with the Inpatient Service 1 or Inpatient Service 2 order set, or placed without use of any order set, 1,221 were excluded based on the exclusion criteria above leaving 5,034 platelet transfusion orders in the analysis. These orders were placed for 472 unique patients, with each patient receiving an average of 10.3 platelet transfusions. Of the included orders, 3,083 were from the baseline period, and 1,951 were from the intervention period. Chart review of suspected platelet ordering errors did not identify any documentation to suggest platelet transfusion was purposefully limited due to clinical concerns of volume overload or any other reason. There was excellent agreement between the two independent physicians' chart review (κ = 0.91). Using the weight-based recommendations in the order sets, 69 errors were identified (1.4% of all platelet orders). Of those errors, 53 resulted in undertransfusion (77%). The remaining 23% of ordering errors were corrected before the transfusion occurred. In every instance of ordering error when an order set was used, the ordered volume corresponded to the expected volume for the weight range below the patient's.

In the baseline time period, we identified significantly more undertransfusion ordering errors on orders placed using the Inpatient Service 1 as compared with the Inpatient Service 2 order set ([Table 1]). Additionally, we identified significantly more undertransfusion ordering errors on orders placed using the Inpatient Service 1 order set as compared with use of no order set. The percentage of platelet undertransfusion ordering errors using the Inpatient Service 1 order set decreased from a mean of 3.1% in the baseline period to a mean of 0.3% in the intervention period ([Fig. 2]). The decrease in the platelet undertransfusion ordering error occurrence coincided with the implementation of the revised order set. After revision, there were significantly fewer errors on orders placed using the Inpatient Service 1 order set as compared with the use of no order set. There was no significant difference between the use of the Inpatient Service 1 or Inpatient Service 2 order set in the intervention period. The percentage of platelet undertransfusion ordering errors using the Inpatient Service 2 order set did not change significantly from the baseline to the intervention period.

Table 1 Platelet transfusion ordering errors using the Inpatient Service 1 and Inpatient Service 2 order sets

Baseline period (1/1/17–12/12/17)	Orders	Errors	Percentage errors	p-Value
Inpatient Service 1 order set (original order set)	1,249	39	3.1%
Inpatient Service 2 order set	740	9	1.2%	< 0.05[a]
No order set	1,094	5	0.46%	< 0.001[a]
Intervention period (12/13/17–6/30/18)
Inpatient Service 1 order set (revised order set)	897	3	0.3%	< 0.001[a]
Inpatient Service 2 order set	335	2	0.6%	0.52[b]
No order set	719	11	1.5%	< 0.05[c]

^a p-Value as compared with Inpatient Service 1 baseline period (original order set).

^b p-Value as compared with Inpatient Service 2 baseline period.

^c p-Value as compared with Inpatient Service 1 intervention period (revised order set).

Fig. 2 Statistical control p-chart: Inpatient Service 1 platelet transfusion order set ordering error rate.

There was no change in the results for the additional subgroup analysis that was done to exclude errors when the ordering weight was within 1 kg of the category below (data not shown).

Discussion

Summary

We found that significantly more platelet undertransfusion ordering errors were made when using the Inpatient Service 1 order set compared with the Inpatient Service 2 order set. Given the display of the original Inpatient Service 1 order set, with each check box flanked by two different weight parameters, we suspect that the increased error frequency was the result of juxtaposition errors, which were less likely with the design of the Inpatient Service 2 order set. This hypothesis is supported by the rapid decrease in undertransfusion ordering errors for orders placed using the Inpatient Service 1 order set during the intervention period. Interestingly, even though the Inpatient Service 2 order set requires users to manually type the platelet volume, this suggests that there were few or no errors caused by users inadvertently hitting the wrong key. We found that the number of orders placed without using an order set increased significantly after implementation of our revised order set. The reason for this change is unclear, and future QI efforts can assess barriers to use of order sets and methods to develop new CDS or workflows that would encourage more use. Additionally, we found that 77% of incorrect orders resulted in actual transfusion of the incorrect volume of platelets, which suggests these errors frequently go unidentified by both blood bank and nursing staff when reviewing and approving the orders. This may be due to lack of awareness of transfusion guidelines, inadequate processes in place to confirm appropriate dosing, or overreliance on technology.[27] At the time of development of many of our institutional order sets, design options were limited within our commercial EHR. When the Inpatient Service 1 order set was developed, the CDS tool placed the actionable item (checkbox) between two lines of text, making it difficult to clearly associate the actionable item with the relevant reference text. At the time of development of both the original and revised order set, we did not have the capability to select a default option based on the most recent weight documented. While our EHR has always provided the ability for weight-based dosing of medications, it has not provided the ability for weight-based dosing of blood products. This capability has subsequently been added, and one of our next steps will be to modify the order sets to include a default selection. Additionally, like many other institutions, the individuals designing these first order sets had no formal training for this process and minimal experience with CPOE. This study demonstrates how CDS and CPOE can lead to unintended consequences that manifest as silent errors, which can be difficult to detect even with careful analysis.[26] [42] At our institution, we did not find any errors reported in our safety event reporting system or mentioned in review of clinical notes during several months of our baseline data review, which suggests these errors may go unrecognized by the clinical team. Moreover, the low overall error ordering rate contributes to the difficulty of recognizing the recurrence of this type of error. We suggest that the use of collections of clinical data may help identify patterns in erroneous ordering, which could otherwise go undetected. As CDS continues to propagate to assist with guideline adherence and QI efforts,[8] designers should recognize the opportunity for error when creating CDS tools.[43] Given that these types of errors can be hard to predict, organizations should utilize usability testing and postimplementation monitoring to mitigate the risk of unintended errors.[26] [42] [44] [45] [46]

The Safety Assurance Factors for Electronic Health Record Resilience guidelines include usability testing and monitoring as recommended practices for institutions using CDS and CPOE.[47] Institutions using CDS should have procedures that require regular review of order sets that includes both a clinical evaluation and patient safety evaluation with data when possible. Based on the success of our intervention, we propose the following framework as one potential method for identifying and tracking the effectiveness and safety of CDS tools ([Fig. 3]). We suggest that there are multiple ways to identify CDS tools for review using this framework, including: review of all new CDS tools, a random or scheduled screening of existing CDS tools, or review of CDS tools when there is concern for an issue (e.g., through error reports from frontline staff). Additionally, as institutions develop new CDS tools it is critical to identify measurable outcomes to monitor for errors prospectively. While this type of testing and monitoring can be costly and time-consuming, it may provide long-term cost savings through improved adherence to guidelines, decreased practice variation, and minimizing adverse events, particularly for high-risk therapies, like blood product administration.

Fig. 3 Framework for tracking effectiveness of clinical decision support tools.

Finally, there is an important role for clinical informaticists as a key part of the development, approval, and monitoring of CDS. Institutions should ensure they have enough clinicians who are knowledgeable about informatics and user-centered design to support safe practices.

Limitations

This study has several limitations. First, we recognize that in our initial baseline data collection period, we compared order sets developed for use in Inpatient Service 1 to order sets for use in Inpatient Service 2 which care for different, though somewhat overlapping, patient populations. While these patient populations are different, the patients receiving platelet transfusions are often the same and the platelet transfusion recommendations are the same. Second, the proposed approach for tracking effectiveness of CDS tools is only applicable if there is a measurable outcome. Of course, there are many types of CDS tools where no simple metric can be created, in which case another type of postimplementation monitoring would be indicated. Third, though it is known that transfusing less than the recommend amount of platelets could increase the risk of hemorrhage or other exposure-related adverse reactions, we did not track the rates of these occurrences and so cannot comment on whether our revised clinical decision order set has reduced the risk of these outcomes. Finally, we recognize that the data reporting infrastructure available at a tertiary care pediatric hospital may not be available at all organizations, which may limit the generalizability of our approach.

Conclusion

This QI intervention successfully reduced platelet transfusion ordering errors placed using an order set which could be easily misinterpreted. A similar initiative could be undertaken for many CDS tools to identify and prevent medical errors and other unintended consequences of CDS. This type of study will likely become increasingly important as CDS becomes more ubiquitous.

Clinical Relevance Statement

This study describes one way that clinical decision support design may contribute to ordering errors. In this study, the order set design was associated with an increased frequency of ordering an inappropriately low volume platelet transfusion. Ordering providers and clinical decision support designers should be aware of the way that design can facilitate or prevent ordering errors. Additionally, this study describes a framework which could be used to identify and track the effectiveness of similar clinical decision support tools, which could ultimately reduce or prevent ordering errors.

Multiple Choice Questions

1. The Health Information Technology for Economic and Clinical Health (HITECH) Act created incentives for which of the following?

   • The meaningful use of the electronic health record to improve health care quality.

   • The use of mobile devices to access and document in the electronic health record.

   • The use of high resolution imaging devices to improve radiological study quality.

   • The use of state-of-the-art laboratory equipment to improve laboratory efficiency.

   Correct Answer: The correct answer is option a, the meaningful use of the electronic health record to improve health care quality. The Health Information Technology for Economic and Clinical Health (HITECH) was enacted in 2009 under the American Recovery and Reinvestment Act. The purpose of the Act was to increase adoption, implementation, and encourage the meaningful use of health information technology.[3]

2. What is a juxtaposition error?

   • Walking to the wrong patient bed when patients are in shared rooms.

   • Inserting a similar sounding but incorrect word when using dictation software.

   • Selecting the wrong ordering option between choices that were adjacent to one another.

   • Placing orders on the wrong patient when multiple tabs are open in the electronic health record.

   Correct Answer: The correct answer is option c. Like the ordering error made in this study, a juxtaposition error is when a user unknowingly selects the wrong option between choices that were adjacent to one another in a selectable list. This is an example of an unanticipated consequence of computerized provider order entry systems.[27] [28]

3. Which of the following are guidelines that recommend best safety practices for organizations using electronic health records?

   • American Institute of Electronic Health Record Safety Guidelines.

   • Safety Assurance Factors for Electronic Health Record Resilience (SAFER) Guides.

   • U.S. Preventative Services Task Force Recommendations on Electronic Health Record Safety.

   • Boston Guidelines for Safe Use of the Electronic Health Record.

   Correct Answer: The correct answer is option b, safety Assurance Factors for Electronic Health Record Resilience (SAFER) Guides. The SAFER guides are self-assessment checklists according to evidence-based practice for safe electronic health record implementation. These guides include a recommendation for usability testing and postimplementation monitoring in organizations using clinical decision support and computerized provider order entry.

Conflict of Interest

None declared.

Acknowledgments

We acknowledge the Boston Children's Hospital business intelligence department and staff for their support and access to the enterprise data warehouse. We acknowledge Chris Wong Quiles for her support in disseminating these findings to the Inpatient Service 1 department. We acknowledge the staff and providers in the oncology department for their guidance and feedback while developing the revised order set.

Protection of Human and Animal Subjects

Quality improvement projects that are designed to improve clinical care to better conform to established or accepted standards are considered exempt by our institutional review board.

• References

• 1 Institute of Medicine Committee on Quality of Health Care in A. In: Kohn LT, Corrigan JM, Donaldson MS. , eds. To Err is Human: Building a Safer Health System. Washington, DC: National Academies Press (US); Copyright 2000 by the National Academy of Sciences. All rights reserved; 2000
• 2 Institute of Medicine Committee on Quality of Health Care in A. Crossing the Quality Chasm. A New Health System for the 21st Century. Washington, DC: National Academies Press (US); Copyright 2001 by the National Academy of Sciences. All rights reserved; 2001
• 3 Fact Sheet. Medicare and Medicaid EHR incentive programs: Title IV of Recovery Act; 2009 . Available at: www.cms.gov/EHRIncentivePrograms . Accessed June 15, 2019
• 4 Wright A, Sittig DF, Ash JS. , et al. Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc 2011; 18 (03) 232-242
• 5 Jaspers MWM, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc 2011; 18 (03) 327-334
• 6 Osheroff JA. , Healthcare Information and Management Systems Society. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed. Chicago, IL: HIMSS; 2005
• 7 Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007; 14 (02) 141-145
• 8 Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform 2016; (Suppl. 01) S103-S116
• 9 Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J. Effect of point-of-care computer reminders on physician behaviour: a systematic review. CMAJ 2010; 182 (05) E216-E225
• 10 Kitchlu A, Abdelshaheed T, Tullis E, Gupta S. Gaps in the inpatient management of chronic obstructive pulmonary disease exacerbation and impact of an evidence-based order set. Can Respir J 2015; 22 (03) 157-162
• 11 Jacobs BR, Hart KW, Rucker DW. Reduction in clinical variance using targeted design changes in computerized provider order entry (CPOE) order sets: impact on hospitalized children with acute asthma exacerbation. Appl Clin Inform 2012; 3 (01) 52-63
• 12 Chaudhry B, Wang J, Wu S. , et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752
• 13 Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med 2001; 345 (13) 965-970
• 14 Scheitel MR, Kessler ME, Shellum JL. , et al. Effect of a novel clinical decision support tool on the efficiency and accuracy of treatment recommendations for cholesterol management. Appl Clin Inform 2017; 8 (01) 124-136
• 15 Utidjian LH, Hogan A, Michel J. , et al. Clinical decision support and palivizumab: a means to protect from respiratory syncytial virus. Appl Clin Inform 2015; 6 (04) 769-784
• 16 Stürzlinger H, Hiebinger C, Pertl D, Traurig P. Computerized physician order entry - effectiveness and efficiency of electronic medication ordering with decision support systems. GMS Health Technol Assess 2009; 5: Doc07
• 17 Devine EB, Hansen RN, Wilson-Norton JL. , et al. The impact of computerized provider order entry on medication errors in a multispecialty group practice. J Am Med Inform Assoc 2010; 17 (01) 78-84
• 18 Wolfstadt JI, Gurwitz JH, Field TS. , et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008; 23 (04) 451-458
• 19 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
• 20 Garg AX, Adhikari NK, McDonald H. , et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238
• 21 Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998; 280 (15) 1339-1346
• 22 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 23 Bright TJ, Wong A, Dhurjati R. , et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 24 Bates DW, Pappius E, Kuperman GJ. , et al. Using information systems to measure and improve quality. Int J Med Inform 1999; 53 (2-3): 115-124
• 25 Sittig DF, Wright A, Simonaitis L. , et al. The state of the art in clinical knowledge management: an inventory of tools and techniques. Int J Med Inform 2010; 79 (01) 44-57
• 26 Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annual Symposium Proceedings. AMIA Symposium; 2007: 26-30
• 27 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006; 13 (05) 547-556
• 28 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-423
• 29 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: the health IT safety framework. BMJ Qual Saf 2016; 25 (04) 226-232
• 30 Koppel R, Metlay JP, Cohen A. , et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-1203
• 31 Koppel R, Chen Y. Unintended consequences: new problems, new solutions. Contributions from 2015. Yearb Med Inform 2016; (01) 87-92
• 32 Computerized Provider Order Entry with Decision Support. The Office of the National Coordinator for Health Information Technology; 2016 . Available at: HealthIT.gov . Accessed June 15, 2019
• 33 Schiffer CA, Bohlke K, Delaney M. , et al. Platelet transfusion for patients with cancer: American Society of Clinical Oncology Clinical Practice guideline update. J Clin Oncol 2018; 36 (03) 283-299
• 34 Roseff S. Pediatric Transfusion: A Physician’s Handbook. 3rd ed. Bethesda, MD: AABB Press; 2009
• 35 Heddle NM, Cook RJ, Tinmouth A. , et al; SToP Study Investigators of the BEST Collaborative. A randomized controlled trial comparing standard- and low-dose strategies for transfusion of platelets (SToP) to patients with thrombocytopenia. Blood 2009; 113 (07) 1564-1573
• 36 Benson AB, Moss M, Silliman CC. Transfusion-related acute lung injury (TRALI): a clinical review with emphasis on the critically ill. Br J Haematol 2009; 147 (04) 431-443
• 37 Bolton-Maggs PHB, Cohen H. Serious Hazards of Transfusion (SHOT) haemovigilance and progress is improving transfusion safety. Br J Haematol 2013; 163 (03) 303-314
• 38 Rothschild JM, McGurk S, Honour M. , et al. Assessment of education and computerized decision support interventions for improving transfusion practice. Transfusion 2007; 47 (02) 228-239
• 39 Chang CS, Lin YC, Wu YC, Yeh CJ, Lin YC. The effects of a computerized transfusion decision support system on physician compliance and its appropriateness for fresh frozen plasma use in a medical center. Am J Clin Pathol 2011; 135 (03) 417-422
• 40 Speroff T, O'Connor GT. Study designs for PDSA quality improvement research. Qual Manag Health Care 2004; 13 (01) 17-32
• 41 McQuillan RF, Silver SA, Harel Z. , et al. How to measure and interpret quality improvement data. Clin J Am Soc Nephrol 2016; 11 (05) 908-914
• 42 Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The value of monitoring clinical decision support interventions. Appl Clin Inform 2018; 9 (01) 163-173
• 43 Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform 2013; 82 (06) 492-503
• 44 Wright A, Sittig DF, Ash JS. , et al. Lessons learned from implementing service-oriented clinical decision support at four sites: a qualitative study. Int J Med Inform 2015; 84 (11) 901-911
• 45 Wright A, Ai A, Ash J. , et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2018; 25 (05) 496-506
• 46 Kannry J, McCullagh L, Kushniruk A, Mann D, Edonyabo D, McGinn T. A framework for usable and effective clinical decision support: experience from the iCPR randomized clinical trial. EGEMS (Wash DC) 2015; 3 (02) 1150
• 47 SAFER Guides. Available at: https://www.healthit.gov/topic/safety/safer-guides . Accessed June 15, 2019