Development of a Standardized Rating Tool for Drug Alerts to Reduce Information Overload^[*]

Summary

Background: A well-known problem in current clinical decision support systems (CDSS) is the high number of alerts, which are often medically incorrect or irrelevant. This may lead to the so-called alert fatigue, an over -riding of alerts, including those that are clinically relevant, and underuse of CDSS in general.

Objectives: The aim of our study was to develop and to apply a standardized tool that allows its users to evaluate the quality of system-generated drug alerts. The users’ ratings can subsequently be used to derive recommendations for developing a filter function to reduce irrelevant alerts.

Methods: We developed a rating tool for drug alerts and performed a web-based evaluation study that also included a user review of alerts. In this study the following categories were evaluated: “data linked correctly”, “medically correct”, “action required”, “medication change”, “critical alert”, “information gained” and “show again”. For this purpose, 20 anonymized clinical cases were randomly selected and displayed in our customized CDSS research prototype, which used the summary of product characteristics (SPC) for alert generation. All the alerts that were provided were evaluated by 13 physicians. The users’ ratings were used to derive a filtering algorithm to reduce overalerting.

Results: In total, our CDSS research prototype generated 399 alerts. In 98 % of all alerts, medication data were rated as linked correctly to drug information; in 93 %, the alerts were assessed as “medically correct”; 19.5 % of all alerts were rated as “show again”. The interrater-agreement was, on average, 68.4 %. After the application of our filtering algorithm, the rate of alerts that should be shown again decreased to 14.8 %.

Conclusions: The new standardized rating tool supports a standardized feedback of user-perceived clinical relevance of CDSS alerts. Overall, the results indicated that physicians may consider the majority of alerts formally correct but clinically irrelevant and override them. Filtering may help to reduce overalerting and increase the specificity of a CDSS.

Keywords

Clinical decision support systems - medication alert system - assessment tool - evaluation study

^* Supplementary material published on our website http://dx.doi.org/10.3414/ME16-01-0003

^** These authors contributed equally to this work

• References

• 1 Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998; 279 (Suppl. 15) 1200-1205.
• 2 Wester K, Jönsson AK, Spigset O, Druid H, Hägg S. Incidence of fatal adverse drug reactions: a population based study. Br J Clin Pharmacol 2008; 65 (Suppl. 04) 573-579.
• 3 WHO.. Medicines: safety of medicines – adverse drug reactions. Fact sheet N°293 2008. Available from: http://www.wiredhealthresources.net/resources/NA/WHO-FS_MedicinesSafetyAdverseReactions.pdf
• 4 Schnurrer JU, Frölich JC. [Incidence and prevention of lethal undesirable drug effects]. Internist (Berl) 2003; 44 (Suppl. 07) 889-895. German.
• 5 Schrappe M, Lessing C, Albers B. Agenda Patientensicherheit 2007. 2008 Available from: http://www.aps-ev.de/fileadmin/fuerRedakteur/PDFs/Agenda_Patientensicherheit/Agenda_2007_mit_Titelblatt.pdf
• 6 Sedlmayr B, Patapovas A, Kirchner M, Sonst A, Müller F, Pfistermeister B. et al. Comparative evaluation of different medication safety measures for the emergency department: physicians’ usage and acceptance of training, poster, checklist and computerized decision support. BMC Med Inform Decis Mak 2013; 13: 79.
• 7 Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (Suppl. 10) 1223-38.
• 8 Ojeleye O, Avery A, Gupta V, Boyd M. The evidence for the effectiveness of safety alerts in electronic patient medication record systems at the point of pharmacy order entry: a systematic review. BMC Med Inform Decis Mak 2013; 13: 69.
• 9 Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014; 23 (Suppl. 09) 773-780.
• 10 Pauly A, Wolf C, Busse M, Strauß A-C, Krebs S, Dörje F. et al. Evaluation of eight drug interaction databases commonly used in the German health-care system. Eur J Hosp Pharm 2014; 22 (Suppl. 03) 165-170.
• 11 Ammenwerth E, Hackl WO, Massari P, Darmoni S. Validation of completeness, correctness, relevance and understandability of the PSIP CDSS for medication safety. Stud Health Technol Inform 2011; 166: 254-259.
• 12 Spina JR, Glassman PA, Belperio P, Cader R, Asch S. Primary Care Investigative Group of the V.. A. Los Angeles Healthcare System. Clinical relevance of automated drug alerts from the perspective of medical providers. Am J Med Qual 2005; 20 (Suppl. 01) 7-14.
• 13 Fritz D, Ceschi A, Curkovic I, Huber M, Egbring M, Kullak-Ublick GA. et al. Comparative evaluation of three clinical decision support systems: prospective screening for medication errors in 100 medical inpatients. Eur J Clin Pharmacol 2012; 68 (Suppl. 08) 1209-1219.
• 14 Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G. et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 (Suppl. 01) 29-40.
• 15 Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B. et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006; 13 (Suppl. 01) 5-11.
• 16 van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008; 15 (Suppl. 04) 439-448.
• 17 Pfistermeister B, Sass A, Criegee-Rieck M, Bürkle T, Fromm MF, Maas R. Inconsistencies and misleading information in officially approved prescribing information from three major drug markets. Clin Pharmacol Ther 2014; 96 (Suppl. 05) 616-624.
• 18 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (Suppl. 02) 138-147.
• 19 Jung M, Riedmann D, Hackl WO, Hoerbst A, Jaspers MW, Ferret L. et al. Physicians’ perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems. BMC Med Inform Decis Mak 2012; 12: 111.
• 20 van der Sijs H. Drug safety alerting in computerized physician order entry – unraveling and counteracting alert fatigue. Rotterdam: Erasmus University; 2009. Available from: http://repub.eur.nl/pub/16936
• 21 Hsieh TC, Kuperman GJ, Jaggi T, Hojnowski-Diaz P, Fiskio J, Williams DH. et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004; 11 (Suppl. 06) 482-491.
• 22 Marcilly R, Ammenwerth E, Roehrer E, Pelayo S, Vasseur F, Beuscart-Zéphir MC. Usability Flaws in Medication Alerting Systems: Impact on Usage and Work System. Yearb Med Inform 2015; 10 (Suppl. 01) 55-67.
• 23 Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform 2013; 82 (Suppl. 06) 492-503.
• 24 Payne TH, Hines LE, Chan RC, Hartman S, Kapusnik-Uner J, Russ AL. et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (Suppl. 06) 1243-1250.
• 25 Ammenwerth E, Aly AF, Bürkle T, Christ P, Dor-mann H, Friesdorf W. et al. Memorandum on the use of information technology to improve medication safety. Methods Inf Med 2014; 53 (Suppl. 05) 336-343.
• 26 Riedmann D, Jung M, Hackl WO, Stühlinger W, van der Sijs H, Ammenwerth E. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak 2011; 11: 35.
• 27 Patapovas A, Pfistermeister B, Tarkhov A, Terfloth L, Maas R, Fromm MF. et al. The effect object paradigm – a means to support medication safety with clinical decision support. Stud Health Technol Inform 2014; 205: 1065-1069.
• 28 Hahn M, Braus DF. [Psychiatric polypharmacy: hazard through drug-drug-interaction and possibilities for prevention]. Versicherungsmedizin 2012; 64 (Suppl. 03) 127-131. German.
• 29 DiPoto JP, Buckley MS, Kane-Gill SL. Evaluation of an automated surveillance system using trigger alerts to prevent adverse drug events in the intensive care unit and general ward. Drug Saf 2015; 38 (Suppl. 03) 311-317.
• 30 Medizinische Medien Informations GmbH. [cited 2015 July 23]. Available from: http://www.mmi.de/
• 31 Qualtrics LLC. [cited 2015 July 24]. Available from: http://www.qualtrics.com/de
• 32 Miyo K, Kawazoe Y, Yamaguchi I, Tatsukawa A, Ohe K. Evaluation of a context-based prescription alert system: a clinical perspective. Stud Health Technol Inform 2013; 192: 1030.
• 33 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (Suppl. 21) 2625-2631.
• 34 Taylor LK, Tamblyn R. Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Inform 2004; 107 Pt 2 1101-1105.
• 35 Andersson ML, Böttiger Y, Bastholm-Rahmner P, Ovesjö ML, Vég A, Eiermann B. Evaluation of usage patterns and user perception of the drug-drug interaction database SFINX. Int J Med Inform 2015; 84 (Suppl. 05) 327-333.
• 36 Tamblyn R, Eguale T, Buckeridge DL, Huang A, Hanley J, Reidel K. et al. The effectiveness of a new generation of computerized drug alerts in reducing the risk of injury from drug side effects: a cluster randomized trial. J Am Med Inform Assoc 2012; 19 (Suppl. 04) 635-643.
• 37 Koutkias VG, McNair P, Kilintzis V, Skovhus Andersen K, Niès J, Sarfati JC. et al. From adverse drug event detection to prevention. A novel clinical decision support framework for medication safety. Methods Inf Med 2014; 53 (Suppl. 06) 482-492.
• 38 Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM. et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013; 13: 111.
• 39 Cruz MR, Martins C, Dias J, Pinto JS. A validation of an intelligent decision-making support system for the nutrition diagnosis of bariatric surgery patients. JMIR Med Inform 2014; 2 (Suppl. 02) e8.
• 40 Seidling HM, Phansalkar S, Seger DL, Paterno MD, Shaykevich S, Haefeli WE. et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011; 18 (Suppl. 04) 479-484.