Appl Clin Inform 2022; 13(03): 560-568
DOI: 10.1055/s-0042-1748856
Review Article

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Juan D. Chaparro
1   Division of Clinical Informatics, Nationwide Children's Hospital, Columbus, Ohio, United States
2   Departments of Pediatrics and Biomedical Informatics, The Ohio State University College of Medicine, Columbus, Ohio, United States
Jonathan M. Beus
3   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
4   Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Adam C. Dziorny
5   Department of Pediatrics, University of Rochester School of Medicine, Rochester, New York, United States
Philip A. Hagedorn
6   Department of Pediatrics, University of Cincinnati, College of Medicine, Cincinnati, Ohio, United States
7   Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
Sean Hernandez
8   Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
9   Department of General Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
Swaminathan Kandaswamy
3   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
Eric S. Kirkendall
8   Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
10   Department of Pediatrics, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
11   Center for Biomedical Informatics, Wake Forest School of Medicine, Winston-Salem NC, United States
Allison B. McCoy
12   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
Naveen Muthu
13   Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
14   Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
Evan W. Orenstein
3   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
4   Children's Healthcare of Atlanta, Atlanta, Georgia, United States
› Author Affiliations
Funding None.


Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Protection of Human and Animal Subjects

There were no human subjects involved in the project.

Publication History

Received: 20 January 2022

Accepted: 21 March 2022

Article published online:
25 May 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Osheroff JA, Teich J, Levick D. et al. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed.. HIMSS; 2012
  • 2 Kwan JL, Lo L, Ferguson J. et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020; 370: m3216
  • 3 Varghese J, Kleine M, Gessner SI, Sandmann S, Dugas M. Effects of computerized decision support system implementations on patient outcomes in inpatient care: a systematic review. J Am Med Inform Assoc 2018; 25 (05) 593-602
  • 4 Bonafide CP, Localio AR, Holmes JH. et al. Video analysis of factors associated with response time to physiologic monitor alarms in a children's hospital. JAMA Pediatr 2017; 171 (06) 524-531
  • 5 ECRI. Special report: top 10 health technology hazards for 2020. Accessed January 11, 2022 at:
  • 6 Escovedo C, Bell D, Cheng E. et al. Noninterruptive clinical decision support decreases ordering of respiratory viral panels during influenza season. Appl Clin Inform 2020; 11 (02) 315-322
  • 7 Gouveia WA, Diamantis C, Barnett GO. Computer applications in the hospital medication system. Am J Hosp Pharm 1969; 26 (03) 141-150
  • 8 Drew BJ, Harris P, Zègre-Hemsey JK. et al. Insights into the problem of alarm fatigue with physiologic monitor devices: a comprehensive observational study of consecutive intensive care unit patients. PLoS One 2014; 9 (10) e110274
  • 9 Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020; 11 (01) 46-58
  • 10 Westbrook JI, Coiera E, Dunsmuir WT. et al. The impact of interruptions on clinical task completion. Qual Saf Health Care 2010; 19 (04) 284-289
  • 11 Grundgeiger T, Sanderson P. Interruptions in healthcare: theoretical views. Int J Med Inform 2009; 78 (05) 293-307
  • 12 Ashcroft DM, Quinlan P, Blenkinsopp A. Prospective study of the incidence, nature and causes of dispensing errors in community pharmacies. Pharmacoepidemiol Drug Saf 2005; 14 (05) 327-332
  • 13 Westbrook JI, Woods A, Rob MI, Dunsmuir WT, Day RO. Association of interruptions with an increased risk and severity of medication administration errors. Arch Intern Med 2010; 170 (08) 683-690
  • 14 Ariosto D. Factors contributing to CPOE opiate allergy alert overrides. AMIA Annu Symp Proc 2014; 2014: 256-265
  • 15 Humphrey K, Jorina M, Harper M, Dodson B, Kim SY, Ozonoff A. An investigation of drug-drug interaction alert overrides at a pediatric hospital. Hosp Pediatr 2018; 8 (05) 293-299
  • 16 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
  • 17 Muthu N, Shelov E, Tobias MC, Karavite DJ, Orenstein EW, Grundmeier RW. Variability in user response to custom alerts in the electronic health record: an observational study. Paper presented at: the AMIA 2019 National Symposium. Washington DC: ; November 16–20, 2019
  • 18 Bliss JP, Gilson RD, Deaton JE. Human probability matching behaviour in response to alarms of varying reliability. Ergonomics 1995; 38 (11) 2300-2312
  • 19 Bliss JP, Dunn MC. Behavioural implications of alarm mistrust as a function of task workload. Ergonomics 2000; 43 (09) 1283-1300
  • 20 Baysari MT, Tariq A, Day RO, Westbrook JI. Alert override as a habitual behavior - a new perspective on a persistent problem. J Am Med Inform Assoc 2017; 24 (02) 409-412
  • 21 Simpao AF, Ahumada LM, Desai BR. et al. Optimization of drug-drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard. J Am Med Inform Assoc 2015; 22 (02) 361-369
  • 22 Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. J Biomed Inform 2018; 78: 134-143
  • 23 Wright A, Hickman TT, McEvoy D. et al. Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016; 23 (06) 1068-1076
  • 24 McGreevey III JD, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing alert burden in electronic health records: state of the art recommendations from four health systems. Appl Clin Inform 2020; 11 (01) 1-12
  • 25 Schreiber R, Gregoire JA, Shaha JE, Shaha SH. Think time: a novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts. Int J Med Inform 2017; 97: 59-67
  • 26 McDaniel RB, Burlison JD, Baker DK. et al. Alert dwell time: introduction of a measure to evaluate interruptive clinical decision support alerts. J Am Med Inform Assoc 2016; 23 (e1): e138-e141
  • 27 Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar AM. Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden. Appl Clin Inform 2019; 10 (05) 909-917
  • 28 McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 (02) 195-202
  • 29 McCoy AB, Waitman LR, Lewis JB. et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012; 19 (03) 346-352
  • 30 Strategies to the five rights of clinical decision support. January 23, 2019. Accessed January 11, 2022 at:
  • 31 Orenstein EW, Kandaswamy S, Muthu N. et al. Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics. J Am Med Inform Assoc 2021; 28 (12) 2654-2660
  • 32 Sinha A, Stevens LA, Su F, Pageler NM, Tawfik DS. Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings. Appl Clin Inform 2021; 12 (04) 737-744
  • 33 Bonafide CP, Lin R, Zander M. et al. Association between exposure to nonactionable physiologic monitor alarms and response time in a children's hospital. J Hosp Med 2015; 10 (06) 345-351
  • 34 Bonafide CP, Miller JM, Localio AR. et al. Association between mobile telephone interruptions and medication administration errors in a pediatric intensive care unit. JAMA Pediatr 2020; 174 (02) 162-169
  • 35 Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005; 83 (04) 691-729
  • 36 Moore GF, Audrey S, Barker M. et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ 2015; 350: h1258
  • 37 Proctor E, Silmere H, Raghavan R. et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38 (02) 65-76
  • 38 Kandaswamy S, Karavite DJ, Muthu N. et al. User and task analysis for evaluation of clinical decision support for quality improvement. Proc Hum Factors Ergon Soc Annu Meet 2020; 64 (01) 750-754
  • 39 Wright A, Ash JS, Aaron S. et al. Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: results of a Delphi study. Int J Med Inform 2018; 118: 78-85
  • 40 Aaron S, McEvoy DS, Ray S, Hickman TT, Wright A. Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc 2019; 26 (01) 37-43
  • 41 Ray S, McEvoy DS, Aaron S, Hickman TT, Wright A. Using statistical anomaly detection models to find clinical decision support malfunctions. J Am Med Inform Assoc 2018; 25 (07) 862-871
  • 42 Kassakian SZ, Yackel TR, Gorman PN, Dorr DA. Clinical decisions support malfunctions in a commercial electronic health record. Appl Clin Inform 2017; 8 (03) 910-923
  • 43 Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The value of monitoring clinical decision support interventions. Appl Clin Inform 2018; 9 (01) 163-173
  • 44 Seidling HM, Phansalkar S, Seger DL. et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011; 18 (04) 479-484
  • 45 Kawamanto K, Flynn MC, Kukhareva P. et al. A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: a case study. AMIA Annu Symp Proc 2018; 2018: 624-633
  • 46 DeRosier J, Stalhandske E, Bagian JP, Nudell T. Using health care failure mode and effect analysis: the VA National Center for Patient Safety's prospective risk analysis system. Jt Comm J Qual Improv 2002; 28 (05) 248-267
  • 47 Electronic Health Record Association. Electronic health record design patterns for patient safety. 2017 . Accessed January 11, 2022 at:
  • 48 Gross B, Dahl D, Nielsen L. Physiologic monitoring alarm load on medical/surgical floors of a community hospital. Biomed Instrum Technol 2011; (Suppl): 29-36
  • 49 Yu D, Obuseh M, DeLaurentis P. Quantifying the impact of infusion alerts and alarms on nursing workflows: a retrospective analysis. Appl Clin Inform 2021; 12 (03) 528-538
  • 50 Ruskin KJ, Hueske-Kraus D. Alarm fatigue: impacts on patient safety. Curr Opin Anaesthesiol 2015; 28 (06) 685-690
  • 51 Hagedorn PA, Singh A, Luo B, Bonafide CP, Simmons JM. Secure text messaging in healthcare: latent threats and opportunities to improve patient safety. J Hosp Med 2020; 15 (06) 378-380
  • 52 Pater CM, Sosa TK, Boyer J. et al. Time series evaluation of improvement interventions to reduce alarm notifications in a paediatric hospital. BMJ Qual Saf 2020; 29 (09) 717-726
  • 53 Tchou MJ, Andersen H, Robinette E. et al. Accelerating initiation of adequate antimicrobial therapy using real-time decision support and microarray testing. Pediatr Qual Saf 2019; 4 (04) e191
  • 54 Singh A, Haque A, Alahi A. et al. Automatic detection of hand hygiene using computer vision technology. J Am Med Inform Assoc 2020; 27 (08) 1316-1320
  • 55 Redelmeier DA, Tibshirani RJ. Association between cellular-telephone calls and motor vehicle collisions. N Engl J Med 1997; 336 (07) 453-458
  • 56 Wright A, Sittig DF, Ash JS. et al. Governance for clinical decision support: case studies and recommended practices from leading institutions. J Am Med Inform Assoc 2011; 18 (02) 187-194
  • 57 Orenstein EW, Muthu N, Weitkamp AO. et al. Towards a maturity model for clinical decision support operations. Appl Clin Inform 2019; 10 (05) 810-819
  • 58 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
  • 59 Zhang J, Walji M. Better EHR: Usability, Workflow & Cognitive Support in Electronic Health Records. 1st ed.. National Center for Cognitive Informatics & Decision Making in Healthcare, University of Texas Health Science Center at Houston & School of Biomedical Informatics; 2014
  • 60 Miller K, Capan M, Weldon D. et al. The design of decisions: matching clinical decision support recommendations to Nielsen's design heuristics. Int J Med Inform 2018; 117: 19-25
  • 61 Kannampallil TG, Kaufman DR, Patel VL. Cognitive Informatics for Biomedicine: Human Computer Interaction in Healthcare. 1st ed.. Springer International Publishing; 2015
  • 62 Rodriguez JA, Clark CR, Bates DW. Digital health equity as a necessity in the 21st century Cures Act era. JAMA 2020; 323 (23) 2381-2382
  • 63 Majumder MA, Guerrini CJ, Bollinger JM, Cook-Deegan R, McGuire AL. Sharing data under the 21st Century Cures Act. Genet Med 2017; 19 (12) 1289-1294
  • 64 Pageler NM, Webber EC, Lund DP. Implications of the 21st Century Cures Act in pediatrics. Pediatrics 2021; 147 (03) e2020034199
  • 65 Gordon WJ, Mandl KD. The 21st Century Cures Act: a competitive apps market and the risk of innovation blocking. J Med Internet Res 2020; 22 (12) e24824
  • 66 Dzau VJ, Cho A, Ellaissi W. et al. Transforming academic health centers for an uncertain future. N Engl J Med 2013; 369 (11) 991-993
  • 67 Horwitz LI, Kuznetsova M, Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med 2019; 381 (12) 1175-1179
  • 68 Friedman C, Rubin J, Brown J. et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc 2015; 22 (01) 43-50
  • 69 Crandall W, Kappelman MD, Colletti RB. et al. ImproveCareNow: The development of a pediatric inflammatory bowel disease improvement network. Inflamm Bowel Dis 2011; 17 (01) 450-457
  • 70 Marsolo K, Margolis PA, Forrest CB, Colletti RB, Hutton JJ. A digital architecture for a network-based learning health system: integrating chronic care management, quality improvement, and research. EGEMS (Wash DC) 2015; 3 (01) 1168