Evaluation of an Antimicrobial Stewardship Decision Support for Pediatric Infections

Abstract

Objectives Clinical decision support (CDS) has promise for the implementation of antimicrobial stewardship programs (ASPs) in the emergency department (ED). We sought to assess the usability of a newly developed automated CDS to improve guideline-adherent antibiotic prescribing for pediatric community-acquired pneumonia (CAP) and urinary tract infection (UTI).

Methods We conducted comparative usability testing between an automated, prototype CDS-enhanced discharge order set and standard order set, for pediatric CAP and UTI antibiotic prescribing. After an extensive user-centered design process, the prototype CDS was integrated into the electronic health record, used passive activation, and embedded locally adapted prescribing guidelines. Participants were randomized to interact with three simulated ED scenarios of children with CAP or UTI, across both systems. Measures included task completion, decision-making and usability errors, clinical actions (order set use and correct antibiotic selection), as well as objective measures of system usability, utility, and workload using the National Aeronautics and Space Administration Task Load Index (NASA-TLX). The prototype CDS was iteratively refined to optimize usability and workflow.

Results Usability testing in 21 ED clinical providers demonstrated that, compared to the standard order sets, providers preferred the prototype CDS, with improvements in domains such as explanations of suggested antibiotic choices (p < 0.001) and provision of additional resources on antibiotic prescription (p < 0.001). Simulated use of the CDS also led to overall improved guideline-adherent prescribing, with a 31% improvement for CAP. A trend was present toward absolute workload reduction. Using the NASA-TLX, workload scores for the current system were median 26, interquartile ranges (IQR): 11 to 41 versus median 25, and IQR: 10.5 to 39.5 for the CDS system (p = 0.117).

Conclusion Our CDS-enhanced discharge order set for ED antibiotic prescribing was strongly preferred by users, improved the accuracy of antibiotic prescribing, and trended toward reduced provider workload. The CDS was optimized for impact on guideline-adherent antibiotic prescribing from the ED and end-user acceptability to support future evaluative trials of ED ASPs.

Keywords

clinical decision support - user testing - antimicrobial stewardship

Protection of Human and Animal Subjects

The current study was deemed exempt by our institutional review board.

Publication History

Received: 07 May 2022

Accepted: 16 November 2022

Article published online:
08 February 2023

• References

• 1 CDC. Antibiotic Resistance Threats in the United States. GA: U.S. Department of Health and Human Services, CDC; 2019
• 2 Fleming-Dutra KE, Hersh AL, Shapiro DJ. et al. Prevalence of inappropriate antibiotic prescriptions among us ambulatory care visits, 2010-2011. JAMA 2016; 315 (17) 1864-1873
• 3 Poole NM, Shapiro DJ, Fleming-Dutra KE, Hicks LA, Hersh AL, Kronman MP. Antibiotic prescribing for children in United States Emergency Departments: 2009-2014. Pediatrics 2019; 143 (02) e20181056
• 4 Sanchez GV, Fleming-Dutra KE, Roberts RM, Hicks LA. Core elements of outpatient antibiotic stewardship. MMWR Recomm Rep 2016; 65 (06) 1-12
• 5 Society for Healthcare Epidemiology of America, Infectious Diseases Society of America, Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). Infect Control Hosp Epidemiol 2012; 33 (04) 322-327
• 6 Spellberg B, Blaser M, Guidos RJ. et al; Infectious Diseases Society of America (IDSA). Combating antimicrobial resistance: policy recommendations to save lives. Clin Infect Dis 2011; 52 (Suppl. 05) S397-S428
• 7 Avdic E, Cushinotto LA, Hughes AH. et al. Impact of an antimicrobial stewardship intervention on shortening the duration of therapy for community-acquired pneumonia. Clin Infect Dis 2012; 54 (11) 1581-1587
• 8 Carling P, Fung T, Killion A, Terrin N, Barza M. Favorable impact of a multidisciplinary antibiotic management program conducted during 7 years. Infect Control Hosp Epidemiol 2003; 24 (09) 699-706
• 9 Ohl CA, Luther VP. Antimicrobial stewardship for inpatient facilities. J Hosp Med 2011; 6 (Suppl. 01) S4-S15
• 10 Fishman N. Antimicrobial stewardship. Am J Infect Control 2006;34(5, suppl 1):S55–S63, discussion S64–S73
• 11 May L, Cosgrove S, L'Archeveque M. et al. A call to action for antimicrobial stewardship in the emergency department: approaches and strategies. Ann Emerg Med 2013; 62 (01) 69-77.e2
• 12 Mistry RD, Dayan PS, Kuppermann N. The battle against antimicrobial resistance: time for the emergency department to join the fight. JAMA Pediatr 2015; 169 (05) 421-422
• 13 Mistry RD, Newland JG, Gerber JS. et al. Current state of antimicrobial stewardship in children's hospital emergency departments. Infect Control Hosp Epidemiol 2017; 38 (04) 469-475
• 14 Chung P, Scandlyn J, Dayan PS, Mistry RD. Working at the intersection of context, culture, and technology: provider perspectives on antimicrobial stewardship in the emergency department using electronic health record clinical decision support. Am J Infect Control 2017; 45 (11) 1198-1202
• 15 Ozkaynak M, Metcalf N, Cohen DM, May LS, Dayan PS, Mistry RD. Considerations for designing EHR-embedded clinical decision support systems for antimicrobial stewardship in pediatric emergency departments. Appl Clin Inform 2020; 11 (04) 589-597
• 16 Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Annu Fall Symp 1997; 218-222
• 17 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
• 18 Hettinger AZ, Melnick ER, Ratwani RM. Advancing electronic health record vendor usability maturity: progress and next steps. J Am Med Inform Assoc 2021; 28 (05) 1029-1031
• 19 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care—an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-549
• 20 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 21 Nguyen OT, Jenkins NJ, Khanna N. et al. A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being. J Am Med Inform Assoc 2021; 28 (05) 974-984
• 22 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 23 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 24 Kouri A, Yamada J, Lam Shin Cheung J, Van de Velde S, Gupta S. Do providers use computerized clinical decision support systems? A systematic review and meta-regression of clinical decision support uptake. Implement Sci 2022; 17 (01) 21
• 25 Jones BE, Collingridge DS, Vines CG. et al. CDS in a learning health care system: identifying physicians' reasons for rejection of best-practice recommendations in pneumonia through computerized clinical decision support. Appl Clin Inform 2019; 10 (01) 1-9
• 26 Bradley JS, Byington CL, Shah SS. et al; Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. The management of community-acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis 2011; 53 (07) e25-e76
• 27 Roberts KB. Subcommittee on Urinary Tract Infection, Steering Committee on Quality Improvement and Management. Urinary tract infection: clinical practice guideline for the diagnosis and management of the initial UTI in febrile infants and children 2 to 24 months. Pediatrics 2011; 128 (03) 595-610
• 28 Osheroff JA. Improving Medication Use and Outcomes with Clinical Decision Support: A Step by Step Guide. 1st ed. Chicago, IL: HIMSS Publishing; 2009
• 29 Rosson MB, Carroll JM. Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Burlington, MA: Morgan Kaufmann Publishers Inc.; 2001
• 30 Russ AL, Saleem JJ. Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inform 2018; 78: 123-133
• 31 Cochran WG. Cox, Gertrude M, Experimental Designs. Hoboken, NJ: John Wiley & Sons; 1957
• 32 Robert Schumacher, Svetlana Lowry. (NISTIR 7741) NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records. Published online November 29, 2010.
• 33 Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In: Hancock PA, Meshkati N. eds. Advances in Psychology. Vol 52. North-Holland; 1988. :139–183
• 34 Hart SG. NASA-task load index (NASA-TLX); 20 years later. Proc Hum Factors Ergon Soc Annu Meet 2006; 50 (09) 904-908
• 35 Said S, Gozdzik M, Roche TR. et al. Validation of the Raw National Aeronautics and Space Administration Task Load Index (NASA-TLX) questionnaire to assess perceived workload in patient monitoring tasks: pooled analysis study using mixed models. J Med Internet Res 2020; 22 (09) e19472
• 36 Karavite DJ, Miller MW, Ramos MJ. et al. User testing an information foraging tool for ambulatory surgical site infection surveillance. Appl Clin Inform 2018; 9 (04) 791-802
• 37 Pollack AH, Pratt W. Association of health record visualizations with physicians' cognitive load when prioritizing hospitalized patients. JAMA Netw Open 2020; 3 (01) e1919301
• 38 NASA. NASA TLX: Task Load Index. Published online 2006. Accessed December 2, 2022 at: http://humansystems.arc.nasa.gov/groups/TLX/
• 39 Lang TA, Secic M. How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers. ACP Press; 2006
• 40 Cohen J. Set correlation and contingency tables. Appl Psychol Meas 1988; 12 (04) 425-434
• 41 Li AC, Kannry JL, Kushniruk A. et al. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform 2012; 81 (11) 761-772
• 42 Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Human Factors 2015; 2 (02) e14
• 43 Richardson S, Mishuris R, O'Connell A. et al. “Think aloud” and “Near live” usability testing of two complex clinical decision support tools. Int J Med Inform 2017; 106: 1-8
• 44 Ward MJ, Chavis B, Banerjee R, Katz S, Anders S. User-centered design in pediatric acute care settings antimicrobial stewardship. Appl Clin Inform 2021; 12 (01) 34-40