Download PDF
Appl Clin Inform 2023; 14(01): 108-118
DOI: 10.1055/s-0042-1760082
Research Article

Evaluation of an Antimicrobial Stewardship Decision Support for Pediatric Infections

Erin A. McGonagle
1   Department of Pediatrics and Emergency Medicine, University of Colorado School of Medicine, Aurora, Colorado, United States
,
Dean J. Karavite
2   Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
,
Robert W. Grundmeier
2   Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
,
Sarah K. Schmidt
1   Department of Pediatrics and Emergency Medicine, University of Colorado School of Medicine, Aurora, Colorado, United States
,
Larissa S. May
3   Department of Emergency Medicine, University of California at Davis School of Medicine, Davis, California, United States
,
Daniel M. Cohen
4   Department of Pediatrics, The Ohio State University School of Medicine, Columbus, Ohio, United States
,
Andrea T. Cruz
5   Department of Pediatrics, Baylor College of Medicine, Houston, Texas, United States
,
Shin-Ping Tu
6   Department of Medicine, University of California at Davis School of Medicine, Davis, California, United States
,
Lalit Bajaj
1   Department of Pediatrics and Emergency Medicine, University of Colorado School of Medicine, Aurora, Colorado, United States
,
Peter S. Dayan
7   Department of Emergency Medicine and Pediatrics, Columbia University College of Physicians and Surgeons, New York, New York, United States
,
Rakesh D. Mistry
8   Department of Pediatrics, University of Colorado School of Medicine, Aurora, Colorado, United States
› Author Affiliations Funding This work was supported by the Division of Intramural Research, National Institute of Allergy and Infectious Diseases R21 Exploratory and Developmental Grant program (grant no.: R21AI139839).
› Further Information
    Permissions and Reprints

Abstract

Objectives Clinical decision support (CDS) has promise for the implementation of antimicrobial stewardship programs (ASPs) in the emergency department (ED). We sought to assess the usability of a newly developed automated CDS to improve guideline-adherent antibiotic prescribing for pediatric community-acquired pneumonia (CAP) and urinary tract infection (UTI).

Methods We conducted comparative usability testing between an automated, prototype CDS-enhanced discharge order set and standard order set, for pediatric CAP and UTI antibiotic prescribing. After an extensive user-centered design process, the prototype CDS was integrated into the electronic health record, used passive activation, and embedded locally adapted prescribing guidelines. Participants were randomized to interact with three simulated ED scenarios of children with CAP or UTI, across both systems. Measures included task completion, decision-making and usability errors, clinical actions (order set use and correct antibiotic selection), as well as objective measures of system usability, utility, and workload using the National Aeronautics and Space Administration Task Load Index (NASA-TLX). The prototype CDS was iteratively refined to optimize usability and workflow.

Results Usability testing in 21 ED clinical providers demonstrated that, compared to the standard order sets, providers preferred the prototype CDS, with improvements in domains such as explanations of suggested antibiotic choices (p < 0.001) and provision of additional resources on antibiotic prescription (p < 0.001). Simulated use of the CDS also led to overall improved guideline-adherent prescribing, with a 31% improvement for CAP. A trend was present toward absolute workload reduction. Using the NASA-TLX, workload scores for the current system were median 26, interquartile ranges (IQR): 11 to 41 versus median 25, and IQR: 10.5 to 39.5 for the CDS system (p = 0.117).

Conclusion Our CDS-enhanced discharge order set for ED antibiotic prescribing was strongly preferred by users, improved the accuracy of antibiotic prescribing, and trended toward reduced provider workload. The CDS was optimized for impact on guideline-adherent antibiotic prescribing from the ED and end-user acceptability to support future evaluative trials of ED ASPs.

Keywords

clinical decision support - user testing - antimicrobial stewardship

Protection of Human and Animal Subjects

The current study was deemed exempt by our institutional review board.


Supplementary Material



Publication History

Received: 07 May 2022

Accepted: 16 November 2022

Article published online:
08 February 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 CDC. Antibiotic Resistance Threats in the United States. GA: U.S. Department of Health and Human Services, CDC; 2019
    Google Scholar
  • 2 Fleming-Dutra KE, Hersh AL, Shapiro DJ. et al. Prevalence of inappropriate antibiotic prescriptions among us ambulatory care visits, 2010-2011. JAMA 2016; 315 (17) 1864-1873
    CrossrefPubMedGoogle Scholar
  • 3 Poole NM, Shapiro DJ, Fleming-Dutra KE, Hicks LA, Hersh AL, Kronman MP. Antibiotic prescribing for children in United States Emergency Departments: 2009-2014. Pediatrics 2019; 143 (02) e20181056
    CrossrefPubMedGoogle Scholar
  • 4 Sanchez GV, Fleming-Dutra KE, Roberts RM, Hicks LA. Core elements of outpatient antibiotic stewardship. MMWR Recomm Rep 2016; 65 (06) 1-12
    CrossrefPubMedGoogle Scholar
  • 5 Society for Healthcare Epidemiology of America, Infectious Diseases Society of America, Pediatric Infectious Diseases Society. Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society (PIDS). Infect Control Hosp Epidemiol 2012; 33 (04) 322-327
    CrossrefPubMedGoogle Scholar
  • 6 Spellberg B, Blaser M, Guidos RJ. et al; Infectious Diseases Society of America (IDSA). Combating antimicrobial resistance: policy recommendations to save lives. Clin Infect Dis 2011; 52 (Suppl. 05) S397-S428
    PubMedGoogle Scholar
  • 7 Avdic E, Cushinotto LA, Hughes AH. et al. Impact of an antimicrobial stewardship intervention on shortening the duration of therapy for community-acquired pneumonia. Clin Infect Dis 2012; 54 (11) 1581-1587
    CrossrefPubMedGoogle Scholar
  • 8 Carling P, Fung T, Killion A, Terrin N, Barza M. Favorable impact of a multidisciplinary antibiotic management program conducted during 7 years. Infect Control Hosp Epidemiol 2003; 24 (09) 699-706
    CrossrefPubMedGoogle Scholar
  • 9 Ohl CA, Luther VP. Antimicrobial stewardship for inpatient facilities. J Hosp Med 2011; 6 (Suppl. 01) S4-S15
    PubMedGoogle Scholar
  • 10 Fishman N. Antimicrobial stewardship. Am J Infect Control 2006;34(5, suppl 1):S55–S63, discussion S64–S73
    PubMedGoogle Scholar
  • 11 May L, Cosgrove S, L'Archeveque M. et al. A call to action for antimicrobial stewardship in the emergency department: approaches and strategies. Ann Emerg Med 2013; 62 (01) 69-77.e2
    CrossrefPubMedGoogle Scholar
  • 12 Mistry RD, Dayan PS, Kuppermann N. The battle against antimicrobial resistance: time for the emergency department to join the fight. JAMA Pediatr 2015; 169 (05) 421-422
    CrossrefPubMedGoogle Scholar
  • 13 Mistry RD, Newland JG, Gerber JS. et al. Current state of antimicrobial stewardship in children's hospital emergency departments. Infect Control Hosp Epidemiol 2017; 38 (04) 469-475
    CrossrefPubMedGoogle Scholar
  • 14 Chung P, Scandlyn J, Dayan PS, Mistry RD. Working at the intersection of context, culture, and technology: provider perspectives on antimicrobial stewardship in the emergency department using electronic health record clinical decision support. Am J Infect Control 2017; 45 (11) 1198-1202
    CrossrefPubMedGoogle Scholar
  • 15 Ozkaynak M, Metcalf N, Cohen DM, May LS, Dayan PS, Mistry RD. Considerations for designing EHR-embedded clinical decision support systems for antimicrobial stewardship in pediatric emergency departments. Appl Clin Inform 2020; 11 (04) 589-597
    Article in Thieme ConnectPubMedGoogle Scholar
  • 16 Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Annu Fall Symp 1997; 218-222
    PubMedGoogle Scholar
  • 17 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
    CrossrefPubMedGoogle Scholar
  • 18 Hettinger AZ, Melnick ER, Ratwani RM. Advancing electronic health record vendor usability maturity: progress and next steps. J Am Med Inform Assoc 2021; 28 (05) 1029-1031
    CrossrefPubMedGoogle Scholar
  • 19 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care—an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-549
    CrossrefPubMedGoogle Scholar
  • 20 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
    CrossrefPubMedGoogle Scholar
  • 21 Nguyen OT, Jenkins NJ, Khanna N. et al. A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being. J Am Med Inform Assoc 2021; 28 (05) 974-984
    CrossrefPubMedGoogle Scholar
  • 22 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
    CrossrefPubMedGoogle Scholar
  • 23 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
    CrossrefPubMedGoogle Scholar
  • 24 Kouri A, Yamada J, Lam Shin Cheung J, Van de Velde S, Gupta S. Do providers use computerized clinical decision support systems? A systematic review and meta-regression of clinical decision support uptake. Implement Sci 2022; 17 (01) 21
    CrossrefPubMedGoogle Scholar
  • 25 Jones BE, Collingridge DS, Vines CG. et al. CDS in a learning health care system: identifying physicians' reasons for rejection of best-practice recommendations in pneumonia through computerized clinical decision support. Appl Clin Inform 2019; 10 (01) 1-9
    Article in Thieme ConnectPubMedGoogle Scholar
  • 26 Bradley JS, Byington CL, Shah SS. et al; Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. The management of community-acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis 2011; 53 (07) e25-e76
    CrossrefPubMedGoogle Scholar
  • 27 Roberts KB. Subcommittee on Urinary Tract Infection, Steering Committee on Quality Improvement and Management. Urinary tract infection: clinical practice guideline for the diagnosis and management of the initial UTI in febrile infants and children 2 to 24 months. Pediatrics 2011; 128 (03) 595-610
    CrossrefPubMedGoogle Scholar
  • 28 Osheroff JA. Improving Medication Use and Outcomes with Clinical Decision Support: A Step by Step Guide. 1st ed. Chicago, IL: HIMSS Publishing; 2009
    Google Scholar
  • 29 Rosson MB, Carroll JM. Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Burlington, MA: Morgan Kaufmann Publishers Inc.; 2001
    Google Scholar
  • 30 Russ AL, Saleem JJ. Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inform 2018; 78: 123-133
    CrossrefPubMedGoogle Scholar
  • 31 Cochran WG. Cox, Gertrude M, Experimental Designs. Hoboken, NJ: John Wiley & Sons; 1957
    Google Scholar
  • 32 Robert Schumacher, Svetlana Lowry. (NISTIR 7741) NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records. Published online November 29, 2010.
    PubMedGoogle Scholar
  • 33 Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In: Hancock PA, Meshkati N. eds. Advances in Psychology. Vol 52. North-Holland; 1988. :139–183
    Google Scholar
  • 34 Hart SG. NASA-task load index (NASA-TLX); 20 years later. Proc Hum Factors Ergon Soc Annu Meet 2006; 50 (09) 904-908
    CrossrefPubMedGoogle Scholar
  • 35 Said S, Gozdzik M, Roche TR. et al. Validation of the Raw National Aeronautics and Space Administration Task Load Index (NASA-TLX) questionnaire to assess perceived workload in patient monitoring tasks: pooled analysis study using mixed models. J Med Internet Res 2020; 22 (09) e19472
    CrossrefPubMedGoogle Scholar
  • 36 Karavite DJ, Miller MW, Ramos MJ. et al. User testing an information foraging tool for ambulatory surgical site infection surveillance. Appl Clin Inform 2018; 9 (04) 791-802
    Article in Thieme ConnectPubMedGoogle Scholar
  • 37 Pollack AH, Pratt W. Association of health record visualizations with physicians' cognitive load when prioritizing hospitalized patients. JAMA Netw Open 2020; 3 (01) e1919301
    CrossrefPubMedGoogle Scholar
  • 38 NASA. NASA TLX: Task Load Index. Published online 2006. Accessed December 2, 2022 at: http://humansystems.arc.nasa.gov/groups/TLX/
    PubMedGoogle Scholar
  • 39 Lang TA, Secic M. How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers. ACP Press; 2006
    Google Scholar
  • 40 Cohen J. Set correlation and contingency tables. Appl Psychol Meas 1988; 12 (04) 425-434
    CrossrefPubMedGoogle Scholar
  • 41 Li AC, Kannry JL, Kushniruk A. et al. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Inform 2012; 81 (11) 761-772
    CrossrefPubMedGoogle Scholar
  • 42 Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Human Factors 2015; 2 (02) e14
    CrossrefPubMedGoogle Scholar
  • 43 Richardson S, Mishuris R, O'Connell A. et al. “Think aloud” and “Near live” usability testing of two complex clinical decision support tools. Int J Med Inform 2017; 106: 1-8
    CrossrefPubMedGoogle Scholar
  • 44 Ward MJ, Chavis B, Banerjee R, Katz S, Anders S. User-centered design in pediatric acute care settings antimicrobial stewardship. Appl Clin Inform 2021; 12 (01) 34-40
    Article in Thieme ConnectPubMedGoogle Scholar