Best Paper Selection

Content Summary of Best Papers for the Decision Support Section of the 2022 IMIA Yearbook

Austrian J, Mendoza F, Szerencsy A, Fenelon L, Horwitz LI, Jones S, Kuznetsova M, Mann DM

Applying A/B testing to clinical decision support: rapid randomized controlled trials

J Med Internet Res 2021 Apr 9;23(4):e16651

In this paper the authors describe the A/B testing approach to evaluate the performance of CDS systems, with a design comparable to randomized controlled trials in terms of internal validity. A/B testing is often associated with websites and apps’ performance evaluations, but in this paper, the authors applied the same principles embedding the A/B testing within a commercial EHR. The authors tested first an inpatient influenza alert, that was intended to improve vaccination rate in eligible patients. The interruptive alert triggered at the time the nurse documented responses to the influenza screening question flowsheets. The third version of the alert resulted in a 64% reduction in firings per patient per day. A second alert was designed to promote counseling and treatment for patients using tobacco products (outpatient tobacco cessation alert). The non-interruptive alert displayed in an alert section for outpatient providers at chart opening when active smokers had not received counseling within the previous three months. In the tobacco alert experiment, the authors tested three potential improvements with varying message framing and in this case, the authors observed that neither the framing method nor the addition of images resulted in significant differences in acceptance rates for the tobacco alert. The authors recognized that the testing of multiple versions of alerts in a short period of time can help to improve the acceptance of the interventions.

Dutta S, McEvoy DS, Stump T, McCabe J, Mahendra-Rajah A, McMurry R, White BA, Rubins D

Clinical decision support reduces unnecessary tetanus vaccinations in the emergency department

Ann Emerg Med 2021 Sep;78(3):370-80

In this paper Dutta et al., implemented and evaluated a CDS alert in an electronic health record to warn providers about a previously documented tetanus vaccine, when they are ordering a new one. In this study, the authors explored the alert efficacy to reduce unnecessary vaccines in the Emergency Department (ED). The study demonstrates a successful implementation of an electronic health record-based reminder system in the ED to decrease an unnecessary preventive intervention.

Orenstein EW, ElSayed-Ali O, Kandaswamy S, Masterson E, Blanco R, Shah P, Lantis P, Kolwaite A, Dawson TE, Ray E, Bryant C, Iyer S, Shane AL, Jernigan S

Evaluation of a clinical decision support strategy to increase seasonal influenza vaccination among hospitalized children before inpatient discharge

JAMA Netw Open 2021 Jul 1;4(7):e2117809

Here the authors analyzed interruptive alert firings at six pediatric hospitals and they compared four metrics: alerts per patient encounter, alerts per inpatient-day, alerts per 100 orders and alerts per unique clinician days. One of the key findings was a substantial variation across sites as to different types of providers experiencing the highest alert burden. For example, when measuring using alerts per 100 orders, alerts per 100 encounters, or alerts per inpatient day, nurses had the highest alert burden on average across all sites. The authors also described a substantial variability with ordering providers having the greatest burden at two of six sites and pharmacists at one site, but with a different metric, pharmacists experienced a much higher alert burden compared to all other provider types. The authors highlighted that alert were consistently responsible for most of the alert burden, this means that individual organization practices and culture for the development of alerts may be a primary driver of alert burden and may present an opportunity to substantially modify alert burden.

Publication History

Article published online:
04 December 2022