Methods Inf Med 2016; 55(06): 507-515
DOI: 10.3414/ME16-01-0003
Original Articles
Schattauer GmbH

Development of a Standardized Rating Tool for Drug Alerts to Reduce Information Overload[*]

Barbara Pfistermeister**
1   Friedrich-Alexander-Universität Erlangen-Nürnberg, Institute of Experimental and Clinical Pharmacology and Toxicology, Erlangen, Germany
,
Brita Sedlmayr**
2   Friedrich-Alexander-Universität Erlangen-Nürnberg, Chair of Medical Informatics, Erlangen, Germany
,
Andrius Patapovas
2   Friedrich-Alexander-Universität Erlangen-Nürnberg, Chair of Medical Informatics, Erlangen, Germany
,
Gerald Suttner
3   Universitätsklinikum Erlangen and Friedrich-Alexander-Universität Erlangen-Nürnberg, Department of Psychiatry and Psychotherapy, Erlangen, Germany
,
Ozan Tektas
3   Universitätsklinikum Erlangen and Friedrich-Alexander-Universität Erlangen-Nürnberg, Department of Psychiatry and Psychotherapy, Erlangen, Germany
,
Aleksey Tarkhov
4   Molecular Networks GmbH, Erlangen, Germany
,
Johannes Kornhuber
3   Universitätsklinikum Erlangen and Friedrich-Alexander-Universität Erlangen-Nürnberg, Department of Psychiatry and Psychotherapy, Erlangen, Germany
,
Martin F. Fromm
1   Friedrich-Alexander-Universität Erlangen-Nürnberg, Institute of Experimental and Clinical Pharmacology and Toxicology, Erlangen, Germany
,
Thomas Bürkle
5   Bern University of Applied Science BFH, Institute for Medical Informatics, Biel, Switzerland
,
Hans-Ulrich Prokosch
2   Friedrich-Alexander-Universität Erlangen-Nürnberg, Chair of Medical Informatics, Erlangen, Germany
,
Renke Maas
1   Friedrich-Alexander-Universität Erlangen-Nürnberg, Institute of Experimental and Clinical Pharmacology and Toxicology, Erlangen, Germany
› Author Affiliations
Funding This study was supported by the German Federal Ministry of Education and Research within the German Leading Edge Cluster Medical Valley [http://www.gesundheitsforschung-bmbf.de/de/4641.php; funding reference number 13EX1015B (JK, MFF, TB, HUP, RM), 13EX1015G (AT)]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Further Information

Publication History

received: 07 January 2016

accepted: 07 July 2016

Publication Date:
08 January 2018 (online)

Summary

Background: A well-known problem in current clinical decision support systems (CDSS) is the high number of alerts, which are often medically incorrect or irrelevant. This may lead to the so-called alert fatigue, an over -riding of alerts, including those that are clinically relevant, and underuse of CDSS in general.

Objectives: The aim of our study was to develop and to apply a standardized tool that allows its users to evaluate the quality of system-generated drug alerts. The users’ ratings can subsequently be used to derive recommendations for developing a filter function to reduce irrelevant alerts.

Methods: We developed a rating tool for drug alerts and performed a web-based evaluation study that also included a user review of alerts. In this study the following categories were evaluated: “data linked correctly”, “medically correct”, “action required”, “medication change”, “critical alert”, “information gained” and “show again”. For this purpose, 20 anonymized clinical cases were randomly selected and displayed in our customized CDSS research prototype, which used the summary of product characteristics (SPC) for alert generation. All the alerts that were provided were evaluated by 13 physicians. The users’ ratings were used to derive a filtering algorithm to reduce overalerting.

Results: In total, our CDSS research prototype generated 399 alerts. In 98 % of all alerts, medication data were rated as linked correctly to drug information; in 93 %, the alerts were assessed as “medically correct”; 19.5 % of all alerts were rated as “show again”. The interrater-agreement was, on average, 68.4 %. After the application of our filtering algorithm, the rate of alerts that should be shown again decreased to 14.8 %.

Conclusions: The new standardized rating tool supports a standardized feedback of user-perceived clinical relevance of CDSS alerts. Overall, the results indicated that physicians may consider the majority of alerts formally correct but clinically irrelevant and override them. Filtering may help to reduce overalerting and increase the specificity of a CDSS.

* Supplementary material published on our website http://dx.doi.org/10.3414/ME16-01-0003


** These authors contributed equally to this work