Appl Clin Inform 2022; 13(05): 1024-1032
DOI: 10.1055/s-0042-1757923
Research Article

Real-Time User Feedback to Support Clinical Decision Support System Improvement

David Rubins
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
Allison B. McCoy
3   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Sayon Dutta
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
4   Department of Emergency Medicine, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, United States
,
Dustin S. McEvoy
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
Lorraine Patterson
5   HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Amy Miller
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
John G. Jackson
5   HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States
,
Gianna Zuccotti
1   Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
2   Digital, Mass General Brigham, Boston, Massachusetts, United States
,
Adam Wright
3   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
› Author Affiliations

Abstract

Objectives To improve clinical decision support (CDS) by allowing users to provide real-time feedback when they interact with CDS tools and by creating processes for responding to and acting on this feedback.

Methods Two organizations implemented similar real-time feedback tools and processes in their electronic health record and gathered data over a 30-month period. At both sites, users could provide feedback by using Likert feedback links embedded in all end-user facing alerts, with results stored outside the electronic health record, and provide feedback as a comment when they overrode an alert. Both systems are monitored daily by clinical informatics teams.

Results The two sites received 2,639 Likert feedback comments and 623,270 override comments over a 30-month period. Through four case studies, we describe our use of end-user feedback to rapidly respond to build errors, as well as identifying inaccurate knowledge management, user-interface issues, and unique workflows.

Conclusion Feedback on CDS tools can be solicited in multiple ways, and it contains valuable and actionable suggestions to improve CDS alerts. Additionally, end users appreciate knowing their feedback is being received and may also make other suggestions to improve the electronic health record. Incorporation of end-user feedback into CDS monitoring, evaluation, and remediation is a way to improve CDS.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. This project was undertaken as a quality improvement initiative at Mass General Brigham, and as such was not formally supervised by the Institutional Review Board per their policies.




Publication History

Received: 28 March 2022

Accepted: 13 September 2022

Article published online:
26 October 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany