Appl Clin Inform 2019; 10(03): 521-527
DOI: 10.1055/s-0039-1693427
Research Article
Georg Thieme Verlag KG Stuttgart · New York

Exploration and Initial Development of Text Classification Models to Identify Health Information Technology Usability-Related Patient Safety Event Reports

Allan Fong
1  National Center for Human Factors in Healthcare, Washington, District of Columbia, United States
,
Tomilayo Komolafe
2  Virginia Polytechnic Institute and State University, Blacksburg, Virginia, United States
,
Katharine T. Adams
1  National Center for Human Factors in Healthcare, Washington, District of Columbia, United States
,
Arman Cohen
3  Allen Institute for Artificial Intelligence, Seattle, Washington, United States
,
Jessica L. Howe
1  National Center for Human Factors in Healthcare, Washington, District of Columbia, United States
,
Raj M. Ratwani
1  National Center for Human Factors in Healthcare, Washington, District of Columbia, United States
4  Georgetown University Medical Center, Washington, District of Columbia, United States
› Author Affiliations
Further Information

Publication History

04 March 2019

06 June 2019

Publication Date:
17 July 2019 (online)

Abstract

Background With the pervasive use of health information technology (HIT) there has been increased concern over the usability and safety of this technology. Identifying HIT usability and safety hazards, mitigating those hazards to prevent patient harm, and using this knowledge to improve future HIT systems are critical to advancing health care.

Purpose The purpose of this work is to demonstrate the feasibility of a modeling approach to identify HIT usability-related patient safety events (PSEs) from the free-text of safety reports and the utility of such models for supporting patient safety analysts in their analysis of event data.

Methods We evaluated three feature representations (bag-of-words [BOWs], topic modeling, and document embeddings) to classify HIT usability-related PSE reports using 5,911 manually annotated reports. Model results were reviewed with patient safety analysts to gather feedback on their usefulness and integration into workflow.

Results The combination of term frequency-inverse document frequency BOWs and document embedding features modeled with support vector machine (SVM) with radial basis function (RBF) had the highest overall precision-recall area under the curve (AUC) and f1-score, 72 and 66%, respectively. Using only document embedding features achieved a similar precision-recall AUC and f1-score performance with the SVM RBF model, 70 and 66%, respectively. Models generally favored specificity and sensitivity over precision. Patient safety analysts found the model results to be useful and offered three suggestions on how it can be integrated into their workflow at the point of report entry, in a visual dashboard layer, and to support data retrievals.

Conclusion Text mining and document embeddings can support identification of HIT usability-related PSE reports. The positive feedback received on the HIT usability model shows its potential utility in real-world applications.

Note

These results are the opinions of MedStar Health researchers and do not reflect in any way an analysis or opinions of the Pennsylvania Patient Safety Authority (the “Authority”). This analysis was not prepared by the Authority. This analysis was conducted by researchers from MedStar Health. Neither the Authority nor its agents, and staff bear any responsibility or liability for the results of MedStar Health's analysis, which are solely the opinion of MedStar Health. The opinions expressed in this document are those of the authors and do not necessarily reflect the official position of the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.


Protection of Human and Animal Subjects

This study was approved by the MedStar Health Research Institute Institutional Review Board (protocol #2014-101).