Exploration and Initial Development of Text Classification Models to Identify Health Information Technology Usability-Related Patient Safety Event Reports

Abstract

Background With the pervasive use of health information technology (HIT) there has been increased concern over the usability and safety of this technology. Identifying HIT usability and safety hazards, mitigating those hazards to prevent patient harm, and using this knowledge to improve future HIT systems are critical to advancing health care.

Purpose The purpose of this work is to demonstrate the feasibility of a modeling approach to identify HIT usability-related patient safety events (PSEs) from the free-text of safety reports and the utility of such models for supporting patient safety analysts in their analysis of event data.

Methods We evaluated three feature representations (bag-of-words [BOWs], topic modeling, and document embeddings) to classify HIT usability-related PSE reports using 5,911 manually annotated reports. Model results were reviewed with patient safety analysts to gather feedback on their usefulness and integration into workflow.

Results The combination of term frequency-inverse document frequency BOWs and document embedding features modeled with support vector machine (SVM) with radial basis function (RBF) had the highest overall precision-recall area under the curve (AUC) and f1-score, 72 and 66%, respectively. Using only document embedding features achieved a similar precision-recall AUC and f1-score performance with the SVM RBF model, 70 and 66%, respectively. Models generally favored specificity and sensitivity over precision. Patient safety analysts found the model results to be useful and offered three suggestions on how it can be integrated into their workflow at the point of report entry, in a visual dashboard layer, and to support data retrievals.

Conclusion Text mining and document embeddings can support identification of HIT usability-related PSE reports. The positive feedback received on the HIT usability model shows its potential utility in real-world applications.

Keywords

natural language processing - interfaces and usability - incident reporting - electronic health records and systems - safety

Note

These results are the opinions of MedStar Health researchers and do not reflect in any way an analysis or opinions of the Pennsylvania Patient Safety Authority (the “Authority”). This analysis was not prepared by the Authority. This analysis was conducted by researchers from MedStar Health. Neither the Authority nor its agents, and staff bear any responsibility or liability for the results of MedStar Health's analysis, which are solely the opinion of MedStar Health. The opinions expressed in this document are those of the authors and do not necessarily reflect the official position of the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

Protection of Human and Animal Subjects

This study was approved by the MedStar Health Research Institute Institutional Review Board (protocol #2014-101).

• References

• 1 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 2 Ratwani RM, Savage E, Will A. , et al. Identifying electronic health record usability and safety challenges in pediatric settings. Health Aff (Millwood) 2018; 37 (11) 1752-1759
• 3 Institute of Medicine. Health IT and Patient Safety Building Safer Systems for Better Care. Washington, DC: National Academies Press; 2012
• 4 Puthumana JS, Fong A, Blumenthal J, Ratwani RM. Making patient safety event data actionable: understanding patient safety analyst needs. J Patient Saf 2017; DOI: 10.1097/PTS.0000000000000400.
• 5 Amato MG, Salazar A, Hickman T-TT. , et al. Computerized prescriber order entry–related patient safety reports: analysis of 2522 medication errors. J Am Med Inform Assoc 2017; 24 (02) 316-322
• 6 Fong A, Harriott N, Walters DM, Foley H, Morrissey R, Ratwani RR. Integrating natural language processing expertise with patient safety event review committees to improve the analysis of medication events. Int J Med Inform 2017; 104: 120-125
• 7 Chai KEK, Anthony S, Coiera E, Magrabi F. Using statistical text classification to identify health information technology incidents. J Am Med Inform Assoc 2013; 20 (05) 980-985
• 8 Ong M-S, Magrabi F, Coiera E. Automated categorisation of clinical incident reports using statistical text classification. Qual Saf Health Care 2010; 19 (06) e55
• 9 Pennington J, Socher R, Manning CD. GloVe: Global Vectors for Word Representation. In: Conference on Empirical Methods in Natural Language Processing; 2014: 1532-1543
• 10 Mikolov T, Chen K, Corrado G, Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Proceedings of the Advances in Neural Information Processing Systems; 2013: 3111-3119
• 11 Chen M, Jin X, Shen D. Short Text Classification Improved by Learning Multi-Granularity Topics. In: International Joint Conference on Artificial Intelligence; 2011: 1776-1781
• 12 Aggarwal CC, Zhai C. A Survey of Text Classification Algorithms. Mining Text Data. Boston, MA: Springer; 2012: 163-222
• 13 Wang Z, Qian X. Text Categorization Based on LDA and SVM. In: International Conference on Computer Science and Software Engineering; 2008: 674-677
• 14 Blei DM. Probabilistic topic models. Commun ACM 2012; 55 (04) 77-84
• 15 Fong A, Ratwani R. An evaluation of patient safety event report categories using unsupervised topic modeling. Methods Inf Med 2015; 54 (04) 338-345
• 16 Blei D, Ng A, Jordan M. Latent Dirichlet allocation. J Mach Learn Res 2003; 3: 993-1022
• 17 Le Q, Mikolov T. Distributed representations of sentences and documents. In: International Conference on Machine Learning; 2014: 1188-1196
• 18 J.H. Lau, T. Baldwin, An empirical evaluation of doc2vec with practical insights into document embedding generation, ArXiv Prepr. ArXiv1607.05368; 2016
• 19 Yang Q, Zimmerman J, Steinfeld A, Carey L, Antaki JF. Investigating the heart pump implant decision process: opportunities for decision support tools to help. ACM Trans Comput Hum Interact 2016; 2016: 4477-4488
• 20 Gong Y, Kang H, Wu X, Hua L. Enhancing patient safety event reporting. A systematic review of system design features. Appl Clin Inform 2017; 8 (03) 893-909