Doctors Identify Hemorrhage Better during Chart Review when Assisted by Artificial Intelligence

Abstract

Objectives This study evaluated if medical doctors could identify more hemorrhage events during chart review in a clinical setting when assisted by an artificial intelligence (AI) model and medical doctors' perception of using the AI model.

Methods To develop the AI model, sentences from 900 electronic health records were labeled as positive or negative for hemorrhage and categorized into one of 12 anatomical locations. The AI model was evaluated on a test cohort consisting of 566 admissions. Using eye-tracking technology, we investigated medical doctors' reading workflow during manual chart review. Moreover, we performed a clinical use study where medical doctors read two admissions with and without AI assistance to evaluate performance when using and perception of using the AI model.

Results The AI model had a sensitivity of 93.7% and a specificity of 98.1% on the test cohort. In the use studies, we found that medical doctors missed more than 33% of relevant sentences when doing chart review without AI assistance. Hemorrhage events described in paragraphs were more often overlooked compared with bullet-pointed hemorrhage mentions. With AI-assisted chart review, medical doctors identified 48 and 49 percentage points more hemorrhage events than without assistance in two admissions, and they were generally positive toward using the AI model as a supporting tool.

Conclusion Medical doctors identified more hemorrhage events with AI-assisted chart review and they were generally positive toward using the AI model.

Keywords

electronic health records - hemorrhage - artificial intelligence - decision support systems - natural language processing

Data Sharing Statement

We cannot make data publicly available due to sensitive information, but we encourage researchers to contact us for sharing possibilities.

Protection of Human and Animal Subjects

This study only included EHR data that were stored and processed on a secure platform in compliance with the General Data Protection Regulations. The study was approved by The Danish Data Protection Agency at the OUH (Journal nr 21/25172) and the regional secretary and Law Department in the Region of Southern Denmark (Journal nr 21/22013). According to section 14(2) of the Danish Act on Research Ethics Review of Health Research Projects, studies using retrospective data that do not involve human biological material do not require ethical approval.

Ethics

The study was approved by The Danish Data Protection Agency at the OUH (Journal nr 21/25172) and the Regional Secretary and Law Department in the Region of Southern Denmark (Journal nr 21/22013).

^* Equal contribution.

Publikationsverlauf

Eingereicht: 04. April 2023

Angenommen: 29. Juni 2023

Accepted Manuscript online:
03. Juli 2023

Artikel online veröffentlicht:
20. September 2023

• References

• 1 Entzeridou E, Markopoulou E, Mollaki V. Public and physician's expectations and ethical concerns about electronic health record: benefits outweigh risks except for information security. Int J Med Inform 2018; 110: 98-107
• 2 Miotto R, Wang F, Wang S, Jiang X, Dudley JT. Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform 2018; 19 (06) 1236-1246
• 3 Tayefi M, Ngo P, Chomutare T. et al. Challenges and opportunities beyond structured data in analysis of electronic health records. Wiley Interdiscip Rev Comput Stat 2021; 13 (06) e1549
• 4 Valkhoff VE, Coloma PM, Masclee GMC. et al; EU-ADR Consortium. Validation study in four health-care databases: upper gastrointestinal bleeding misclassification affects precision but not magnitude of drug-related upper gastrointestinal bleeding risk. J Clin Epidemiol 2014; 67 (08) 921-931
• 5 Øie LR, Madsbu MA, Giannadakis C. et al. Validation of intracranial hemorrhage in the Norwegian Patient Registry. Brain Behav 2018; 8 (02) e00900
• 6 Delekta J, Hansen SM, AlZuhairi KS, Bork CS, Joensen AM. The validity of the diagnosis of heart failure (I50.0-I50.9) in the Danish National Patient Register. Dan Med J 2018; 65 (04) A5470
• 7 Higgins TL, Deshpande A, Zilberberg MD. et al. Assessment of the accuracy of using ICD-9 diagnosis codes to identify pneumonia etiology in patients hospitalized with pneumonia. JAMA Network Open 2020; 3 (07) e207750-e207750
• 8 Wabe N, Li L, Lindeman R. et al. Evaluation of the accuracy of diagnostic coding for influenza compared to laboratory results: the availability of test results before hospital discharge facilitates improved coding accuracy. BMC Med Inform Decis Mak 2021; 21 (01) 168
• 9 Lee HJ, Jiang M, Wu Y. et al. A comparative study of different methods for automatic identification of clopidogrel-induced bleedings in electronic health records. AMIA Jt Summits Transl Sci Proc 2017; 2017: 185-192
• 10 Taggart M, Chapman WW, Steinberg BA. et al. Comparison of 2 natural language processing methods for identification of bleeding among critically ill patients. JAMA Netw Open 2018; 1 (06) e183451-e183451
• 11 Li R, Hu B, Liu F. et al. Detection of bleeding events in electronic health record notes using convolutional neural network models enhanced with recurrent neural network autoencoders: deep learning approach. JMIR Med Inform 2019; 7 (01) e10788
• 12 Elkin PL, Mullin S, Mardekian J. et al. Using artificial intelligence with natural language processing to combine electronic health record's structured and free text data to identify nonvalvular atrial fibrillation to decrease strokes and death: evaluation and case-control study. J Med Internet Res 2021; 23 (11) e28946
• 13 Mitra A, Rawat BPS, McManus D, Kapoor A, Yu H. Bleeding entity recognition in electronic health records: a comprehensive analysis of end-to-end systems. In: AMIA Annual Symposium Proceedings. Vol. 2020. American Medical Informatics Association; 2020: 860
• 14 Mitra A, Rawat BPS, McManus DD, Yu H. Relation classification for bleeding events from electronic health records using deep learning systems: an empirical study. JMIR Med Inform 2021; 9 (07) e27527
• 15 Shung D, Tsay C, Laine L. et al. Early identification of patients with acute gastrointestinal bleeding using natural language processing and decision rules. J Gastroenterol Hepatol 2021; 36 (06) 1590-1597
• 16 Pedersen JS, Laursen MS, Rajeeth Savarimuthu T. et al. Deep learning detects and visualizes bleeding events in electronic health records. Res Pract Thromb Haemost 2021; 5 (04) e12505
• 17 Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med 2022; 28 (01) 31-38
• 18 Decousus H, Tapson VF, Bergmann JF. et al; IMPROVE Investigators. Factors at admission associated with bleeding risk in medical patients: findings from the IMPROVE investigators. Chest 2011; 139 (01) 69-79
• 19 National Guideline Centre #x0028;UK#x0029;. Venous thromboembolism in over 16s: Reducing the risk of hospital-acquired deep vein thrombosis or pulmonary embolism. London: National Institute for Health and Care Excellence #x0028;NICE#x0029;; March 2018.
• 20 Amin A, Stemkowski S, Lin J, Yang G. Thromboprophylaxis rates in US medical centers: success or failure?. J Thromb Haemost 2007; 5 (08) 1610-1616
• 21 Rwabihama JP, Audureau E, Laurent M. et al; améliorer la prophylaxie de la Maladie ThromboEmbolique veineuse en milieu gériatrique (MATEV) Study Group. Prophylaxis of venous thromboembolism in geriatric settings: a cluster-randomized multicomponent interventional trial. J Am Med Dir Assoc 2018; 19 (06) 497-503
• 22 Amin A, Stemkowski S, Lin J, Yang G. Appropriate thromboprophylaxis in hospitalized cancer patients. Clin Adv Hematol Oncol 2008; 6 (12) 910-920
• 23 Matthiesen S, Meboldt M, Ruckpaul A, Mussgnug M. et al. Eye tracking, a method for engineering design research on engineers' behavior while analyzing technical systems. In: DS 75-7: Proceedings of the 19th International Conference on Engineering Design (ICED13), Design for Harmonies, Vol. 7: Human Behaviour in Design,. Seoul, Korea, August 19–22, 2013: 277-286
• 24 Clark K, Luong MT, Le QV, Manning CD. ELECTRA: pre-training text encoders as discriminators rather than generators. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,. April 26–30, 2020 . OpenReview.net. https://openreview.net/forum?id=r1xMH1BtvB
• 25 Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S. GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP Workshop BLackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics; 2018: 353-355
• 26 Wang A, Pruksachatkun Y, Nangia N. et al. SuperGLUE: a stickier benchmark for general-purpose language understanding systems. In: Wallach H, Larochelle H, Beygelzimer A, d'Alché-Buc F, Fox E, Garnett R. eds. Advances in Neural Information Processing Systems. Vol 32. Curran Associates, Inc.; 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf
• 27 Sun T, Gaut A, Tang S. et al. Mitigating gender bias in natural language processing: literature review. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 1630-1640
• 28 Pedersen JS, Laursen MS, Vinholt PJ, Alnor AB, Savarimuthu TR. Investigating anatomical bias in clinical machine learning algorithms. In: Findings of the European Chapter of the Association for Computational Linguistics: EACL 2023. Association for Computational Linguistics; 2023
• 29 Rogers EM, Singhal A, Quinlan MM. Diffusion of innovations: an integrated approach to communication theory and research. Routledge; 2014: 432-448
• 30 Kahneman D. Thinking, Fast and Slow. Macmillan; 2011: 19-31
• 31 Pedersen JS, Laursen MS, Soguero-Ruiz C, Savarimuthu TR, Hansen RS, Vinholt PJ. Domain over size: clinical ELECTRA surpasses general BERT for bleeding site classification in the free text of electronic health records. In: 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE; 2022: 1-4
• 32 Bojanowski P, Grave E, Joulin A, Mikolov T. Enriching word vectors with subword information. Trans Assoc Comput Linguist 2017; 5: 135-146
• 33 Laursen MS, Pedersen JS, Vinholt PJ, Hansen RS, Savarimuthu TR. Benchmark for evaluation of Danish clinical word embeddings. North Eur J Lang Technol. 2023;9(01):
• 34 Salton G, Wong A, Yang CS. A vector space model for automatic indexing. Commun ACM 1975; 18 (11) 613-620