PDF icon PDF herunterladen
Open Access
CC BY 4.0 · Appl Clin Inform 2025; 16(03): 718-731
DOI: 10.1055/a-2565-9155
Research Article

Automating Responses to Patient Portal Messages Using Generative AI

Amarpreet Kaur
1   Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania
,
Alexander Budko
1   Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania
,
Katrina Liu
2   School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, Pennsylvania
,
Eric Eaton
2   School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, Pennsylvania
,
Bryan D. Steitz
3   Vanderbilt University Medical Center, Vanderbilt University, Nashville, Tennessee
,
Kevin B. Johnson
1   Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania
2   School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, Pennsylvania
› Institutsangaben
Funding None.
› Weitere Informationen
Auch verfügbar aufeRef
  Lizenzen und Reprints
Preview

Abstract

Background

Patient portals bridge patient and provider communications but exacerbate physician and nursing burnout. Large language models (LLMs) can generate message responses that are viewed favorably by health care professionals/providers (HCPs); however, these studies have not included diverse message types or new prompt-engineering strategies.

Objectives

Our goal is to investigate and compare the quality and precision of GPT-generated message responses versus real doctor responses across the spectrum of message types within a patient portal.

Methods

We used prompt engineering techniques to craft synthetic provider responses tailored to adult primary care patients. We enrolled a sample of primary care providers in a cross-sectional study to compare authentic with synthetic patient portal message responses generated by GPT-3.5-turbo, July 2023 version (GPT). The survey assessed each response's empathy, relevance, medical accuracy, and readability on a scale from 0 to 5. Respondents were asked to identify responses that were GPT-generated versus provider-generated. Mean scores for all metrics were computed for subsequent analysis.

Results

A total of 49 HCPs participated in the survey (59% completion rate), comprising 16 physicians and 32 advanced practice providers (APPs). In comparison to responses generated by real doctors, GPT-generated responses scored statistically significantly higher than doctors in two of the four parameters: empathy (p < 0.05) and readability (p < 0.05). However, no statistically significant difference was observed for relevance and accuracy (p > 0.05). Although readability scores were significantly different, the absolute difference was small, and the clinical significance of this finding remains uncertain.

Conclusion

Our findings affirm the potential of GPT-generated message responses to achieve comparable levels of empathy, relevance, and readability to those found in typical responses crafted by HCPs. Additional studies should be done within provider workflows and with careful evaluation of patient attitudes and concerns related to the ethics as well as the quality of generated responses in all settings.

Keywords

patient web portal - physician–patient interaction - artificial intelligence - communication - health - medical informatics - large language models

Protection of Human Subjects

The University of Pennsylvania Human Research Protection Program, under study No. 854147, granted approval for this research project. Participant consent was not deemed necessary as the study involved secondary data analysis of patient portal messages sourced through a meticulously crafted pipeline. Furthermore, the protocol for this research, also approved under study No. 854147, granted approval for retrieving the initial set of patient portal messages from a repository at Vanderbilt University Medical Center (VUMC), which were later used to create synthetic patient portal messages used in the study. The utilization of patient portal messages from VUMC was conducted in compliance with ethical guidelines. This study did not require patient consent for using the patient portal messages retrieved from VUMC, as the data used in this study underwent a rigorous de-identification process, rendering it impossible to trace any information back to individual patients. Thus, our research respects and upholds the principles of confidentiality and anonymity, ensuring the protection of participants' privacy rights in accordance with established ethical standards.




Publikationsverlauf

Eingereicht: 12. August 2024

Angenommen: 11. Februar 2025

Accepted Manuscript online:
25. März 2025

Artikel online veröffentlicht:
30. Juli 2025

© 2025. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, and reproduction so long as the original work is properly cited. (https://creativecommons.org/licenses/by/4.0/)

Georg Thieme Verlag KG
Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany