Download PDF
Appl Clin Inform 2026; 17(02): 172-176
DOI: 10.1055/a-2818-1611
Research Article

Evaluation of ChatGPT and Gemini in Answering Patient Questions after Gynecologic Surgery

Authors

  • Petra C. Voigt 

    1   Division of Minimally Invasive Gynecologic Surgery, Department of Obstetrics and Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States

  • Rhea S. Sharma

    2   Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States

  • Angela Chaudhari

    1   Division of Minimally Invasive Gynecologic Surgery, Department of Obstetrics and Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States

  • Susan Tsai

    1   Division of Minimally Invasive Gynecologic Surgery, Department of Obstetrics and Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States

  • Magdy P. Milad

    1   Division of Minimally Invasive Gynecologic Surgery, Department of Obstetrics and Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States

  • Linda C. Yang

    1   Division of Minimally Invasive Gynecologic Surgery, Department of Obstetrics and Gynecology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States

Funding Information No grants or industry funding was used in preparation of this manuscript.
Further Information
Also available at
   Permissions and Reprints

Abstract

Objective

This study aimed to explore the performance of ChatGPT version 4.0 (GPT-4) and Gemini Advanced (Gemini) large language models (LLMs) in addressing common patient questions after gynecology surgery with regards to accuracy, relevance, helpfulness to the average patient, and readability.

Methods

In this cross-sectional study, the two LLMs were prompted to generate answers to postoperative patient questions after gynecologic surgery. Postoperative patient questions were developed to simulate common patient questions after gynecologic surgery, based on expert opinion and compiled from anonymous posters on Reddit (r/endometriosis). Questions were focused on six topics: endometriosis, vaginal bleeding, bowel/bladder function, incision care, resumption of activities, and sexual function. Questions were asked in a systematic three-step submission process with the memory reset after each query. Responses were then blinded and independently assessed for accuracy and relevance on a 5-Point Likert scale by four board-certified gynecologic surgeons with fellowship training in gynecologic surgery. Readability of the answers was calculated with the Flesch Kincaid grade level calculator. Responses were also evaluated by three clinic nurses.

Results

A total of 41 questions were posed to GPT-4 and Gemini three times. The responses were independently evaluated by four surgeons and three nurses leading to a total of 1,968 evaluations for accuracy, relevance, helpfulness to the average patient, and readability. Surgeons and nurses graded Gemini responses as more accurate (4.23 vs. 4.03, p = 0.015) and helpful (4.37 vs. 4.21, p = 0.025) than GPT-4 responses. Responses from both models were similarly found to be relevant or very relevant (4.45 vs. 4.36, p = 0.2). Most responses by GPT-4 (85%) and Gemini (87%) were consistent across all questions. The average reading level for GPT-4 and Gemini responses were 11th and 10th grade, above the recommended 6th grade reading level for patient information.

Conclusion

GPT-4 and Gemini provided overall accurate, relevant, and helpful responses to common postoperative patient questions for gynecologic surgery. Gemini outperformed GPT-4 in both accuracy and helpfulness and had objectively more readable responses.

Keywords

large language models - patient questions - postoperative instructions - patient–provider communication

Declaration of GenAI Use

No generative artificial intelligence (AI) or manuscript preparation assistance was used in the writing of this work.


Protection of Human and Animal Subjects

This study was reviewed by the Institutional Review Board of the Northwestern University Feinberg School of Medicine and determined to be exempt. Human and/or animal subjects were not included in this study.


Note

No additional persons contributed to the work reported in the manuscript outside of the manuscript authors.

This study was presented as a Rapid Oral Poster at the Society for Gynecology Surgeons Annual Meeting 2025, March 30–April 2, 2025, in Palm Springs, California, United States.


Supplementary Material



Publication History

Received: 04 August 2025

Accepted after revision: 19 February 2026

Accepted Manuscript online:
24 February 2026

Article published online:
30 March 2026

© 2026. Thieme. All rights reserved.

Georg Thieme Verlag KG
Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany