Download PDF
Eur J Pediatr Surg
DOI: 10.1055/a-2551-2131
Original Article

Solving Complex Pediatric Surgical Case Studies: A Comparative Analysis of Copilot, ChatGPT-4, and Experienced Pediatric Surgeons' Performance

Richard Gnatzy
1   Department of Pediatric Surgery, Leipzig University, Leipzig, Germany
,
Martin Lacher
1   Department of Pediatric Surgery, Leipzig University, Leipzig, Germany
,
Michael Berger
2   Department of Pediatric Surgery, University Hospital Essen, Essen, Germany
,
Michael Boettcher
3   Department of Pediatric Surgery, University Medical Centre Mannheim, Mannheim, Germany
,
Oliver J. Deffaa
1   Department of Pediatric Surgery, Leipzig University, Leipzig, Germany
,
Joachim Kübler
4   Department of Pediatric Surgery, Hospital Bremen-Mitte, Bremen, Germany
,
Omid Madadi-Sanjani
5   Department of Pediatric Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
,
Illya Martynov
6   Centre for Pediatric Surgery, Department of Pediatric Surgery and Urology, University Hospital Giessen-Marburg, Baldingerstraße, Marburg, Germany
7   Centre for Pediatric Surgery, Department of Pediatric Surgery, University Hospital Giessen-Marburg, Giessen, Germany
,
Steffi Mayer
1   Department of Pediatric Surgery, Leipzig University, Leipzig, Germany
,
Mikko P. Pakarinen
8   Department of Pediatric Surgery, University of Helsinki Children's Hospital Unit of Pediatric Surgery, Helsinki, Finland
,
Richard Wagner 
1   Department of Pediatric Surgery, Leipzig University, Leipzig, Germany
,
Tomas Wester
9   Department of Pediatric Surgery, Karolinska University Hospital, Stockholm, Sweden
10   Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden
,
Augusto Zani
11   Department of Surgery, Division of Pediatric Surgery, Washington University School of Medicine, St. Louis, Missouri, United States
,
Ophelia Aubert 
1   Department of Pediatric Surgery, Leipzig University, Leipzig, Germany
3   Department of Pediatric Surgery, University Medical Centre Mannheim, Mannheim, Germany
› Author Affiliations
› Further Information
Also available at
   Permissions and Reprints

Abstract

Introduction

The emergence of large language models (LLMs) has led to notable advancements across multiple sectors, including medicine. Yet, their effect in pediatric surgery remains largely unexplored. This study aims to assess the ability of the artificial intelligence (AI) models ChatGPT-4 and Microsoft Copilot to propose diagnostic procedures, primary and differential diagnoses, as well as answer clinical questions using complex clinical case vignettes of classic pediatric surgical diseases.

Methods

We conducted the study in April 2024. We evaluated the performance of LLMs using 13 complex clinical case vignettes of pediatric surgical diseases and compared responses to a human cohort of experienced pediatric surgeons. Additionally, pediatric surgeons rated the diagnostic recommendations of LLMs for completeness and accuracy. To determine differences in performance, we performed statistical analyses.

Results

ChatGPT-4 achieved a higher test score (52.1%) compared to Copilot (47.9%) but less than pediatric surgeons (68.8%). Overall differences in performance between ChatGPT-4, Copilot, and pediatric surgeons were found to be statistically significant (p < 0.01). ChatGPT-4 demonstrated superior performance in generating differential diagnoses compared to Copilot (p < 0.05). No statistically significant differences were found between the AI models regarding suggestions for diagnostics and primary diagnosis. Overall, the recommendations of LLMs were rated as average by pediatric surgeons.

Conclusion

This study reveals significant limitations in the performance of AI models in pediatric surgery. Although LLMs exhibit potential across various areas, their reliability and accuracy in handling clinical decision-making tasks is limited. Further research is needed to improve AI capabilities and establish its usefulness in the clinical setting.

Keywords

large language models - natural language processing - case studies - pediatric surgery - artificial intelligence

Supplementary Material



Publication History

Received: 11 February 2025

Accepted: 04 March 2025

Accepted Manuscript online:
05 March 2025

Article published online:
02 April 2025

© 2025. Thieme. All rights reserved.

Georg Thieme Verlag KG
Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany

 

  • References

  • 1 Wu S, Koo M, Blum L. et al. Benchmarking open-source Large Language Models, GPT-4 and Claude 2 on multiple-choice questions in nephrology. NEJM AI 2024; 1 (02) AIdbp2300092
    PubMedSearch in Google Scholar
  • 2 Lim ZW, Pushpanathan K, Yew SME. et al. Benchmarking large language models' performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine 2023; 95: 104770
    PubMedSearch in Google Scholar
  • 3 González R, Poenaru D, Woo R. et al.; Pediatric Surgery ChatGPT Collaborative Group. ChatGPT: What every pediatric surgeon should know about its potential uses and pitfalls. J Pediatr Surg 2024; 59 (05) 941-947
    PubMedSearch in Google Scholar
  • 4 Xiao D, Meyers P, Upperman JS, Robinson JR. Revolutionizing Healthcare with ChatGPT: An early exploration of an AI language model's impact on medicine at large and its role in pediatric surgery. J Pediatr Surg 2023; 58 (12) 2410-2415
    PubMedSearch in Google Scholar
  • 5 OpenAI. GPT-4 technical report. Updated November 22, 2022
    PubMed
  • 6 Oztermeli AD, Oztermeli A. ChatGPT performance in the medical specialty exam: An observational study. Medicine (Baltimore) 2023; 102 (32) e34673
    PubMedSearch in Google Scholar
  • 7 Barbour AB, Barbour TA. A radiation oncology board exam of ChatGPT. Cureus 2023; 15 (09) e44541
    PubMedSearch in Google Scholar
  • 8 LV K, Yang Y, Liu T, Gao Q, Guo Q, Qiu X. Full parameter fine-tuning for large language models with limited resources. arXiv preprint 2023; arXiv:2306:09782
    PubMed
  • 9 Zhao WX, Zhou K, Li J. et al. A survey of large language models. arXiv preprint 2023; arXiv:2303:18223
    PubMed
  • 10 Ebel S, Ehrengut C, Denecke T, Gößmann H, Beeskow AB. GPT-4o's competency in answering the simulated written European Board of Interventional Radiology exam compared to a medical student and experts in Germany and its ability to generate exam items on interventional radiology: a descriptive study. J Educ Eval Health Prof 2024; 21: 21
    PubMedSearch in Google Scholar
  • 11 Marcinkevics R, Reis Wolfertstetter P, Wellmann S, Knorr C, Vogt JE. Using machine learning to predict the diagnosis, management and severity of pediatric appendicitis. Front Pediatr 2021; 9: 662183
    CrossrefPubMedSearch in Google Scholar
  • 12 Lure AC, Du X, Black EW. et al. Using machine learning analysis to assist in differentiating between necrotizing enterocolitis and spontaneous intestinal perforation: A novel predictive analytic tool. J Pediatr Surg 2021; 56 (10) 1703-1710
    CrossrefPubMedSearch in Google Scholar
  • 13 Khondker A, Kwong JCC, Rickard M. et al. Application of STREAM-URO and APPRAISE-AI reporting standards for artificial intelligence studies in pediatric urology: A case example with pediatric hydronephrosis. J Pediatr Urol 2024; 20 (03) 455-467
    PubMedSearch in Google Scholar
  • 14 Hsu FR, Dai ST, Chou CM, Huang SY. The application of artificial intelligence to support biliary atresia screening by ultrasound images: A study based on deep learning models. PLoS One 2022; 17 (10) e0276278
    PubMedSearch in Google Scholar
  • 15 Troesch VL, Wald M, Bonnett MA, Storm DW, Lockwood GM, Cooper CS. The additive impact of the distal ureteral diameter ratio in predicting early breakthrough urinary tract infections in children with vesicoureteral reflux. J Pediatr Urol 2021; 17 (02) 208.e1-208.e5
    PubMedSearch in Google Scholar
  • 16 Elahmedi M, Sawhney R, Guadagno E, Botelho F, Poenaru D. The state of artificial intelligence in pediatric surgery: A systematic review. J Pediatr Surg 2024; 59 (05) 774-782
    PubMedSearch in Google Scholar
  • 17 Weegar R. Applying natural language processing to electronic medical records for estimating healthy life expectancy. Lancet Reg Health West Pac 2021; 9: 100132
    PubMedSearch in Google Scholar
  • 18 Borjali A, Magnéli M, Shin D, Malchau H, Muratoglu OK, Varadarajan KM. Natural language processing with deep learning for medical adverse event detection from free-text medical narratives: A case study of detecting total hip replacement dislocation. Comput Biol Med 2021; 129: 104140
    PubMedSearch in Google Scholar
  • 19 Fonferko-Shadrach B, Lacey AS, Roberts A. et al. Using natural language processing to extract structured epilepsy data from unstructured clinic letters: development and validation of the ExECT (extraction of epilepsy clinical text) system. BMJ Open 2019; 9 (04) e023232
    PubMedSearch in Google Scholar
  • 20 Bucher BT, Shi J, Ferraro JP. et al. Portable automated surveillance of surgical site infections using natural language processing: development and validation. Ann Surg 2020; 272 (04) 629-636
    CrossrefPubMedSearch in Google Scholar
  • 21 Li Y, Li Z, Zhang K, Dan R, Jiang S, Zhang Y. ChatDoctor: A medical chat model fine-tuned on a Large Language Model Meta-AI (LLaMA) using medical domain knowledge. Cureus 2023; 15 (06) e40895
    PubMedSearch in Google Scholar
  • 22 Parmanto B, Aryoyudanta B, Soekinto TW. et al. A reliable and accessible Caregiving Language Model (CaLM) to support tools for caregivers: Development and evaluation study. JMIR Form Res 2024; 8: e54633
    PubMedSearch in Google Scholar
  • 23 Reis M, Reis F, Kunde W. Influence of believed AI involvement on the perception of digital medical advice. Nat Med 2024; 30 (11) 3098-3100
    PubMedSearch in Google Scholar
  • 24 Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med 2023; 6 (01) 120
    CrossrefPubMedSearch in Google Scholar
  • 25 Freyer O, Wiest IC, Kather JN, Gilbert S. A future role for health applications of large language models depends on regulators enforcing safety standards. Lancet Digit Health 2024; 6 (09) e662-e672
    PubMedSearch in Google Scholar
  • 26 Wen Z, Tian Z, Jian Z. et al. Perception of knowledge boundary for large language models through semi-open-ended question answering. arXiv preprint 2024; arXiv:2405:14383
    PubMed