Impact of standard enhancement settings of endoscopy systems on performance of endoscopic artificial intelligence systems

Abstract

Background Artificial intelligence (AI) systems in endoscopy are predominantly developed and tested using high-quality imagery from expert centers. However, their performance may be different when applied in clinical practice, partly due to the diversity in post-processing enhancement settings used in endoscopy units. We evaluated the impact of post-processing enhancement settings on AI performance and tested specific data augmentation strategies to mitigate performance loss.

Methods We used a computer-aided detection (CADe) system for Barrett’s neoplasia (6223 images, 906 patients) and a computer-aided diagnosis (CADx) system for colorectal polyps (3288 images, 969 patients), both trained on datasets acquired with Olympus equipment and with limited variability in enhancement settings. The CAD systems were then tested across a wide range of test sets, which comprised the same images, but displayed with different enhancement settings. Both CAD systems were then retrained using image enhancement-based data augmentation. The performance of the adjusted CAD systems was evaluated on the same test sets.

Results Both systems displayed substantial performance variability over a range of enhancement settings (CADe: 83 %–92 % sensitivity, 84 %–91 % specificity; CADx: 78 %–85 % sensitivity, 45 %–63 % specificity). After retraining, variability in sensitivity and specificity was reduced to 2 % (P < 0.001) and 1 %, respectively (P = 0.003) for CADe, and 2 % (P = 0.03) and 8 %, respectively (P = 0.19) for CADx.

Conclusion The performance of endoscopic AI systems can vary substantially depending on post-processing enhancement settings of the endoscopy unit. Specific data augmentation can mitigate this performance loss.

^‡ Co-first authors.

^* All members and collaborators of the Barrett’s Oesophagus Imaging for Artificial Intelligence (BONS-AI) Consortium are listed in the Supplementary material.

Publikationsverlauf

Eingereicht: 03. September 2024

Angenommen nach Revision: 29. Januar 2025

Accepted Manuscript online:
30. Januar 2025

Artikel online veröffentlicht:
28. Februar 2025

• References

• 1 Wallace MB, Sharma P, Bhandari P. et al. Impact of artificial intelligence on miss rate of colorectal neoplasia. Gastroenterology 2022; 163: 295-304.e5
• 2 Fockens N, Jong MR, Jukema JB. et al. A deep learning system for detection of early Barrett’s neoplasia: a model development and validation study. Lancet Digit Health 2023; 5: 905-921
• 3 Finlayson SG, Subbaswamy A, Singh K. et al. The clinician and dataset shift in artificial intelligence. New Engl J Med 2021; 385: 283-286
• 4 Subbaswamy A, Saria S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics 2020; 21: 345-352
• 5 DeGrave AJ, Janizek JD, Lee SI. AI for radiographic COVID-19 detection selects shortcuts over signal. Nat Mach Intell 2021; 3: 610-619
• 6 Ugurlu D, Puyol-Anton E, Ruijsink B. et al. The impact of domain shift on left and right ventricle segmentation in short axis cardiac MR images. arXiv [Preprint] 2021;
• 7 Pooch EHP, Ballester PL, Barros RC. Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification. arXiv [Preprint] 2019;
• 8 Perez L, Wang J. The effectiveness of data augmentation in image classification using deep learning. arXiv [Preprint] 2017;
• 9 Fockens KN, Jukema JB, Boers T. et al. Towards a robust and compact deep learning system for primary detection of early Barrett’s neoplasia: initial image-based results of training on a multi-center retrospectively collected data set. United European Gastroenterol J 2023; 11: 324-336
• 10 Houwen BB, Hazewinkel Y, Giotis I. et al. Real-time computer-aided diagnosis system for optical diagnosis of diminutive colorectal polyps including sessile serrated lesions: a prospective, multicenter study with benchmarking against screening endoscopists. Endoscopy 2022; 54: eP133
• 11 Yu AC, Mohajer B, Eng J. External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiol Artif Intell 2022; 4: e210064
• 12 Karsenti D, Tharsis G, Perrot B. et al. Effect of real-time computer-aided detection of colorectal adenoma in routine colonoscopy (COLO-GENIUS): a single-centre randomised controlled trial. Lancet Gastroenterol Hepatol 2023; 8: 726-734
• 13 Levy I, Bruckmayer L, Klang E. et al. Artificial intelligence-aided colonoscopy does not increase adenoma detection rate in routine clinical practice. Am J Gastroenterol 2022; 117: 1871-1873