CC BY 4.0 · Endoscopy 2021; 53(09): 893-901
DOI: 10.1055/a-1306-7590
Original article

Establishing key research questions for the implementation of artificial intelligence in colonoscopy: a modified Delphi method

 1   Wellcome/EPSRC Centre for Interventional & Surgical Sciences, University College London, London, UK
 2   Digestive Disease Center, Showa University Northern Yokohama Hospital, Yokohama, Japan
 3   Clinical Effectiveness Research Group, Institute of Health and Society, University of Oslo, Oslo, Norway
Masashi Misawa
 2   Digestive Disease Center, Showa University Northern Yokohama Hospital, Yokohama, Japan
 2   Digestive Disease Center, Showa University Northern Yokohama Hospital, Yokohama, Japan
 4   Department of Gastroenterology, Gloucestershire Hospitals NHS Foundation Trust, Gloucester, UK
Jorge Bernal
 5   Computer Science Department, Universitat Autonoma de Barcelona and Computer Vision Center, Barcelona, Spain
Tyler M. Berzin
 6   Center for Advanced Endoscopy, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
 7   Department of Gastroenterology and Hepatology, University Hospitals Leuven, TARGID KU Leuven, Leuven, Belgium
Michael F. Byrne
 8   Division of Gastroenterology, Vancouver General Hospital, University of British Columbia, Vancouver, British Columbia, Canada
 9   Division of Gastroenterology, Tri-Service General Hospital, National Defense Medical Center, Taipei, Taiwan
10   Translational Gastroenterology Unit, John Radcliffe Hospital, Oxford, UK
11   Oxford NIHR Biomedical Research Centre, University of Oxford, Oxford, UK
Tom Eelbode
12   Medical Imaging Research Center, ESAT/PSI, KU Leuven, Leuven, Belgium
Daniel S. Elson
13   Hamlyn Centre for Robotic Surgery, Institute of Global Health Innovation, Imperial College London, London, UK
14   Department of Surgery and Cancer, Imperial College London, London, UK
Suryakanth R. Gurudu
15   Division of Gastroenterology and Hepatology, Mayo Clinic, Scottsdale, Arizona, USA
16   ETIS, Universite de Cergy-Pointoise, ENSEA, CNRS, Cergy-Pointoise Cedex, France
William E. Karnes
17   H. H. Chao Comprehensive Digestive Disease Center, Division of Gastroenterology & Hepatology, Department of Medicine, University of California, Irvine, California, USA
Alessandro Repici
18   Department of Gastroenterology, Humanitas Clinical and Research Center, IRCCS, Rozzano, Milan, Italy
19   Humanitas University, Department of Biomedical Sciences, Pieve Emanuele, Milan, Italy
Rajvinder Singh
20   Department of Gastroenterology and Hepatology, Lyell McEwan Hospital, Adelaide, South Australia, Australia
Pietro Valdastri
21   School of Electronics and Electrical Engineering, University of Leeds, Leeds, UK
22   Division of Gastroenterology & Hepatology, Mayo Clinic, Jacksonville, Florida, USA
Pu Wang
23   Department of Gastroenterology, Sichuan Academy of Medical Sciences and Sichuan Provincial People’s Hospital, Chengdu, China
Danail Stoyanov
 1   Wellcome/EPSRC Centre for Interventional & Surgical Sciences, University College London, London, UK
Laurence B. Lovat
 1   Wellcome/EPSRC Centre for Interventional & Surgical Sciences, University College London, London, UK
24   Gastrointestinal Services, University College London Hospital, London, UK
› Author Affiliations
Supported by: Wellcome Trust 203145Z/16/Z
Supported by: Engineering and Physical Sciences Research Council 203145Z/16/Z
Supported by: Engineering and Physical Sciences Research Council EP/P027938/1


Background Artificial intelligence (AI) research in colonoscopy is progressing rapidly but widespread clinical implementation is not yet a reality. We aimed to identify the top implementation research priorities.

Methods An established modified Delphi approach for research priority setting was used. Fifteen international experts, including endoscopists and translational computer scientists/engineers, from nine countries participated in an online survey over 9 months. Questions related to AI implementation in colonoscopy were generated as a long-list in the first round, and then scored in two subsequent rounds to identify the top 10 research questions.

Results The top 10 ranked questions were categorized into five themes. Theme 1: clinical trial design/end points (4 questions), related to optimum trial designs for polyp detection and characterization, determining the optimal end points for evaluation of AI, and demonstrating impact on interval cancer rates. Theme 2: technological developments (3 questions), including improving detection of more challenging and advanced lesions, reduction of false-positive rates, and minimizing latency. Theme 3: clinical adoption/integration (1 question), concerning the effective combination of detection and characterization into one workflow. Theme 4: data access/annotation (1 question), concerning more efficient or automated data annotation methods to reduce the burden on human experts. Theme 5: regulatory approval (1 question), related to making regulatory approval processes more efficient.

Conclusions This is the first reported international research priority setting exercise for AI in colonoscopy. The study findings should be used as a framework to guide future research with key stakeholders to accelerate the clinical implementation of AI in endoscopy.

Appendix 1s, 2s, Table 1s

Publication History

Received: 24 June 2020

Accepted: 09 November 2020

Accepted Manuscript online:
09 November 2020

Article published online:
13 January 2021

© 2020. © 2020. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, and reproduction so long as the original work is properly cited. (

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Kudo S, Mori Y, Misawa M. et al. Artificial intelligence and colonoscopy: Current status and future perspectives. Dig Endosc 2019; 31: 363-371
  • 2 Ahmad OF, Soares AS, Mazomenos E. et al. Artificial intelligence and computer-aided diagnosis in colonoscopy: current evidence and future directions. Lancet Gastroenterol Hepatol 2019; 4: 71-80
  • 3 Mori Y, Kudo S, Berzin TM. et al. Computer-aided diagnosis for colonoscopy. Endoscopy 2017; 49: 813-819
  • 4 Wang P, Berzin TM, Glissen Brown JR. et al. Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut 2019; 68: 1813-1819
  • 5 Wang P, Liu X, Berzin TM. et al. Effect of a deep-learning computer-aided detection system on adenoma detection during colonoscopy (CADe-DB trial): a double-blind randomised study. Lancet Gastroenterol Hepatol 2020; 5: 343-351
  • 6 Su J-R, Li Z, Shao X-J. et al. Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: a prospective randomized controlled study (with videos). Gastrointest Endosc 2020; 91: 415-424.e4
  • 7 Liu W-N, Zhang Y-Y, Bian X-Q. et al. Study on detection rate of polyps and adenomas in artificial-intelligence-aided colonoscopy. Saudi J Gastroenterol 2020; 26: 13-19
  • 8 Repici A, Badalamenti M, Maselli R. et al. Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial. Gastroenterology 2020; 159: 512-520.e7
  • 9 He J, Baxter SL, Xu J. et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019; 25: 30-36
  • 10 Langlotz CP, Allen B, Erickson BJ. et al. A roadmap for foundational research on artificial intelligence in medical imaging: From the 2018 NIH/RSNA/ACR/The Academy Workshop. Radiology 2019; 291: 781-791
  • 11 Vinsard DG, Mori Y, Misawa M. et al. Quality assurance of computer-aided detection and diagnosis in colonoscopy. Gastrointest Endosc 2019; 90: 55-63
  • 12 Rees CJ, Ngu WS, Regula J. et al. European Society of Gastrointestinal Endoscopy – Establishing the key unanswered research questions within gastrointestinal endoscopy. Endoscopy 2016; 48: 884-891
  • 13 Francis N, Kazaryan AM, Pietrabissa A. et al. A research agenda for the European Association for Endoscopic Surgeons (EAES). Surg Endosc 2017; 31: 2042-2049
  • 14 Burt CG, Cima RR, Koltun WA. et al. Developing a research agenda for the American Society of Colon and Rectal Surgeons: Results of a Delphi approach. Dis Colon Rectum 2009; 52: 898-905
  • 15 Hart AL, Lomer M, Verjee A. et al. What are the top 10 research questions in the treatment of inflammatory bowel disease? A priority setting partnership with the James Lind Alliance. J Crohns Colitis 2017; 11: 204-211
  • 16 Liu X, Cruz Rivera S, Moher D. et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med 2020; 26: 1364-1374
  • 17 Wang P, Liu P, Glissen Brown JR. et al. Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study. Gastroenterology 2020; 159: 1252-1261.e5
  • 18 Mori Y, Kudo S, Misawa M. et al. Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: A prospective study. Ann Intern Med 2018; 169: 357-366
  • 19 Hassan C, Repici A. The FUSE enigma: Wide-angle or wide-minded?. Gastrointest Endosc 2018; 88: 865-867
  • 20 East JE, Rees CJ. Making optical biopsy a clinical reality in colonoscopy. Lancet Gastroenterol Hepatol 2018; 3: 10-12
  • 21 Bisschops R, East JE, Hassan C. et al. Advanced imaging for detection and differentiation of colorectal neoplasia: European Society of Gastrointestinal Endoscopy (ESGE) Guideline – Update 2019. Endoscopy 2019; 51: 1155-1179
  • 22 Barua I, Vinsard D, Jodal H. et al. Artificial intelligence for polyp detection during colonoscopy: a systematic review and meta-analysis. Endoscopy 2020; DOI: 10.1055/a-1201-7165.
  • 23 Hassan C, Bhandari P, Antonelli G. et al. Artificial intelligence for non-polypoid colorectal neoplasms. Dig Endosc 2020; DOI: 10.1111/den.13807.
  • 24 Bernal J, Tajkbaksh N, Sanchez FJ. et al. Comparative validation of polyp detection methods in video colonoscopy: Results from the MICCAI 2015 Endoscopic Vision Challenge. IEEE Trans Med Imaging 2017; 36: 1231-1249
  • 25 Xu S, Perez M, Yang K. et al. Determination of the latency effects on surgical performance and the acceptable latency levels in telesurgery using the dV-Trainer simulator. Surg Endosc 2014; 28: 2569-2576
  • 26 Byrne MF, Chapados N, Soudan F. et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut 2019; 68: 94-100
  • 27 Mori Y, Kudo S-E, Misawa M. et al. Simultaneous detection and characterization of diminutive polyps with the use of artificial intelligence during colonoscopy. VideoGIE 2019; 4: 7-10
  • 28 Guizard N, Ghalehjegh SH, Henkel M. et al. Artificial intelligence for real-time multiple polyp detection with identification, tracking, and optical biopsy during colonoscopy. Gastroenterology 2019; 156: S48-S49
  • 29 Sánchez-Montes C, Sánchez FJ, Bernal J. et al. Computer-aided prediction of polyp histology on white light colonoscopy using surface pattern analysis. Endoscopy 2019; 51: 261-265
  • 30 Lutnick B, Ginley B, Govind D. et al. An integrated iterative annotation technique for easing neural network training in medical image analysis. Nat Mach Intell 2019; 1: 112-119
  • 31 Borgli H, Thambawita V, Smedsrud PH. et al. HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci Data 2020; 7: 283
  • 32 Ahmad OF, Stoyanov D, Lovat LB. Barriers and pitfalls for artificial intelligence in gastroenterology: Ethical and regulatory issues. Tech Innov Gastrointest Endosc 2020; 22: 80-84
  • 33 Mori Y, Kudo S, East JE. et al. Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video). Gastrointest Endosc 2020; 92: 905-911.e1