Accuracy of self-assessment in gastrointestinal endoscopy: a systematic review and meta-analysis

Abstract

Background Assessment is necessary to ensure both attainment and maintenance of competency in gastrointestinal (GI) endoscopy, and this can be accomplished through self-assessment. We conducted a systematic review with meta-analysis to evaluate the accuracy of self-assessment among GI endoscopists.

Methods This was an individual participant data meta-analysis of studies that investigated self-assessment of endoscopic competency. We performed a systematic search of the following databases: Ovid MEDLINE, Ovid EMBASE, Wiley Cochrane CENTRAL, and ProQuest Education Resources Information Center. We included studies if they were primary investigations of self-assessment accuracy in GI endoscopy that used statistical analyses to determine accuracy. We conducted a meta-analysis of studies using a limits of agreement (LoA) approach to meta-analysis of Bland–Altman studies.

Results After removing duplicate entries, we screened 7138 records. After full-text review, we included 16 studies for qualitative analysis and three for meta-analysis. In the meta-analysis, we found that the LoA were wide (−41.0 % to 34.0 %) and beyond the clinically acceptable difference. Subgroup analyses found that both novice and intermediate endoscopists had wide LoA (−45.0 % to 35.1 % and −54.7 % to 46.5 %, respectively) and expert endoscopists had narrow LoA (−14.2 % to 21.4 %).

Conclusions GI endoscopists are inaccurate in self-assessment of their endoscopic competency. Subgroup analyses demonstrated that novice and intermediate endoscopists were inaccurate, while expert endoscopists have accurate self-assessment. While we advise against the sole use of self-assessment among novice and intermediate endoscopists, expert endoscopists may wish to integrate it into their practice.

^* Equal first authors

Publication History

Received: 06 November 2021

Accepted after revision: 20 June 2022

Article published online:
26 September 2022

• References

• 1 Scaffidi MA, Khan R, Grover SC. et al. Self-assessment of competence in endoscopy: challenges and insights. J Can Assoc Gastroenterol 2020; 4: 151-157
• 2 Panadero E. A review of self-regulated learning: Six models and four directions for research. Front Psychol 2017; 8: 1-28
• 3 Panadero E, Jonsson A, Botella J. Effects of self-assessment on self-regulated learning and self-efficacy: Four meta-analyses. Educ Res Rev 2017; 22: 74-98
• 4 Davis DA, Mazmanian PE, Fordis M. et al. Accuracy of physician self-assessment compared a systematic review. JAMA 2006; 296: 1094-1102
• 5 Blanch-Hartigan D. Medical students’ self-assessment of performance: Results from three meta-analyses. Patient Educ Couns 2011; 84: 3-9
• 6 Nayar SK, Musto L, Baruah G. et al. Self-assessment of surgical skills: a systematic review. J Surg Educ 2020; 77: 348-361
• 7 Shamseer L, Moher D, Clarke M. et al. Preferred reporting items for systematic review and meta-analysis protocols (PRIMSA-P) 2015: Elaboration and explanation. BMJ 2015; 349: 1-25
• 8 McGowan J, Sampson M, Salzwedel DM. et al. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 2016; 75: 40-46
• 9 Bramer WM, Giustini D, De Jonge GB . et al. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc 2016; 104: 240-243
• 10 National Institute for Health Research. PROSPERO International prospective register of systematic reviews [Internet]. Accessed: 17 August 2020. Available from: www.crd.york.ac.uk/PROSPERO
• 11 World Health Organization. International Clinical Trials Registry Platform [Internet]. Accessed: 17 August 2020. Available from: apps.who.int/trialsearch
• 12 Walsh CM, Ling SC, Khanna N. et al. Gastrointestinal endoscopy competency assessment tool: Reliability and validity evidence. Gastrointest Endosc 2015; 81: 1417-1424
• 13 Khan R, Zheng E, Wani S. et al. Colonoscopy competence assessment tools: A systematic review of validity evidence. Endoscopy 2021; 53: 1235-1245
• 14 Whiting PF, Rutjes AWS, Westwood ME. et al. QUADAS-2: A revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011; 155: 529-536
• 15 Tipton E, Shuster J. A framework for the meta-analysis of Bland-Altman studies based on a limits of agreement approach. Stat Med 2017; 176: 139-148
• 16 Walsh CM, Ling SC, Mamula P. et al. The gastrointestinal endoscopy competency assessment tool for pediatric colonoscopy. J Pediatr Gastroenterol Nutr 2015; 60: 474-480
• 17 Al-Mansour MR, Tyler KM, Kutayli ZN. et al. Concurrent performance in virtual reality and clinical endoscopy. SAGES conference 2012. Accessed: 11 July 2022. Available from: https://www.sages.org/meetings/annual-meeting/abstracts-archive/?search=P171&meeting=2012
• 18 Amadio JM, Walsh CM, Scaffidi MA. et al. Su1553: Can novice endoscopists accurately self-assess performance during their initial clinical colonoscopies? a prospective, cross-sectional study. . Gastrointest Endosc 2015; 81: AB325
• 19 Ansell J, Hurley JJ, Horwood J. et al. Can endoscopists accurately self-assess performance during simulated colonoscopic polypectomy? A prospective, cross-sectional study. . Am J Surg 2014; 207: 32-38
• 20 Figueroa Barojas Barojas, Sobrino-Cossio SR, Hernandez-Guerrero A. et al. M1412: Endoscopic inanimate biological simulators for training in endoscopic mucosal dissection. Gastrointest Endosc 2010; 71: AB214
• 21 Cookson TA, Beilman CL, Kohansal AR. et al. A231: Endoscopy trainees are able to accurately self-assess procedural skills. J Can Assoc Gastroenterol 2019; 2 (Suppl. 02) 451-452
• 22 Lee RX, Smith M, Samuels S. et al. WP38: Accuracy of clinician-reported adenoma detection rate in colonoscopy. Colorectal Dis 2019; 21 (Suppl. 03) 19-32
• 23 Moritz V, Holme O, Leblanc M. et al. An explorative study from the Norwegian Quality Register Gastronet comparing self-estimated versus registered quality in colonoscopy performance. Endosc Int Open 2016; 04: E326-E332
• 24 Muething L, Kaltenbach TR, Nguyen-vu T. et al. Fr051: Moderate concordance between trainee self-assessment and expert assessment of cold snare polypectomy competence: results from a randomized controlled study. Gastroenterology 2020; 160: 197
• 25 Neumann M, Hahn C, Horbach T. et al. Score card endoscopy: A multicenter study to evalaute learning curves in 1-week courses using the Elangen Endo-Trainer. Endoscopy 2003; 35: 515-520
• 26 Sedlack RE, Kolars JC, Alexander JA. Computer simulation training enhances patient comfort during endoscopy. Clin Gastroenterol Hepatol 2004; 2: 348-352
• 27 Vassiliou MC, Kaneva PA, Poulose BK. et al. Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc 2010; 24: 1834-1841
• 28 Vyasa P, Willis RE, Dunkin BJ. et al. Are general surgery residents accurate assessors of their own flexible endoscopy skills?. J Surg Educ 2016; 74: 23-29
• 29 Wani S, Muthusamy VR, Wang AY. et al. Su1541: Interobserver agreement between trainers and trainees: results from a multicenter study evaluating learning curves and competency in ERCP. Gastrointest Endosc 2014; 79: AB314-AB315
• 30 Scaffidi MA, Grover SC, Carnahan H. et al. Impact of experience on self-assessment accuracy of clinical colonoscopy competence. Gastrointest Endosc 2018; 87: 688-694.e2
• 31 Scaffidi MA, Khan R, Carnahan H. et al. Can pediatric endoscopists accurately assess their clinical competency? A comparison across skill levels. . J Pediatr Gastroenterol Nutr 2019; 68: 311-317
• 32 Scaffidi M, Walsh C, Khan R. et al. Influence of video-based feedback on self-assessment accuracy of endoscopic skills: a randomized controlled trial. Endosc Int Open 2019; 07: E678-E684
• 33 Dunning D, Johnson K, Ehrlinger J. et al. Why people fail to recognize their own incompetence. Curr Dir Psychol Sci 2003; 12: 83-87
• 34 Zaki R, Bulgiba A, Ismail R. et al. Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: A systematic review. PLoS One 2012; 7: 1-7
• 35 Jeyalingam T, Walsh CM. Characterizing the development of colonoscopy competence using assessment data. Am J Gastroenterol 2020; 115: 214-215