UK Joint Advisory Group consensus statements for training and certification in endoscopic retrograde cholangiopancreatography

Introduction

Endoscopic retrograde cholangiopancreatography (ERCP) is a complex and technically demanding procedure. Over the last three decades, the role of ERCP has shifted to one which is almost exclusively therapeutic. Of all the widely performed endoscopic modalities, ERCP carries the greatest risk of serious complications with a recognized complication rate of between 10 and 14 %, and a death rate of 0.1 to 1 % [1] [2]. Adverse events have been attributed to patient selection, procedure type, and ERCPist experience [3] [4] [5]. In 2004, the National Confidential Enquiry into Patient Outcome and Death audit and other documents, acknowledged that some trainees who accredited in gastroenterology, but with limited training in ERCP, reported their intention to perform this procedure without additional supervision or further training [6] [7] [8]. In 2014, the ERCP Standards Framework was published by the British Society of Gastroenterology (BSG) in an attempt to define quality standards and recommendations for training in ERCP, but stopped short of specifying how and when trainees should be credentialed for independent practice [9].

Invariably, high-quality ERCP is reliant on high-quality training. Prolonged training time is required for novices to transition into a competent independent practitioner capable of delivering a safe and high-quality service. Although guidance on ERCP credentialing exists outside the UK [10] [11], these are typically based on minimum procedural numbers and key performance indicators (KPIs), with variable uptake within training systems [12]. In the era of competency-based education, there is an urgent need for national consensus and implementation of an ERCP credentialing pathway, which accommodates the differences in the training environment and learning curves between trainees. These are necessary to signpost trainees and training programs with clear and verifiable competency endpoints to define the minimum standards required to perform ERCP independently.

In the UK, the Joint Advisory Group on Gastrointestinal Endoscopy (JAG) oversees endoscopy training and certification [13]. Certification is a national, standardized process which formally credentials a trainee for independent and unsupervised endoscopy in the UK. Since 2011, JAG certification has been awarded for gastroscopy, flexible sigmoidoscopy and colonoscopy [14]. In recent years, there have been calls for certification to be extended to ERCP with the intention of quality assuring training and to improve UK ERCP standards [15]. In response, and following consultation with UK Specialist Advisory Committees (SACs), an expert committee was assembled by JAG and its stakeholders, including BSG and the Association of Upper gastrointestinal Surgeons (AUGIS), to develop evidence and consensus-based recommendations relevant to training and certification in ERCP.

Aims and scope

The aim of this Delphi process was to develop a robust set of recommendations which would form the framework of ERCP certification within the UK. This would apply to all endoscopist who wish to begin or pursue ERCP training in the UK, regardless of grade or specialty. Specifically, recommendations were made on the following areas:

• Definition of competence in trainees

• Acquisition of competence

• Assessment of competence

• Post-certification support

The following aspects were not included within the scope of this guideline:

• Advanced procedures (Schutz 3 and 4 procedures)

• Advanced therapeutics (e. g. pancreatic therapy, cholangioscopy, ampullectomy, endoscopic ultrasound (EUS)-assisted ERCP)

• Pediatric ERCP

• Up-skilling for established independent endoscopists

• Trainees or practitioners in whom the majority of ERCP training has been undertaken outside the UK or before implementation of this document

Methods

Guideline development

A modified Delphi process was commissioned by the JAG Quality Assurance of Training Working Group, with inclusion of JAG, BSG, AUGIS, training leads, trainee members, and representation from England, Wales, Scotland and Northern Ireland. Through a series of teleconferences, participants were allocated to four working groups based on the scope of the guideline. Each working group was tasked with framing questions relevant to training and certification, using a Population, Intervention, Comparator, Outcome (PICO) format where possible. Literature searches were then conducted by independent working groups on major databases including Embase, Medline and the Cochrane Database of Systematic Reviews. Results were collated and summarized into recommendation statements. Working groups were allowed to formulate statements relevant to training in the absence of evidence if recommendations were felt to be important to training and certification. These were appraised using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework [16]. While it was recognized that evidence in the ERCP training literature was limited, GRADE methodology was necessary to facilitate evidence review and discussion among stakeholders. The level of evidence and strength of recommendation were provided for each statement. Although it is standard practice to align recommendations with the level of evidence, statements could receive discordant recommendations (e. g. strong recommendation for low quality evidence) if the perceived benefit in clinical practice outweighed the paucity of available evidence. In a teleconference prior to the face-to-face voting process, statements and supporting evidence were peer-reviewed by the guideline development group to maximize efficiency of the consensus process.

Consensus process

An anonymized, electronic voting process was undertaken during a face-to-face meeting to measure consensus with recommendation statements. Five Likert scale responses were provided for each statement (strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree). It is accepted that “agree” and “strongly agree” indicate agreement with a statement. Eighty percent or more agreement was the specified a priori threshold to accept a statement. For statements that were not accepted, up to three rounds of revisions and re-voting were permitted before they were rejected. Upon collation of the accepted statements, the document was ratified by relevant stakeholder groups and SACs for review. Statements were then included in the final ERCP certification pathway.

Recommendation statements

In total, 27 recommendation statements were generated for the following domains: 1) Definition of competence (9 statements); 2) Acquisition of competence (8 statements); 3) Assessment of competence (6 statements); and 4) Post-certification support (4 statements). The full list of statements are shown in [Table 1] and an illustrated summary provided in [Fig. 1].

Table 1 Summary of recommendations for training and certification in endoscopic retrograde cholangiopancreatography (ERCP).

Recommendation statement
1.1	ERCP competence should be defined as the ability to independently carry out effective procedures across a spectrum of case difficulty and case contexts with acceptable safety
1.2	ERCP is an advanced therapeutic procedure which is operator-dependent and requires specific knowledge and skills-based training to achieve competence
1.3	The rate of successful selective deep cannulation of duct of interest is an important determinant of competency and correlates with improved performance, but it should not serve as the sole marker of competency
1.4	ERCP training should take place within a structured training programme to achieve the requisite knowledge and skill-based competencies
1.5	Trainees are required to demonstrate non-technical skills of ERCP (i. e. communication skills, situational awareness, leadership and judgement)
1.6	The modified Schutz score should be used to grade the difficulty of ERCP procedures
1.7	Successful completion of an ERCP is defined as the completion of therapeutic intent in ERCPs of grade 1 and 2 complexity. This should be achieved without any trainer assistance in ≥ 80 % of cases by the end of training, and before a mentored period of practice.
1.8	When performed by trainees, ERCPs of grade 3 and 4 complexity can be used to count towards lifetime procedure numbers and completion metrics, e. g. deep cannulation rates, but should be excluded from other key performance indicators (e. g. therapeutic success)
1.9	Trainees should be able to demonstrate an overall 30-day post-ERCP pancreatitis rate of 5 % or less of their Schutz 1 and 2 ERCP cases
2.1	Trainees should be competent in diagnostic esophagogastroduodenoscopy and have experience of upper gastrointestinal endoscopic therapy before commencing ERCP training.
2.2	Trainees should demonstrate the desire and commitment to practice ERCP at consultant-level
2.3	It is desirable but not mandatory for trainees to train in both ERCP and endoscopic ultrasound (EUS)
2.4	For ERCP certification, UK trainees are required to attend a JAG accredited basic ERCP course (with simulation and lectures) in the early stages of their ERCP training. JAG-accredited intermediate and upskilling courses are encouraged but not mandatory
2.5	Trainees are recommended to use digital resources and attend live endoscopy courses and conferences to become familiar with ERCP techniques and accessories
2.6	Trainees are required to show evidence of attendance at hepatobiliary multidiscliplinary team meetings and contribute to the care of inpatients and outpatients with pancreaticobiliary disease
2.7	Trainers delivering training in ERCP should have undertaken an endoscopy-specific train-the-trainers course
2.8	All trainees should have evidence of experience of a minimum 300 ERCP cases prior to certification
3.1	Formative DOPS assessments during ERCP training should be used to track progression in technical and non-technical skills, and to support trainee feedback
3.2	Formative DOPS assessment should be performed regularly (i. e. at least 1 DOPS per every 10 hands-on training procedures during training) to provide objective evidence of skills acquisition and targeted feedback
3.3	Self-assessment is an acceptable method of monitoring competency development which should be used in conjunction with objective assessment tools. Trainees should log all training procedures onto the JETS e-portfolio
3.4	Trainees must demonstrate the following key performance indicators to be eligible for summative assessment for certification: Complete stone clearance in 70 % Stenting of distal biliary strictures 75 % Native papilla cannulation rate 80 % Unassisted in 80 % of cases in last 3 months (minimum 15 cases)
3.5	Formative ERCP DOPS assessments should be used in conjunction with other supporting certification criteria to assess eligibility for summative assessment. To undertake summative assessment, trainees should be rated as “ready for independent practice” in ≥ 85 % of the individual items of 5 recent formative DOPS (minimum of 3 DOPS assessments on cases with a native papilla within the past 6 months), and with no items rated as requiring “maximum supervision”
3.6	For successful completion of the summative DOPS assessment, the trainee should be rated as “ready for independent practice” in all items within two DOPS assessments, by two different assessors, one of whom is not based at their current endoscopy unit
4.1	Newly certified ERCP practitioners should have a defined period of mentorship lasting a minimum period of 2 years, with provisions available for regular progress reviews, e. g. at 3-month intervals
4.2	The ongoing training requirements of newly accredited ERCP practitioners should be identified and should be encouraged to attend further training opportunities, e. g. up-skilling courses
4.3	Clinicians who have recently certified in ERCP should have systems in place to ensure appropriate case load selection: regular vetting of cases or through weekly HPB MDT/triage meetings
4.4	There should be appropriate mechanisms in place for performance monitoring and review during the agreed transition period, e. g. at 3-month intervals

ERCP, endoscopic retrograde cholangiopancreatography; JAG, Joint Advisory Group on Gastrointestinal Endoscopy; DOPS, direct observation of procedural skills; HPB, hepatobiliary; MDT, multidisciplinary team.

Fig. 1 Proposed Joint Advisory Group (JAG) pathway for training and certification in endoscopic retrograde cholangiopancreatography (ERCP) in the United Kingdom. DOPS, direct observation of procedure skills; EGD: esophagogastroduodenoscopy; PD, program director; JETS e-portfolio, Joint Advisory Group Endoscopy Training System e-portfolio.

1. Definition of Competence in ERCP

1.1: ERCP competence should be defined as the ability to independently carry out effective procedures across a spectrum of case difficulty and case contexts with acceptable safety (strong recommendation, low quality evidence).

Consensus: 100 %

Competence in endoscopy may be defined as the ability to independently carry out procedures in a safe and effective manner, and across a spectrum of case difficulties and case contexts. The American Society of Gastrointestinal Endoscopy (ASGE) recommends for ERCP credentialing decisions to be based on the achievement of selective cannulation in at least 90 % of procedures, accurately interpreting endoscopic and radiologic images, and successful sphincterotomy and stent placement when necessary [10].

1.2: ERCP is an advanced therapeutic procedure which is operator-dependent and requires specific knowledge and skills-based training to achieve competence (strong recommendation, low quality evidence).

Consensus: 100 %

Trainees in ERCP should possess the knowledge and skills related to:

• Procedural indications and contraindications

• Radiation safety

• Instruments and accessories

• Pre-procedure optimization of the acutely ill patient

• ERCP skills: technical and non-technical

• Procedure outcomes, adverse events and their management

• Alternative approaches in the case of a failed procedure

1.3: The rate of successful selective deep cannulation of duct of interest is an important determinant of competency and correlates with improved performance, but it should not serve as the sole marker of competency (strong recommendation, high quality evidence).

Consensus: 100 %

Selective common bile duct cannulation (CBD) is often used as a surrogate marker for competency rather than those that relate to sphincterotomy, stent placement and stone extraction. Furthermore, it is known that substantial variation in learning curves may exist within a trainee cohort [17]. ERCP is a demanding procedure that requires both technical and cognitive skills, with both having distinct learning curves [18] [19] [20] [21] [22] [23] [24].

1.4: ERCP training should take place within a structured training program to achieve the requisite knowledge and skill-based competencies (strong recommendation, low quality evidence).

Consensus: 100 %

While there are no studies comparing training within and outside a structured training program, two studies by Wani et al in 2016 and 2018 which evaluated advanced fellowship (EUS/ERCP) training programs in the USA showed high levels of cognitive competency at the end of a 1-year training [17] [25]. This was the case for various levels of exposure to ERCP prior to beginning the training program, including those with no procedures prior to the program. The EUS and ERCP Skills Assessment Tool (TEESAT), a validated TEESAT cognitive aspect scores was used to assess the participants. These validate the use of dedicated ERCP/EUS training programs.

1.5: Trainees are required to demonstrate non-technical skills of ERCP (i. e. communication skills, situational awareness, leadership and judgment) (strong recommendation, low quality evidence).

Consensus: 100 %

Endoscopic non-technical skills are generic skills encompassing communication skills, teamwork, situational awareness, clinical judgment, decision-making and leadership [26] [27] [28]. These are considered essential for safe and effective ERCP, and are associated with positive effects on team performance and clinical outcomes [28] [29].

1.6: The modified Schutz score should be used to grade the difficulty of ERCP procedures (strong recommendation, low quality evidence).

Consensus: 100 %

The modified Schutz score ( [Table 2]) is a well-recognized tool to grade ERCP difficulty and has been validated previously by the ASGE working group [30]. It is commonly used in studies assessing the success rates of ERCPs based on difficulty.

Table 2 Modified Schutz scale for grading complexity in endoscopic retrograde cholangiopancreatography (ERCP) [30].

Grade[1]	Procedure
1	Deep cannulation of duct of interest, main papilla, or sampling Biliary stent removal or exchange
2	Biliary stone extraction < 10 mm Treatment of biliary leaks Treatment of extrahepatic strictures (benign or malignant) Placement of prophylactic pancreatic stents
3	Biliary stone extraction > 10 mm Minor papilla cannulation in divisum, and therapy Removal of internally migrated biliary stents Intraductal imaging, biopsy or fine needle aspiration Management of acute or recurrent pancreatitis Treatment of pancreatic strictures Removal of pancreatic stones that are mobile and < 5 mm Treatment of hilar tumours Treatment of benign biliary strictures, hilum and above Management of suspect sphincter of Oddi dysfunction
4	Removal of internally migrated pancreatic stents Intraductal image-guided therapy (e. g. lithrotripsy) Removal of pancreatic stones that are impacted and/or > 5 mm Removal of intrahepatic stones Pseudocyst drainage or necrosectomy Ampullectomy ERCP after Whipple’s or Roux-en-Y bariatric surgery

ERCP, endoscopic retrograde cholangiopancreatography.

¹ Add one grade (for a maximum grade of 4) for procedures performed after normal working hours, in post-Bilroth II gastrectomy patients, or for procedures that had been previously unsuccessful.

1.7: Successful completion of an ERCP is defined as the completion of therapeutic intent in ERCPs of Schutz grade 1 and 2 complexity. This should be achieved without any trainer assistance in ≥ 80 % of cases by the end of training, and before a mentored period of practice (strong recommendation, low quality evidence).

Consensus: 100 %

Procedure success, defined as achieving therapeutic intent in the first procedure, varies with procedural difficulty. In highly skilled hands, previous performance audits have reported ERCP success rates exceeding 97 % [13]. The ERCP Quality Network study by Cotton et al presented self-reported small stone extraction rates of 100 %, falling to 91–96 % for stones > 10 mm [31].

Successful completion of therapeutic intent during training is not well represented in the literature. However, difficulty achieving a consistent cannulation rate in the context of time-limited biliary cannulation attempts for trainees is demonstrated [10] [14] [16]. The consensus recommendation for the minimum standard of a competent ERCP endoscopist for completion of therapeutic intent without assistance has been set at >80 %, owing to the real-world nature of ERCP training where time for cannulation and number of attempts may be limited by the trainer. This also represents the importance of a high chance of success by the trainee once they enter into independent practice and reflects the expectation for continued skills development after certification. The likelihood of further improvement in ERCP competence has been evidenced by Wani et al in a multicenter cohort study of advanced endoscopy training fellows across the United States [32].

1.8: When performed by trainees, ERCPs of Schutz grade 3 and 4 complexity can be used to count toward lifetime procedure numbers and completion metrics, e. g. deep cannulation rates, but should be excluded from other key performance indicators (e. g. therapeutic success) (strong recommendation, very low quality evidence).

Consensus: 100 %

Hands-on exposure to more complex procedures (Schutz grade 3–4) may be beneficial for trainees for maximizing training exposure. Even in established ERCP endoscopists, lower rates of successful completion have been reported (grade 3: 86.7 %, grade 4: 46.7 %) 13. For this reason, overall lifetime procedure numbers and individual completion metrics such as deep cannulation could be considered in e-portfolio outputs, but standards of therapeutic success will be reserved for less complex procedures (grades 1 and 2), where therapeutic success rates in capable ERCP endoscopists can reach in excess of 97 % [20] [31].

1.9: Trainees should be able to demonstrate an overall 30-day post-ERCP pancreatitis rate of 5 % or less of their Schutz 1 and 2 ERCP cases (weak recommendation, low quality evidence).

Consensus: 100 %

Pancreatitis is a common and potentially avoidable complication of ERCP. In meta-analyses, the incidence of post-ERCP pancreatitis varied from 3.5 % (21 studies) to 9.7 % (108 RCTs) [33] [34]. Although the risk may be mitigated by medical and technical interventions, this metric is also endoscopist dependent and inversely correlates with procedural volume and ERCPist experience [35] [36].

In the meta-analysis by Andriulli et al, in addition to the 3.5 % rate for pancreatitis, other major adverse event rates comprised infection (1.4 %), bleeding (1.3 %) and perforation (0.6%) [34]. Although the acceptable rates of these complications for trainees are not stipulated in this document, trainees should proactively audit their 30-day major complication rates and discuss these with their trainer as part of the governance process.

2. Acquisition of Competence in ERCP

2.1: Trainees should be competent in diagnostic oesophagogastroduodenoscopy and have experience of upper gastrointestinal endoscopic therapy before commencing ERCP training (strong recommendation, very low quality evidence).

Consensus: 100 %

It is accepted that trainees commencing ERCP training should be competent at upper gastrointestinal endoscopy to ensure they understand the principles of scope handling and upper gastrointestinal anatomy, even though this will require further attention in early training due to the differences of handling a side-viewing duodenoscope. Some experience of therapeutic upper gastrointestinal endoscopy is also desirable. Competence in colonoscopy, or concurrent training in colonoscopy is not considered mandatory prior to ERCP training. Intuitively, a competent colonoscopist may have scope handling skills that translate to a shortened learning curve for the handling of the duodenoscope, but there is no evidence that this is also true of all the other aspects of an ERCP. Given the minimum number of ERCP cases expected during training (Statement 2.8), and that trainees may also be training in EUS, there is an argument for trainees to pursue ERCP and EUS training somewhat earlier, rather than spending considerable training time developing colonoscopy skills [14].

2.2: Trainees should demonstrate the desire and commitment to practice ERCP at consultant-level (strong recommendation, very low quality evidence).

Consensus: 100 %

Places on ERCP training programs are limited. Efforts should be made by program directors to ensure trainees committed to a career that will include ERCP practice are appointed to these posts.

2.3: It is desirable but not mandatory for trainees to train in both ERCP and EUS (strong recommendation, very low quality evidence).

Consensus: 84 %

An endoscopist can be an expert in delivering a safe and effective ERCP service without undertaking EUS. Patients with the possibility of bile duct stones can be safely and efficiently managed with a confirmatory EUS before proceeding with a same-session ERCP and duct clearance. Moreover, EUS guided tissue acquisition during the same session as an ERCP and biliary stent in obstructive jaundice from a distal malignant biliary obstruction expedites definitive management for the patient. Also, therapeutic EUS guided biliary drainage is continuing to evolve as a discipline that can assist a failed ERCP cannulation (e. g. EUS-assisted rendezvous in failed ERCP cannulation) and provide an effective alternative to ERCP or a percutaneous transhepatic drain in certain circumstances altogether (e. g. choledochoduodenostomy). Trainees who are competent in ERCP will be well placed to learn other interventional EUS procedures that are becoming established such as transmural gallbladder drainage, as these interventions can involve the use of wires, stents and dilatation balloons. Hence new trainees entering ERCP training should be strongly encouraged to train in EUS.

2.4: For ERCP certification, UK trainees are required to attend a JAG-accredited basic ERCP course (with simulation and lectures) in the early stages of their ERCP training. JAG-accredited intermediate and up-skilling courses are encouraged but not mandatory (strong recommendation, low quality evidence).

Consensus: 100 %

Attendance of the JAG Basic Skills course is mandatory for certification in upper and lower gastrointestinal endoscopy. These courses are procedure-specific and are intended to equip trainees with the core theory and hands-on skills in a standardized manner [37]. For trainees at early stages of training, JAG-accredited basic skills in ERCP course covers simulation-based teaching in an environment without risk to patients. Mechanical simulation models have been shown to demonstrate acceptable face validity, realism, and appear superior to ex-vivo tissue models [38] [39] [40]. There is moderate quality evidence from two randomized controlled trials linking mechanical simulator training with improved trainee performance, reduced procedure time and improved cannulation rates [41] [42] [43], especially when paired with coaching and feedback [43]. In independent practitioners, attendance of a 2-day hands-on ERCP workshop involving mechanical models led to improved post-course confidence in sphincterotomy, stone extraction, mechanical lithotripsy and metal stent placement [44]. As such, simulation-based training may also have a role in the training and up-skilling of therapeutic interventions.

2.5: Trainees are recommended to use digital resources and attend live endoscopy courses and conferences to become familiar with ERCP techniques and accessories (strong recommendation, very low quality evidence).

Consensus:100 %

Endoscopic demonstrations, either live via a video to a conference proceeding, or as a pre-recorded demonstration, can teach endoscopy technique in a safe and effective manner. A real-time demonstration adds value as it allows the delegate to observe the endoscopist’s team management skills, room set up, decision-making process, and on potentially on the management of complications. There is consensus that such events are of educational value, as long as patient safety remains the primary focus of the procedure [45] [46], but do not replace the need for hands-on ERCP training.

2.6: Trainees are required to show evidence of attendance at hepatobiliary multidiscliplinary team meetings and contribute to the care of inpatients and outpatients with pancreaticobiliary disease (strong recommendation, very low quality evidence).

Consensus: 85 %

An important aspect of safe and effective ERCP is the decision-making on the role of ERCP, the appraisal of procedural risks and benefits, and the therapeutic strategies are likely to be employed. This often requires correlation with radiology, multidisciplinary consensus and detailed discussions with patients and their advocates. Complications of ERCP may also require input from other disciplines. It is therefore advisable for trainees in ERCP to maximize exposure to pancreaticobiliary medicine and attend hepatobiliary (HPB) multidisciplinary team (MDT) meetings.

2.7: Trainers delivering training in ERCP should have undertaken an endoscopy-specific train-the-trainers course (weak recommendation, very low quality evidence).

Consensus: 100 %

Defining the standards of an effective day-to-day training environment for ERCP is beyond the remit of this consensus document. However, ERCP trainers should have completed an endoscopy-specific train-the-trainers course [47]. Train-the-trainers cover the principles of adult learning with the emphasis being on developing the trainer’s skills in teaching endoscopic skills. Specific Train-the-ERCP-Trainer courses have been developed in the UK and trainers are encouraged to attend one of these.

2.8: All trainees should have a minimum of 300 hands-on ERCP cases prior to certification (strong recommendation, moderate quality evidence)

Consensus: 100 %

In 2004, the NCEPOD report was highly critical on training for ERCP, mainly due to insufficient numbers of ERCP procedures performed by trainees before moving in to independent practice [6]. Time spent training and procedure numbers have been used as surrogate markers of competence in ERCP. Given that the majority of ERCPs are performed with therapeutic intent, the endpoint of selective deep biliary cannulation in patients with native papilla arguably provides the best ‘global’ measure of procedural competence.

Recently, Siau et al reported learning curves on 818 ERCP DOPS assessments in all UK ERCP trainees for more than a 2-year period up to October 2018 [28]. Competency in selective cannulation was achieved after 300 procedures (mean 89 %, 95 % CI 80 %-95 %) [28]. Ekkelenkamp et al showed successful common bile duct (CBD) cannulation of a virgin papilla of only 68 % after 180 ERCPs [21]. Only one out of 15 trainees reached 85 % successful cannulation rates at 200 procedures. Shahidi et al performed a systematic review of the learning curve for ERCP and included nine studies (137 trainees and 17,100 ERCPs) [48]. Depending on the definition of competency, the outcome was achieved after 70 to 400 ERCPs. In the two studies that evaluated pancreatic duct (PD) cannulation rate, competency was achieved after 70 to 160 ERCPs; of the five studies which measured selective duct cannulation, competency was achieved by 79 to 300 ERCPs and where the endpoint was CBD cannulation, the learning curve was 16 to 400 procedures. When stratified according to deep cannulation of a native papilla, only one single-operator study achieved competency after 350 to 400 procedures [19].

Wani et al reported outcomes of 24 trainees who completed a 1-year advanced endoscopy training program [32]. Not all attained the requisite competency measures. Specifically, only 17 of 24 trainees achieved native papilla cannulation rates of > 90 % after a median number of 361 cases. The same trainees were followed through their first year of independent practice, and by the end, were achieving high rates of CBD cannulation (94.9 %) [32].

No study provides certainty with regard to the minimum number of cases required for the majority of trainees to meet acceptable KPIs. This group’s consensus was that most trainees are likely to require ≥ 300 ERCPs to reach the CBD cannulation rate for native papillae of 85 %. It is therefore the recommendation that trainees should have accumulated a minimum of 300 hands-on procedures (with acceptable KPIs) before being eligible for summative assessment in the ERCP certification process.

2.9: Trainees looking to practice independently in advanced ERCP (Schutz 3 and 4, and cholangioscopy) will benefit from a further period of focused training and/or mentorship (strong recommendation, low quality evidence)

Consensus: 100 %

Many trainees will be seeking to advance their ERCP skills to be able to undertake more complex procedures independently, particularly if they are looking toward a position in a tertiary referral HPB unit. ERCP cases with Schutz 3 and 4 complexity include pancreatic endoscopic therapy; ERCP in surgically altered anatomy and management of proximal biliary obstruction. Cholangioscopy and associated endobiliary therapy is not itemized in Cotton’s definition of ERCP from 2011 but is considered an advanced ERCP intervention [30]. Advanced ERCP should be undertaken in units that have the on-site support of interventional endoscopy and HPB surgery, and it is encouraged that these cases are discussed within the confines of a dedicated HPB multidisciplinary meeting. It is advised that ERCP trainers will supervise trainees undertaking more advanced cases as they proceed through training once they are confident that the trainee has mastered ERCP duodenoscope handling, and the safe and effective use of wires and accessories. While evidence is lacking, it is likely that progression to independence in advanced ERCP will require a further period of focused advanced ERCP training either alongside Schutz 1 and 2 training or after. Competence in advanced ERCP may not be achieved by the time the trainee has undertaken the 300 ERCP procedures required to practice to Schutz level 1 and 2 cases independently. The learning curve for each advanced ERCP procedure is likely to vary, but for cholangioscopy the initial learning curve has been estimated to be approximately nine procedures, with a steady improvement in competence, after that [49]. It is not envisaged that there will be certification in advanced ERCP interventions, but trainees are encouraged to continue to record a contemporaneous record of all ERCP procedures they undertake, including the case complexity and the any related complications. It follows that an individual can undertake advanced ERCPs independently once they have ERCP accreditation and they have undergone further training in a high volume ERCP center, either in a formal training post or as a mentee following certification. They would also be expected to demonstrate an acceptable case volume, success rate and complication rate for each indication for advanced intervention.

3. Assessment of Competence

3.1: Formative DOPS assessments during ERCP training should be used to track progression in technical and non-technical skills, and to support trainee feedback (strong recommendation, very low quality evidence).

Consensus: 100 %

Formative assessments are performed with the objective of complementing training by highlighting procedure-specific strengths and weaknesses [26] [28] [50]. The use of objective ERCP formative DOPS assessments are available in electronic format within the JETS (JAG Endoscopy Training System) e-portfolio [51]. These assess 27 procedural competencies which are grouped within six domains, thereby enabling the assessment of specific technical and non-technical skills. The TEESAT assessment tool has been validated in North American fellowship programs [17] [18] [52], but is not currently supported on the JETS e-portfolio.

3.2: Formative DOPS assessments should be performed regularly (i. e. at least 1 DOPS per every 10n hands-on training procedures during training) to provide objective evidence of skills acquisition and targeted feedback (strong recommendation, low quality evidence).

Consensus: 100 %

Increasing the frequency of assessment enhances the validity and reliability of competency estimation [53]. Serial formative assessments can provide an indication of a trainee’s progress, direct performance enhancing feedback, and indicate readiness for summative assessment and unsupervised practice. Greater engagement in formative DOPS has been identified as an independent predictor of competence in DOPS assessments [28]. Formative DOPS assessments are currently supported in electronic format on the JETS e-portfolio and may be independently verified by central JAG assessors as part of the sign-off process.

3.3: Self-assessment is an acceptable method of monitoring competency development which should be used in conjunction with objective assessment tools. Trainees should log all training procedures onto the JETS e-portfolio (strong recommendation, low quality evidence).

Consensus: 100 %

The JETS e-portfolio provides a framework for the electronic documentation of ERCP procedural experience which is recognized by all UK endoscopy trainees and trainers. Validity is supported from other training modalities [51]. JETS enables the recording of specific trainee extents for diagnostic and therapeutic ERCP elements, which enables the formulation of unassisted KPIs which are embedded into ERCP certification criteria. Validity evidence also exists in support of the use of the Rotterdam self-assessment ERCP form (RAF-E) [21] [54].

3.4: Trainees must demonstrate the following key performance indicators to be eligible for summative assessment for certification:

• Native papilla cannulation rate of 80 %

• Complete bile duct clearance (in cases where largest calculus ≤ 1 cm) in 70 %

• Successful stenting of distal biliary strictures in 75 %

• Unassisted in 80 % of cases in last 3 months (min 15 cases) (strong recommendation, very low quality evidence).

Consensus: 92 %

The BSG standards document provides KPI targets for competent independent practice [9]. These include a ≥ 85 % cannulation rate for native papillae in Schutz 1–2 procedures, CBD stone clearance for ≥ 75 % of those undergoing first ever ERCP, and stenting (with cytology/histology where appropriate) in ≥80 % [9]. There was consensus by the panel to allow a reduction of 5 % in unassisted success rates for trainees compared to the BSG standards document, which reflects the limitations achievable by a trainee due to the occasions that a trainer takes over aspects of a case to advance the procedure, e. g. due to time pressures on ad hoc training lists or sedation-related factors. These metrics should apply to Schutz 1–2 procedures.

3.5: Formative ERCP DOPS assessments should be used in conjunction with other supporting certification criteria to assess eligibility for summative assessment. To undertake summative assessment, trainees should be rated as “ready for independent practice” in ≥ 85 % of the individual items of five recent formative DOPS (minimum of 3 DOPS assessments on cases with a native papilla within the past 6 months), and with no items rated as requiring “maximum supervision” (strong recommendation, low quality evidence).

Consensus: 100 %

Formative DOPS assessments are typically used to objectively evaluate competency development during training. In a study of 818 ERCP DOPS assessments, Siau et al demonstrated using contrasting groups analysis that the attainment of competence in 87 % of assessed items per DOPS provided the optimal competency benchmark (pass-fail threshold) in this cohort of trainees [28]. In more advanced trainees, formative DOPS can be used in a pseudo-summative context, i. e. to gauge readiness to undertake summative assessment for independent practice. Objective assessment within formative DOPS may be used to complement KPI data to inform whether technical and non-technical competencies have been acquired. We recommend that each trainee should have demonstrable evidence of adequate performance in their last five recent DOPS (of which at least three should be on native papilla) within the preceding 6 months of training.

3.6: For successful completion of the summative DOPS assessment, the trainee should be rated as “ready for independent practice” in all items within two DOPS assessments, by two different assessors, one of whom is not based at their current endoscopy unit (weak recommendation, very low quality evidence).

Consensus: 93 %

The concept of summative assessment is embedded in the JAG certification process; this is required to ensure objectivity of competence assessment prior to certification [14]. As with other JAG certification procedures, trainees undertake a summative assessment process in order to provide robust and objective evidence of competence prior to certification for independent practice. To mitigate bias, we recommend that trainees should perform a total of two summative ERCP DOPS and be rated competent in all items by two separate assessors, of which one of these assessors should not be a current trainer based at the trainee’s unit. The summative assessment cases should take place at an endoscopy unit chosen by the trainee (usually their current or recent training unit) such that endoscopy equipment and environment are familiar to the trainee. At least one of the assessors should have been formally trained in assessing JAG ERCP summative DOPS or have attended an ERCP train the trainer course which includes training in the use of assessment tools.

4. Post-Certification Support

4.1: Newly certified ERCP practitioners should have a defined period of mentorship lasting a minimum period of 2 years, with provisions available for regular progress reviews, e. g. at 3-month intervals (strong recommendation, very low quality evidence).

Consensus: 100 %

ERCP certification signifies that an endoscopist has reached the minimum standards required for independent practice. It is acknowledged that performance will continue to improve during the early period of independent practice before aspirational standards may be reached [19] [32]. It therefore follows that there should be provisions for mentorship and regular performance review should be made available for recently certified ERCPists in accordance with the 2014 BSG standards document [9]. In “Coaching and Mentoring at Work,” Connor and Pokora define Coaching and Mentoring as “learning relationships which help people to take charge of their own development, to release their potential and to achieve results which they value” [55]. Although a universal understanding of mentorship has been historically elusive, it is now increasingly recognized in healthcare [56] [57] [58] [59]. “ERCP mentorship” may be defined as the process by which an experienced colleague who performs high quality ERCP engages with a new colleague to foster their development and expertise in ERCP. A period of 2 years is suggested to enable sufficient time to support and nurture a practitioner into one who can provide a high quality ERCP service. Focusing merely on technical skills can miss the opportunity to develop wider expertise, e. g. developing insight into one’s abilities, multidisciplinary team working, and supporting service development. Although the mentor should be an experienced ERCPist, additional training may be required to develop specific mentorship expertise. For mentorship to flourish, both mentor and mentee should have time to invest in the relationship, ideally with time put aside for regular scheduled meetings.

The early induction meeting between mentor and the newly certified ERCP clinician is important to define and agree the mentoring process, establish expectations, set timelines, and agree the duration for which a mentor is going to be present for the mentee’s ERCP lists. There may well be benefit for the mentee to attend or partake in the mentor’s ERCP lists for a period of time, and an important aspect of the mentoring process is to encourage an openness to discuss any potential adverse events that the mentee will inevitably encounter. Mentoring schemes can be organized within regional support networks if they exist.

4.2: The ongoing training requirements of newly accredited ERCP practitioners should be identified and should be encouraged to attend further training opportunities, e. g. up-skilling courses (strong recommendation, very low quality evidence).

Consensus: 100 %

Newly accredited ERCP practitioners will be recommended to maintain a procedural log of their procedures. The benefits are multiple: accountability, audit, contributing to morbidity and mortality meetings, and providing insights into practice. A review of data and discussion with a mentor could facilitate discussions on future training requirements. These should be identified, discussed with the ERCP mentor, who can support these training requirements, e. g. up-skilling course or regional/national ERCP meetings if deemed suitable, and to discuss exposure to more complex indications and therapeutic cases.

4.3: Clinicians who have recently certified in ERCP should have systems in place to ensure appropriate case load selection: regular vetting of cases or through weekly HPB MDT/triage meetings (strong recommendation, very low quality evidence).

Consensus: 100 %

It is recognized that following the commencement of independent practice, performance continues to improve over the first year [9]. It can be concluded therefore that the ERCP cases should be appropriate for the newly accredited ERCP practitioner, which requires an effective system of triage of ERCP referrals.

4.4: There should be appropriate mechanisms in place for performance monitoring and review during the agreed transition period, e. g. at 3-month intervals (strong recommendation, very low quality evidence).

Consensus: 100 %

Performance review is separate to mentorship and is equally important. All ERCP practitioners should be subject to the monitoring of their ERCP KPIs, which will be facilitated in the UK with the roll-out of the National Endoscopy Database (NED). Based on colonoscopy data, it may be inferred that practitioners will have a drop in performance during the newly independent period [60]. The mentor will be expected to review the performance data of the ERCP practitioner during the mentorship period, which can facilitate the identification of underperformance and inform the need for supportive arrangements [61].

Discussion

ERCP is a technically demanding procedure that carries the greatest risk of complications of all the commonly undertaken endoscopic procedures and there is a pressing need to define the standards trainees are expected to attain prior to independent practice. This paper outlines the consensus views on UK ERCP training from a group of experienced ERCP practitioners and trainees and defines competence in ERCP, how this should best be achieved, and how competence should be evidenced. Recommendations have also been made about how newly certified ERCP practitioners might be best supported via a mentoring process in the crucial and challenging early stages of independent practice.

This document recommends that a trainee must have evidence of a career ERCP experience of 300 procedures before certification to practice independently can be attained, and this recommendation is arguably supported by the strongest evidence [21] [28]. This recommendation has the potential to generate debate and is likely to affect the structure and organization of training in the UK. The concept of defining a certain number of procedures as a benchmark of competence has been criticized as outdated by international experts [62] [63]. The case against advocating minimum threshold numbers is understandable, and is based on the acknowledgment that some trainees will reach competence earlier than others. This variation can be explained by many factors, such as: the prior experience of the trainee; the time the trainee spends on training cases and training away from the endoscopy room (such as in hands-on courses and using ERCP simulation); and the quality of the trainer and training environment. It may be that trainees can move up the learning curve with fewer cases if the other recommendations in this document are delivered, such as engagement with simulation training and formal courses, and ensuring that trainers are well instructed in how to train ERCP. With the formal evaluation of trainee KPIs and objective assessment of ERCP competence recommended in this document, this may support the case for the abolition of minimum procedural numbers as a criterion for certification in future. In the meantime, the panel recognizes that ERCP is a diverse procedure with the indication, patient anatomy and therapeutic strategy varying between cases, with the practitioner needing to lead the endoscopy team effectively. As such, the case for procedure experience stands as it correlates with successful outcomes. Furthermore, training program directors and trainees require an indication of the procedural numbers required during training so that certification is likely to be successful, such that appropriate training programs can be structured to deliver these targets.

It is essential that JETS e-portfolio data are once again used to determine the rates of attainment of performance measures and learning curves so that these training recommendations can be revisited. It is expected that performance measures for ERCP within JETS will be derived from the NED and so represent accurate outcomes. These data will be particularly important when the current recommendations for cannulation rates, minimum procedure numbers and rates of successful completion of therapeutic intervention in Schutz level 1 and 2 cases are revisited. The process and outcomes of the summative assessment of ERCP will also need to be validated.

It should be emphasized that this document outlines the training pathway ([Fig. 1]) for trainees to be accredited in Schutz 1 and 2 levels of complexity (stone extraction ≤ 1cm; stenting of distal biliary obstruction and management of biliary leaks). Trainees wishing to undertake more complex interventions (Schutz 3 and 4 complexity including pancreatic endoscopic therapy and cholangioscopy) should only do so once they are able to demonstrate that they have had specific training in these complex interventions in a high volume ERCP center and have acceptable rates of both procedural success and complications. Advanced ERCP should be undertaken within units with access to HPB surgery and interventional radiology, and involvement in a dedicated HPB MDT is encouraged.

Some limitations of the Delphi process should be acknowledged. The recommendations have been made with reference to the best available published evidence which have been extrapolated from different training contexts, the majority of which rely on observational data which is vulnerable to bias. As a result, the statements are based on low or very low quality evidence, which reflects the paucity of evidence in the literature relating to training and credentialing in ERCP. We also acknowledge that the statements were formulated by a UK panel to support UK ERCP training using existing JAG infrastructure, such as basic skills course, JETS e-portfolio, and DOPS assessments. Therefore, not all elements will be applicable to international trainees.

Conclusions

Gastroenterology training programs in the UK should ensure adequate training provision, including the availability of fellowships, or continued access to training after completing gastroenterology specialist training. Meeting the training, assessment and mentoring standards outlined in this document will inevitably be a challenge for ERCP trainees, endoscopy trainers and training program directors. However, there is sufficient published evidence to suggest that with effective training programs and support, the recommendations stated within this document can be met and serve to ensure that patients requiring an ERCP receive the highest quality of care regardless of the training route of the endoscopist.

• References

• 1 Cohen S, Bacon BR, Berlin JA. et al. National Institutes of Health State-of-the-Science Conference Statement: ERCP for diagnosis and therapy, January 14-16, 2002. Gastrointest Endosc 2002; 56: 803-809
• 2 Vandervoort J, Soetikno RM, Tham TC. et al. Risk factors for complications after performance of ERCP. Gastrointest Endosc 2002; 56: 652-656
• 3 Freeman ML. Adverse outcomes of endoscopic retrograde cholangiopancreatography. Rev Gastroenterol Disord 2002; 2: 147-168
• 4 Freeman ML, Nelson DB, Sherman S. et al. Complications of endoscopic biliary sphincterotomy. New Engl J Med 1996; 335: 909-918
• 5 Loperfido S, Angelini G, Benedetti G. et al. Major early complications from diagnostic and therapeutic ERCP: a prospective multicenter study. Gastrointest Endosc 1998; 48: 1-10
• 6 Cullinane M, Gray AJG, Hargraves CMK. et al. Scoping our practice: the 2004 report of the confidential enquiry into patient outcome and death. 2004 http://www.ncepod.org.uk/2004report/
• 7 Allison MC, Ramanaden DN, Fouweather MG. et al. Provision of ERCP services and training in the United Kingdom. Endoscopy 2000; 32: 693-699
• 8 Kowalski T, Kanchana T, Pungpapong S. Perceptions of gastroenterology fellows regarding ERCP competency and training. Gastrointest Endosc 2003; 58: 345-349
• 9 British Society of Gastroenterology ERCP Working Party. ERCP – The Way Forward, A Standards Framework. 2014 https://www.bsg.org.uk/asset/341DCD67-426A-44F4-910DD392C8A39606
• 10 Faulx AL, Lightdale JR, Acosta RD. et al. Guidelines for privileging, credentialing, and proctoring to perform gastrointestinal endoscopy. Gastrointest Endosc 2017; 85: 273-281
• 11 Springer J, Enns R, Romagnuolo J. et al. Canadian credentialing guidelines for endoscopic retrograde cholangiopancreatography. Can J Gastroenterol 2008; 22: 547-551
• 12 Cotton PB, Feussner D, Dufault D. et al. A survey of credentialing for ERCP in the United States. Gastrointest Endosc 2017; 86: 866-869
• 13 Siau K, Green JT, Hawkes ND. et al. Impact of the Joint Advisory Group on Gastrointest Endosc (JAG) on endoscopy services in the UK and beyond. Frontline Gastroenterol 2019; 10: 93-106
• 14 Siau K, Anderson JT, Valori R. et al. Certification of UK gastrointestinal endoscopists and variations between trainee specialties: results from the JETS e-portfolio. Endosc Int Open 2019; 7: E551-E560
• 15 Bekkali NL, Johnson GJ. Training in ERCP and EUS in the UK anno 2017. Frontline Gastroenterol 2017; 8: 124-128
• 16 Balshem H, Helfand M, Schünemann HJ. et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011; 64: 401-406
• 17 Wani S, Hall M, Wang AY. et al. Variation in learning curves and competence for ERCP among advanced endoscopy trainees by using cumulative sum analysis. Gastrointest Endosc 2016; 83: 711-719.e711
• 18 Wani S, Keswani R, Hall M. et al. A Prospective multicenter study evaluating learning curves and competence in endoscopic ultrasound and endoscopic retrograde cholangiopancreatography among advanced endoscopy trainees: The Rapid Assessment of Trainee Endoscopy Skills Study. Clin Gastroenterol Hepatol 2017; 15: 1758-1767.e1711
• 19 Verma D, Gostout CJ, Petersen BT. et al. Establishing a true assessment of endoscopic competence in ERCP during training and beyond: a single-operator learning curve for deep biliary cannulation in patients with native papillary anatomy. Gastrointest Endosc 2007; 65: 394-400
• 20 Torun S, Parlak E, Yildiz H. et al. Assessment of the endoscopic retrograde cholangiopancreatography grading system: A prospective study from a tertiary care center. Turk J Gastroenterol 2016; 27: 187-191
• 21 Ekkelenkamp VE, Koch AD, Rauws EA. et al. Competence development in ERCP: the learning curve of novice trainees. Endoscopy 2014; 46: 949-955
• 22 Schlup MM, Williams SM, Barbezat GO. ERCP: a review of technical competency and workload in a small unit. Gastrointest Endosc 1997; 46: 48-52
• 23 Voiosu T, Bălănescu P, Voiosu A. et al. Measuring trainee competence in performing endoscopic retrograde cholangiopancreatography: A systematic review of the literature. United European Gastroenterol J 2019; 7: 239-249
• 24 Voiosu T, Puscasu C, Orlandini B. et al. Motion training on a validated mechanical ERCP simulator improves novice endoscopist performance of selective cannulation: a multicenter trial. Endosc Int Open 2021; 9: E145-E151
• 25 Wani S, Keswani RN, Han S. et al. Competence in endoscopic ultrasound and endoscopic retrograde cholangiopancreatography, from training through independent practice. Gastroenterology 2018; 155: 1483-1494
• 26 Siau K, Hawkes ND, Dunckley P. Training in endoscopy. Curr Treat Options Gastroenterol 2018; 16: 345-361
• 27 Walsh CM. In-training gastrointestinal endoscopy competency assessment tools: Types of tools, validation and impact. Best Pract Res Clin Gastroenterol 2016; 30: 357-374
• 28 Siau K, Dunckley P, Feeney M. et al. ERCP assessment tool: evidence of validity and competency development during training. Endoscopy 2019; 51: 1017-1026
• 29 Hitchins CR, Metzner M, Edworthy J. et al. Non-technical skills and gastrointestinal endoscopy: a review of the literature. Frontline Gastroenterol 2018; 9: 129-134
• 30 Cotton PB, Eisen G, Romagnuolo J. et al. Grading the complexity of endoscopic procedures: results of an ASGE working party. Gastrointest Endosc 2011; 73: 868-874
• 31 Cotton PB, Romagnuolo J, Faigel DO. et al. The ERCP Quality Network: A Pilot Study of Benchmarking Practice and Performance. Am J Med Quality 2012; 28: 256-260
• 32 Wani S, Keswani RN, Han S. et al. Competence in endoscopic ultrasound and endoscopic retrograde cholangiopancreatography, from training through independent practice. Gastroenterology 2018; 155: 1483-1494.e1487
• 33 Kochar B, Akshintala VS, Afghani E. et al. Incidence, severity, and mortality of post-ERCP pancreatitis: a systematic review by using randomized, controlled trials. Gastrointest Endosc 2015; 81: 143-149.e149
• 34 Andriulli A, Loperfido S, Napolitano G. et al. Incidence rates of post-ERCP complications: a systematic survey of prospective studies. Am J Gastroenterol 2007; 102: 1781-1788
• 35 Lee HJ, Cho CM, Heo J. et al. Impact of hospital volume and the experience of endoscopist on adverse events related to endoscopic retrograde cholangiopancreatography: a prospective observational study. Gut Liver 2020; 14: 257-264
• 36 Li JW, Ang TL, Kam JW. et al. The learning curve for needle knife precut sphincterotomy revisited. United European Gastroenterol J 2017; 5: 1116-1122
• 37 Siau K, Hodson J, Anderson JT. et al. Impact of a national basic skills in colonoscopy course on trainee performance: An interrupted time series analysis. World Journal of Gastroenterology 2020; 26: 3283-3292
• 38 Leung JW, Wang D, Hu B. et al. A head-to-head hands-on comparison of ERCP mechanical simulator (EMS) and Ex-vivo porcine stomach model (PSM). J Interv Gastroenterol 2011; 1: 108-113
• 39 van der Wiel SE, Koch AD, Bruno MJ. Face and construct validity of a novel mechanical ERCP simulator. Endosc Int Open 2018; 6: E758-E765
• 40 van der Wiel SE, Koch AD, Bruno MJ. Face validity of a synthetic papilla designed for biliary sphincterotomy training. Endosc Int Open 2019; 7: E757-E761
• 41 Lim BS, Leung JW, Lee J. et al. Effect of ERCP mechanical simulator (EMS) practice on trainees' ERCP performance in the early learning period: US multicenter randomized controlled trial. Am J Gastroenterol 2011; 106: 300-306
• 42 Meng W, Leung JW, Yue P. et al. Sa1238 Practice With ERCP mechanical simulator (EMS) improves basic ERCP skills of novice surgical trainees. Gastrointest Endosc 2016; 83: AB267-AB268
• 43 Liao WC, Leung JW, Wang HP. et al. Coached practice using ERCP mechanical simulator improves traineesʼ ERCP performance: a randomized controlled trial. Endoscopy 2013; 45: 799-805
• 44 Sedlack RE, Petersen BT, Kolars JC. The impact of a hands-on ERCP workshop on clinical practice. Gastrointest Endosc 2005; 61: 67-71
• 45 Loren DE, Azar R, Charles RJ. et al. Updated guidelines for live endoscopy demonstrations. Gastrointest Endosc 2010; 71: 1105-1107
• 46 Dinis-Ribeiro M, Hassan C, Meining A. et al. Live endoscopy events (LEEs): European Society of Gastrointest Endosc position statement - Update 2014. Endoscopy 2015; 47: 80-86
• 47 Waschke KA, Anderson J, Valori RM. et al. ASGE principles of endoscopic training. Gastrointest Endosc 2019; 90: 27-34
• 48 Shahidi N, Ou G, Telford J. et al. When trainees reach competency in performing ERCP: a systematic review. Gastrointest Endosc 2015; 81: 1337-1342
• 49 Weigt J, Kandulski A, Malfertheiner P. Technical improvement using ultra-slim gastroscopes for direct peroral cholangioscopy: analysis of the initial learning phase. J Hepatobiliary Pancreat Sci 2015; 22: 74-78
• 50 Siau K, Dunckley P, Valori R. et al. Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment. Endoscopy 2018; 50: 770-778
• 51 Mehta T, Dowler K, McKaig BC. et al. Development and roll out of the JETS e-portfolio: a web based electronic portfolio for endoscopists. Frontline Gastroenterol 2011; 2: 35-42
• 52 Wani S, Keswani RN, Petersen B. et al. Training in EUS and ERCP: standardizing methods to assess competence. Gastrointest Endosc 2018; 87: 1371-1382
• 53 Barton JR, Corbett S, van der Vleuten CP. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 2012; 75: 591-597
• 54 Ekkelenkamp VE, Koch AD, Haringsma J. et al. Quality evaluation through self-assessment: a novel method to gain insight into ERCP performance. Frontline Gastroenterol 2014; 5: 10-16
• 55 Connor M, Pokora J. Coaching and mentoring at work: Developing effective practice: Developing effective practice: McGraw-Hill Education (UK). 2012
• 56 Standing Committee on Postgraduate Medical Dental Education. Supporting doctors and dentists at work: an enquiry into mentoring. London: SCOPME; 1998
• 57 General Medical Council. Leadership And Management For All Doctors. 2012 [cited 12th December 2019]. https://www.gmc-uk.org/-/media/documents/Leadership_and_management_for_all_doctors___English_1015.pdf_48903400.pdf
• 58 Royal College of Surgeons. Mentoring: a guide to good practice. 2018 [cited 12th December 2019]. https://www.rcseng.ac.uk/standards-and-research/standards-and-guidance/good-practice-guides/mentoring/
• 59 Smith KH, Hallett RJ, Wilkinson-Smith V. et al. Results of the British Society of Gastroenterology supporting women in gastroenterology mentoring scheme pilot. Frontline Gastroenterol 2019; 10: 50-55
• 60 Siau K, Hodson J, Valori RM. et al. Performance indicators in colonoscopy after certification for independent practice: outcomes and predictors of competence. Gastrointest Endosc 2019; 89: 482-492.e482
• 61 Joint Advisory Group on Gastrointestinal Endoscopy. A framework for managing underperformance and supporting endoscopists – a JAG perspective. 2019 https://www.thejag.org.uk/CMS/UploadedDocuments/Scheme/Scheme5/STG190917%20-%20guidance%20-%20UK%20underperformance%20for%20JAG%20v2.0.pdf
• 62 Jones DB, Hunter JG, Townsend CM. et al. SAGES rebuttal. Gastrointest Endosc 2017; 86: 751-754
• 63 Koch AD. “To measure is to know” certainly applies to ERCP training. Endoscopy 2019; 51: 1013-1014