ERCP competence assessment: Miles to go before standardization

Introduction

Over the past decade, increasing emphasis has been placed on quality metrics and competency assessment in health care. The goals of high-quality endoscopy – appropriate patient selection (indicated procedure), accurate diagnosis, and appropriate implementation – should be achieved with minimal patient risk and performed by properly trained and competent endoscopists [1]. The ultimate goals should be improvement in patient outcome and satisfaction [2] [3]. Competency, an important element of quality endoscopy, should be defined as the consistent ability to meet technical goals of the intended procedure and to correctly perform cognitive aspects of the procedure. In a constantly changing environment and with the introduction of new techniques and technologies, competency is crucial to endoscopic practice.

Endoscopic retrograde cholangiopancreatography (ERCP) is one of the most technically demanding and high-risk procedures performed by gastrointestinal endoscopists, requiring significant focused training and experience to maximize success and patient outcome. It has a steep learning curve in both physical skills and judgment/interpretation, and increasingly more sophisticated diagnostic and therapeutic techniques are employed. Goals of an endoscopy training program include ensuring that core motor and cognitive skills necessary to successfully and competently perform ERCP are acquired. Ensuring competence in ERCP has recently emerged as an area of intense scrutiny as training programs and hospital credentialing committees attempt to produce and approve, respectively, adequately trained endoscopists. Despite this, universally accepted standards for competence in ERCP have not been established. It is commonly assumed that competence is achieved when a minimum number of ERCP procedures have been performed. For most training programs this may only be in the range of 100 to 200 cases [4]. With the appreciation that individual trainees develop endoscopic skills at different rates, [5] there has been a shift towards competency-based training and certification. An assessment of individual performance is probably more robust than the use of minimum numbers for defining competence. Accurate and validated ERCP performance measures are necessary in order to effectively train practitioners using a competency-based curriculum. The endoscopic trainee must be able to achieve a standard rate of technical success, but must also be able to recognize abnormal pathology, identify it correctly, and decide upon the appropriate course of action.

Recognizing the limitations of the current models of training and assessment of competence, the Accreditation Council for Graduate Medical Education (ACGME) replaced the reporting system with the Next Accreditation System focusing on competency-based medical education (CBME). Ideally, ERCP training programs will move toward the adoption of CBME and demonstrate that trainees have achieved competence and attained the technical and cognitive skills required for safe and effective unsupervised practice in advanced endoscopy [6] [7] [8].

Quality metrics

The demand for quality assessment in endoscopic procedures is increasing, but to date, there is still no gold standard to assess ERCP quality. The American Society for Gastrointestinal Endoscopy (ASGE) Committee on Outcomes Research has proposed several quality metrics to establish competence and help define areas of continuous quality improvement [9] [10]. Many of these quality metrics are not yet validated, and include assessment of pre-, intra-, and post-procedural periods. Pre-procedural considerations include documentation of appropriate indication(s), obtaining complete informed consent, appropriate assessment of procedural difficulty and the appropriate use of prophylactic antibiotics and rectally administered nonsteroidal agents for prevention of post-ERCP pancreatitis. Peri-procedural factors, relative to endoscopist technical performance include selective cannulation success rates of the duct of interest, and technical success rates of subsequent interventions, such as extraction of bile duct stones and biliary stent placement, as well as documentation of fluoroscopic time and radiation dose. ASGE guidelines suggest competency in ERCP is demonstrated by the ability to consistently perform (without assistance) cannulation and opacification of the desired duct, sphincterotomy, stone extraction and stent placement [11]. Post-procedural factors to be monitored consist of procedure-related adverse events (AEs), including those related to sedation (e. g. cardiopulmonary depression) and local events (e. g. pancreatitis, bleeding and perforation).

Several publications have shown that AEs are markers of quality endoscopic, with identification of predisposing factors and approaches to minimize them [12]. However, procedural success or patient-related outcomes are less often described [13]. The ASGE recommends quality indicators as appropriate ERCP indication, cannulation rate, stone extraction success rate, stent insertion success rate and frequency of post-ERCP pancreatitis [9]. For each of these indicators, reaching the recommended performance target is strongly associated with important clinical outcomes. It is seldom possible for a trainee to be as good as an experienced practitioner at the completion of formal training. The ASGE suggests trainees should demonstrate a minimum deep cannulation rate of the duct of interest of 80 % to 85 % by the end of ERCP training. Expert endoscopists have been shown to successfully cannulate the common bile duct (CBD) at a rate of > 95 %. [9] The ASGE also recommends deep cannulation of the duct of interest in patients with native papillae without surgically altered anatomy, clearance of CBD stones < 1 cm and stent placement for biliary obstruction in patients with normal anatomy and with obstruction below the bifurcation should be achieved in > 90 % of cases. [9] The British Society of Gastroenterology (BSG) recommends a cannulation rate of ≥ 90 % for native papillae, CBD stone clearance for ≥ 80 % of those patients who are ERCP-naïve, and for patients with an extrahepatic stricture, successful stent placement with tissue sampling, as appropriate in ≥ 85 % of ERCP-naïve patients. Both success rates and AEs should be routinely assessed [14] [15]. The current rate of post-ERCP pancreatitis in clinical practice ranges from 1 % to 7 %, varies by patient selection, operator skill and experience as well as the type of ERCP performed, and, for that reason, it is difficult to set a single performance target for all ERCPs for this indicator. Perforation and clinically significant hemorrhage after sphincterotomy or sphincteroplasty rates should be ≤ 0.2 % and ≤ 1 %, respectively. [9]

Assessment of competence

To ensure that future endoscopists practice high-quality endoscopy, the issue of quality needs to be introduced and practiced from the onset of training. This may require a change in the culture of training programs and their sponsor institutions. A constraint to trainee’learning is the difficulty in allowing trainees enough independent time (observation without instruction), especially in a busy practice with high case volumes and time constraints that may be set by anesthesia. Even in a relaxed learning environment that may allow for the additional time necessary for trainees to perform procedures, different trainers may have different teaching methodologies, being more or less proactive.

To assess and document competence in ERCP, a number of assessment tools have been developed in the past 3 years. They are intended to facilitate training programs to meet the new ACGME reporting requirements and, more importantly, to help program directors identify specific skill deficiencies early in training, thus allowing for the development of tailored, individualized remediation. To meet these endpoints, it must be stressed that assessment using these tools must be done in a continuous fashion to allow differentiation of learning curve progression from premature plateauing of skills. Ideally, it should also allow a balanced evaluation of both cognitive and motor skills. Self-assessment of procedural performance by the trainee should also be considered in these tools, once they are a successful device to provide insight into quality of ERCPs, and might enhance quality by stimulating active reflection of one’s actions [16]. Furthermore, the comparison of subjective scores given by trainees and trainers may provide additional insight on the value of self-reflection. The concept of comprehensive competency must be highlighted, since once assessment becomes more robust and refined, it has been suggested that procedural thresholds will quickly rise above traditional estimates [17].

In 2013 Ekkelenkamp et al. [4] proposed a self-assessment evaluation tool (Rotterdam Assessment Form) comprising the proposed quality indicators for ERCP. It consists of 3 parts, with the first part covering objective parameters such as procedural indication, degree of technical difficulty, presence of a native papilla or previous sphincterotomy and previous ERCP failure; the second part encompasses success or failure options for different parts of the procedure such as CBD cannulation, sphincterotomy, stone extraction and stent placement (scored as successfully completed, partial success, or failure) and the third part consisting of an improvement plan proposal after every 10 procedures (identification of the deficiencies, proposed solution(s) and improvement strategies). The form also invites the endoscopist to rate his/her own performance using a 10-point visual analogue scale for each technical skill. The value of the subjective assessment is the creation of self-awareness to enable reflection on performance, rather than providing evidence for quality measurements. The time required for participating endoscopists to complete the form was approximately 1 minute, with adherence to completion of the self-assessment ranging from 82.9 % to 89.6 % [4] [14]. The main reasons cited for lack of form completion were lack of time due to practice demands.

Recently, the BSG ERCP working party proposed the Direct Observation of Procedural Skills (DOPS) used by the Joint Advisory Group (JAG) for Gastrointestinal Endoscopy in the United Kingdom [18]. It consists of pre-, intra-, and post-procedural components. The first part covered the grade of difficulty, as well as the presence of a native papilla. A 4-point scale was used to grade the individual endpoints (1 = trainer undertakes the majority of the tasks/decisions and delivers constant verbal prompts; 2 = trainee undertakes tasks requiring frequent supervisor input and verbal prompts; 3 = trainee undertakes tasks requiring occasional supervisor input and verbal prompts; 4 = no supervision required). The pre-procedural component comprises indication, risk, preparation, equipment check, consent and sedation and monitoring. The intra-procedural component covers performance of basic maneuvers (intubation, visualization and position relative to ampulla, and patient comfort) as well as technical aspects (cannulation and imaging, selective cannulation, wire management, fluoroscopy quality and interpretation, execution of selected therapy, decisions on appropriate therapy, sphincterotomy, sphincteroplasty, stone therapy, tissue sampling, stent placement). It also includes trainees’ actions to minimize pancreatitis, and documents procedural-related AEs. The post-procedural component covers report writing and management plan. Non-technical skills are also evaluated (communication and teamwork, situation awareness, leadership, judgment and decision). There is also a comment section for each of the components, as well as learning objectives for future cases, to be added to the trainee’s personal development plan once the DOPS is completed.

In 2016 Wani et al. [19] proposed an assessment form, which evaluates individual technical and cognitive aspects of ERCP, distinguishing biliary from pancreatic procedures. A 4-point scoring system was used to grade the individual endpoints (1 = achieves without instruction; 2 = achieves with minimal verbal cues; 3 = achieves with multiple verbal cues or hands-on assistance; 4 = unable to complete and requires trainer to take over). The first part covers the indication for ERCP and the grade of difficulty, as well as performance of basic maneuvers (intubation, achieving the short position and identification of the papilla) and the presence or absence of a previous sphincterotomy. The second part evaluates technical aspects such as performance of deep cannulation of the desired duct, sphincterotomy, stone clearance, and stent placement (when applicable). The time allowed to attempt cannulation, whether cannulation was achieved by the trainer when the trainee was unsuccessful, as well as the need for advanced cannulation techniques (placement of a pancreatic duct stent, use of double-wire technique, or precut sphincterotomy) are also considered. The third part grades cognitive aspects of ERCP that focus on a clear understanding of the indication and appropriate use of fluoroscopy, proficient use of real-time cholangiography/pancreatography interpretation, and the ability to formulate a logical plan based on the procedural findings. The fourth part documents procedural AEs (pain requiring hospitalization, pancreatitis and severity, immediate or delayed post-sphincterotomy bleeding, perforation, mortality and cardiopulmonary AE). In addition to grading specific endpoints, an overall assessment was provided by trainers using a 10-point scoring system.

An important aspect of these type of forms, intended to be completed after every procedure, is the balance between evaluation of the most important procedural quality parameters and completion of the forms in an acceptably brief length of time. A major consideration is the inverse correlation between complexity of the forms and their completion. Previous studies, however, state that once familiar with the forms, staff typically require less than 1 minute to complete the assessment tool form [20]. Periodic spot-checking at specific steps of training may be used instead of completion of forms for all procedures. The ASGE recommends a minimum assessment with each of these tools on a periodic basis so that approximately 10 % of the total procedures performed by a trainee have completed evaluation forms. [19] Understandably, the more forms completed, the more precise the performance profile of a specific trainee, and which allows training directors to quickly identify those who are meeting or surpassing the expected milestones as well as those who are in need of remediation [19]. Minimal competency thresholds or endpoints that need to be achieved must be defined. The only competency threshold data currently available are based on the Mayo Colonoscopy Skills Assessment Tool (MCSAT), which suggest that achieving average scores of 3.5 or higher for each specific core skill correlates with achievement of minimal competence criteria [21]. The form by Ekkelenkamp et al. [4] focuses primarily on a limited number of motor skills, with minimal inclusion procedure-related cognitive skill assessment. The forms created by Wani et al. [19] and the BSG DOPS [18] assess cognitive and motor skills in a balanced manner, however, only the latter considers endoscopic non-technical skills. A strategy plan based on procedural findings as well as when technical difficulty arises is also another important issue that should be assessed by these forms. It can be argued that procedural outcome and AE may be difficult to determine at the time of the procedure, as many AEs are delayed. In our opinion, such kind of form should be completed independently by both the trainee and the trainer, with the purpose of comparing different subjective scores. It is anticipated that this will allow insight for the trainee and promote self-reflection, and increase comprehensive competency of the trainer. Deconstructing ERCP and its techniques and maneuvers in may allow address various deficiencies in ERCP training and teaching to be addressed in a more granular fashion.

Conclusion

Pathways to achieve competence and measurement of competence continue to evolve with the goal of increasing quality delivery of health care by ensuring technical success and minimizing AEs of ERCP. Efforts should be made to improve quality above the proposed minimum thresholds. Standardization of the performance, definition of competence in ERCP, and demonstrating competence at the end of training are critical to improve patient outcome.

Given that training in medicine has undergone major transformations, emphasis needs to shift from absolute numbers of procedures performed toward well-defined and validated competency thresholds. The quality of health care can be measured by comparing the performance of an individual or a group of individuals with an ideal or benchmark [22]. Tools to assess ERCP competence acquisition will facilitate training programs to document progress of individual trainees. Real-time feedback can be provided on a trainee’s performance and specific skill deficiencies can be detected earlier in training, allowing restructuring and provision of additional training. It will also help to increase comprehensive competency in the trainer, “teaching” him/her to teach the trainer. Assessment of traineesʼ individual performance by means of learning curves is much more robust and relevant for the individual trainee than simple threshold numbers. We believe these are the first steps to be taken in the miles to go before standardization.

Competing interests

None

• References

• 1 Faigel DO, Pike IM, Baron TH. et al. Quality indicators for gastrointestinal endoscopic procedures: an introduction. Gastrointest Endosc 2006; 63: S3-S9
• 2 Donabedian A. The quality of care. How can it be assessed?. JAMA 1988; 260: 1743-1748
• 3 Campbell SM, Roland MO, Buetow SA. Defining quality of care. Soc Sci Med 2000; 51: 1611-1625
• 4 Ekkelenkamp VE, Koch AD, Rauws EA. et al. Competence development in ERCP: the learning curve of novice trainees. Endoscopy 2014; 46: 949-955
• 5 Lim BS, Leung JW, Lee J. et al. Effect of ERCP mechanical simulator (EMS) practice on trainees' ERCP performance in the early learning period: US multicenter randomized controlled trial. Am J Gastroenterol 2011; 106: 300-306
• 6 Wani S, Hall M, Keswani RN. et al. Variation in aptitude of trainees in endoscopic ultrasonography, based on cumulative sum analysis. Clin Gastroenterol Hepatol 2015; 13: 1318-1325
• 7 Iobst WF, Caverzagie KJ. Milestones and competency-based medical education. Gastroenterology 2013; 145: 921-924
• 8 Nasca TJ, Philibert I, Brigham T. et al. The next GME accreditation system–rationale and benefits. N Engl J Med 2012; 366: 1051-1056
• 9 Baron TH, Petersen BT, Mergener K. et al. Quality indicators for endoscopic retrograde cholangiopancreatography. Am J Gastroenterol 2006; 101: 892-897
• 10 Adler DG, Lieb 2nd JG, Cohen J. et al. Quality indicators for ERCP. Gastrointest Endosc 2015; 81: 54-66
• 11 Adler DG, Bakis G, Coyle WJ. ASGE Training Committee. et al. Principles of training in GI endoscopy. Gastrointest Endosc 2012; 75: 231-235
• 12 Rochester JS, Jaffe DL. Minimizing complications in endoscopic retrograde cholangiopancreatography and sphincterotomy. Gastrointest Endosc Clin N Am 2007; 17: 105-127
• 13 Ekkelenkamp VE, Dowler K, Valori RM. et al. Patient comfort and quality in colonoscopy. World J Gastroenterol 2013; 19: 2355-2361
• 14 Srivastava S, Sharma BC, Puri AS. et al. Impact of completion of primary biliary procedure on outcome of endoscopic retrograde cholangiopancreatographic related perforation. Endoscop Int Open 2017; 05: 706-709
• 15 Navaneethan U, Lourdusamy D, Gutierrez NG. et al. New approach to decrease post-ERCP adverse events in patients with primary sclerosing cholangitis. Endoscop Int Open 2017; 05: 710-717
• 16 Ekkelenkamp VE, Koch AD, Haringsma J. et al. Quality evaluation through self-assessment: a novel method to gain insight into ERCP performance. Frontline Gastroenterol 2014; 5: 10-16
• 17 Shahidi N, Ou G, Telford J. et al. Establishing the learning curve for achieving competency in performing colonoscopy: a systematic review. Gastrointest Endosc 2014; 80: 410-416
• 18 Gastroenterology. BSo. ERCP - The way forward, a standards framework (June 2014). Available at: http://www.bsg.org.uk/clinical-guidance/endoscopy/ercp-%E2%80%93-the-way-forward-a-standards-framework.html Accessed October 14, 2016.
• 19 Wani S, Hall M, Wang AY. et al. Variation in learning curves and competence for ERCP among advanced endoscopy trainees by using cumulative sum analysis. Gastrointest Endosc 2016; 83: 711-9.e11
• 20 Sedlack RE. Validation process for new endoscopy teaching tools. Tech Gastrointest Endosc 2011; 13: 151-154
• 21 Sedlack RE, Coyle WJ, Obstein KL. ASGE Training Committee. et al. ASGE's assessment of competency in endoscopy evaluation tools for colonoscopy and EGD. Gastrointest Endosc 2014; 79: 1-7
• 22 Chassin MR, Galvin RW. The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. JAMA 1998; 280: 1000-1005