The intended and unintended consequences of performance monitoring in colonoscopy

Performance measurement is now routine in many areas of medicine and is relatively easy to apply to colonoscopy practice as there are several variables that can be quantified.

The oldest and universally reported performance indicator is the cecal intubation rate (CIR). It is globally agreed that the standard for cecal intubation should be at least 90 %, which is to say that the endoscopist should be able to pass the colonoscope to the cecum in at least 90 % of cases. A lower CIR is associated with poorer outcomes [1]. For decades this single parameter was the focus of audit, training and performance assessment.

As a single parameter, however, the CIR is now considered to be an inadequate reflection of global competency and additional performance measures are now routinely reported as markers of quality colonoscopy. Razzak et al. in this journal describe other key performance indicators (KPIs) that are accepted as measures of colonoscopy quality and explore whether active monitoring of these has an impact on performance.

One such KPI, the Adenoma Detection Rate (ADR) is generally accepted as the best surrogate marker for colonoscopic diagnostic accuracy having a proven association with cancer pick up and interval rates [2] . It is therefore appropriate to monitor this parameter. In Kaminski’s study, an ADR of < 20 % in a screening population was significantly associated with an increased rate of interval cancer.

Razzak et al. report their study, which monitored 2 KPIs: the ADR and negative Cecal Withdrawal Time (CWT) (the time taken to extubate the colon in cases where no polyps were removed). They found no significant impact on ADR or CWT across each study phase.

This study showed a direct relation between high ADR performers and increased CWT in keeping with other studies [3] [4] and it is now generally accepted that a mean (negative) withdrawal time > 6 minutes should be employed during screening colonoscopy in order to optimize ADR.

Although Razzak’s study population was broader than Kaminski’s, [2] the baseline ADR of > 30 % is respectable, indicating that the endoscopists studied were already performing at a reasonably high level. Razzak demonstrated a range of ADR, with some endoscopists having almost double the rate of others. This variability is a feature of other studies [5]. The design of their study was simply to inform the endoscopists that they were being monitored without providing a target range for CWT and without any feedback to the group. Of note the only information they were given regarding ADR was that there would be a performance incentive for those with an ADR > 25 %, however, because all endoscopists were performing at a higher level than that anyway, there may have been little incentive to improve.

It is important to consider whether there are unintended consequences of monitoring performance. Razzak showed some negative outcomes of monitoring: When CWT alone was monitored, fewer adenomas were removed per examination. When the ADR was the focus, more adenomas were removed. One explanation for this could be that without the context for why CWT was monitored, the endoscopists may have rushed as they may have felt they should complete the procedure more quickly. Their data suggests that during the period of CWT and ADR monitoring, more insignificant polyps were removed and there was a nonsignificant rise in post-polypectomy adverse events. Searching for and removing low-risk or no-risk lesions, particularly in elderly patients, may be an adverse unintended consequence of monitoring endoscopists [6] as “gaming” behavior may occur in order to satisfy the requisite measures. It is perhaps worth considering adjusting KPIs depending on the patient population so that elderly patients are not put at unnecessary risk.

Another notable finding was the rise in the number of cases where bowel preparation was reported as “inadequate” during the ADR monitoring phase. Previous studies have shown an association between ADR and judgment of quality of bowel preparation where higher ADR performers tend to be more critical of quality of bowel preparation and conversely lower performers are more accepting of poor bowel preparation [7]. An unintended consequence of this could be an increased number of repeated procedures with the inherent risks and costs entailed.

In the improvement science literature the term used for monitoring KPIs and reporting back to health professionals is “Audit and Feedback.” A Cochrane review showed that the “effect of audit and feedback on professional behaviour and on patient outcomes ranges from little or no effect to a substantial effect” [8]. This reflects the heterogeneity of the studies and their interventions. Audit and feedback is most likely to have a significant impact on performance and outcomes if, as a minimum, the target parameters are clearly defined.

It is perhaps unsurprising, therefore, that this study’s findings suggest that monitoring quality metrics did not have a significant effect on the performance of the small group of endoscopists studied. In order to effect change, monitoring alone without feedback is unlikely to be successful. The clinical context in which parameters are measured, feeding back the results, and ensuring the evidence base for standards are made clear are a minimum.

Audit and feedback have the potential to identify and correct flaws in technique responsible for suboptimal performance [9] but may also be able to develop a culture shift through targeted training with a wider effect through role modelling and behavioral change to a broader group of endoscopists [10]. The use of appropriate targets and action plans to rectify deficiencies has also been shown to improve outcomes. A combination of these approaches is likely to have the most long lasting and broadest impact, but exactly which interventions are most effective needs further study.

High performance in colonoscopy is difficult to define but it is clear that several measures can predict important outcomes such as low interval cancer rates and low complication rates, both primary aims of screening colonoscopy. Although technical skills can be taught and measured using validated tools [11] [12], the so-called Endoscopic Non-Technical Skills (ENTS) and specifically attitudes are much more difficult to teach and monitor. ENTS are undoubtedly very important, however, not only for high-level performance but will also impact how an endoscopist responds to being monitored. Measurement for “measurement sake” should be avoided. Training and feedback should be in the context of the clinical situation, with targeted training interventions available to encourage professional development. Future studies should consider how monitoring is presented to individuals in order to achieve the most useful outcomes.

• References

• 1 Brenner H, Chang-Claude J, Seiler CM et al. Interval cancers after negative colonoscopy: population-based case-control study. Gut 2012; 61: 156-1582
• 2 Kaminski MF, Regula J, Kraszewska E et al. Quality Indicators for colonoscopy and the risk of Interval Cancer. NEJM 2010; 362: 1795-1803
• 3 Lee TJ, Regula J, Kraszewska E et al. Longer mean colonoscopy withdrawal time is associated with increased adenoma detection: evidence from the Bowel Cancer Screening Programme in England. Endoscopy 2013; 45: 20-26
• 4 Barclay RL, Vicari JJ, Doughty AS et al. Colonoscopic withdrawal times and adenoma detection during screening colonoscopy. NEJM 2006; 355: 2533-2541
• 5 Adler A, Wegscheider K, Lieberman D et al. Factors determining the quality of screening colonoscopy: a prospective study on adenoma detection rates from 12 134 examinations. Gut 2013; 62: 236-241
• 6 Baker G. Learning from adverse outcomes: guidelines on colonoscopic polypectomy in patients aged 85 years and older Frontline. Gastro-enterology 2016; 7: 199-201
• 7 Thomas-Gibson S, Rogers P, Cooper S et al. Judgement of the quality of bowel preparation at screening flexible sigmoidoscopy is associated with variability in adenoma detection rates. Endoscopy 2006; 38: 456-460
• 8 Ivers N, Jamtvedt G, Flottorp S et al. Audit and Feedback: effects on professional practice and healthcare outcomes (Review). Cochrane Database of Systematic Reviews 2012; Issue 6. Art. No.: CD000259. DOI: 10.1002/14651858.CD000259.pub.
• 9 Coe SG, Crook JE, Diehl NN et al. An endoscopic quality improvement program improves detection of colorectal adenomas. Am J Gastroenterol 2013; 108: 219-226
• 10 Kaminski MJ, Anderson J, Valori R et al. Leadership training to improve adenoma detection rate in screening colonoscopy: a randomized trial. Gut 2016; 65: 616-624
• 11 Barton JR, Corbett S, van der Vleuten CP et al. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. GIE 2012; 75: 591-597
• 12 Gupta S, Anderson J, Bhandari P et al. Development and validation of a novel method for assessing competency in polypectomy: direct observation of polypectomy skills. GIE 2011; 73: 1232-1239