Factors defining expertise in screening colonoscopy

Introduction

Bowel cancer screening (BCS) has been successfully rolled out across England, with stringent quality requirements for units undertaking screening and individuals performing colonoscopy within the program. There is a rigorous colonoscopist assessment process, which includes both factual knowledge and practical ability. After commencing screening, performance is measured regularly across a broad range of key performance indicators (KPIs) and individual performance is compared to others within the same region.

There has been long-standing interest in factors affecting performance in colonoscopy [1]. In particular, the correlation of higher adenoma detection rates (ADR) in medium-risk patients with a reduction in risk of interval cancer [2] has prompted special interest in this performance metric. Numerous studies have been performed looking at methods of improving ADR, including increasing colonoscopic withdrawal times [3], position change during extubation [4], use of hyoscine [5], chromendoscopy [6] and other novel endoscopic techniques.

Analysis of data from the Bowel Cancer Screening Programme (BCSP) has found that the vast majority of individuals perform above the prescribed minimum standards [7]. However, as expected in any population, some individuals perform consistently higher than others, even within the already selected group of BCS colonoscopists. The reasons for these differences are unclear but are not accounted for by known factors affecting adenoma detection rate.

There is very little literature defining characteristics of expert endoscopists. One small study examined factors contributing to high-quality colonoscopy by using a Delphi survey [8]. The vast majority of published material concerns training in endoscopy [9] [10] [11] [12], with some limited literature on assessment of technical endoscopic ability using simulators in experts [13], but often as a comparator to unskilled endoscopists. There are some data on the correlation between technical and non-technical skills in anesthetists, but the relative importance of these attributes in endoscopy is unknown [14].

It is hypothesised that previously undetermined human factors correlate with KPIs in screening colonoscopists. The aim of this study was to determine factors contributing towards expertise in screening colonoscopy.

Materials and methods

Research into human factors lends itself to qualitative rather than quantitative analysis, as qualitative research can offer a more in-depth understanding of defined topic areas [15] [16]. Qualitative research with thematic analysis is a well-validated method to answer questions such as “What constitutes an expert?” [17]. It is well recognised in research that using a single method of enquiry is likely to result in inadequate data collection, and using multiple methods is much more likely to produce an accurate representation of the important human factors in individual disciplines [18].

The study was therefore planned in several steps ([Fig. 1]) to ensure that important factors were captured. The study proposal was peer reviewed by the Bowel Cancer Screening Programme Research Committee, who gave formal permission for the study to recruit colonoscopists from the Programme.

Fig. 1 Schematic of methodology to highlight factors responsible for high performance in colonoscopy.

Ethical approval

The study was evaluated by the local ethics department and deemed not to need formal ethical approval. The work was carried out in accordance with the Declaration of Helsinki including, but not limited to, there being no potential harm to participants, that the anonymity of participants was guaranteed, and that informed consent of participants was obtained.

Attribute identification focus group

This semi-structured group discussion was designed to determine whether published themes were pertinent to expertise in endoscopy or whether other criteria should be included. This was based on a ‘brainstorming’ exercise aimed at identifying which skills or behaviors experts in the field consider important for expert endoscopy. Participants were asked to identify skills that they considered to be relevant to high performance in endoscopy.

A group of endoscopists including BCS screeners from several centers and endoscopy staff (nurse endoscopists, support staff) were asked to participate in an initial focus group. For convenience, the focus group comprised staff from a single city hospital, including screening endoscopists, gastroenterologists with a specialist interest in endoscopy, a nurse consultant endoscopist, trainee gastroenterologists, nurse endoscopists, endoscopy nurses, secretarial and administration staff. This sample was chosen as it encompassed a wide group of professionals with an interest in the subject topic.

Key themes from the focus group were informally recorded on a chart, initially for discussion with the group, and then to create a list of factors thought to be important.

Rating task – initial iteration

This task was designed for 3 purposes: to stratify identified factors in terms of importance, to ascertain whether other factors had been omitted, and to ensure that the suggested factors were not biased.

An independent group of 39 BCS endoscopists, none of whom were present during the focus group, were polled at a meeting for an unrelated training session. They were asked to rank themes derived from the initial focus group, as well as to provide suggestions about any omitted themes.

This sample was chosen to try to validate the themes identified initially by overcoming institutional bias by including different BCS endoscopists from across England from a variety of different units.

Semi-structured interviews

All BCS endoscopists currently practicing colonoscopy in England were emailed directly about the proposed research. They were asked to reply if they did not wish to participate in the research. An information sheet about the interview process was provided. It was made clear that there was no compunction to take part and that all information would be anonymized.

After an interval to permit any colonoscopists to withdraw, further email contact was made by inviting screeners to provide their contact details if they wished to contribute. Interviews lasting up to 60 minutes were scheduled with the first 20 respondents. These interviews were recorded with consent.

The interviews comprised several parts. First, a participant was asked to describe a case study based on his or her experience of a time that required the skills of an expert endoscopist.

The interviewee was asked to recount the case in detail, describing his or her thoughts, decisions, actions and communications with colleagues. The case study was chosen because this methodology sometimes allows abstract concepts to be put into a real-life perspective [19], which facilitates exploration of the issues, and it was envisaged that this would allow deeper understanding of participants’ views of expertise [20].

The second part was the skill identification exercise. Endoscopists were asked directly about the skills they felt were important in defining an expert endoscopist. The discussion was then expanded to ask about how skills are currently developed in training. Given feedback from the earlier stages in the research, questions were asked about the specific differences between the skills needed for diagnostic and therapeutic colonoscopy.

Independent rating task – second iteration

The final part was the attribute identification and rating task, which was performed for a second time using the themes identified from the focus group and independent raters.

To avoid bias, only after the interview was complete was an interviewee asked to perform an online rating task, by assigning an importance to each of the themes previously identified from the previous stages of the research. Each theme was rated from 1 (most important) to 5 (least important).

Data analysis

The semi-structured interviews were recorded and anonymized. The audio file was then edited, enhanced to improve the sound quality, and then professionally transcribed.

The initial interviews were performed in conjunction with a psychologist trained in qualitative research to ensure that they were appropriately conducted. The psychologist gave feedback after each interview to improve the performance of the principal interviewer. Only after the psychologist was comfortable that the interviews were conducted to a high standard were the interviews conducted by a sole interviewer.

All interviews were coded using specialist software by the main investigators (QSR NVivo quantitative analysis software). To avoid investigator bias, the initial 2 transcripts were independently double coded by the independent psychologist and the main investigator. The coding of the 2 researchers was compared and discussed to ensure multiple perspectives in the analysis.

The transcripts were thematically analyzed to develop a thematic framework including key themes. A preliminary taxonomy related to expertise was developed by the lead investigator in collaboration with the psychologist. A sample of the transcribed interviews were re-coded according to the preliminary taxonomy using an iterative approach whereby the thematic framework kept being reviewed as new themes and subthemes emerged. This method of cross-checking data as themes evolve has been successfully used in previous studies [21].

Results

Attribute identification – focus group

A broad variety of factors were thought to be important by participants.

Technical ability ranked highly in participants’ perception of experts. The ability to “do what other endoscopists couldn’t” routinely do as well as the ability to “deal with the unexpected” were perceived as important characteristics. A focus on quality was deemed a defining characteristic by some, especially the importance of the adenoma detection rate in screening colonoscopy. Another theme emerging from the group included how experts possessed greater experience than others in terms of numbers of cases completed.

One participant thought peer recognition was important, stating “I’d let them scope me.” How this recognition was achieved, whether self-declared or independently recognised by colleagues was discussed, with 1 endoscopist considering that true experts could be defined partly by their academic publication record.

Non-technical qualities of experts were also featured. Self-insight was also thought to be important with expert endoscopists’ knowledge of their own competence and awareness of their limits discussed. The relevance of judgement in difficult situations was another theme, especially in dealing with complications. The importance of good interactions with patients and staff were also considered by some to be essential characteristics of experts.

Some group members highlighted how different skills were relevant to diagnostic and therapeutic colonoscopy.

The themes were then summarized to encompass the comments that had been received by all participants ([Table 1]).

Table 1 Themes highlighted by focus group.

Ability to deal with complications

Ability to tackle cases others won’t

Academic publication record

Adenoma detection rate

Communication skills

Declaration of expertise by others

Inter-personal skills with staff

Lifetime experience

Low complication rates

Manner with patients

Self-declaration of expertise

Staying calm under pressure

Usage of novel endoscopic techniques

Rating task

In total, 36 responses were received from individual anonymous BCS endoscopists, a response rate of 92 %.

Each individual item was ranked 1 to 13 for both diagnostic and therapeutic colonoscopy, with a score of 1 relating to the item the endoscopists felt was most important and 13 the least. No additional themes were suggested not already included in the list derived from the focus group. The consensus views in order of importance are shown in [Table 2].

Table 2 Ranked themes from bowel cancer screeners.

Rank	Diagnostic colonoscopy	Therapeutic colonoscopy
1	Low complication rates	Ability to deal with complications
2	Adenoma detection rate	Staying calm under pressure
3	Inter-personal skills with staff	Low complication rates
4	Communication skills	Communication skills
5	Manner with patients	Inter-personal skills with staff
6	Staying calm under pressure	Ability to tackle cases others won’t
7	Lifetime experience	Manner with patients
8	Ability to deal with complications	Adenoma detection rate
9	Declaration of expertise by others	Lifetime experience
10	Ability to tackle cases others won’t	Usage of novel endoscopic techniques
11	Usage of novel endoscopic techniques	Declaration of expertise by others
12	Self-declaration of expertise	Self-declaration of expertise
13	Academic publication record	Academic publication record

Semi-structured interviews

In total, 267 BCS endoscopists were invited to participate. There were 21 responses, a response rate of 7.9 %. Interviews with the first 20 respondents were conducted during the study period. The interviewees comprised 14 gastroenterologists, 4 surgeons and 2 nurses. The sample size was deemed adequate as after the first 6 interviews no new themes emerged. This methodology is recognized as consistent with previous work in this field [20].

The principal themes and subthemes relating to expertise are listed in [Table 3].

Table 3 Overarching themes relating to expertise.

Theme	Number of participants (total 20)
Technical skills	20
Endoscopic mucosal resection	20
Detection of polyps	17
Lifting polyps	14
High cecal intubation rate	11
Comfortable examination	7
Detection of cancer	3
Motor skills	3
Importance of hand-eye coordination	2
Speedy examination	1
Previous experience	19
Judgement / decision-making	18
Awareness of own limitations	11
Forward planning	6
Insight into own ability	6
Adequate knowledge	4
Awareness of alternative treatment options	4
Willingness to take risks	3
Attention to detail	3
Following instinct/heuristics	2
Communication	18
With nursing staff	8
With patient to keep comfortable	7
Instilling confidence in patient	3
Explaining procedure	2
Teamwork	15
Able to predict instructions	6
Having the same staff	5
Resources	13
Staff	13
Familiarity with equipment	9
Leadership	8
In control during procedure	3

Technical skills

Technical skills were mentioned by all 20 interviewees. All interviewees chose to discuss a case of difficult EMR as the scenario they felt required an expert. Whether diagnostic and therapeutic skills were different was contested. Some drew a distinction between diagnostic and therapeutic skills:

I think you can distinguish ... there’s the technical ability to get round the colon in an efficient, pain-free manner consistently, that’s one set of skills, and then a second set of skills is the therapy, so the judgement of knowledge and then the endoscopic fine motor skills and so on to manipulate this and to remove the polyp safely.

I think there’s clearly a bit of overlap between them but I think you can be a very competent diagnostic colonoscopist without being an expert therapeutic colonoscopist. So I think there are some attributes that make a therapeutic colonoscopist that aren’t necessarily found in every diagnostic colonoscopist. I think attitude is important, attitude towards risk I think is hugely important, and being prepared to perhaps approach things with a more surgical mentality would be a feature of the most advanced expert therapeutic colonoscopists that wouldn’t be seen in expert diagnostic colonoscopists.

Participant 11, gastroenterologist

Others however felt therapeutic colonoscopy involved an evolution of the skills required for diagnostic procedures rather than being fundamentally different:

Different is the wrong word. You’ve got to be able to have all the diagnostic skills to do therapeutic skills because otherwise you can’t get there. It’s the foundations and the first step. You don’t build the second floor without the first floor. You can build buildings without foundations, they’ll fall down but you can do it, but you can’t build a second floor without a first floor. And to do the therapeutic skills, which are more advanced, you’ve got to be able to do the therapeutic stuff first. You’ve got to walk before you can run.

Participant 4, surgeon

Interestingly, 3 participants reported during the interviews that they performed colonoscopy on their colleagues and rated this as a marker of their expertise and their technical proficiency.

Previous experience

The value of experience when attempting a case needing an expert was almost universally mentioned by interviewees (19/20). The number of cases interviewees had tackled during their lifetimes ranged from 2000 to “10 s of thousands.”

Experts said that they relied on their previous experience “completely” or “heavily.” One endoscopist questioned the conscious value of the experience they had gained as they had been “having been doing this sort of thing for an awfully long period of time one probably takes it for granted” (participant 14, gastroenterologist).

The incremental value of training experience over the years was also felt to contribute positively towards performance and tackling more difficult lesions:

The sheer number of polyps and sheer number of patients that one has scoped during the years puts you in a position to be able to take on the more difficult stuff that experts take on.”

Participant 16, gastroenterologist

The process of becoming a bowel cancer screening endoscopist itself was also mentioned to impact positively upon individual performance:

So I think when you start as a bowel cancer screening colonoscopist it’s quite scary because polyps are much bigger than you’re used to…but as you do more and more then your confidence grows and your skills improve. My skills, certainly in polypectomy, improved enormously when I started bowel cancer screening.

Participant 5, nurse endoscopist

Judgement/decision making

The role of good judgement in expert colonoscopy was mentioned as frequently as that of experience.

Expressions such as “do I think I can do this?” were commonly encountered during the interviews. Expert colonoscopists often seemed to question themselves about whether the current treatment was correct, including during procedures. A good example of this was described by a colonoscopist when performing a difficult EMR:

And at each stage I was thinking, is it safe to proceed, is it safe to proceed? Would this man be better and safer if I stopped and put him through another pathway? Because this was a big polyp and frankly at every stage during this I was thinking, can I do this? Can anybody do it? Can somebody do it better than me? And that was my thought process throughout most of the management that I had to do with him.

Participant 8, gastroenterologist

Communication/teamwork

Teamwork and communication were rated highly by most interviewees. Factors such as non-verbal communication and the ability to predict instructions before being asked were considered strong features of a good team by 6 respondents.

One endoscopist said:

You almost catch them out of the corner of your eye going to get something and it’s only when you ask and it’s there waiting for you”

Participant 16, gastroenterologist.

Another noted:

I did another EMR this morning as well and it was a case of – I’m putting the snare in, I’m saying, “Open” and even before I’m saying it, it’s opening. I’m saying, “Close.” Even before I’m saying it, they’re saying, “It’s closing.” I’m saying – because I find that everybody says, “Oh, you should always close it by yourself.”

Participant 11, surgeon

This aspect of staff working closely together and communicating efficiently yet often silently was a recurrent theme:

And the best you can say about a team is when the team works smoothly and nobody really notices the fact there’s a team going on, because if you notice there’s a team it’s usually because somebody’s done something you weren’t expecting or hasn’t done something you were expecting. If a team works smoothly nobody notices.

Participant 8, gastroenterologist

Clear communication with the nursing staff was highlighted by 8 interviewees. Half of the interviewees also emphasised communication with the patient, in terms of keeping them comfortable (7/20), instilling confidence (3/20) and explaining the procedure (2/20).

Resources

Interviewees’ view of the resources that were important to them fell into two broad categories.

The majority (13/20) mentioned staff as a key resource and “that the staff that are supporting you, your endoscopy assistant is someone who you’re confident in” (participant 2, gastroenterologist).

The second category of important resources was additional equipment, such as snares, lifting solution and diathermy machines. A broad range of equipment was not deemed to be essential: “it doesn’t need to be a very wide variety, it just needs to be the right things” (participant 4, surgeon). Familiarity with the equipment was deemed crucial by 9/20 respondents, with availability of the correct equipment instilling confidence in colonoscopists. One endoscopist commented: “I would never attempt to perform this sort of procedure with the other bit of equipment that I’m less comfortable with” (participant 1, gastroenterologist).

Rating task – second iteration

Nineteen of 20 interviewees completed the online rating task, ranking each previously defined attribute from 1 (most important) to 5 (least important). The median scores given for each attribute for both diagnostic and therapeutic colonoscopy are shown in [Table 4] below, in descending order of importance.

Table 4 Relative importance of predetermined themes by interviewees.

	Diagnostic	Therapeutic
Low complication rates	1	1
Adenoma detection rate	1	2
Manner with patients	1	2
Ability to deal with complications	2	1
Communication skills	2	1
Inter-personal skills with staff	2	1
Staying calm under pressure	2	1
Lifetime experience	2	1.5
Ability to tackle cases others won’t	2.5	2
Declaration of expertise by others	2.5	2
Self-declaration of expertise	3	3
Usage of novel endoscopic techniques	3	3
Academic publication record	4.5	4.5

This ranking was largely similar to the order identified in the first iteration with themes such as academic publication record, usage of novel endoscopic techniques, self-declaration of expertise and declaration of expertise by others appearing at the bottom of both lists.

Discussion

It is unsurprising that technical ability rates highly in each of the phases of this work. Colonoscopy is by its very nature a practical skill and without a certain degree of ability, safe, comfortable and effective colonoscopy is not possible.

The differences in the perception of skills needed for diagnostic and therapeutic colonoscopy were interesting. Although some did view the procedures as entirely different, others took a more nuanced view and thought that proficiency in diagnostic procedures was the “foundation” for competent therapeutic colonoscopy. Interestingly however, when asked to rate the themes at the end of the interview, the scores given by colonoscopists in each of these 2 domains were largely similar.

Although technical ability was the most common theme identified, other non-technical skills appeared very frequently. Judgement, communication, teamwork and leadership were all integral parts of experts’ views of qualities that they and other expert colonoscopists possessed.

The relevance of non-technical skills was confirmed by ratings given by interviewees in the second iteration of the scoring task. Of the 7 highest-ranked qualities scoring 1 or 2, 4 were related to non-technical skills, including a good patient manner, communication skills, interpersonal skills with staff and staying calm under pressure. Previous research has shown the importance of communication skills in improving patient satisfaction, adherence to treatments and overall outcomes [23] [24].

These findings have not been shown to date in endoscopy with no published studies correlating endoscopic outcomes with non-technical skills. However, in other areas such as surgery, some studies have shown a correlation between non-technical performance and technical outcome [25]. When surgical teams were assessed for their non-technical abilities and number of mistakes made during laparoscopic cholecystectomy, it was found that there was a negative correlation between surgeons’ situational awareness and their error rates [25]. Other studies have mirrored these findings with poorer non-technical skills associated with higher rates of technical errors in surgeons [26] [27] [28].

It is in some ways predictable that these findings could be translated through to endoscopy, especially as with more complex procedures performed by experts the endoscopy room increasingly takes on certain characteristics of the formal operating suite.

If this is the case, the challenge is to develop methods of training that can reliably imbue new endoscopists with these skills in a less haphazard way than the simple experiential learning of the past. All of the respondents in this study had performed thousands of procedures; indeed some stated that they had performed so many over decades that they had stopped counting altogether.

There is some evidence that non-technical skills training can improve surgical outcomes, although the effect size has been small [29]. In the aviation industry, crew resource management training has been embraced for several decades to improve the way in which rapidly changing teams work together. Even in this field, partly as a consequence of the low numbers of adverse events, the overall effect in improving safety is still controversial [30].

Team work was recognized as an important theme contributing towards expertise by most (15/20) respondents. It has been shown that effective teams have common characteristics including shared goals, behavioural norms, defined roles, flexible leadership, good communication, and common shared resources [31]. Although interviewees were heavily reliant on their individual teams for their own performance, no endoscopist mentioned how team performance could be improved as a whole. It has been shown that formal team training can be more effective than the team-building that naturally occurs from individuals working collaboratively together [32] [33].

The logical next step is to formulate interventions that could improve technical and non-technical skills and then assess whether the desired effects are seen in clinical practice. This is likely to be difficult however as, as in other arenas both in and out of medicine, the influence of any intervention is likely to be small and the difference therefore difficult to measure and conclusively prove. One study has shown that a 1-day course training multidisciplinary endoscopy teams improved awareness of patient safety knowledge and attitudes [34], but whether this has an effect on real patient care remains to be seen.

Although this research was confined to the Bowel Cancer Screening Programme, it is likely that the findings can be translated into general clinical endoscopic practice. Studies have shown that regular feedback, particularly in regard to the adenoma detection rate, can in itself improve performance [35] [36]. The importance of non-technical skills alongside those targeting technical performance metrics is being increasingly acknowledged [37]. Whereas all endoscopists would acknowledge the importance of technical proficiency, the relevance of non-technical skills such as communication in a highly skilled examination is likely to be less well recognized. It may be that a combination of training in technical and non-technical skills is the most effective way of improving expertise generally in all endoscopists, although how this can be most efficiently achieved remains unclear, with several models hypothesized [37]. The imminent introduction of a national endoscopic database in the United Kingdom is likely to highlight local differences in performance with greater ease than in the past and may itself drive standards towards those achieved in the best-performing centres.

Strengths and limitations

This is a large study with several different methodologies used to ascertain features of expert endoscopy. Data was collected from several different sources independently. As the participants were all volunteers and the response rate to the email invitation low, there is a chance that the results are affected by selection bias. Of course, it would not have been practical to interview unwilling participants; however, use of a nationally recruited group of endoscopists is likely to have counteracted this hypothetical issue.

Additional factors that strengthen the interview cohort include their diversity, comprising endoscopists with backgrounds in medicine, surgery and nursing, as well as the early saturation of themes. This suggests the results are likely to have been similar had there been a higher initial response rate to the email.

Conclusion

Both technical and non-technical abilities are considered essential components of expertise by experienced colonoscopists. Further research into targeted interventions to improve the rate of acquisition of these skills when training endoscopists may be useful for improving performance.

• References

• 1 Rex DK, Bond JH, Winawer S. et al. Quality in the technical performance of colonoscopy and the continuous quality improvement process for colonoscopy: recommendations of the US Multi-Society Task Force on Colorectal Cancer. Am J Gastroenterol 2002; 97: 1296-1308
• 2 Corley DA, Jensen CD, Marks AR. et al. Adenoma detection rate and risk of colorectal cancer and death. N Engl J Med 2014; 370: 1298-1306
• 3 Gellad ZF, Weiss DG, Ahnen DJ. et al. Colonoscopy withdrawal time and risk of neoplasia at 5 years: results from VA Cooperative Studies Program 380. Am J Gastroenterol 2010; 105: 1746-1752
• 4 Koksal AS, Kalkan IH, Torun S. et al. A simple method to improve adenoma detection rate during colonoscopy: altering patient position. Can J Gastroenterol 2013; 27: 509-512
• 5 Corte C, Dahlenburg L, Selby W. et al. Hyoscine butylbromide administered at the cecum increases polyp detection: a randomized double-blind placebo-controlled trial. Endoscopy 2012; 44: 917-922
• 6 Hlavaty T, Huorka M, Koller T. et al. Colorectal cancer screening in patients with ulcerative and Crohn's colitis with use of colonoscopy, chromoendoscopy and confocal endomicroscopy. Eur J Gastroenterol Hepatol 2011; 23: 680-689
• 7 Patel KP, Thomas-Gibson S, Faiz O. et al. Tu1319 An Evaluation of Screening Colonoscopists' Performance After a Structured Accreditation Process. Gastrointest Endosc 2013; 77: AB497-AB498
• 8 Ekkelenkamp VE, Koch AD, Haringsma J. et al. Endoscopist-related factors contributing to high-quality colonoscopy: results of a Delphi survey. Perspect Med Educ 2014; 3: 31-40
• 9 Ferlitsch A, Glauninger P, Gupper A. et al. Evaluation of a virtual endoscopy simulator for training in gastrointestinal endoscopy. Endoscopy 2002; 34: 698-702
• 10 Haycock A, Koch AD, Familiari P. et al. Training and transfer of colonoscopy skills: a multinational, randomized, blinded, controlled trial of simulator versus bedside training. Gastrointest Endosc 2010; 71: 298-307
• 11 Grantcharov TP, Carstensen L, Schulze S. Objective assessment of gastrointestinal endoscopy skills using a virtual reality simulator. JSLS, Journal of the Society of Laparoendoscopic Surgeons 2005; 9: 130-133
• 12 Sedlack RE, Baron TH, Downing SM. et al. Validation of a colonoscopy simulation model for skills assessment. Am J Gastroenterol 2007; 102: 64-74
• 13 Koch A, Haringsma J, Schoon E. et al. A second-generation virtual reality simulator for colonoscopy: validation and initial experience. Endoscopy 2008; 40: 735
• 14 Riem N, Boet S, Bould M. et al. Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth 2012; 109: 723-728
• 15 Arbous MS, Grobbee DE, van Kleef JW. et al. Dutch case-control study of anaesthesia-related morbidity and mortality. Rationale and methods. Anaesthesia 1998; 53: 162-168
• 16 Caelli K, Ray L, Mill J. ‘Clear as mud’: toward greater clarity in generic qualitative research. International journal of qualitative methods 2003; 2: 1-13
• 17 Green J, Britten N. Qualitative research and evidence based medicine. BMJ 1998; 316: 1230
• 18 Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res 1999; 34: 1189
• 19 Kohlbacher F. The use of qualitative content analysis in case study research. In, Forum Qualitative Sozialforschung/Forum: Qualitative Social Research; 2006
• 20 Larkin M, Thompson A. Interpretative phenomenological analysis. Qualitative research methods in mental health and psychotherapy: A guide for students and practitioners. 2003: 101-116
• 21 Pope C, Ziebland S, Mays N. Analysing qualitative data. Qualitative Research in Health Care. Third Edition 2007: 63-81
• Joffe H, Yardley L. 4 Content And Thematic Analysis. In: Marks D, Yardley L. Research methods for clinical and health psychology. London: Sage; 2004: 56-68
• 23 Charlton CR, Dearing KS, Berry JA. et al. Nurse practitioners’ communication styles and their impact on patient outcomes: an integrated literature review. J Am Acad Nurse Pract 2008; 20: 382-388
• 24 Street RL, Makoul G, Arora NK. et al. How does communication heal? Pathways linking clinician–patient communication to health outcomes. Patient Educ Couns 2009; 74: 295-301
• 25 Mishra A, Catchpole K, Dale T. et al. The influence of non-technical performance on technical outcome in laparoscopic cholecystectomy. Surg Endosc 2008; 22: 68-73
• 26 Catchpole K, Mishra A, Handa A. et al. Teamwork and error in the operating room: analysis of skills and roles. Ann Surg 2008; 247: 699-706
• 27 ElBardissi AW, Wiegmann DA, Henrickson S. et al. Identifying methods to improve heart surgery: an operative approach and strategy for implementation on an organizational level. Eur J Cardiothorac Surg 2008; 34: 1027-1033
• 28 Hull L, Arora S, Aggarwal R. et al. The impact of nontechnical skills on technical performance in surgery: a systematic review. J Am Coll Surg 2012; 214: 214-230
• 29 McCulloch P, Mishra A, Handa A. et al. The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Quality and Safety in Health Care 2009; 18: 109-115
• 30 Salas E, Wilson KA, Burke CS. et al. Does crew resource management training work? An update, an extension, and some critical needs. Human Factors 2006; 48: 392-412
• 31 Clark PR. Teamwork: building healthier workplaces and providing safer patient care. Crit Care Nurs Q 2009; 32: 221-231
• 32 Leedom DK, Simon R. Improving team coordination: a case for behavior-based training. Mil Psychol 1995; 7: 109
• 33 Stout RJ, Salas E, Fowlkes JE. Enhancing teamwork in complex environments through team training. Group Dyn 1997; 1: 169
• 34 Matharoo M, Haycock A, Sevdalis N. et al. Endoscopic non-technical skills team training: The next step in quality assurance of endoscopy training. World journal of gastroenterology: WJG 2014; 20: 17507
• 35 Kahi CJ, Ballard D, Shah AS. et al. Impact of a quarterly report card on colonoscopy quality measures. Gastrointest Endosc 2013; 77: 925-931
• 36 Chaptini L, Laine L. Can I improve my adenoma detection rate?. J Clin Gastroenterol 2015; 49: 270-281
• 37 Anderson J. The future of gastroenterology training: instruction in technical skills. Frontline Gastroenterology 2012; 3: i13-i18