Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer

Background and study aims: Fine-needle aspiration (FNA) guided by endoscopic ultrasonography (EUS) is important in mediastinal staging of non-small cell lung cancer (NSCLC). Training standards and implementation strategies of this technique are currently under discussion. The aim of this study was to explore the reliability and validity of a newly developed EUS Assessment Tool (EUSAT) designed to measure competence in EUS – FNA for mediastinal staging of NSCLC.

Patients and methods: A total of 30 patients with proven or suspected NSCLC underwent EUS – FNA for mediastinal staging by three trainees and three experienced physicians. Their performances were assessed prospectively by three experts in EUS under direct observation and again 2 months later in a blinded fashion using digital video-recordings. Based on the assessments, intra-rater reliability, inter-rater reliability, and construct validity were explored.

Results: The intra-rater reliability was good (Cronbach’s α = 0.80), but comparison of results based on direct observations and blinded video-recordings indicated a significant bias favoring consultants (P = 0.022). Inter-rater reliability was very good (Cronbach’s α = 0.93). However, one rater assessing five procedures or two raters each assessing four procedures were necessary to secure a generalizability coefficient of 0.80. The assessment tool demonstrated construct validity by discriminating between trainees and experienced physicians (P = 0.034).

Conclusions: Competency in mediastinal staging of NSCLC using EUS and EUS – FNA can be assessed in a reliable and valid way using the EUSAT assessment tool. Measuring and defining competency and training requirements could improve EUS quality and benefit patient care.

• References

• 1 Youlden DR, Cramb SM, Baade PD. The international epidemiology of lung cancer: geographical distribution and secular trends. J Thorac Oncol 2008; 3: 819-831
• 2 Chansky K, Sculier JP, Crowley JJ et al. The International Association for the Study of Lung Cancer Staging Project: prognostic factors and pathologic TNM stage in surgically managed non-small cell lung cancer. J Thorac Oncol 2009; 4: 792-801
• 3 Scott WJ, Howington J, Feigenberg S et al. Treatment of non-small cell lung cancer stage I and stage II: ACCP evidence-based clinical practice guidelines (2nd edition). Chest 2007; 132: 234-242
• 4 Tomaszek SC, Wigle DA. Surgical management of lung cancer. Semin Respir Crit Care Med 2011; 32: 69-77
• 5 De LP, Lardinois D, Van Schil PE et al. ESTS guidelines for preoperative lymph node staging for non-small cell lung cancer. Eur J Cardiothorac Surg 2007; 32: 1-8
• 6 Detterbeck FC, Jantz MA, Wallace M et al. Invasive mediastinal staging of lung cancer: ACCP evidence-based clinical practice guidelines (2nd edition). Chest 2007; 132: 202-220
• 7 Annema JT, van Meerbeeck JP, Rintoul RC et al. Mediastinoscopy vs endosonography for mediastinal nodal staging of lung cancer: a randomized trial. JAMA 2010; 304: 2245-2252
• 8 Sanders MK, Faigel DO. Training and simulators. In: Hawes RH, Fockens P, eds. Endosonography. 2nd. edn. Philadelphia: Elsevier Saunders; 2011: 22-28
• 9 Eisen GM, Dominitz JA, Faigel DO et al. Guidelines for credentialing and granting privileges for endoscopic ultrasound. Gastrointest Endosc 2001; 54: 811-814
• 10 Vilmann P, Saftoiu A. Endoscopic ultrasound-guided fine needle aspiration biopsy: equipment and technique. J Gastroenterol Hepatol 2006; 21: 1646-1655
• 11 Annema JT, Bohoslavsky R, Burgers S et al. Implementation of endoscopic ultrasound for lung cancer staging. Gastrointest Endosc 2010; 71: 64-70
• 12 Reznick RK, MacRae H. Teaching surgical skills – changes in the wind. N Engl J Med 2006; 355: 2664-2669
• 13 Rosch T. State of the art lecture: endoscopic ultrasonography: training and competence. Endoscopy 2006; 38: 69-S72
• 14 Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA 2002; 287: 226-235
• 15 Reznick R, Regehr G, MacRae H et al. Testing technical skill via an innovative “bench station” examination. Am J Surg 1997; 173: 226-230
• 16 Brennan RL. Generalizability theory. 1st. edn. New York: Springer; 2001
• 17 Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. The dependability of behavioral measurements. 1st. edn. New York: John Wiley and Sons; 1972
• 18 Vassiliou MC, Feldman LS, Fraser SA et al. Evaluating intraoperative laparoscopic skill: direct observation versus blinded videotaped performances. Surg Innov 2007; 14: 211-216
• 19 Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004; 38: 1006-1012
• 20 McGaghie WC, Butter J, Kaye M. Observational assessment. In: Downing SM, Yudkowsky R, eds. Assessment in health professions education. 1st. edn. New York: Routledge; 2009: 185-215
• 21 Konge L, Larsen KR, Clementsen P et al. Reliable and valid assessment of clinical bronchoscopy performance. Respiration 2012; 83: 53-60
• 22 Konge L, Arendrup H, von Buchwald C et al. Using performance in multiple simulated scenarios to assess bronchoscopy skills. Respiration 2011; 81: 483-490
• 23 Davoudi M, Osann K, Colt HG. Validation of two instruments to assess technical bronchoscopic skill using virtual reality simulation. Respiration 2008; 76: 92-101
• 24 Larsen CR, Grantcharov T, Schouenborg L et al. Objective assessment of surgical competence in gynaecological laparoscopy: development and validation of a procedure-specific rating scale. BJOG 2008; 115: 908-916
• 25 Ringsted C, Hodges B, Scherpbier A. “The research compass”: an introduction to research in medical education: AMEE Guide No. 56.. Med Teach 2011; 33: 695-709
• 26 Magill RA. The stages of learning. Motor learning and control. 8th. edn. New York: McGraw-Hill; 2007: 263-289
• 27 Eloubeidi MA, Tamhane A. EUS-guided FNA of solid pancreatic masses: a learning curve with 300 consecutive procedures. Gastrointest Endosc 2005; 61: 700-708
• 28 Mertz H, Gautam S. The learning curve for EUS-guided FNA of pancreatic cancer. Gastrointest Endosc 2004; 59: 33-37
• 29 Polkowski M, Larghi A, Weynand B et al. Learning, techniques, and complications of endoscopic ultrasound (EUS)-guided sampling in gastroenterology: European Society of Gastrointestinal Endoscopy (ESGE) Technical Guideline. Endoscopy 2012; 44: 190-206
• 30 Konge L, Clementsen P, Larsen KR et al. Establishing pass/fail criteria for bronchoscopy performance. Respiration 2012; 83: 140-146