Simple Rules, Not So Simple: The Use of International Ovarian Tumor Analysis (IOTA) Terminology and Simple Rules in Inexperienced Hands in a Prospective Multicenter Cohort Study

Abstract

Objectives To analyze how well untrained examiners – without experience in the use of International Ovarian Tumor Analysis (IOTA) terminology or simple ultrasound-based rules (simple rules) – are able to apply IOTA terminology and simple rules and to assess the level of agreement between non-experts and an expert.

Methods This prospective multicenter cohort study enrolled women with ovarian masses. Ultrasound was performed by non-expert examiners and an expert. Ultrasound features were recorded using IOTA nomenclature, and used for classifying the mass by simple rules. Interobserver agreement was evaluated with Fleiss’ kappa and percentage agreement between observers.

Results 50 consecutive women were included. We observed 46 discrepancies in the description of ovarian masses when non-experts utilized IOTA terminology. Tumor type was misclassified often (n = 22), resulting in poor interobserver agreement between the non-experts and the expert (kappa = 0.39, 95 %-CI 0.244 – 0.529, percentage of agreement = 52.0 %). Misinterpretation of simple rules by non-experts was observed 57 times, resulting in an erroneous diagnosis in 15 patients (30 %). The agreement for classifying the mass as benign, malignant or inconclusive by simple rules was only moderate between the non-experts and the expert (kappa = 0.50, 95 %-CI 0.300 – 0.704, percentage of agreement = 70.0 %). The level of agreement for all 10 simple rules features varied greatly (kappa index range: -0.08 – 0.74, percentage of agreement 66 – 94 %).

Conclusion Although simple rules are useful to distinguish benign from malignant adnexal masses, they are not that simple for untrained examiners. Training with both IOTA terminology and simple rules is necessary before simple rules can be introduced into guidelines and daily clinical practice.

Zusammenfassung

Ziel Analyse, ob ungeübte Anwender – ohne Erfahrung mit der „International Ovarian Tumor Analysis“ (IOTA)- Terminologie oder einfache ultraschallbasierten Regeln („Simple Rules“) – in der Lage sind, IOTA-Kriterien und „Simple Rules“ anzuwenden. Auch wird der Grad der Übereinstimmung zwischen Nichtexperten und Experten bewertet.

Methoden Diese prospektive multizentrische Kohortenstudie nahm Frauen mit ovarialen Raumforderungen auf. Die Sonografie wurde von Nichtexperten und einem Experten durchgeführt. Ultraschall-Kriterien wurden mittels IOTA-Nomenklatur dokumentiert und dann für die Klassifizierung der Raumforderung mittels „Simple Rules“ verwendet. Die Intraobserver-Übereinstimmung zwischen den Beobachtern wurde durch Fleiss-Kappa und prozentualer Übereinstimmung bewertet.

Ergebnisse Eingeschlossen wurden 50 aufeinander folgende Frauen. Wir beobachteten 46 Diskrepanzen bei der Beschreibung der ovarialen Raumforderungen, wenn Nichtexperten die IOTA-Terminologie benutzten. Der Tumortyp wurde häufig falsch klassifiziert (n = 22), was zu einer schlechten Interobserver-Übereinstimmung zwischen Nichtexperten und Experten führte (Kappa = 0,39; 95 %-CI 0,244 – 0,529; prozentuale Übereinstimmung = 52,0 %). Eine Falschinterpretation der „Simple Rules“ durch Nichtexperten wurde 57 Mal beobachtet und führte bei 15 Patienten (30 %) zu einer Falschdiagnose. Die Übereinstimmung bei der Klassifizierung einer Raumforderung als gutartig, maligne oder nicht eindeutig durch die „Simple Rules“ war zwischen Nichtexperten und Experten nur mittelmäßig (Kappa = 0,50; 95 %-CI 0,300 – 0,704; prozentuale Übereinstimmung = 70,0 %). Der Grad der Übereinstimmung bei allen 10 „Simple Rules“-Kriterien variierte enorm (Kappa-Index-Bereich: -0,08 – 0,74; prozentuale Übereinstimmung 66 – 94 %).

Schlussfolgerung Obwohl die „Simple Rules“ nützlich sind, um benigne und maligne adnexale Raumforderungen zu unterscheiden, sind diese für ungeübte Untersucher nicht so einfach zu handhaben. Schulungen die sowohl IOTA-Terminologie als auch „Simple Rules“, zum Inhalt haben sind notwendig, noch ehe „Simple Rules“ in Leitlinien und den Praxisalltag Eingang finden.

Key words

ultrasound - adnexal mass - IOTA simple rules - reproducibility - ovarian cancer

• References

• 1 Meys EMJ, Kaijser J, Kruitwagen RF. et al. Subjective assessment versus ultrasound based models to diagnose ovarian cancer: a systematic review and meta-analysis. Eur J Cancer 2016; 58: 17-29
• 2 Timmerman D, Testa AC, Bourne T. et al. Simple ultrasound-based rules for the diagnosis of ovarian cancer. Ultrasound Obstet Gynecol 2008; 31: 681-690
• 3 Timmerman D, Ameye L, Fischerova D. et al. Simple ultrasound rules to distinguish between benign and malignant adnexal masses before surgery: prospective validation by IOTA group. BMJ 2010; 341: c6839
• 4 Fathallah K, Huchon C, Bats AS. et al. [External validation of simple ultrasound rules of Timmerman on 122 ovarian tumors]. Gynecol Obstet Fertil 2011; 39: 477-481
• 5 Hartman CA, Juliato CR, Sarian LO. et al. Ultrasound criteria and CA 125 as predictive variables of ovarian cancer in women with adnexal tumors. Ultrasound Obstet Gynecol 2012; 40: 360-366
• 6 Alcazar JL, Pascual MA, Olartecoechea B. et al. IOTA simple rules for discriminating between benign and malignant adnexal masses: prospective external validation. Ultrasound Obstet Gynecol 2013; 42: 467-471
• 7 Nunes N, Ambler G, Foo X. et al. Use of IOTA simple rules for diagnosis of ovarian cancer: meta-analysis. Ultrasound Obstet Gynecol 2014; 44: 503-514
• 8 Testa A, Kaijser J, Wynants L. et al. Strategies to diagnose ovarian cancer: new evidence from phase 3 of the multicentre international IOTA study. Br J Cancer 2014; 111: 680-688
• 9 Tantipalakorn C, Wanapirak C, Khunamornpong S. et al. IOTA simple rules in differentiating between benign and malignant ovarian tumors. Asian Pac J Cancer Prev 2014; 15: 5123-5126
• 10 Ruiz de Gauna B, Rodriguez D, Olartecoechea B. et al. Diagnostic performance of IOTA simple rules for adnexal masses classification: a comparison between two centers with different ovarian cancer prevalence. Eur J Obstet Gynecol Reprod Biol 2015; 191: 10-14
• 11 Knafel A, Banas T, Nocun A. et al. The Prospective External Validation of International Ovarian Tumor Analysis (IOTA) Simple Rules in the Hands of Level I and II Examiners. Ultraschall in Med 2016; 37(5): 516-523
• 12 Silvestre L, Martins WP, Candido-Dos-Reis FJ. Limitations of three-dimensional power Doppler angiography in preoperative evaluation of ovarian tumors. J Ovarian Res 2015; 8: 47
• 13 Tinnangwattana D, Vichak-Ururote L, Tontivuthikul P. et al. IOTA Simple Rules in Differentiating between Benign and Malignant Adnexal Masses by Non-expert Examiners. Asian Pac J Cancer Prev 2015; 16: 3835-3838
• 14 Timmerman D, Valentin L, Bourne TH. et al. Terms, definitions and measurements to describe the sonographic features of adnexal tumors: a consensus opinion from the International Ovarian Tumor Analysis (IOTA) Group. Ultrasound Obstet Gynecol 2000; 16: 500-505
• 15 Meys EM, Rutten IJ, Kruitwagen RF. et al. Investigating the performance and cost-effectiveness of the simple ultrasound-based rules compared to the risk of malignancy index in the diagnosis of ovarian cancer (SUBSONiC-study): protocol of a prospective multicenter cohort study in the Netherlands. BMC Cancer 2015; 15: 482
• 16 Bossuyt PM, Reitsma JB, Bruns DE. et al. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Ann Intern Med 2003; 138: 40-44
• 17 European Federation of Societies for Ultrasound in Medicine, Biology, Education, Committee PS. Minimum training recommendations for the practice of medical ultrasound. Ultraschall in Med 2006; 27: 79-105
• 18 Brennan P, Silman A. Statistical methods for assessing observer variability in clinical measures. BMJ 1992; 304: 1491-1494
• 19 Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990; 43: 543-549
• 20 Freelon DG. ReCal: Intercoder reliability calculation as a web service. International Journal of Internet Science 2010; 5: 20-23
• 21 Ruiz de Gauna B, Sanchez P, Pineda L. et al. Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional ultrasound volumes and digital clips. Ultrasound Obstet Gynecol 2014; 44: 95-99
• 22 Sayasneh A, Kaijser J, Preisler J. et al. A multicenter prospective external validation of the diagnostic performance of IOTA simple descriptors and rules to characterize ovarian masses. Gynecol Oncol 2013; 130: 140-146
• 23 Sayasneh A, Wynants L, Preisler J. et al. Multicentre external validation of IOTA prediction models and RMI by operators with varied training. Br J Cancer 2013; 108: 2448-2454
• 24 Alcazar JL, Pascual MA, Graupera B. et al. External validation of IOTA simple descriptors and simple rules for classifying adnexal masses. Ultrasound Obstet Gynecol 2016; 48: 397-402
• 25 Sladkevicius P, Valentin L. Interobserver agreement in describing the ultrasound appearance of adnexal masses and in calculating the risk of malignancy using logistic regression models. Clin Cancer Res 2015; 21: 594-601
• 26 Zannoni L, Savelli L, Jokubkiene L. et al. Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven observers. Ultrasound Obstet Gynecol 2014; 44: 100-108
• 27 Zannoni L, Savelli L, Jokubkiene L. et al. Intra- and interobserver reproducibility of assessment of Doppler ultrasound findings in adnexal masses. Ultrasound Obstet Gynecol 2013; 42: 93-101
• 28 Peces RamaA, Llanos LlanosMC, Sanchez FerrerML. et al. Simple descriptors and simple rules of the International Ovarian Tumor Analysis (IOTA) Group: a prospective study of combined use for the description of adnexal masses. Eur J Obstet Gynecol Reprod Biol 2015; 195: 7-11
• 29 Van Holsbeke C, Daemen A, Yazbek J. et al. Ultrasound methods to distinguish between malignant and benign adnexal masses in the hands of examiners with different levels of experience. Ultrasound Obstet Gynecol 2009; 34: 454-461
• 30 Van Calster B, Van Hoorde K, Froyman W. et al. Practical guidance for applying the ADNEX model from the IOTA group to discriminate between different subtypes of adnexal tumors. Facts Views Vis Obgyn 2015; 7: 32-41
• 31 Guerriero S, Saba L, Ajossa S. et al. Assessing the reproducibility of the IOTA simple ultrasound rules for classifying adnexal masses as benign or malignant using stored 3D volumes. Eur J Obstet Gynecol Reprod Biol 2013; 171: 157-160
• 32 Blehar DJ, Barton B, Gaspari RJ. Learning curves in emergency ultrasound education. Acad Emerg Med 2015; 22: 574-582
• 33 Green J, Kahan M, Wong S. Obstetric and Gynecologic Resident Ultrasound Education Project: Is the Current Level of Gynecologic Ultrasound Training in Canada Meeting the Needs of Residents and Faculty?. J Ultrasound Med 2015; 34: 1583-1589
• 34 Fischerova D, Zikan M, Dundr P. et al. Diagnosis, treatment, and follow-up of borderline ovarian tumors. Oncologist 2012; 17: 1515-1533
• 35 Gaspari RJ, Dickman E, Blehar D. Learning curve of bedside ultrasound of the gallbladder. J Emerg Med 2009; 37: 51-56
• 36 Jang T, Aubin C, Naunheim R. Minimum training for right upper quadrant ultrasonography. Am J Emerg Med 2004; 22: 439-443
• 37 Heegeman DJ, Kieke Jr B. Learning curves, credentialing, and the need for ultrasound fellowships. Acad Emerg Med 2003; 10: 404-405
• 38 Kendall JL, Shimp RJ. Performance and interpretation of focused right upper quadrant ultrasound by emergency physicians. J Emerg Med 2001; 21: 7-13
• 39 Alcazar JL, Diaz L, Florez P. et al. Intensive training program for ultrasound diagnosis of adnexal masses: protocol and preliminary results. Ultrasound Obstet Gynecol 2013; 42: 218-223
• 40 Tolsgaard MG, Rasmussen MB, Tappert C. et al. Which factors are associated with trainees' confidence in performing obstetric and gynecological ultrasound examinations?. Ultrasound Obstet Gynecol 2014; 43: 444-451
• 41 Patel H, Chandrasekaran D, Myriokefalitaki E. et al. The Role of Ultrasound Simulation in Obstetrics and Gynecology Training: A UK Trainees' Perspective. Simul Healthc 2016; 11: 340-344