Download PDF
Methods Inf Med 2015; 54(01): 93-102
DOI: 10.3414/ME13-01-0081
Original Articles
Schattauer GmbH

Analysis of Underlying Causes of Inter-expert Disagreement in Retinopathy of Prematurity Diagnosis

Application of Machine Learning Principles
E. Ataer-Cansizoglu
1   Cognitive Systems Laboratory, Northeastern University, Boston, MA, USA
,
J. Kalpathy-Cramer
2   Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, MA, USA
,
S. You
1   Cognitive Systems Laboratory, Northeastern University, Boston, MA, USA
,
K. Keck
3   Department of Ophthalmology & Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR, USA
,
D. Erdogmus
1   Cognitive Systems Laboratory, Northeastern University, Boston, MA, USA
,
M. F. Chiang
3   Department of Ophthalmology & Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR, USA
› Author Affiliations
Further Information

Publication History

received: 17 July 2013

accepted: 02 July 2014

Publication Date:
22 January 2018 (online)

    Permissions and Reprints

Summary

Objective: Inter-expert variability in image-based clinical diagnosis has been demonstrated in many diseases including retinopathy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.

Methods: The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts. The best three features for each observer were selected by an exhaustive search on all possible feature subsets and considering joint MI as a relevance criterion. We also compared our results with the results of Cohen’s Kappa [36] as an inter-rater reliability measure.

Results: The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.

Conclusion: Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.

Keywords

Inter-expert disagreement - feature selection - retinopathy of prematurity - kernel density estimation
 

  • References

  • 1 Elmore JG, Feinstein AR. A bibliography of publications on observer variability. Journal of Clinical Epidemiology 1992; 4 (06) 567-580.
    PubMedGoogle Scholar
  • 2 Munzenmayer C, Kage A, Wittenberg T, Muhldorfer S. Computer-assisted diagnosis for precancerous lesions in the esophagus. Methods Inf Med 2009; 48 (04) 324.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 3 Elmore JG, Brenner RJ. The more eyes, the better to see? from double to quadruple reading of screening mammograms. Journal of the National Cancer Institute. 2007: 1141-1143.
    Google Scholar
  • 4 Chiang MF, Jiang L, Gelman R, Du Y. Inter expert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol 2007; 125 (07) 875-880.
    CrossrefPubMedGoogle Scholar
  • 5 Van Ast JF, Talmon JL, Renier WO, Hasman A. An approach to knowledge base construction based on expert opinions. Methods Inf Med 2004; 43 (04) 427-432.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 6 Early Treatment For ROP Cooperative Group. Revised indications for the treatment of retinopathy of prematurity; results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol 2003; 121: 1684-1694.
    CrossrefPubMedGoogle Scholar
  • 7 Gilbert C, Foster A. Childhood blindness in the context of VISION 2020-the right to sight. Bull World Health Organ 2001; 79 (03) 227-232.
    PubMedGoogle Scholar
  • 8 Gilbert C, Fielder A, Gordillo L, Quinn G, Semiglia R, Visintin P. et al. Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate, and high levels of development: Implications for screening programs. Pediatrics 2003; 115 (05) 518-525.
    PubMedGoogle Scholar
  • 9 Kemper A, Wallace D. Neonatologists practices and experiences in arranging retinopathy of prematurity screening services. Pediatrics 2007; 120 (03) 527-531.
    CrossrefPubMedGoogle Scholar
  • 10 Javitt J. Dei CR, Chiang YP. Cost-effectiveness of screening and Cryotherapy for threshold retinopathy of prematurity. Pediatrics 1993; 91 (05) 859-866.
    PubMedGoogle Scholar
  • 11 The Committee for the Classiӿcation of Retinopathy of Prematurity. An international classification of retinopathy of prematurity. Arch Ophthalmol 1984; 1130: 1134.
    PubMedGoogle Scholar
  • 12 The Committee for the Classiӿcation of Retinopathy of Prematurity. The international classiӿcation of retinopathy of prematurity revisited. Arch Ophthalmol 2005; 991: 999.
    PubMedGoogle Scholar
  • 13 Chiang MF, Jiang L, Gelman R, Du Y. Inter expert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol 2007; 125 (07) 875-880.
    CrossrefPubMedGoogle Scholar
  • 14 Wallace D, Quinn G, Freedman S, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS 2008; 12 (04) 352-356.
    CrossrefPubMedGoogle Scholar
  • 15 Rao R, Jonsson NJ, Ventura C, Gelman R, Lindquist MA, Casper DS. et al. Plus disease in retinopathy of prematurity: diagnostic impact of field of view. Retina 2012; 32 (06) 1148-1155.
    CrossrefPubMedGoogle Scholar
  • 16 Thyparampil PJ, Park Y, Martinez-Perez ME, Lee TC, Weissgold DJ, Berrocal AM. et al. Plus disease in retinopathy of prematurity: quantitative analysis of vascular change. American Journal of Ophthalmology 2010; 150 (04) 468-575.
    CrossrefPubMedGoogle Scholar
  • 17 Wittenberg LA, Jonsson NJ, Paul Chan RV, Chiang MF. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity. J Pediatr Ophthalmol Strabismus 2012; 49 (01) 11-19.
    CrossrefPubMedGoogle Scholar
  • 18 Guyon I. André Elisseeff. An introduction to variable and feature selection. The Journal of Machine Learning Research 2003; 1157: 1182.
    PubMedGoogle Scholar
  • 19 Molina LC, Lluís B, Nebot A. Feature selection algorithms: A survey and experimental evaluation. IEEE International Conference on Data Mining (ICDM). 2002: 306-313.
    Google Scholar
  • 20 Rong J, Ghahramani Z. Learning with multiple labels. Advances in Neural Information Processing Systems. 2002: 897-904.
    Google Scholar
  • 21 Raykar VC, Shipeng Y, Linda HZ, Gerardo HV, Florin C, Bogoni L. et al. Learning from crowds. The Journal of Machine Learning Research 2010; 99: 1297-1322.
    PubMedGoogle Scholar
  • 22 Wallace DK, Jomier J, Aylward SR, Landers III MB. Computer-automated quantiӿcation of plus disease in retinopathy of prematurity. Journal of American Association for Pediatric Ophthalmology and Strabismus 2003; 7 (02) 126-130.
    CrossrefPubMedGoogle Scholar
  • 23 Gelman R, Martinez-Perez M, Vanderveen D, Moskowitz A, Fulton A. Diagnosis of plus disease in retinopathy of prematurity using retinal image multiscale analysis. Investigative ophthalmology & visual science 2005; 46 (12) 4734-4738.
    CrossrefPubMedGoogle Scholar
  • 24 Wilson C, Cocker K, Moseley M, Paterson C, Clay S, Schulenburg W. et al. Computerized analysis of retinal vessel width and tortuosity in premature infants. Investigative ophthalmology & visual science 2008; 49 (08) 3577-3585.
    CrossrefPubMedGoogle Scholar
  • 25 Shah D, Wilson C, Ying G, Karp K, Fielder A, Ng J. et al. Semiautomated digital image analysis of posterior pole vessels in retinopathy of prematurity. Journal of American Association for Pediatric Ophthalmology and Strabismus 2009; 13 (05) 504-506.
    CrossrefPubMedGoogle Scholar
  • 26 Gelman R, Jiang L, Du Y, Martinez-Perez M, Flynn J, Chiang MF. Plus disease in retinopathy of prematurity: pilot study of computer-based and expert diagnosis. Journal of American Association for Pediatric Ophthalmology and Strabismus. 2007; 11 (06) 532-540.
    CrossrefPubMedGoogle Scholar
  • 27 Koreen S, Gelman R, Martinez-Perez M, Jiang L, Berrocal A, Hess D. et al. Evaluation of a computer-based system for plus disease diagnosis in retinopathy of prematurity. Ophthalmology 2007; 114 (12) e59-e67.
    CrossrefPubMedGoogle Scholar
  • 28 Wallace D, Freedman S, Zhao Z, Jung S. Accuracy of roptool vs individual examiners in assessing retinal vascular tortuosity. Archives of ophthalmology 2007; 125 (11) 1523.
    CrossrefPubMedGoogle Scholar
  • 29 Cryotherapy for Retinopathy of Prematurity Cooperative Group. Multicenter trial of cryotherapy for retinopathy of prematurity. Preliminary results. Arch Ophthalmol 1988; 106: 471-479.
    CrossrefPubMedGoogle Scholar
  • 30 Van Uitert R, Bitter I. Subvoxel precise skeletons of volumetric data based on fast marching methods. Medical physics 2007; 34: 627.
    CrossrefPubMedGoogle Scholar
  • 31 Douglas D, Peucker T. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Canadian Cartographer 1973; 10 (02) 112-122.
    CrossrefPubMedGoogle Scholar
  • 32 De Boor C. A Practical Guide to Splines. New York: Springer Verlag; 1978
    Google Scholar
  • 33 Sibson R. SLINK: an optimally efficient algorithm for the single-link cluster method. The Computer Journal 1973; 16 (01) 30-34.
    CrossrefPubMedGoogle Scholar
  • 34 Ataer-Cansizoglu E, You S, Kalpathy-Cramer J, Keck K, Chiang MF, Erdogmus D. Observer and feature analysis on diagnosis of retinopathy of prematurity. IEEE International Workshop on Machine Learning for Signal Processing (MLSP). 2012: 1-6.
    Google Scholar
  • 35 Finn JT. Use of the average mutual information index in evaluating classification error and consistency. International Journal of Information Systems 1993; 7 (04) 349-366.
    PubMedGoogle Scholar
  • 36 Cohen J. A coefficient of agreement for nominal scales. Educational and psychological measurement 1960; 20 (01) 37-46.
    CrossrefPubMedGoogle Scholar
  • 37 Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical therapy 2005; 85 (03) 257-268.
    PubMedGoogle Scholar
  • 38 Klemens B. Mutual Information as measure of Intercoder Agreement. Journal of Official Statistics 2012; 28 (03) 395-412.
    PubMedGoogle Scholar