Download PDF
Appl Clin Inform 2020; 11(04): 606-616
DOI: 10.1055/s-0040-1715892
Research Article

A Web Application for Adrenal Incidentaloma Identification, Tracking, and Management Using Machine Learning

Wasif Bala
1   Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
,
Jackson Steinkamp
1   Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
,
Timothy Feeney
1   Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
,
Avneesh Gupta
1   Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
,
Abhinav Sharma
2   Department of Family Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, Ontario, Canada
,
Jake Kantrowitz
3   Department of Internal Medicine, Kent Hospital, Brown University Alpert Medical School, Warwick, Rhode Island, United States
,
Nicholas Cordella
1   Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
,
James Moses
1   Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
,
Frederick Thurston Drake
1   Boston Medical Center, One Boston Medical Center Pl, Boston, Massachusetts, United States
› Author Affiliations
› Further Information
   Permissions and Reprints

Abstract

Background Incidental radiographic findings, such as adrenal nodules, are commonly identified in imaging studies and documented in radiology reports. However, patients with such findings frequently do not receive appropriate follow-up, partially due to the lack of tools for the management of such findings and the time required to maintain up-to-date lists. Natural language processing (NLP) is capable of extracting information from free-text clinical documents and could provide the basis for software solutions that do not require changes to clinical workflows.

Objectives In this manuscript we present (1) a machine learning algorithm we trained to identify radiology reports documenting the presence of a newly discovered adrenal incidentaloma, and (2) the web application and results database we developed to manage these clinical findings.

Methods We manually annotated a training corpus of 4,090 radiology reports from across our institution with a binary label indicating whether or not a report contains a newly discovered adrenal incidentaloma. We trained a convolutional neural network to perform this text classification task. Over the NLP backbone we built a web application that allows users to coordinate clinical management of adrenal incidentalomas in real time.

Results The annotated dataset included 404 positive (9.9%) and 3,686 (90.1%) negative reports. Our model achieved a sensitivity of 92.9% (95% confidence interval: 80.9–97.5%), a positive predictive value of 83.0% (69.9–91.1)%, a specificity of 97.8% (95.8–98.9)%, and an F1 score of 87.6%. We developed a front-end web application based on the model's output.

Conclusion Developing an NLP-enabled custom web application for tracking and management of high-risk adrenal incidentalomas is feasible in a resource constrained, safety net hospital. Such applications can be used by an institution's quality department or its primary care providers and can easily be generalized to other types of clinical findings.

Keywords

incidental findings - software tools - natural language processing - electronic medical record

Protection of Human and Animal Subjects

This study was reviewed and approved by the Boston Medical Center and Boston University Medical Campus Institutional Review Board (IRB).




Publication History

Received: 12 April 2020

Accepted: 22 July 2020

Article published online:
16 September 2020

© 2020. Thieme. All rights reserved.

Georg Thieme Verlag KG
Stuttgart · New York

 

  • References

  • 1 Young Jr WF. Clinical practice. The incidentally discovered adrenal mass. N Engl J Med 2007; 356 (06) 601-610
    CrossrefPubMedGoogle Scholar
  • 2 Terzolo M, Stigliano A, Chiodini I. , et al; Italian Association of Clinical Endocrinologists. AME position statement on adrenal incidentaloma. Eur J Endocrinol 2011; 164 (06) 851-870
    CrossrefPubMedGoogle Scholar
  • 3 Young Jr WF. Management approaches to adrenal incidentalomas. A view from Rochester, Minnesota. Endocrinol Metab Clin North Am 2000; 29 (01) 159-185
    CrossrefPubMedGoogle Scholar
  • 4 Mantero F, Terzolo M, Arnaldi G. , et al; Study Group on Adrenal Tumors of the Italian Society of Endocrinology. A survey on adrenal incidentaloma in Italy. J Clin Endocrinol Metab 2000; 85 (02) 637-644
    PubMedGoogle Scholar
  • 5 Cawood TJ, Hunt PJ, O'Shea D, Cole D, Soule S. Recommended evaluation of adrenal incidentalomas is costly, has high false-positive rates and confers a risk of fatal cancer that is similar to the risk of the adrenal lesion becoming malignant; time for a rethink?. Eur J Endocrinol 2009; 161 (04) 513-527
    CrossrefPubMedGoogle Scholar
  • 6 Mabotuwana T, Hombal V, Dalal S, Hall CS, Gunn M. Determining adherence to follow-up imaging recommendations. J Am Coll Radiol 2018; 15 (3 Pt A) 422-428
    CrossrefPubMedGoogle Scholar
  • 7 Dang PA, Kalra MK, Blake MA. , et al. Natural language processing using online analytic processing for assessing recommendations in radiology reports. J Am Coll Radiol 2008; 5 (03) 197-204
    CrossrefPubMedGoogle Scholar
  • 8 Sistrom CL, Dreyer KJ, Dang PP. , et al. Recommendations for additional imaging in radiology reports: multifactorial analysis of 5.9 million examinations. Radiology 2009; 253 (02) 453-461
    CrossrefPubMedGoogle Scholar
  • 9 Bhargavan M, Sunshine JH. Utilization of radiology services in the United States: levels and trends in modalities, regions, and populations. Radiology 2005; 234 (03) 824-832
    CrossrefPubMedGoogle Scholar
  • 10 Weingart SN, Yaghi O, Barnhart L. , et al. Preventing diagnostic errors in ambulatory care: an electronic notification tool for incomplete radiology tests. Appl Clin Inform 2020; 11 (02) 276-285
    Article in Thieme ConnectPubMedGoogle Scholar
  • 11 O'Connor SD, Khorasani R, Pochebit SM, Lacson R, Andriole KP, Dalal AK. Semiautomated system for nonurgent, clinically significant pathology results. Appl Clin Inform 2018; 9 (02) 411-421
    Article in Thieme ConnectPubMedGoogle Scholar
  • 12 Bovio S, Cataldi A, Reimondo G. , et al. Prevalence of adrenal incidentaloma in a contemporary computerized tomography series. J Endocrinol Invest 2006; 29 (04) 298-302
    CrossrefPubMedGoogle Scholar
  • 13 Ortman JM, Velkoff VA, Hogan H. An Aging Nation: The Older Population in the United States. Suitland-Silver Hill, MD: United States Census Bureau; 2014
    Google Scholar
  • 14 Grumbach MM, Biller BM, Braunstein GD. , et al. Management of the clinically inapparent adrenal mass (“incidentaloma”). Ann Intern Med 2003; 138 (05) 424-429
    CrossrefPubMedGoogle Scholar
  • 15 Nieman LK. Approach to the patient with an adrenal incidentaloma. J Clin Endocrinol Metab 2010; 95 (09) 4106-4113
    CrossrefPubMedGoogle Scholar
  • 16 Zeiger MA, Thompson GB, Duh Q-Y. , et al; American Association of Clinical Endocrinologists; American Association of Endocrine Surgeons. The American Association of Clinical Endocrinologists and American Association of Endocrine Surgeons medical guidelines for the management of adrenal incidentalomas. Endocr Pract 2009; 15 (Suppl. 01) 1-20
    CrossrefPubMedGoogle Scholar
  • 17 Fassnacht M, Arlt W, Bancos I. , et al. Management of adrenal incidentalomas: European Society of Endocrinology Clinical Practice Guideline in collaboration with the European Network for the Study of Adrenal Tumors. Eur J Endocrinol 2016; 175 (02) G1-G34
    CrossrefPubMedGoogle Scholar
  • 18 Kapoor A, Morris T, Rebello R. Guidelines for the management of the incidentally discovered adrenal mass. Can Urol Assoc J 2011; 5 (04) 241-247
    CrossrefPubMedGoogle Scholar
  • 19 Lee JM, Kim MK, Ko SH. , et al; Korean Endocrine Society, Committee for Clinical Practice Guidelines. Clinical guidelines for the management of adrenal incidentaloma. Endocrinol Metab (Seoul) 2017; 32 (02) 200-218
    CrossrefPubMedGoogle Scholar
  • 20 Feeney T, Talutis S, Janeway M. , et al. Evaluation of incidental adrenal masses at a tertiary referral and trauma center. Surgery 2020; 167 (05) 868-875
    CrossrefPubMedGoogle Scholar
  • 21 Becker J, Woloszyn J, Bold R, Campbell MJ. The adrenal incidentaloma: an opportunity to improve patient care. J Gen Intern Med 2018; 33 (03) 256-257
    CrossrefPubMedGoogle Scholar
  • 22 Kim Y. Convolutional neural networks for sentence classification. Paper presented at: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: Association for Computational Linguistics; 2014: 1746-1751
    PubMed
  • 23 Trivedi G, Hong C, Dadashzadeh ER, Handzel RM, Hochheiser H, Visweswaran S. Identifying incidental findings from radiology reports of trauma patients: an evaluation of automated feature representation methods. Int J Med Inform 2019; 129: 81-87
    CrossrefPubMedGoogle Scholar
  • 24 Shi S, Wang Q, Xu P, Chu X. Benchmarking state-of-the-art deep learning software tools. Paper presented at: 2016 7th International Conference on Cloud Computing and Big Data (CCBD). Macau: IEEE; 2016
    PubMedGoogle Scholar
  • 25 Pennington J, Socher R, Manning C. Glove: global vectors for word representation. Paper presented at: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP);. 2014. Doha: Association for Computational Linguistics;
  • 26 spaCy. Industrial-strength natural language processing in python. Available at: https://spacy.io/. Accessed September 7, 2019
    PubMedGoogle Scholar
  • 27 Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv [csLG]. December 2014. Available at: http://arxiv.org/abs/1412.6980
    PubMedGoogle Scholar
  • 28 Ahmad MA, Teredesai A, Eckert C. Interpretable machine learning in healthcare. Paper presented at: 2018 IEEE International Conference on Healthcare Informatics (ICHI);. 2018 . Hong Kong: IEEE Computer Society:447–447
    PubMed
  • 29 Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: visual explanations from deep networks via gradient-based localization. arXiv [csCV]; October 2016. Available at: http://arxiv.org/abs/1610.02391. Accessed June 23, 2020
    PubMedGoogle Scholar
  • 30 Hanneke S. Rates of convergence in active learning. Ann Stat 2011; 39 (01) 333-361
    CrossrefPubMedGoogle Scholar
  • 31 Lim PS, Schneider D, Sternlieb J. , et al. Process improvement for follow-up radiology report recommendations of lung nodules. BMJ Open Qual 2019; 8 (02) e000370
    CrossrefPubMedGoogle Scholar
  • 32 Pons E, Braun LMM, Hunink MGM, Kors JA. Natural language processing in radiology: a systematic review. Radiology 2016; 279 (02) 329-343
    CrossrefPubMedGoogle Scholar
  • 33 Zingmond D, Lenert LA. Monitoring free-text data using medical language processing. Comput Biomed Res 1993; 26 (05) 467-481
    CrossrefPubMedGoogle Scholar
  • 34 Garla V, Taylor C, Brandt C. Semi-supervised clinical text classification with Laplacian SVMs: an application to cancer case management. J Biomed Inform 2013; 46 (05) 869-875
    CrossrefPubMedGoogle Scholar
  • 35 Dutta S, Long WJ, Brown DFM, Reisner AT. Automated detection using natural language processing of radiologists recommendations for additional imaging of incidental findings. Ann Emerg Med 2013; 62 (02) 162-169
    CrossrefPubMedGoogle Scholar
  • 36 Yetisgen-Yildiz M, Gunn ML, Xia F, Payne TH. Automatic identification of critical follow-up recommendation sentences in radiology reports. AMIA Annu Symp Proc 2011; 2011: 1593-1602
    PubMedGoogle Scholar
  • 37 Meng X, Ganoe CH, Sieberg RT, Cheung YY, Hassanpour S. Assisting radiologists with reporting urgent findings to referring physicians: a machine learning approach to identify cases for prompt communication. J Biomed Inform 2019; 93: 103169
    CrossrefPubMedGoogle Scholar
  • 38 Meng X, Heinz MV, Ganoe CH, Sieberg RT, Cheung YY, Hassanpour S. Understanding urgency in radiology reporting: identifying associations between clinical findings in radiology reports and their prompt communication to referring physicians. Stud Health Technol Inform 2019; 264: 1546-1547
    PubMedGoogle Scholar
  • 39 Pham A-D, Névéol A, Lavergne T. , et al. Natural language processing of radiology reports for the detection of thromboembolic diseases and clinically relevant incidental findings. BMC Bioinformatics 2014; 15: 266
    CrossrefPubMedGoogle Scholar
  • 40 Hassanpour S, Bay G, Langlotz CP. Characterization of change and significance for clinical findings in radiology reports through natural language processing. J Digit Imaging 2017; 30 (03) 314-322
    CrossrefPubMedGoogle Scholar
  • 41 Mabotuwana T, Hall CS, Dalal S, Tieder J, Gunn ML. Extracting follow-up recommendations and associated anatomy from radiology reports. Stud Health Technol Inform 2017; 245: 1090-1094
    PubMedGoogle Scholar
  • 42 Dalal S, Hombal V, Weng W-H. , et al. Determining follow-up imaging study using radiology reports. J Digit Imaging 2020; 33 (01) 121-130
    CrossrefPubMedGoogle Scholar
  • 43 Grundmeier RW, Masino AJ, Casper TC. , et al; Pediatric Emergency Care Applied Research Network. Identification of long bone fractures in radiology reports using natural language processing to support healthcare quality improvement. Appl Clin Inform 2016; 7 (04) 1051-1068
    Article in Thieme ConnectPubMedGoogle Scholar
  • 44 Sevenster M, Buurman J, Liu P, Peters JF, Chang PJ. Natural language processing techniques for extracting and categorizing finding measurements in narrative radiology reports. Appl Clin Inform 2015; 6 (03) 600-110
    Article in Thieme ConnectPubMedGoogle Scholar
  • 45 Trivedi G, Dadashzadeh ER, Handzel RM, Chapman WW, Visweswaran S, Hochheiser H. Interactive NLP in clinical care: identifying incidental findings in radiology reports. Appl Clin Inform 2019; 10 (04) 655-669
    Article in Thieme ConnectPubMedGoogle Scholar