Download PDF
Open Access
CC BY-NC-ND 4.0 · Methods Inf Med 2022; 61(S 02): e51-e63
DOI: 10.1055/a-1862-0421
Original Article

A Systematic Approach to Configuring MetaMap for Optimal Performance

Authors

  • Xia Jing

    1   Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson University, Clemson, South Carolina, United States

  • Akash Indani

    2   School of Computing, College of Engineering, Computing and Applied Sciences, Clemson University, Clemson, South Carolina, United States

  • Nina Hubig

    2   School of Computing, College of Engineering, Computing and Applied Sciences, Clemson University, Clemson, South Carolina, United States

  • Hua Min

    3   Department of Health Administration and Policy, College of Health and Human Services, George Mason University, Fairfax, Virginia, United States

  • Yang Gong

    4   School of Biomedical Informatics, The University of Texas Health Sciences Center at Houston, Houston, Texas, United States

  • James J. Cimino

    5   Informatics Institute, The University of Alabama at Birmingham, Birmingham, Alabama, United States

  • Dean F. Sittig

    4   School of Biomedical Informatics, The University of Texas Health Sciences Center at Houston, Houston, Texas, United States

  • Lior Rennert

    1   Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson University, Clemson, South Carolina, United States

  • David Robinson

    6   Independent Consultant, Cumbria, United kingdom

  • Paul Biondich

    7   Department of Pediatrics, Clem McDonald Biomedical Informatics Center, Regenstrief Institute, Indiana University School of Medicine, Indianapolis, Indiana, United States

  • Adam Wright

    8   Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States

  • Christian Nøhr

    9   Department of Planning, Faculty of Engineering, Aalborg University, Aalborg, Denmark

  • Timothy Law

    10   Ohio Musculoskeletal and Neurologic Institute, Ohio University, Athens, Ohio, United States

  • Arild Faxvaag

    11   Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology, Trondheim, Norway

  • Ronald Gimbel

    1   Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson University, Clemson, South Carolina, United States

Funding This work is supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award No. R01GM138589 and partially under P20 GM121342. We acknowledge Clemson University for the generous allotment of computing time on the Palmetto Cluster.
Further Information
  Permissions and Reprints
Preview

Abstract

Background MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward.

Objective To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance.

Methods MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated.

Results The percentages of exact matches and missing gold-standard terms were 0.6–0.79 and 0.09–0.3 for one behavior option, and 0.56–0.8 and 0.09–0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively.

Conclusion We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.

Keywords

MetaMap - natural language processing - clinical decision support system - configuration and optimization - performance

Supplementary Material



Publication History

Accepted Manuscript online:
25 May 2022

Article published online:
19 September 2022

© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany