Appl Clin Inform 2017; 08(02): 560-580
DOI: 10.4338/ACI-2016-12-RA-0211
Research Article
Schattauer GmbH

The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance

Jeffrey P. Ferraro
Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
Intermountain Healthcare, Salt Lake City, Utah, USA
,
Ye Ye
Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
Intelligent Systems Program, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
,
Per H. Gesteland
Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
Department of Pediatrics, University of Utah, Salt Lake City, Utah, USA
,
Peter J. Haug
Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
Intermountain Healthcare, Salt Lake City, Utah, USA
,
Fuchiang Tsui
Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
Intelligent Systems Program, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
,
Gregory F. Cooper
Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
,
Rudy Van Bree
Intermountain Healthcare, Salt Lake City, Utah, USA
,
Thomas Ginter
VA Salt Lake City Healthcare System, Salt Lake City, Utah
,
Andrew J. Nowalk
Department of Pediatrics, Children‘s Hospital of Pittsburgh of University of Pittsburgh, Pittsburgh, Pennsylvania, USA
,
Michael Wagner
Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
Intelligent Systems Program, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
› Author Affiliations
Funding Research reported in this publication was supported by grant R01LM011370 from the National Library of Medicine.
Further Information

Publication History

received: 31 December 2016

accepted: 11 March 2017

Publication Date:
21 December 2017 (online)

Summary

Objectives: This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance.

Methods: We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) ‘other’ diagnosis.

Results: On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser).

Conclusion: In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.

Citation: Ferraro JP, Ye Y, Gesteland PH, Haug PJ, Tsui F(R), Cooper GF, Van Bree R, Ginter T, Nowalk AJ, Wagner M. The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance. Appl Clin Inform 2017; 8: 560–580 https://doi.org/10.4338/ACI-2016-12-RA-0211