Download PDF
Methods Inf Med 2020; 59(06): 219-226
DOI: 10.1055/s-0041-1729951
Original Article

Accuracy of Asthma Computable Phenotypes to Identify Pediatric Asthma at an Academic Institution

Mindy K. Ross   #
1   Department of Pediatrics, University of California Los Angeles, Los Angeles, California, United States
,
Henry Zheng   #
2   Department of Radiological Sciences, University of California Los Angeles, Los Angeles, California, United States
,
Bing Zhu
2   Department of Radiological Sciences, University of California Los Angeles, Los Angeles, California, United States
,
Ailina Lao
3   University of California Los Angeles, Los Angeles, California, United States
,
Hyejin Hong
3   University of California Los Angeles, Los Angeles, California, United States
,
Alamelu Natesan
1   Department of Pediatrics, University of California Los Angeles, Los Angeles, California, United States
,
Melina Radparvar
1   Department of Pediatrics, University of California Los Angeles, Los Angeles, California, United States
,
Alex A.T. Bui
2   Department of Radiological Sciences, University of California Los Angeles, Los Angeles, California, United States
› Author Affiliations Funding Javier Sanz and the UCLA Clinical and Translational Science Institute (CTSI) Data Core; Dr. Sleiman, CHOP. The authors would like to acknowledge the UCLA CTSI (UL TR001881) and the Institute for Precision Health (IPH) for their support in this project.
› Further Information
    Permissions and Reprints

Abstract

Objectives Asthma is a heterogenous condition with significant diagnostic complexity, including variations in symptoms and temporal criteria. The disease can be difficult for clinicians to diagnose accurately. Properly identifying asthma patients from the electronic health record is consequently challenging as current algorithms (computable phenotypes) rely on diagnostic codes (e.g., International Classification of Disease, ICD) in addition to other criteria (e.g., inhaler medications)—but presume an accurate diagnosis. As such, there is no universally accepted or rigorously tested computable phenotype for asthma.

Methods We compared two established asthma computable phenotypes: the Chicago Area Patient-Outcomes Research Network (CAPriCORN) and Phenotype KnowledgeBase (PheKB). We established a large-scale, consensus gold standard (n = 1,365) from the University of California, Los Angeles Health System's clinical data warehouse for patients 5 to 17 years old. Results were manually reviewed and predictive performance (positive predictive value [PPV], sensitivity/specificity, F1-score) determined. We then examined the classification errors to gain insight for future algorithm optimizations.

Results As applied to our final cohort of 1,365 expert-defined gold standard patients, the CAPriCORN algorithms performed with a balanced PPV = 95.8% (95% CI: 94.4–97.2%), sensitivity = 85.7% (95% CI: 83.9–87.5%), and harmonized F1 = 90.4% (95% CI: 89.2–91.7%). The PheKB algorithm was performed with a balanced PPV = 83.1% (95% CI: 80.5–85.7%), sensitivity = 69.4% (95% CI: 66.3–72.5%), and F1 = 75.4% (95% CI: 73.1–77.8%). Four categories of errors were identified related to method limitations, disease definition, human error, and design implementation.

Conclusion The performance of the CAPriCORN and PheKB algorithms was lower than previously reported as applied to pediatric data (PPV = 97.7 and 96%, respectively). There is room to improve the performance of current methods, including targeted use of natural language processing and clinical feature engineering.

Keywords

asthma - computable phenotype - datasets - electronic health records - pediatrics

Ethical Considerations

A waiver of consent was obtained from the UCLA Institutional Review Board (IRB) committee (IRB #18–002015).


# Co-first authors.


Supplementary Material



Publication History

Received: 21 December 2020

Accepted: 15 April 2021

Article published online:
14 July 2021

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Shivade C, Raghavan P, Fosler-Lussier E. et al. A review of approaches to identifying patient phenotype cohorts using electronic health records. J Am Med Inform Assoc 2014; 21 (02) 221-230
    CrossrefPubMedGoogle Scholar
  • 2 Shang N, Liu C, Rasmussen LV. et al. Making work visible for electronic phenotype implementation: lessons learned from the eMERGE network. J Biomed Inform 2019; 99: 103293
    CrossrefPubMedGoogle Scholar
  • 3 Mo H, Thompson WK, Rasmussen LV. et al. Desiderata for computable representations of electronic health records-driven phenotype algorithms. J Am Med Inform Assoc 2015; 22 (06) 1220-1230
    CrossrefPubMedGoogle Scholar
  • 4 Kirby JC, Speltz P, Rasmussen LV. et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 2016; 23 (06) 1046-1052
    CrossrefPubMedGoogle Scholar
  • 5 NCQA. National Centers for Quality Assurance. HEDIS and Performance Measurement. Accessed December 6, 2020 at: https://www.ncqa.org/hedis/
    PubMedGoogle Scholar
  • 6 Centers for Medicare and Medicaid Services. Chronic Conditions Data Warehouse. Accessed December 6, 2020 at: http://www.ccwdata.org/index.htm
    PubMedGoogle Scholar
  • 7 AHRQ. Agency for Healthcare Research and Quality. Healthcare Cost and Utilization Project.
    PubMedGoogle Scholar
  • 8 Thompson WK, Rasmussen LV, Pacheco JA. et al. An evaluation of the NQF Quality Data Model for representing electronic health record driven phenotyping algorithms. AMIA Annu Symp Proc 2012; 2012: 911-920
    PubMedGoogle Scholar
  • 9 Jiang G, Kiefer RC, Rasmussen LV. et al. Developing a data element repository to support EHR-driven phenotype algorithm authoring and execution. J Biomed Inform 2016; 62: 232-242
    CrossrefPubMedGoogle Scholar
  • 10 Pletcher MJ, Forrest CB, Carton TW. PCORnet's collaborative research groups. Patient Relat Outcome Meas 2018; 9: 91-95
    CrossrefPubMedGoogle Scholar
  • 11 OHDSI. Observational Health Data Sciences and Informatics. OHDSI Gold Standard Phenotype Library Architecture.. Accessed January 6, 2020 at: https://www.ohdsi.org/2019-us-symposium-showcase-105/
    PubMedGoogle Scholar
  • 12 Pacheco JA, Avila PC, Thompson JA. et al. A highly specific algorithm for identifying asthma cases and controls for genome-wide association studies. AMIA Annu Symp Proc 2009; 2009: 497-501
    PubMedGoogle Scholar
  • 13 Geva A, Gronsbell JL, Cai T. et al; Pediatric Pulmonary Hypertension Network and National Heart, Lung, and Blood Institute Pediatric Pulmonary Vascular Disease Outcomes Bioinformatics Clinical Coordinating Center Investigators. A computable phenotype improves cohort ascertainment in a pediatric pulmonary hypertension registry. J Pediatr 2017; 188: 224-231.e5
    CrossrefPubMedGoogle Scholar
  • 14 Kashyap R, Sarvottam K, Wilson GA, Jentzer JC, Seisa MO, Kashani KB. Derivation and validation of a computable phenotype for acute decompensated heart failure in hospitalized patients. BMC Med Inform Decis Mak 2020; 20 (01) 85
    CrossrefPubMedGoogle Scholar
  • 15 Trivedi M, Denton E. Asthma in children and adults—what are the differences and what can they tell us about asthma?. Front Pediatr 2019; 7: 256
    CrossrefPubMedGoogle Scholar
  • 16 Amado MC, Portnoy JM. Diagnosing asthma in young children. Curr Opin Allergy Clin Immunol 2006; 6 (02) 101-105
    CrossrefPubMedGoogle Scholar
  • 17 National Asthma Education and Prevention Program. Expert panel report 3 (EPR-3): guidelines for the diagnosis and management of asthma—summary report 2007. J Allergy Clin Immunol 2007; 120 (Suppl. 05) S94-S138
    CrossrefPubMedGoogle Scholar
  • 18 Reddel HK, FitzGerald JM, Bateman ED. et al. GINA 2019: a fundamental change in asthma management: treatment of asthma with short-acting bronchodilators alone is no longer recommended for adults and adolescents. Eur Respir J 2019; 53 (06) 1901046
    CrossrefPubMedGoogle Scholar
  • 19 Cabana MD, Slish KK, Nan B, Clark NM. Limits of the HEDIS criteria in determining asthma severity for children. Pediatrics 2004; 114 (04) 1049-1055
    CrossrefPubMedGoogle Scholar
  • 20 CDC. Centers for Disease Control. Most Recent National Asthma Data. 2017 . Available at: https://www.cdc.gov/asthma/most_recent_national_asthma_data.htm
    PubMedGoogle Scholar
  • 21 Nurmagambetov T, Kuwahara R, Garbe P. The economic burden of asthma in the United States, 2008-2013. Ann Am Thorac Soc 2018; 15 (03) 348-356
    CrossrefPubMedGoogle Scholar
  • 22 Collaco JM, Abman SH. Evolving challenges in pediatric pulmonary medicine. new opportunities to reinvigorate the field. Am J Respir Crit Care Med 2018; 198 (06) 724-729
    CrossrefPubMedGoogle Scholar
  • 23 Afshar M, Press VG, Robison RG. et al. A computable phenotype for asthma case identification in adult and pediatric patients: external validation in the Chicago Area Patient-Outcomes Research Network (CAPriCORN). J Asthma 2018; 55: 1035-1042
    CrossrefPubMedGoogle Scholar
  • 24 Almoguera B, Vazquez L, Mentch F. et al. Identification of four novel loci in asthma in European American and African American Populations. Am J Respir Crit Care Med 2017; 195 (04) 456-463
    CrossrefPubMedGoogle Scholar
  • 25 Medicare Cf, Services M. Statistics NCfH. ICD-10-CM official guidelines for coding and reporting FY 2018. 2019
    PubMedGoogle Scholar
  • 26 “Global strategy for asthma management and prevention: GINA executive summary.” E.D. Bateman, S.S. Hurd, P.J. Barnes, J. Bousquet, J.M. Drazen, J.M. FitzGerald, P. Gibson, K. Ohta, P. O'Byrne, S.E. Pedersen, E. Pizzichini, S.D. Sullivan, S.E. Wenzel and H.J. Zar. Eur Respir J 2008; 31: 143-178. Eur Respir J 2018; 51 (02) 0751387
    CrossrefPubMedGoogle Scholar
  • 27 Xi N, Wallace R, Agarwal G, Chan D, Gershon A, Gupta S. Identifying patients with asthma in primary care electronic medical record systems Chart analysis-based electronic algorithm validation study. Can Fam Physician 2015; 61 (10) e474-e483
    PubMedGoogle Scholar
  • 28 Engelkes M, Afzal Z, Janssens H. et al. Automated identification of asthma patients within an electronical medical record database using machine learning. Eur Respir J 2012; 40: 4655
    PubMedGoogle Scholar
  • 29 Afzal Z, Engelkes M, Verhamme KMC. et al. Automatic generation of case-detection algorithms to identify children with asthma from large electronic health record databases. Pharmacoepidemiol Drug Saf 2013; 22 (08) 826-833
    CrossrefPubMedGoogle Scholar
  • 30 Dexheimer JW, Abramo TJ, Arnold DH. et al. Implementation and evaluation of an integrated computerized asthma management system in a pediatric emergency department: a randomized clinical trial. Int J Med Inform 2014; 83 (11) 805-813
    CrossrefPubMedGoogle Scholar
  • 31 Wu ST, Sohn S, Ravikumar KE. et al. Automated chart review for asthma cohort identification using natural language processing: an exploratory study. Ann Allergy Asthma Immunol 2013; 111 (05) 364-369
    CrossrefPubMedGoogle Scholar
  • 32 Kozyrskyj AL, HayGlass KT, Sandford AJ, Paré PD, Chan-Yeung M, Becker AB. A novel study design to investigate the early-life origins of asthma in children (SAGE study). Allergy 2009; 64 (08) 1185-1193
    CrossrefPubMedGoogle Scholar
  • 33 Vollmer WM, O'Connor EA, Heumann M. et al. Searching multiple clinical information systems for longer time periods found more prevalent cases of asthma. J Clin Epidemiol 2004; 57 (04) 392-397
    CrossrefPubMedGoogle Scholar
  • 34 Donahue JG, Weiss ST, Goetsch MA, Livingston JM, Greineder DK, Platt R. Assessment of asthma using automated and full-text medical records. J Asthma 1997; 34 (04) 273-281
    CrossrefPubMedGoogle Scholar
  • 35 Premaratne UN, Marks GB, Austin EJ, Burney PG. A reliable method to retrieve accident & emergency data stored on a free-text basis. Respir Med 1997; 91 (02) 61-66
    CrossrefPubMedGoogle Scholar
  • 36 Richesson RL, Sun J, Pathak J, Kho AN, Denny JC. Clinical phenotyping in selected national networks: demonstrating the need for high-throughput, portable, and computational methods. Artif Intell Med 2016; 71: 57-61
    CrossrefPubMedGoogle Scholar
  • 37 Aaron SD, Boulet LP, Reddel HK, Gershon AS. Underdiagnosis and overdiagnosis of asthma. Am J Respir Crit Care Med 2018; 198 (08) 1012-1020
    CrossrefPubMedGoogle Scholar