Subscribe to RSS
Accuracy of Asthma Computable Phenotypes to Identify Pediatric Asthma at an Academic InstitutionFunding Javier Sanz and the UCLA Clinical and Translational Science Institute (CTSI) Data Core; Dr. Sleiman, CHOP. The authors would like to acknowledge the UCLA CTSI (UL TR001881) and the Institute for Precision Health (IPH) for their support in this project.
Objectives Asthma is a heterogenous condition with significant diagnostic complexity, including variations in symptoms and temporal criteria. The disease can be difficult for clinicians to diagnose accurately. Properly identifying asthma patients from the electronic health record is consequently challenging as current algorithms (computable phenotypes) rely on diagnostic codes (e.g., International Classification of Disease, ICD) in addition to other criteria (e.g., inhaler medications)—but presume an accurate diagnosis. As such, there is no universally accepted or rigorously tested computable phenotype for asthma.
Methods We compared two established asthma computable phenotypes: the Chicago Area Patient-Outcomes Research Network (CAPriCORN) and Phenotype KnowledgeBase (PheKB). We established a large-scale, consensus gold standard (n = 1,365) from the University of California, Los Angeles Health System's clinical data warehouse for patients 5 to 17 years old. Results were manually reviewed and predictive performance (positive predictive value [PPV], sensitivity/specificity, F1-score) determined. We then examined the classification errors to gain insight for future algorithm optimizations.
Results As applied to our final cohort of 1,365 expert-defined gold standard patients, the CAPriCORN algorithms performed with a balanced PPV = 95.8% (95% CI: 94.4–97.2%), sensitivity = 85.7% (95% CI: 83.9–87.5%), and harmonized F1 = 90.4% (95% CI: 89.2–91.7%). The PheKB algorithm was performed with a balanced PPV = 83.1% (95% CI: 80.5–85.7%), sensitivity = 69.4% (95% CI: 66.3–72.5%), and F1 = 75.4% (95% CI: 73.1–77.8%). Four categories of errors were identified related to method limitations, disease definition, human error, and design implementation.
Conclusion The performance of the CAPriCORN and PheKB algorithms was lower than previously reported as applied to pediatric data (PPV = 97.7 and 96%, respectively). There is room to improve the performance of current methods, including targeted use of natural language processing and clinical feature engineering.
A waiver of consent was obtained from the UCLA Institutional Review Board (IRB) committee (IRB #18–002015).
# Co-first authors.
Received: 21 December 2020
Accepted: 15 April 2021
14 July 2021 (online)
© 2021. Thieme. All rights reserved.
Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany
- 1 Shivade C, Raghavan P, Fosler-Lussier E. et al. A review of approaches to identifying patient phenotype cohorts using electronic health records. J Am Med Inform Assoc 2014; 21 (02) 221-230
- 2 Shang N, Liu C, Rasmussen LV. et al. Making work visible for electronic phenotype implementation: lessons learned from the eMERGE network. J Biomed Inform 2019; 99: 103293
- 3 Mo H, Thompson WK, Rasmussen LV. et al. Desiderata for computable representations of electronic health records-driven phenotype algorithms. J Am Med Inform Assoc 2015; 22 (06) 1220-1230
- 4 Kirby JC, Speltz P, Rasmussen LV. et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 2016; 23 (06) 1046-1052
- 5 NCQA. National Centers for Quality Assurance. HEDIS and Performance Measurement. Accessed December 6, 2020 at: https://www.ncqa.org/hedis/
- 6 Centers for Medicare and Medicaid Services. Chronic Conditions Data Warehouse. Accessed December 6, 2020 at: http://www.ccwdata.org/index.htm
- 7 AHRQ. Agency for Healthcare Research and Quality. Healthcare Cost and Utilization Project.
- 8 Thompson WK, Rasmussen LV, Pacheco JA. et al. An evaluation of the NQF Quality Data Model for representing electronic health record driven phenotyping algorithms. AMIA Annu Symp Proc 2012; 2012: 911-920
- 9 Jiang G, Kiefer RC, Rasmussen LV. et al. Developing a data element repository to support EHR-driven phenotype algorithm authoring and execution. J Biomed Inform 2016; 62: 232-242
- 10 Pletcher MJ, Forrest CB, Carton TW. PCORnet's collaborative research groups. Patient Relat Outcome Meas 2018; 9: 91-95
- 11 OHDSI. Observational Health Data Sciences and Informatics. OHDSI Gold Standard Phenotype Library Architecture.. Accessed January 6, 2020 at: https://www.ohdsi.org/2019-us-symposium-showcase-105/
- 12 Pacheco JA, Avila PC, Thompson JA. et al. A highly specific algorithm for identifying asthma cases and controls for genome-wide association studies. AMIA Annu Symp Proc 2009; 2009: 497-501
- 13 Geva A, Gronsbell JL, Cai T. et al; Pediatric Pulmonary Hypertension Network and National Heart, Lung, and Blood Institute Pediatric Pulmonary Vascular Disease Outcomes Bioinformatics Clinical Coordinating Center Investigators. A computable phenotype improves cohort ascertainment in a pediatric pulmonary hypertension registry. J Pediatr 2017; 188: 224-231.e5
- 14 Kashyap R, Sarvottam K, Wilson GA, Jentzer JC, Seisa MO, Kashani KB. Derivation and validation of a computable phenotype for acute decompensated heart failure in hospitalized patients. BMC Med Inform Decis Mak 2020; 20 (01) 85
- 15 Trivedi M, Denton E. Asthma in children and adults—what are the differences and what can they tell us about asthma?. Front Pediatr 2019; 7: 256
- 16 Amado MC, Portnoy JM. Diagnosing asthma in young children. Curr Opin Allergy Clin Immunol 2006; 6 (02) 101-105
- 17 National Asthma Education and Prevention Program. Expert panel report 3 (EPR-3): guidelines for the diagnosis and management of asthma—summary report 2007. J Allergy Clin Immunol 2007; 120 (Suppl. 05) S94-S138
- 18 Reddel HK, FitzGerald JM, Bateman ED. et al. GINA 2019: a fundamental change in asthma management: treatment of asthma with short-acting bronchodilators alone is no longer recommended for adults and adolescents. Eur Respir J 2019; 53 (06) 1901046
- 19 Cabana MD, Slish KK, Nan B, Clark NM. Limits of the HEDIS criteria in determining asthma severity for children. Pediatrics 2004; 114 (04) 1049-1055
- 20 CDC. Centers for Disease Control. Most Recent National Asthma Data. 2017 . Available at: https://www.cdc.gov/asthma/most_recent_national_asthma_data.htm
- 21 Nurmagambetov T, Kuwahara R, Garbe P. The economic burden of asthma in the United States, 2008-2013. Ann Am Thorac Soc 2018; 15 (03) 348-356
- 22 Collaco JM, Abman SH. Evolving challenges in pediatric pulmonary medicine. new opportunities to reinvigorate the field. Am J Respir Crit Care Med 2018; 198 (06) 724-729
- 23 Afshar M, Press VG, Robison RG. et al. A computable phenotype for asthma case identification in adult and pediatric patients: external validation in the Chicago Area Patient-Outcomes Research Network (CAPriCORN). J Asthma 2018; 55: 1035-1042
- 24 Almoguera B, Vazquez L, Mentch F. et al. Identification of four novel loci in asthma in European American and African American Populations. Am J Respir Crit Care Med 2017; 195 (04) 456-463
- 25 Medicare Cf, Services M. Statistics NCfH. ICD-10-CM official guidelines for coding and reporting FY 2018. 2019
- 26 “Global strategy for asthma management and prevention: GINA executive summary.” E.D. Bateman, S.S. Hurd, P.J. Barnes, J. Bousquet, J.M. Drazen, J.M. FitzGerald, P. Gibson, K. Ohta, P. O'Byrne, S.E. Pedersen, E. Pizzichini, S.D. Sullivan, S.E. Wenzel and H.J. Zar. Eur Respir J 2008; 31: 143-178. Eur Respir J 2018; 51 (02) 0751387
- 27 Xi N, Wallace R, Agarwal G, Chan D, Gershon A, Gupta S. Identifying patients with asthma in primary care electronic medical record systems Chart analysis-based electronic algorithm validation study. Can Fam Physician 2015; 61 (10) e474-e483
- 28 Engelkes M, Afzal Z, Janssens H. et al. Automated identification of asthma patients within an electronical medical record database using machine learning. Eur Respir J 2012; 40: 4655
- 29 Afzal Z, Engelkes M, Verhamme KMC. et al. Automatic generation of case-detection algorithms to identify children with asthma from large electronic health record databases. Pharmacoepidemiol Drug Saf 2013; 22 (08) 826-833
- 30 Dexheimer JW, Abramo TJ, Arnold DH. et al. Implementation and evaluation of an integrated computerized asthma management system in a pediatric emergency department: a randomized clinical trial. Int J Med Inform 2014; 83 (11) 805-813
- 31 Wu ST, Sohn S, Ravikumar KE. et al. Automated chart review for asthma cohort identification using natural language processing: an exploratory study. Ann Allergy Asthma Immunol 2013; 111 (05) 364-369
- 32 Kozyrskyj AL, HayGlass KT, Sandford AJ, Paré PD, Chan-Yeung M, Becker AB. A novel study design to investigate the early-life origins of asthma in children (SAGE study). Allergy 2009; 64 (08) 1185-1193
- 33 Vollmer WM, O'Connor EA, Heumann M. et al. Searching multiple clinical information systems for longer time periods found more prevalent cases of asthma. J Clin Epidemiol 2004; 57 (04) 392-397
- 34 Donahue JG, Weiss ST, Goetsch MA, Livingston JM, Greineder DK, Platt R. Assessment of asthma using automated and full-text medical records. J Asthma 1997; 34 (04) 273-281
- 35 Premaratne UN, Marks GB, Austin EJ, Burney PG. A reliable method to retrieve accident & emergency data stored on a free-text basis. Respir Med 1997; 91 (02) 61-66
- 36 Richesson RL, Sun J, Pathak J, Kho AN, Denny JC. Clinical phenotyping in selected national networks: demonstrating the need for high-throughput, portable, and computational methods. Artif Intell Med 2016; 71: 57-61
- 37 Aaron SD, Boulet LP, Reddel HK, Gershon AS. Underdiagnosis and overdiagnosis of asthma. Am J Respir Crit Care Med 2018; 198 (08) 1012-1020