Appl Clin Inform 2023; 14(05): 932-943
DOI: 10.1055/a-2184-6481
Research Article

Effectiveness of a Vendor Predictive Model for the Risk of Pediatric Asthma Exacerbation: A Difference-in-Differences Analysis

Avinash Murugan
1   Department of Medicine, Yale New Haven Hospital, New Haven, Connecticut, United States
Swaminathan Kandaswamy
2   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
Edwin Ray
3   Information Services and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Scott Gillespie
2   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
Evan Orenstein
2   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
3   Information Services and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
› Author Affiliations


Background Asthma is a common cause of morbidity and mortality in children. Predictive models may help providers tailor asthma therapies to an individual's exacerbation risk. The effectiveness of asthma risk scores on provider behavior and pediatric asthma outcomes remains unknown.

Objective Determine the impact of an electronic health record (EHR) vendor-released model on outcomes for children with asthma.

Methods The Epic Systems Risk of Pediatric Asthma Exacerbation model was implemented on February 24, 2021, for volunteer pediatric allergy and pulmonology providers as a noninterruptive risk score visible in the patient schedule view. Asthma hospitalizations, emergency department (ED) visits, or oral steroid courses within 90 days of the index visit were compared from February 24, 2019, to February 23, 2022, using a difference-in-differences design with a control group of visits to providers in the same departments. Volunteer providers were interviewed to identify barriers and facilitators to model use.

Results In the intervention group, asthma hospitalizations within 90 days decreased from 1.4% (54/3,842) to 0.7% (14/2,165) after implementation with no significant change in the control group (0.9% [171/19,865] preimplementation to 1.0% [105/10,743] post). ED visits in the intervention group decreased from 5.8% (222/3,842) to 5.5% (118/2,164) but increased from 5.5% (1,099/19,865) to 6.8% (727/10,743) in the control group. The adjusted difference-in-differences estimators for hospitalization, ED visit, and oral steroid outcomes were −0.9% (95% confidence interval [CI]: −1.6 to −0.3), –2.4% (−3.9 to −0.8), and –1.9% (−4.3 to 0.5). In qualitative analysis, providers understood the purpose of the model and felt it was useful to flag high exacerbation risk. Trust in the model was calibrated against providers' own clinical judgement.

Conclusion This EHR vendor model implementation was associated with a significant decrease in asthma hospitalization and ED visits within 90 days of pediatric allergy and pulmonology clinic visits, but not oral steroid courses.

Protection of Human and Animal Subjects

This project was deemed to be nonhuman subjects research as a quality improvement initiative by the Children's Healthcare of Atlanta IRB (identifier: STUDY00000905).

Authors' Contributions

A.M. and E.O. conceptualized the study design.

A.M., S.K., E.R., and E.O. implemented the study.

A.M., S.G., and E.O. designed and performed the statistical analysis.

A.M., S.K., E.R., S.G., and E.O. wrote and edited the manuscript.

Publication History

Received: 13 June 2023

Accepted: 28 September 2023

Accepted Manuscript online:
29 September 2023

Article published online:
29 November 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Ferrante G, La Grutta S. The burden of pediatric asthma. Front Pediatr 2018; 6: 186
  • 2 Perry R, Braileanu G, Palmer T, Stevens P. The economic burden of pediatric asthma in the United States: literature review of current evidence. PharmacoEconomics 2019; 37 (02) 155-167
  • 3 Hughes HK, Matsui EC, Tschudy MM, Pollack CE, Keet CA. Pediatric asthma health disparities: race, hardship, housing, and asthma in a national survey. Acad Pediatr 2017; 17 (02) 127-134
  • 4 Avery C, Perrin EM, Lang JE. Updates to the pediatrics asthma management guidelines. JAMA Pediatr 2021; 175 (09) 966-967
  • 5 Schuler M, Faller H, Wittmann M, Schultz K. Asthma Control Test and Asthma Control Questionnaire: factorial validity, reliability and correspondence in assessing status and change in asthma control. J Asthma 2016; 53 (04) 438-445
  • 6 Jia CE, Zhang HP, Lv Y. et al. The Asthma Control Test and Asthma Control Questionnaire for assessing asthma control: systematic review and meta-analysis. J Allergy Clin Immunol 2013; 131 (03) 695-703
  • 7 Tsai CL, Clark S, Camargo Jr CA. Risk stratification for hospitalization in acute asthma: the CHOP classification tree. Am J Emerg Med 2010; 28 (07) 803-808
  • 8 Lieu TA, Quesenberry CP, Sorel ME, Mendoza GR, Leong AB. Computer-based models to identify high-risk children with asthma. Am J Respir Crit Care Med 1998; 157 (4, Pt 1): 1173-1180
  • 9 Hanson JR, Lee BR, Williams DD. et al. Developing a risk stratification model for predicting future health care use in asthmatic children. Ann Allergy Asthma Immunol 2016; 116 (01) 26-30
  • 10 Martin A, Bauer V, Datta A. et al. Development and validation of an asthma exacerbation prediction model using electronic health record (EHR) data. J Asthma 2020; 57 (12) 1339-1346
  • 11 Niu C, Xu Y, Schuler CL. et al. Evaluation of risk scores to predict pediatric severe asthma exacerbations. J Allergy Clin Immunol Pract 2021; 9 (12) 4393-4401.e8
  • 12 Patel D, Hall GL, Broadhurst D, Smith A, Schultz A, Foong RE. Does machine learning have a role in the prediction of asthma in children?. Paediatr Respir Rev 2022; 41: 51-60
  • 13 Seol HY, Shrestha P, Muth JF. et al. Artificial intelligence-assisted clinical decision support for childhood asthma management: a randomized clinical trial. PLoS One 2021; 16 (08) e0255261
  • 14 Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med 2018; 178 (11) 1544-1547
  • 15 Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366 (6464): 447-453
  • 16 Wong A, Otles E, Donnelly JP. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. [published correction appears in JAMA Intern Med. 2021 Aug 1;181(8):1144] JAMA Intern Med 2021; 181 (08) 1065-1070
  • 17 Lu JH, Callahan A, Patel BS, Morse KE, Dash D, Pfeffer MA, Shah NH. Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor: A Systematic Review. JAMA Netw Open 2022; 5 (08) e2227779
  • 18 Galaxy - cognitive computing model brief: risk of pediatric asthma exacerbation [Internet]. Accessed November 15, 2022 at:!68!95!100041319
  • 19 Colonescu C. 7.7 The difference-in-differences estimator [Internet]. In: Principles of Econometrics with R. 2016. Accessed November 15, 2022 at:
  • 20 Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4: 50
  • 21 Mcdermott P, Dominguez C, Kasdaglis N, Ryan M, Trahan I, Nelson A. Human-machine teaming systems engineering guide | MITRE [Internet]. 2018. Accessed November 15, 2022 at:
  • 22 Major VJ, Jones SA, Razavian N. et al. Evaluating the effect of a COVID-19 predictive model to facilitate discharge: a randomized controlled trial. Appl Clin Inform 2022; 13 (03) 632-640
  • 23 Wu AC, Tantisira K, Li L, Schuemann B, Weiss ST, Fuhlbrigge AL. Childhood Asthma Management Program Research Group. Predictors of symptoms are different from predictors of severe exacerbations from asthma in children. Chest 2011; 140 (01) 100-107
  • 24 Suruki RY, Daugherty JB, Boudiaf N, Albers FC. The frequency of asthma exacerbations and healthcare utilization in patients with asthma from the UK and USA. BMC Pulm Med 2017; 17 (01) 74
  • 25 Correa-Agudelo E, Ding L, Beck AF. et al. Understanding racial disparities in childhood asthma using individual- and neighborhood-level risk factors. J Allergy Clin Immunol 2022; 150 (06) 1427-1436.e5
  • 26 Kaufmann J, Marino M, Lucas J. et al. Racial and ethnic disparities in acute care use for pediatric asthma. Ann Fam Med 2022; 20 (02) 116-122
  • 27 Schatz M, Clark S, Emond JA, Schreiber D, Camargo Jr CA. Sex differences among children 2-13 years of age presenting at the emergency department with acute asthma. Pediatr Pulmonol 2004; 37 (06) 523-529
  • 28 Plombon S, S Rudin R, Sulca Flores J. et al. Assessing equitable recruitment in a digital health trial for asthma. Appl Clin Inform 2023; 14 (04) 620-631
  • 29 Gao E, Radparvar I, Dieu H, Ross MK. User experience design for adoption of asthma clinical decision support tools. Appl Clin Inform 2022; 13 (04) 971-982
  • 30 Tarabichi Y, Cheng A, Bar-Shain D. et al. Improving timeliness of antibiotic administration using a provider and pharmacist facing sepsis early warning system in the emergency department setting: a randomized controlled quality improvement initiative. Crit Care Med 2022; 50 (03) 418-427
  • 31 Li RC, Asch SM, Shah NH. Developing a delivery science for artificial intelligence in healthcare. NPJ Digit Med 2020; 3: 107