Comparing predictions made by a prediction model, clinical score, and physiciansPediatric asthma exacerbations in the emergency department
27 April 2013
accepted: 19 July 2013
20 December 2017 (online)
Background: Asthma exacerbations are one of the most common medical reasons for children to be brought to the hospital emergency department (ED). Various prediction models have been proposed to support diagnosis of exacerbations and evaluation of their severity.
Objectives: First, to evaluate prediction models constructed from data using machine learning techniques and to select the best performing model. Second, to compare predictions from the selected model with predictions from the Pediatric Respiratory Assessment Measure (PRAM) score, and predictions made by ED physicians.
Design: A two-phase study conducted in the ED of an academic pediatric hospital. In phase 1 data collected prospectively using paper forms was used to construct and evaluate five prediction models, and the best performing model was selected. In phase 2 data collected prospectively using a mobile system was used to compare the predictions of the selected prediction model with those from PRAM and ED physicians.
Measurements: Area under the receiver operating characteristic curve and accuracy in phase 1; accuracy, sensitivity, specificity, positive and negative predictive values in phase 2.
Results: In phase 1 prediction models were derived from a data set of 240 patients and evaluated using 10-fold cross validation. A naive Bayes (NB) model demonstrated the best performance and it was selected for phase 2. Evaluation in phase 2 was conducted on data from 82 patients. Predictions made by the NB model were less accurate than the PRAM score and physicians (accuracy of 70.7%, 73.2% and 78.0% respectively), however, according to McNemar’s test it is not possible to conclude that the differences between predictions are statistically significant.
Conclusion: Both the PRAM score and the NB model were less accurate than physicians. The NB model can handle incomplete patient data and as such may complement the PRAM score. However, it requires further research to improve its accuracy.
- 1 Dexheimer JW, Abramo TJ, Arnold DH. et al. An asthma management system in a pediatric emergency department.. Int J Med Inform 2013; Apr 82 (Suppl. 04) 230-238.
- 2 National Asthma Education and Prevention Program.. Expert Panel Report 3 (EPR-3): Guidelines for the Diagnosis and Management of Asthma-Summary Report 2007.. J Allergy Clin Immunol 2007; 120 (Suppl. 05) S94-S138.
- 3 Chu S, Tan J, Seabrook JA. et al. Paediatric asthma severity score and length of stay in patients presenting to a paediatric emergency department.. Paediatr Perinat Drug Ther 2008; 8 (04) 150-153.
- 4 Zemek R, Plint A, Osmond MH. et al. Triage nurse initiation of corticosteroids in pediatric asthma is associated with improved emergency department efficiency.. Pediatrics 2012; 129 (04) 671-680.
- 5 Sanders DL, Aronsky D. Biomedical informatics applications for asthma care: a systematic review.. J Am Med Inform Assoc 2006; 13 (04) 418-427.
- 6 Caruana R, Niculescu-Mizil A. An empirical comparison of supervised learning algorithms.. Proceedings of the 23rd International Conference on Machine Learning.. ACM, Pittsburgh, PA; 2006: 161-168.
- 7 Lim T-S, Loh W-Y, Shih Y-S. A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms.. Mach Learn 2000; 40 (03) 203-228.
- 8 Perlich C, Provost F, Simonoff JS. Tree induction vs. logistic regression: A learning-curve analysis.. J Mach Learn Res 2003; 4: 211-255.
- 9 Xue J-H, Titterington DM. Comment on ,,On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes“.. Neural Process Lett 2008; 28 (03) 169-187.
- 10 Farion K, Michalowski W, Wilk S. et al. A tree-based decision model to support prediction of the severity of asthma exacerbations in children.. J Med Syst 2010; 34 (04) 551-562.
- 11 Dexheimer JW, Brown LE, Leegon J. et al. Comparing decision support methodologies for identifying asthma exacerbations.. Stud Health Technol Inform 2007; 129 Pt 2 880-884.
- 12 Sanders DL, Aronsky D. Prospective evaluation of a Bayesian Network for detecting asthma exacerbations in a Pediatric Emergency Department.. AMIA Annu Symp Proc; 2006: 1085.
- 13 Ducharme FM, Chalut D, Plotnick L. et al. The Pediatric Respiratory Assessment Measure: a valid clinical score for assessing acute asthma severity from toddlers to teenagers.. J Pediatr 2008; 152 (04) 476-480.
- 14 Sanders DL, Aronsky D. Detecting asthma exacerbations in a pediatric emergency department using a Bayesian network.. AMIA Annu Symp Proc 2006: 684-688.
- 15 Sanders DL, Gregg W, Aronsky D. Identifying asthma exacerbations in a pediatric emergency department: a feasibility study.. Int J Med Inform 2007; 76 (07) 557-564.
- 16 Powell CV, Kelly AM, Kerr D. Lack of agreement in classification of the severity of acute asthma between emergency physician assessment and classification using the National Asthma Council Australia guidelines (1998).. Emerg Med 2003; 15 (01) 49-53.
- 17 Eccles M, McColl E, Steen N. et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.. BMJ 2002; 325 7370 941.
- 18 Hoeksema LJ, Bazzy-Asaad A, Lomotan EA. et al. Accuracy of a computerized clinical decision-support system for asthma assessment and management.. J Am Med Inform Assoc 2011; 18 (03) 243-250.
- 19 Birken CS, Parkin PC, Macarthur C. Asthma severity scores for preschoolers displayed weaknesses in reliability, validity, and responsiveness.. J Clin Epidemiol 2004; 57 (11) 1177-1181.
- 20 Gorelick MH, Stevens MW, Schultz TR. et al. Performance of a novel clinical score, the Pediatric Asthma Severity Score (PASS), in the evaluation of acute asthma.. Acad Emerg Med 2004; 11 (01) 10-8.
- 21 Gouin S, Robidas I, Gravel J. et al. Prospective evaluation of two clinical scores for acute asthma in children 18 months to 7 years of age.. Acad Emerg Med 2010; 17 (06) 598-603.
- 22 Zolnoori M, Zarandi MH, Moin M. Application of intelligent systems in asthma disease: designing a fuzzy rule-based system for evaluating level of asthma exacerbation.. J Med Syst 2012; 36 (04) 2071-283.
- 23 Sefion I, Ennaji A, Gailhardou M. et al. ADEMA: a decision support system for asthma health care.. Stud Health Technol Inform 2003; 95: 623-628.
- 24 Bellazzi R, Zupan B. Predictive data mining in clinical medicine: current issues and guidelines.. Int J Med Inform 2008; 77 (02) 81-97.
- 25 Weka 3: Data Mining Software in Java [cited 2013 Jun 23]. Available from: http://www.cs.waikato.ac.nzml/weka/.
- 26 Witten IH, Frank E, Hall MA. Data mining: practical machine learning tools and techniques.. 3rd ed. Morgan Kaufmann; 2011
- 27 Dietterich TG. Approximate statistical test for comparing supervised classification learning algorithms.. Neural Comput 1998; 10 (07) 1895-1923.
- 28 The R Project for Statistical Computing [cited 2013 Jun 23]. Available from: http://www.r-project.org.
- 29 Chalut DS, Ducharme FM, Davis GM. The Preschool Respiratory Assessment Measure (PRAM): a responsive index of acute asthma severity.. J Pediatr 2000; 137 (06) 762-768.
- 30 Wilk S, Michalowski W, Farion K. et al. MET3-AE system to support management of pediatric asthma exacerbation in the emergency department.. Stud Health Technol Inform 2010; 160 Pt 2 841-845.
- 31 Wilk S, Michalowski W, O’Sullivan D. et al. A task-based support architecture for developing point-of-care clinical decision support systems for the emergency department.. Methods Inf Med 2013; 52 (01) 18-32.
- 32 Sayyad Shirabad J, Wilk S, Michalowski W. et al. Implementing an integrative multi-agent clinical decision support system with open source software.. J Med Syst 2012; 36 (01) 123-137.
- 33 O’Sullivan D, Doyle JS, Michalowski W. et al. Assessing the motivation of MDs to use computer-based support at the point-of-care in the emergency department.. AMIA Annu Symp Proc 2011: 1045-54.
- 34 Sajda P. Machine learning for detection and diagnosis of disease.. Annu Rev Biomed Eng 2006; 8: 537-565.