Download PDF
Appl Clin Inform 2013; 04(03): 376-391
DOI: 10.4338/ACI-2013-04-RA-0029
Research Article
Schattauer GmbH

Comparing predictions made by a prediction model, clinical score, and physicians

Pediatric asthma exacerbations in the emergency department
K.J. Farion
1   Division of Emergency Medicine, Children’s Hospital of Eastern Ontario, 401 Smyth Rd., Ottawa, Ontario, K1H 8L1 Canada
2   Departments of Pediatrics and Emergency Medicine, University of Ottawa, 401 Smyth Rd., Ottawa, Ontario, K1H 8L1 Canada
3   MET Research Group, Telfer School of Management, University of Ottawa, 55 Laurier Ave. E., Ottawa, Ontario, K1N 6N5 Canada
,
S. Wilk
3   MET Research Group, Telfer School of Management, University of Ottawa, 55 Laurier Ave. E., Ottawa, Ontario, K1N 6N5 Canada
4   Institute of Computing Science, Poznan University of Technology, Piotrowo 2, 60–965 Poznan, Poland
,
W. Michalowski
3   MET Research Group, Telfer School of Management, University of Ottawa, 55 Laurier Ave. E., Ottawa, Ontario, K1N 6N5 Canada
,
D. O’Sullivan
3   MET Research Group, Telfer School of Management, University of Ottawa, 55 Laurier Ave. E., Ottawa, Ontario, K1N 6N5 Canada
5   Center for Health Informatics, School of Informatics, City University London, Northampton Square, London, EC1V 0HB, United Kingdom
,
J. Sayyad-Shirabad
6   School of Electrical Engineering and Computer Science, University of Ottawa, 800 King Edward Ave., Ottawa, Ontario, K1N 6N5, Canada
› Author Affiliations
Further Information

Publication History

received: 27 April 2013

accepted: 19 July 2013

Publication Date:
20 December 2017 (online)

   Permissions and Reprints

Summary

Background: Asthma exacerbations are one of the most common medical reasons for children to be brought to the hospital emergency department (ED). Various prediction models have been proposed to support diagnosis of exacerbations and evaluation of their severity.

Objectives: First, to evaluate prediction models constructed from data using machine learning techniques and to select the best performing model. Second, to compare predictions from the selected model with predictions from the Pediatric Respiratory Assessment Measure (PRAM) score, and predictions made by ED physicians.

Design: A two-phase study conducted in the ED of an academic pediatric hospital. In phase 1 data collected prospectively using paper forms was used to construct and evaluate five prediction models, and the best performing model was selected. In phase 2 data collected prospectively using a mobile system was used to compare the predictions of the selected prediction model with those from PRAM and ED physicians.

Measurements: Area under the receiver operating characteristic curve and accuracy in phase 1; accuracy, sensitivity, specificity, positive and negative predictive values in phase 2.

Results: In phase 1 prediction models were derived from a data set of 240 patients and evaluated using 10-fold cross validation. A naive Bayes (NB) model demonstrated the best performance and it was selected for phase 2. Evaluation in phase 2 was conducted on data from 82 patients. Predictions made by the NB model were less accurate than the PRAM score and physicians (accuracy of 70.7%, 73.2% and 78.0% respectively), however, according to McNemar’s test it is not possible to conclude that the differences between predictions are statistically significant.

Conclusion: Both the PRAM score and the NB model were less accurate than physicians. The NB model can handle incomplete patient data and as such may complement the PRAM score. However, it requires further research to improve its accuracy.

Keywords

Asthma - pediatrics - emergency medical services - decision support techniques
 

  • References

  • 1 Dexheimer JW, Abramo TJ, Arnold DH. et al. An asthma management system in a pediatric emergency department.. Int J Med Inform 2013; Apr 82 (Suppl. 04) 230-238.
    CrossrefPubMedGoogle Scholar
  • 2 National Asthma Education and Prevention Program.. Expert Panel Report 3 (EPR-3): Guidelines for the Diagnosis and Management of Asthma-Summary Report 2007.. J Allergy Clin Immunol 2007; 120 (Suppl. 05) S94-S138.
    CrossrefPubMedGoogle Scholar
  • 3 Chu S, Tan J, Seabrook JA. et al. Paediatric asthma severity score and length of stay in patients presenting to a paediatric emergency department.. Paediatr Perinat Drug Ther 2008; 8 (04) 150-153.
    CrossrefPubMedGoogle Scholar
  • 4 Zemek R, Plint A, Osmond MH. et al. Triage nurse initiation of corticosteroids in pediatric asthma is associated with improved emergency department efficiency.. Pediatrics 2012; 129 (04) 671-680.
    CrossrefPubMedGoogle Scholar
  • 5 Sanders DL, Aronsky D. Biomedical informatics applications for asthma care: a systematic review.. J Am Med Inform Assoc 2006; 13 (04) 418-427.
    CrossrefPubMedGoogle Scholar
  • 6 Caruana R, Niculescu-Mizil A. An empirical comparison of supervised learning algorithms.. Proceedings of the 23rd International Conference on Machine Learning.. ACM, Pittsburgh, PA; 2006: 161-168.
    PubMedGoogle Scholar
  • 7 Lim T-S, Loh W-Y, Shih Y-S. A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms.. Mach Learn 2000; 40 (03) 203-228.
    CrossrefPubMedGoogle Scholar
  • 8 Perlich C, Provost F, Simonoff JS. Tree induction vs. logistic regression: A learning-curve analysis.. J Mach Learn Res 2003; 4: 211-255.
    PubMedGoogle Scholar
  • 9 Xue J-H, Titterington DM. Comment on ,,On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes“.. Neural Process Lett 2008; 28 (03) 169-187.
    CrossrefPubMedGoogle Scholar
  • 10 Farion K, Michalowski W, Wilk S. et al. A tree-based decision model to support prediction of the severity of asthma exacerbations in children.. J Med Syst 2010; 34 (04) 551-562.
    CrossrefPubMedGoogle Scholar
  • 11 Dexheimer JW, Brown LE, Leegon J. et al. Comparing decision support methodologies for identifying asthma exacerbations.. Stud Health Technol Inform 2007; 129 Pt 2 880-884.
    PubMedGoogle Scholar
  • 12 Sanders DL, Aronsky D. Prospective evaluation of a Bayesian Network for detecting asthma exacerbations in a Pediatric Emergency Department.. AMIA Annu Symp Proc; 2006: 1085.
    PubMedGoogle Scholar
  • 13 Ducharme FM, Chalut D, Plotnick L. et al. The Pediatric Respiratory Assessment Measure: a valid clinical score for assessing acute asthma severity from toddlers to teenagers.. J Pediatr 2008; 152 (04) 476-480.
    CrossrefPubMedGoogle Scholar
  • 14 Sanders DL, Aronsky D. Detecting asthma exacerbations in a pediatric emergency department using a Bayesian network.. AMIA Annu Symp Proc 2006: 684-688.
    PubMedGoogle Scholar
  • 15 Sanders DL, Gregg W, Aronsky D. Identifying asthma exacerbations in a pediatric emergency department: a feasibility study.. Int J Med Inform 2007; 76 (07) 557-564.
    CrossrefPubMedGoogle Scholar
  • 16 Powell CV, Kelly AM, Kerr D. Lack of agreement in classification of the severity of acute asthma between emergency physician assessment and classification using the National Asthma Council Australia guidelines (1998).. Emerg Med 2003; 15 (01) 49-53.
    CrossrefPubMedGoogle Scholar
  • 17 Eccles M, McColl E, Steen N. et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.. BMJ 2002; 325 7370 941.
    CrossrefPubMedGoogle Scholar
  • 18 Hoeksema LJ, Bazzy-Asaad A, Lomotan EA. et al. Accuracy of a computerized clinical decision-support system for asthma assessment and management.. J Am Med Inform Assoc 2011; 18 (03) 243-250.
    CrossrefPubMedGoogle Scholar
  • 19 Birken CS, Parkin PC, Macarthur C. Asthma severity scores for preschoolers displayed weaknesses in reliability, validity, and responsiveness.. J Clin Epidemiol 2004; 57 (11) 1177-1181.
    CrossrefPubMedGoogle Scholar
  • 20 Gorelick MH, Stevens MW, Schultz TR. et al. Performance of a novel clinical score, the Pediatric Asthma Severity Score (PASS), in the evaluation of acute asthma.. Acad Emerg Med 2004; 11 (01) 10-8.
    CrossrefPubMedGoogle Scholar
  • 21 Gouin S, Robidas I, Gravel J. et al. Prospective evaluation of two clinical scores for acute asthma in children 18 months to 7 years of age.. Acad Emerg Med 2010; 17 (06) 598-603.
    CrossrefPubMedGoogle Scholar
  • 22 Zolnoori M, Zarandi MH, Moin M. Application of intelligent systems in asthma disease: designing a fuzzy rule-based system for evaluating level of asthma exacerbation.. J Med Syst 2012; 36 (04) 2071-283.
    CrossrefPubMedGoogle Scholar
  • 23 Sefion I, Ennaji A, Gailhardou M. et al. ADEMA: a decision support system for asthma health care.. Stud Health Technol Inform 2003; 95: 623-628.
    PubMedGoogle Scholar
  • 24 Bellazzi R, Zupan B. Predictive data mining in clinical medicine: current issues and guidelines.. Int J Med Inform 2008; 77 (02) 81-97.
    CrossrefPubMedGoogle Scholar
  • 25 Weka 3: Data Mining Software in Java [cited 2013 Jun 23]. Available from: http://www.cs.waikato.ac.nzml/weka/.
    PubMed
  • 26 Witten IH, Frank E, Hall MA. Data mining: practical machine learning tools and techniques.. 3rd ed. Morgan Kaufmann; 2011
    PubMedGoogle Scholar
  • 27 Dietterich TG. Approximate statistical test for comparing supervised classification learning algorithms.. Neural Comput 1998; 10 (07) 1895-1923.
    CrossrefPubMedGoogle Scholar
  • 28 The R Project for Statistical Computing [cited 2013 Jun 23]. Available from: http://www.r-project.org.
    PubMed
  • 29 Chalut DS, Ducharme FM, Davis GM. The Preschool Respiratory Assessment Measure (PRAM): a responsive index of acute asthma severity.. J Pediatr 2000; 137 (06) 762-768.
    CrossrefPubMedGoogle Scholar
  • 30 Wilk S, Michalowski W, Farion K. et al. MET3-AE system to support management of pediatric asthma exacerbation in the emergency department.. Stud Health Technol Inform 2010; 160 Pt 2 841-845.
    PubMedGoogle Scholar
  • 31 Wilk S, Michalowski W, O’Sullivan D. et al. A task-based support architecture for developing point-of-care clinical decision support systems for the emergency department.. Methods Inf Med 2013; 52 (01) 18-32.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 32 Sayyad Shirabad J, Wilk S, Michalowski W. et al. Implementing an integrative multi-agent clinical decision support system with open source software.. J Med Syst 2012; 36 (01) 123-137.
    CrossrefPubMedGoogle Scholar
  • 33 O’Sullivan D, Doyle JS, Michalowski W. et al. Assessing the motivation of MDs to use computer-based support at the point-of-care in the emergency department.. AMIA Annu Symp Proc 2011: 1045-54.
    PubMedGoogle Scholar
  • 34 Sajda P. Machine learning for detection and diagnosis of disease.. Annu Rev Biomed Eng 2006; 8: 537-565.
    CrossrefPubMedGoogle Scholar