Comparison of illness severity scoring systems for mortality prediction in Neurointensive Care Unit in India

Abstract

Background: Illness severity scoring systems (SSs) are increasingly being used to provide information about patients’ severity of illness and outcome in terms of mortality or length of Intensive care Unit (ICU) and hospital stay. In this retrospective study, we compared the predictive power of Acute Physiology and Chronic Health Evaluation (APACHE) II and IV, Simplified Acute Physiology Score (SAPS), Mortality Prediction Model at 24 h and Glasgow Coma Scale (GCS) with actual in-hospital 28 day mortality in patients admitted to neuro-ICU over a period of 6 months. Methods: The data required for calculation of above scores was retrieved from medical records. The 28-day post-admission outcome including in-hospital mortality was measured by Glasgow Outcome Scale (GOS). Logistic regression was used to determine the mortality prediction power of each SS. Results: A total of 197 adult patients with varied neurological diagnosis were included in this study. The in-hospital 28-day mortality rate was 19.8%, and the scores of all the SSs correlated significantly with GOS (P < 0.001). All the scores were significantly different between survivors and non-survivors. The accuracy of all the SSs to predict survival and non-survival was more than 80%. The highest accuracy rate was seen for GCS and SAPS (84.3% and 83.8%, respectively). Conclusions: The SSs used in this study had good predictive power, and they had good discriminative ability between survivors and non-survivors. GCS and SAPS have the highest predictive ability, GCS having added advantage of being simple and practical.

Keywords

Acute Physiology and Chronic Health Evaluation II and IV - Glasgow Coma Scale - mortality prediction model - Neurointensive Care Unit - scoring system - Simplified Acute Physiology Score

INTRODUCTION

Illness severity scoring systems (SSs) are used to provide information about the patients’ severity of illness, to facilitate resource allocation, for comparing outcomes in different Intensive Care Units (ICUs) and to predict possible patient outcomes in terms of mortality and ICU/hospital length of stay (LOS). Recently, the utility of the illness severity SS to predict outcome in ICUs has increased. These scorings systems are derived from patients’ characteristics, physiologic measurements, severity of acute illness, chronic health status and the intensity of treatment received in ICU. The majority of the SS require a substantial amount of effort for data collection, especially if it is done manually. SSs, which are commonly used, include Acute Physiology and Chronic Health Evaluation (APACHE) II and IV, Mortality Prediction Model at 24 h (MPM₂₄) and Simplified Acute Physiology Score (SAPS). Furthermore, using these models, alternative models have been devised to cater to the specific patient population.[1]

Glasgow Coma Scale (GCS) is commonly used in neuro-ICU (NICU) to describe the level of patient consciousness after brain injury.[2] It has been shown to have high inter-rater reliability even amongst registered nurses.[3] Although described initially for a head injury, it is generally applied to all types of neurosurgical population. Most SSs use GCS as one of the parameters in their model. Numerous studies have demonstrated a correlation between the GCS score on admission and long-term patient outcome.[4]

In this study, we wanted to assess the predictive power of GCS when compared to other SSs. We routinely use only GCS for neurological assessment and prognostication of our patients in NICU as it is simple and easy to use in an ICU like ours, where there is a high turnover of patients. Moreover, neurological patients sometimes require augmentation of their systemic blood pressure with inotropes even if they are normotensive, in situations such as vasospasm after aneurysmal subarachnoid haemorrhage or in traumatic brain injury (TBI) to maintain their cerebral perfusion pressure. The use of SS in such scenarios may cause a bias towards the unfavourable outcome. In addition, respiratory insufficiency and tachypnoea can occur as a result of neurological insult, without any respiratory system pathology, thereby altering the information from SS. Moreover, we do not have electronic data entry in our NICU, which makes the calculation of APACHE or other SS tedious and time consuming. Therefore, we wanted to test whether GCS alone can be used for predicting patients’ prognosis in our NICU and other similar centres in our country or whether we need to apply other SS to evaluate patient prognosis. This retrospective study was designed to compare the predictive power of APACHE II and IV, SAPS, MPM₂₄ and GCS with the actual in-hospital 28-day mortality of neurologically ill patients (both surgical and non-surgical) admitted to our NICU over a period of 6 months. In addition, we compared these SSs between patients who had good outcome and poor outcome based on Glasgow Outcome Scale (GOS). We hypothesised that predictive power of GCS is comparable to other SSs, in spite of the fact that it accounts for only neurological status and does not take into consideration any systemic illnesses.

SUBJECTS AND METHODS

The study was approved by the Institute Ethics Committee, and waiver of patient consent was obtained.

Medical records of all patients ≥18 years of age admitted to NICU from April 2012 to September 2012 were used for data collection. Exclusion criteria included pregnant patients and any patient who got discharged from ICU within 6 h of admission. The yearly turnover is around 500 patients. We mainly use GCS in our ICU for prognostication. Our nurse to patient ratio is usually 3:1 and functions as a closed unit under the full medical responsibility of the ICU staff in close concert with the referring medical specialists.

The following data were collected for each patient:

1. Characteristics: Age, sex, elective or emergency admission, any surgery done before ICU admission, any surgical service provided within first 24 h of ICU admission

2. Chronic health status: The Presence or absence of chronic renal failure, liver insufficiency, underlying malignancy, immunosuppression, any admission to ICU in the previous 6 months (defined in APACHE II SS). Patients with brain oedema are usually treated with steroids (4 mg 6^th hourly) as an anti-oedema measure and hence, such patients were considered as immune-compromised (dose >0.2 mg/kg/day)

3. Physiologic measurements (values recorded in the first 24 h of admission to ICU): The lowest and the highest values of the following: GCS, systolic blood pressure (SBP), diastolic blood pressure (DBP), heart rate, respiratory rate including both spontaneous and controlled breaths in intubated patients, temperature, urine output

4. Laboratory investigations (within first 24 h): Serum sodium, potassium, blood urea, (blood urea nitrogen was calculated as blood urea/3), serum creatinine, blood glucose, haematocrit, total leucocyte count, platelet count, arterial pH, partial pressure of oxygen in arterial blood_, fraction of inspired oxygen, partial pressure of carbon dioxide in arterial blood, blood bicarbonate levels and prothrombin time (PT)

5. Intensity of treatment: Number of invasive lines in place (endotracheal tube, arterial line, urinary catheter, central venous catheter, peripheral intravenous access, external ventricular drain, lumbar drain), cardiopulmonary resuscitation (CPR) before admission to ICU, any evidence of infection at admission or subsequently proven infection in 24 h, shock in the first 24 h and total number of hours of mechanical ventilation

6. Outcome of patients: As in-hospital mortality within 28 days of hospital admission or as GOS at the end of 28 days from admission or at discharge whichever is earlier.

All the above values were fed to an online open source calculator (URL: http://www.mecriticalcare.net/icu_scores/index.php from Middle East critical care assembly website www.mecriticalcare.net) for calculating the APACHE II and IV, SAPS II and MPM₂₄ scores. As the calculator would not calculate the score, if the value of even one variable was not entered, the missing data values were substituted with normal values. Data which were typically missing were either the PT or values derived from arterial blood gas. Based on the judgement from other data, the values were substituted with normal values. The mortality rate (in percentage) predicted by different SSs was compared with the actual 28-day mortality rate of patients.

Statistical analysis

Continuous variables are described in mean ± standard deviation (SD) and categorical variables are described in numbers and percentages. Non-parametric (Spearman) correlation was used to find out the relation between the scores of different SSs and GOS. A logistic regression was used for each SS to determine if the SS could be used to predict survival or death. Mann-Whitney test was used to compare the illness severity SSs between survivors and non-survivors and between those with favourable and unfavourable outcome. Model discrimination was analysed using the area under the curve (AUC). Discrimination was classified as perfect, excellent, very good, good, moderate and poor if the AUCs, were 1.0, 0.9–0.99, 0.8–0.89, 0.7–0.79, 0.6–0.69 or 0.6, respectively.[5] Calibration for each model was assessed using the Hosmer–Lemeshow goodness-of-fit C statistic. A non-significant P value was considered evidence of good calibration. Individually, each variable was again compared between survivors and non-survivors using Chi-square test and the variables which were significant were entered into multivariate analysis. Data analyses were performed using SPSS software for windows (SPSS Inc. Version 15.0., Chicago, IL, USA).

RESULTS

A total of 197 adult patients were admitted to the NICU during the study. The mean age of our study population was 44.9 ± 14.8 years. The disease pattern was as follows: 63 patients (32%) had cerebral vascular anomalies, 56 (28.42%) had intracranial tumours, 25 (12.7%) had TBI, 21 (10.65%) had Guillain–Barre syndrome (GBS) and myasthenia gravis, 10 (5%) had spine pathology, 7 (3%) had stroke/cortical venous thrombosis and 15 (7.6%) had miscellaneous diagnosis. Of the 10,835 variables required for the calculation, 374 variables were missing and were substituted with normal values. The mean value (SD) for each illness severity SS is given in [Table 1]. The 28-day mortality rate was 19.8% (GOS 1). [Table 2] shows the GOS of the study population. Mortality rate predicted by the SSs correlated negatively with GOS [Table 3], while GCS had significant positive correlation with GOS. When non-survivors were compared with survivors (i.e., GOS 1 [n = 39] vs. GOS of 2–5 [n = 158]), all the illness severity scores such as GCS, APACHE II, APACHE II predicted mortality rate (PMR), APACHE IV, APACHE IV PMR, MPM PMR, SAPS II and SAPS II PMR, were significantly different between the two groups as shown in [Table 4]. Similarly, patients with unfavourable outcome (GOS 1–2; n = 58) had higher APACHE II, APACHE IV, MPM and SAPS score and PMR compared to patients with favourable outcome (GOS 3–5; n = 139) [Table 5]. [Figure 1] depicts the receiver operating characteristic curves which were constructed to determine the discriminative power of the SSs and [Table 6] gives the actual AUC values. Of the 39 patients who died, 10 patients’ outcome was correctly predicted (by logistic regression) by GCS, 8 by APACHE II and APACHE II PMR, 4 by APACHE IV, 9 by APACHE IV PMR, 8 by MPM PMR and SAPS and 12 by SAPS PMR. Similarly, of the 158 who actually survived, survival was predicted (by logistic regression) correctly for 156 patients by GCS (84.3%) and APACHE IV (81.2%), 153 by APACHE II (81.7%), APACHE II PMR (81.7%) and SAPS II PMR (83.8%), 154 by APACHE IV PMR (82.7%) and 155 by MPM PMR (82.7%) and SAPS (82.7%). Hosmer–Lemeshow statistic for all the SSs as shown in [Table 7] was >0.05 indicating that the models fit the data.

Figure 1: The receiver operating characteristic curves of Acute Physiology and Chronic Health Evaluation II and IV, Simplified Acute Physiology Score, Mortality Prediction Model at 24 h and Glasgow Coma Scale

Table 1 Mean value (SD) for each illness severity scoring system

Scoring system	Score value Mean (SD)
GCS=Glasgow Coma Scale, APACHE=Acute Physiology And Chronic Health Evaluation, MPM=Mortality Prediction Model, PMR=Predicted Mortality Rate, SAPS=Simplified Acute physiology Score, GOS=Glasgow Outcome Scale
GCS	10.96 (3.56)
APACHE II
Score	15.15 (6.87)
PMR (%)	25.08 (17.84)
APACHE IV
Score	30.56 (14.1)
PMR (%)	5.91 (7.50)
MPM PMR (%)	13.42 (20.74)
SAPS II
Score	25.13 (14.01)
PMR (%)	12.32 (17.04)
GOS	3.30 (1.48)

Table 2 Glasgow Outcome Score (GOS) of study population

GOS score	No. of patients (%)
GOS 1=Dead, GOS 2=Vegetative state, GOS 3=Severe disability, GOS 4=Moderate disability and GOS 5=Mild disability
1	39 (19.8)
2	19 (9.6)
3	42 (21.3)
4	37 (18.8)
5	60 (30.5)
Total	197 (100)

Table 3 Correlation coefficient between each scoring system and GOS

Scoring system	Correlation coefficient (with GOS)	P
*Correlation is significant at P value of 0.01 (2 tailed)
GCS	0.551	0.001
APACHE II score	−0.535	0.001
APACHE II PMR%	−0.537	0.001
APACHE IV score	−0.310	0.001
APACHE IV PMR%	−0.502	0.001
MPM PMR%	−0.459	0.001
SAPS score	−0.471	0.001
SAPS PMR%	−0.516	0.001

Table 4 Mean value (SD) of each scoring system between survivors and non survivors

Severity score	Non-survivors Mean (SD) n=39	Survivors Mean (SD) n=158	P
*P value significant at 0.05
GCS	8.28 (3.8)	11.63 (3.17)	0.001
APACHE II score	20.31 (7.36)	13.87 (6.13)	0.001
APACHE II PMR%	38.66 (22.17)	21.73 (14.88)	0.001
APACHE IV score	39.90 (18.46)	28.25 (11.77)	0.001
APACHE IV PMR%	12.17 (11.29)	4.36 (5.22)	0.001
MPM PMR%	29.72 (32.30)	9.39 (14.21)	0.001
SAPS score	35.41 (18.73)	22.59 (11.29)	0.001
SAPS PMR%	28.06 (26.93)	8.43 (10.51)	0.001

Table 5 Mean value (SD) of each scoring system between patients with favourable versus unfavourable outcome

Severity score	Unfavourable outcome Mean (SD) n=58	Favourable outcome Mean (SD) n=139	P
*P value significant at 0.05 level
GCS	8.41 (3.52)	12.03 (2.99)	0.001
APACHE II score	20.38 (6.98)	12.97 (5.54)	0.001
APACHE II PMR%	38.96 (20.9)	19.29 (12.53)	0.001
APACHE IV score	39.64 (17.7)	26.77 (10.19)	0.001
APACHE IV PMR%	11.50 (10.64)	3.57 (3.81)	0.001
MPM PMR%	27.90 (30.19)	7.37 (10.46)	0.001
SAPS score	34.60 (16.99)	21.17 (10.30)	0.001
SAPS PMR%	25.26 (24.57)	6.92 (7.98)	0.001

Table 6 Area under the curve (AUC) for each scoring system along with 95% confidence intervals

Scoring system	AUC	Standard error	Significance	95% Confidence interval
GCS	0.255	0.046	0.001	0.165-0.345
APACHE II score	0.749	0.042	0.001	0.667-0.831
APACHE II PMR	0.751	0.042	0.001	0.669-0.833
APACHE IV score	0.691	0.050	0.001	0.593-0.789
APACHE IV PMR	0.778	0.039	0.001	0.701-0.855
MPM PMR	0.762	0.041	0.001	0.682-0.842
SAPS score	0.714	0.049	0.001	0.619-0.810
SAPS PMR	0.768	0.043	0.001	0.683-0.853

Table 7 Hosmer Lemeshow Chi square value for each scoring system

Scoring system	H-L χ 2	P
GCS	6.9	0.229
APACHE II	10	0.188
APACHE II PMR	9.8	0.278
APACHE IV	10.65	0.155
APACHE IV PMR	8.26	0.408
MPM PMR	15	0.06
SAPS	11	0.204
SAPS PMR	13.314	0.101

We found that lowest SBP and DBP, type of admission (emergency), presence of shock and requirement of CPR within first 24 h of admission to ICU correlated with poor outcome (with P values 0.004, 0.001, 0.006, 0.001 and 0.001, respectively). However, when all the above variables were entered into logistic regression model, only CPR in the preceding 24 h of ICU admission turned out to be a significant factor between survivors and non-survivors with Odd’s ratio of 7.6 and Nagalkerke R ² of 0.066.

DISCUSSION

In this single centre retrospective study, we found that the mortality rate predicted by different illness severity SSs correlated negatively with GOS, that is, as the score value of APACHE II, APACHE IV, MPM and SAPS increased, patient outcome worsened and vice-versa. On the other hand, for GCS, the higher the GCS, the better was the outcome. The correlation between the scores of different SSs and mortality was significant. The AUC for the most of the SSs was more than 0.7 indicating good discriminative ability. The ability of all the SSs to predict survival and non-survival was more than 80%. The highest predictability was seen for GCS and SAPS (84.3% and 83.8%, respectively).

In NICU, not only mortality but also the functional outcome is equally important and is a determinant of quality of life. Most SS predict only mortality except GOS, which also tells about functional disability. Since we use only GCS in our NICU, we wanted to see if GCS could help in the differentiating favourable outcome and unfavourable outcome. We found that all the SS including GCS had good discriminative ability to differentiate the two groups.

A number of studies have been done in the past comparing the various SSs to predict mortality in the ICU. Cho and Wang,[6] compared APACHE III, APACHE II and GCS in patients with acute head injury for prediction of mortality and functional outcome. In their study, early hospital mortality prediction was similar between APACHE and GCS SSs. However, for late mortality prediction, APACHE II and III scored better than GCS. Though GCS was initially described to assess patients’ level of consciousness and not to predict mortality, in our study, we found that GCS had very good predictability for 28-day in-hospital mortality rate. Unlike the Cho’s study, which included only head injured patients, our study included not only patients with neurosurgical pathology (both traumatic and non-traumatic), but also patients with non-surgical diagnoses like GBS, and myasthenia gravis. In spite of this wide variety of neurological population, GCS turned out to be a significant predictor of mortality. Ting et al.,[7] compared APACHE II and SAPS II with multiple linear regression model of GCS (GCS-mr), to predict mortality in 154 neurosurgical patients. They demonstrated that the power of GCS-mr was similar to APACHE II and SAPS II in predicting mortality. The advantage of APACHE II over SAPS II is that it has special disease calibration and hence better mortality prediction with different disease groups. A retrospective study by Nejmi et al.,[8] compared APACHE II and SAPS II in 225 patients with TBI. The end point for prognostic analysis was in-hospital mortality. Although both APACHE II and SAPS II were accurate in predicting mortality, the authors observed better discriminative power with APACHE II as compared to SAPS II. However, in the current study, SAPS II had better discrimination ability. One possibility could be that in SAPS II, emergency admission is given more weightage. This is important as the target population in this study were patients with head injuries or other neurological/neurosurgical pathology who require emergency ICU care. This may be contributing to slightly better discriminative ability of SAPS II as compared to APACHE II in our study.

Zali et al.[9] compared APACHE II and GCS in 93 TBI patients with multiple trauma and found APACHE II to be superior to GCS. In this study, all the trauma patients studied had isolated head injury. Hence, we observed better discriminative power with GCS. In patients with neurological illness, GCS performed the best, probably because of its diagnostic specificity.

All the SSs studied in the present data include GCS as one of the components, and the contribution of GCS varies in each SS. In APACHE II SS, the contribution of GCS is 16.9%, and in APACHE III, it is 19%.[7]

Although many parameters were significantly different between survivors and non-survivors, in the multivariate analysis, only CPR within the first 24 h of admission turned out to be a significant and independent predictor of mortality. In addition, in a study conducted by Cuthbertson et al.,[10] the authors found that requirement of CPR in the 24 h preceding ICU admission was predictive of outcome in their ICU and it is a late sign of deterioration. All the previously published studies, which compared the SSs in patients with neurological diagnosis, have included a very limited variety of neurological diseases such as either only head injury or elective neurosurgical population. We, in our study, included a diverse neurological population such as elective spine and brain tumours, patients with neurotrauma, and patients with myasthenia, GBS and cortical venous thrombosis. We do not follow the policy of withdrawal of care.

In a study by Brandner et al.,[11] the authors introduced a new score called Clinical Course Score (CCS) defined as the difference between the GOS grade (assessed at 6 months after discharge) and the reduced GCS grade (five-grade scale) and applied it to 248 patients with neurotrauma and SAH. CCS ranges from −4 to +4. A positive value would represent an improving clinical course of the patient during the post-traumatic phase, whereas a negative CCS value would represent a deteriorating clinical course. It can be easily calculated retrospectively, can be evaluated at various points in time following neurotrauma, and provides better insight into the assessment of the beneficial effects of various therapeutic procedures in neurotrauma.

In a study similar to ours, by Handschu et al.,[12] the authors tested the performance of SAPS I/II and GCS in ninety patients with acute stroke to predict mortality at 10 days, 90 days and 1 year. The authors found that all the three scores had good predictive value and were independent predictors for predicting short-term and long-term mortality in stroke patients. They also concluded that GCS and even its motor subscore provided the same prognostic information as the other complex severity of illness scores and therefore recommended its routine use in critically ill stroke patients.

In this study, we found that GCS and SAPS II have the highest predictive ability amongst all the scores, although all the other models also performed fairly well. One of the advantages of GCS over other SSs is that it is simple, practical and most importantly does not require enormous effort for data collection and calculation.

Our study has some limitations. First, as the study design was retrospective, a 3.5% of observations were missing, and the missing values were substituted with normal values for the calculator to calculate the scores. As the missing values were <10%, we assume that the results would not have been different but needs validation. Second, as this was a single centre study, the applicability of results to a larger population needs to be tested. Third, it has been shown that SAPS II model fail for patients who stay in the ICU for more than a week. As we did not record ICU or hospital LOS, high predictive power of SAPS II results should be interpreted with caution.

CONCLUSIONS

We found that the GCS and SAPS II PMR SSs had higher chances of predicting mortality in patients with neurological illnesses. The use of GCS can be continued in view of its simplicity.

Acknowledgements

The authors would like to acknowledge the Middle East critical care assembly for providing free access to the ICU scoring web page.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

No conflict of interest has been declared by the author(s).

• REFERENCES

• 1 Keegan MT, Gajic O, Afessa B. Severity of illness scoring systems in the Intensive Care Unit. Crit Care Med 2011; 39: 163-9
• 2 Obrist WD, Langfitt TW, Jaggi JL, Cruz J, Gennarelli TA. Cerebral blood flow and metabolism in comatose patients with acute head injury. Relationship to intracranial hypertension. J Neurosurg 1984; 61: 241-53
• 3 Fielding K, Rowley G. Reliability of assessments by skilled observers using the Glasgow Coma Scale. Aust J Adv Nurs 1990; 7: 13-7
• 4 Udekwu P, Kromhout-Schiro S, Vaslef S, Baker C, Oller D. Glasgow Coma Scale score, mortality, and functional outcome in head-injured patients. J Trauma 2004; 56: 1084-9
• 5 Afessa B, Gajic O, Keegan MT. Severity of illness and organ failure assessment in adult Intensive Care Units. Crit Care Clin 2007; 23: 639-58
• 6 Cho DY, Wang YC. Comparison of the APACHE III, APACHE II and Glasgow Coma Scale in acute head injury for prediction of mortality and functional outcome. Intensive Care Med 1997; 23: 77-84
• 7 Ting HW, Chen MS, Hsieh YC, Chan CL. Good mortality prediction by Glasgow Coma Scale for neurosurgical patients. J Chin Med Assoc 2010; 73: 139-43
• 8 Nejmi H, Rebahi H, Ejlaidi A, Abouelhassan T, Samkaoui MA. The ability of two scoring systems to predict in-hospital mortality of patients with moderate and severe traumatic brain injuries in a Moroccan Intensive Care Unit. Indian J Crit Care Med 2014; 18: 369-75
• 9 Zali AR, Seddighi AS, Seddighi A, Ashrafi F. Comparison of the acute physiology and chronic health evaluation score (APACHE) II with GCS in predicting hospital mortality of neurosurgical Intensive Care Unit patients. Glob J Health Sci 2012; 4: 179-84
• 10 Cuthbertson BH, Rajalingam Y, Harrison S, McKirdy F. The outcome of haematological malignancy in Scottish Intensive Care Units. J Intensive Care Soc 2008; 9: 135-40
• 11 Brandner S, Kellermann I, Hore N, Bozhkov Y, Buchfelder M. Clinical Course Score (CCS): A new clinical score to evaluate efficacy of neurotrauma treatment in traumatic brain injury and subarachnoid hemorrhage. J Neurosurg Anesthesiol 2015; 27: 26-30
• 12 Handschu R, Haslbeck M, Hartmann A, Fellgiebel A, Kolominsky-Rabas P, Schneider D. et al. Mortality prediction in critical care for acute stroke: Severity of illness-score or coma-scale?. J Neurol 2005; 252: 1249-54