Implementing Obstetric Early Warning Systems

Abstract

Severe maternal morbidity and mortality are often preventable and obstetric early warning systems that alert care providers of potential impending critical illness may improve maternal safety. While literature on outcomes and test characteristics of maternal early warning systems is evolving, there is limited guidance on implementation. Given current interest in early warning systems and their potential role in care, the 2017 Society for Maternal-Fetal Medicine (SMFM) Annual Meeting dedicated a session to exploring early warning implementation across a wide range of hospital settings. This manuscript reports on key points from this session. While implementation experiences varied based on factors specific to individual sites, common themes relevant to all hospitals presenting were identified. Successful implementation of early warnings systems requires administrative and leadership support, dedication of resources, improved coordination between nurses, providers, and ancillary staff, optimization of information technology, effective education, evaluation of and change in hospital culture and practices, and support in provider decision-making. Evolving data on outcomes on early warning systems suggest that maternal risk may be reduced. To effectively reduce maternal, risk early warning systems that capture deterioration from a broad range of conditions may be required in addition to bundles tailored to specific conditions such as hemorrhage, thromboembolism, and hypertension.

Keywords

Maternal Early Warning Triggers - modified early obstetric warning system - Modified Early Warning Criteria - maternal morbidity - maternal mortality

The Centers for Disease Control and Prevention (CDC) estimated that from 2011 to 2013 2,009 pregnancy-related deaths occurred in the United States. Since the CDC's Pregnancy Mortality Surveillance System (PMSS) was implemented, pregnancy-related deaths rose steadily from 7.2 in 1987 to 17.8 per 100,000 deliveries in 2009, and have since remained at highs since the 1980s with death rates of 17.8, 15.9, and 17.3 in 2011, 2012, and 2013, respectively.[1] Maternal death and severe morbidity—key indicators of population health—not only remain common but are actually increasing in the United States.[2] [3] [4] [5] The CDC now estimates that more than 65,000 pregnant and recently postpartum women suffer major morbidity annually.[3] [6]

Major maternal morbidity and mortality may be preventable in many cases[7] [8] [9] and use of obstetric early warning systems that alert care providers of potential impending critical illness may improve maternal safety and outcomes. Both the Saving Mothers' Lives report, a comprehensive review of maternal deaths in the United Kingdom, and the National Partnership for Maternal Safety, a multistakeholder leadership organization in the United States, have advocated for use of early warning systems ([Tables 1] [2] [3]).[7] [10] Mortality reviews from the United States and the United Kingdom have demonstrated failure to recognize abnormal maternal vital signs that occurs in many cases of preventable maternal death.[7] [11]

Table 1 A modified early obstetric warning system (MEOWS)[12]

Physiological parameters	Yellow alert	Red alert
Respiration rate	21–30	< 10 or > 30
Oxygen saturation		< 95
Temperature	35–36	< 35 or > 38
Systolic blood pressure	150–160 or 90–100	< 90 or > 160
Diastolic blood pressure	90–100	> 100
Heart rate	100–120 or 40–50	> 120 or < 40
Pain score	2–3
Neurological response	Voice	Unresponsive, pain

Note: Respiration rate (breaths per minute); oxygen saturation (%); temperature (°C); systolic blood pressure (mm HG); heart rate (beats per minute); level of consciousness is based on the Alert Voice Pain Unresponsive (AVPU) scale which assesses four possible outcomes to measure and record a patient's level of consciousness; pain scores (0 = no pain, 1 = slight pain on movement, 2 = intermittent pain at rest/moderate pain on movement). A single red score or two yellow scores triggers an evaluation.

Table 2 Maternal Early Warning Criteria (MEWC)[10]

Systolic BP; mm Hg	< 90 or > 160
Diastolic BP; mm Hg	> 100
Heart rate; beats per min	< 50 or > 120
Respiratory rate; breaths per min	< 10 or > 30
Oxygen saturation; % on room air	< 95
Oliguria; mL/h for ≥2 h	< 35

Abbreviation: BP, blood pressure.

Note: Neurologic: Maternal agitation, confusion, or unresponsiveness; Patient with preeclampsia reporting a nonremitting headache or shortness of breath. Legend: The presence of any of the abnormal parameters above necessitates the prompt evaluation of the patient by a provider.

Table 3 Maternal Early Warning Trigger (MEWT)

“Yellow” triggers
Systolic BP; mm Hg	< 80 or 156–160
Diastolic BP; mm Hg	< 45 or 106–110
Heart rate; beats per min	< 50 or 111–130
Respiratory rate; breaths per min	< 12 or 25–30
Temperature, degrees centigrade	≤36
Oxygen saturation; % on room air	90–93
Altered mental status
“Red” triggers
Nursing clinically uncomfortable with patient status
Temperature, degrees centigrade	≥38
Respiratory rate; breaths per min	> 30
Oxygen saturation; % on room air	< 90%
Heart rate; beats per min	> 130
Systolic BP; mm Hg	> 160
Respiratory rate; breaths per min	> 30
Diastolic BP; mm Hg	> 110
Mean arterial pressure; mm HG	< 55

Abbreviation: BP, blood pressure.

Note: A single red trigger or two yellow triggers requires evaluation by provider. Abnormal vital signs must be sustained over at least 20 min to be considered triggers.

While there is evolving literature on outcomes and test characteristics of maternal early warning systems,[12] [13] there is limited guidance on implementation. Compared with other obstetric safety improvement initiatives such as expanding venous thromboembolism (VTE) prophylaxis and encouraging timely administration of antihypertensive medications,[14] [15] early warning implementation may involve more complex systems issues. Successful use of early warnings systems will require administrative and leadership support, dedication of resources, improved coordination between nurses, providers, and ancillary staff, optimization of information technology (IT), evaluation of and change in hospital culture and practices, and support in provider decision-making.

Because of current interest in early warning systems and their potential role in improving maternal safety, the Society for Maternal-Fetal Medicine's (SMFM) 2017 Annual Meeting dedicated a session to exploring issues with early warning implementation across a wide range of hospital settings. This article reports on key points from this session and includes three sections on: (1) research evidence supporting maternal early warnings, (2) implementation and optimization of early warning systems, and (3) aligning IT and electronic medical records (EMRs) with maternal alerts.

Evidence on Maternal Early Warning Systems

Outcomes evidence on early warning systems is encouraging and optimizing alert system performance is a critical goal; however, data on most early warning systems are limited. Early warning systems that result in a large number of false positives may function as a “nuisance alarm,” worsening clinical care and contributing to “alarm fatigue.” “Alarm fatigue” occurs when clinical providers are overwhelmed and desensitized by alerts of minimal or no clinical usefulness (“nuisance alarms”) and is a recognized source of medical errors.[16] [17] As different early warning systems utilize varying parameters to trigger an evaluation, there is the possibility that some warning systems perform better than others. Major early warning systems include: (1) the modified early obstetric warning system (MEOWS) proposed by the UK Saving Mothers' Lives report,[7] (2) the Maternal Early Warning Criteria (MERC) proposed by the National Partnership for Maternal Safety,[10] and (3) the Maternal Early Warning Trigger (MEWT) tool utilized in the Dignity Health System and at other hospitals in the United States. With MEOWS, two moderately abnormal parameters (yellow alerts) or one severely abnormal parameter (red alert) trigger a clinical response to urgently assess the patient's status and make a follow-up surveillance plan ([Table 1]). The MEWC parameters represent a simplified early warning system adapted from MEOWS ([Table 2]). The MEWT tool differs from MEOWS and MERC in that it categorizes alerts into diagnostic pathways for conditions such as sepsis, cardiopulmonary conditions, hypertensive disorders, and obstetric hemorrhage, and provides diagnostic and management recommendations based on parameters ([Table 3]).[13] [Table 4] demonstrates major differences between MEOWS, MERC, and MEWT.

Table 4 Comparison of early warning systems

	MEOWS	MEWT	MERC
Criteria for evaluation	One “red” trigger or two “yellow” triggers	One “red” trigger or two “yellow” triggers	One “red” trigger (no “yellow” triggers)
Decision support and escalation guidance	No	Yes, “clinical pathways”	No
Data on test characteristics	Yes	Yes	No
Outcomes data	No	Yes	No

Abbreviations: MEOWS, modified early obstetric warning system; MERC, Maternal Early Warning Criteria; MEWT, Maternal Early Warning Trigger.

Limited data are currently available for MEOWS and MERC. Singh et al assessed MEOWS parameters in an obstetric population specifically evaluating test characteristics for detecting high-risk clinical scenarios and severe maternal morbidity including obstetric hemorrhage, severe preeclampsia, infection, and thromboembolism among other diagnoses. The screen positive rate for 673 obstetric admissions was 30% (200 women) with 13% of women developing the composite adverse outcome. MEOWS was overall 89% sensitive and 79% specific with a positive predictive value of 39%.[12] Another study evaluating MEOWS demonstrated similar test characteristics.[18] While these trials demonstrated reasonable test characteristics for MEOWS, they did not assess efficacy in terms of (1) identifying undetected impending critical illness, (2) optimizing process measures of management processes (e.g., time to administration of antihypertensives), or (3) improvements in clinically meaningful outcomes.

Data on the MEWT tool comes from analysis of early warning implementation in six hospitals in the Dignity Health system.[13] Outcomes for 12,611 patients over 13 months preimplementation were compared with outcomes for 24,221 patients over 24 months postimplementation. Overall, there was a significant reduction in severe maternal morbidity as defined by criteria from the CDC from 2.0% preimplementation to 1.6% postimplementation.[6] The screen positive rate was much lower with MEWT (2.3%) than with MEOWS. This differential may be secondary to: (1) requirements for abnormal parameters in the MEWT tool to be sustained prior to an alert being triggered, and (2) stricter thresholds for single-parameter activation. Retrospective data used to establish the MEWT criteria suggested that using sustained vital signs, the alarm rate would be decreased from 20 to 4%. When these criteria were tested prospectively the alarm rate was 2.3%.[19]

Implementation and Optimization

The SMFM early warning session on implementing early warning systems included presentations from a variety of clinical settings including: EvergreenHealth Medical Center, a community hospital in Kirkland, Washington, Eastside Medical Center, a community hospital in Snellville, Georgia, Columbia University Medical Center, a tertiary teaching hospital in New York City, New York, and Dignity Health, a health care system including 39 hospitals in California, Arizona, and Nevada. While implementation experiences varied based on factors specific to individual sites, common themes relevant to all hospitals were identified. Key barriers to implementation included the following:

• Lack of multidisciplinary coordination and buy-in.

• Inadequate education.

• Suboptimal integration within hospital culture and practices.

• Lack of leadership support.

• Suboptimal alignment with other quality and safety initiatives.

Multidisciplinary Coordination and Buy-In

Timely activation of a maternal alert system requires appropriate communication and decision-making by the medical team responsible for providing care for a patient. This team will include the person responsible for taking the patient's vital signs, the nurse caring for the patient, and the provider responsible for management and diagnostic decision-making. A successful maternal alert response begins with a series of abnormal vital signs being taken and communicated to providers in a timely fashion. If a nursing assistant performs vital sign assessments, an abnormal alert parameter must be recognized and then quickly communicated to the responsible nurse who in turn must notify the provider. One of the primary functions of the early warning system is to have the provider quickly evaluate the patient, create an appropriate management plan, and follow-up closely. If a patient deteriorates, timely escalation of care is critical. If there is a breakdown at any point in the alert system from collecting and recognition of abnormal vital signs to prompt notification of the provider to an appropriate clinical evaluation occurring, an opportunity to use an “early warning” to prevent or mitigate onset of critical illness may be lost. An effective alerts system relies on processes that facilitate reliable communication including integration into the EMR system, and the optimal means of relaying information between nursing assistants, nurses, and providers should be identified prior to implementation. Additionally, soliciting qualitative feedback from the entire care team including providers and nursing after implementation may be necessary to successfully tailor an early warning system to a specific hospital environment.

Education

Given that early warning systems are dependent on all members of the team caring for a patient, appropriate education for nursing assistants, nurses, and providers is critical. If a maternal early warning system is being implemented, the best mechanisms for education of each type of team member within the hospital must be identified and utilized. Education should focus on imparting the rationale and purpose for implementing an alert system—to improve maternal safety and avoid adverse outcomes—to encourage buy-in. It should also focus on critical processes within the alert system and what the role is of each team member. Because many obstetric services have large staffs of caregivers, a given amount of turnover will be a constant; after a roll out of an alert system, ongoing education will be required for newly hired employees or those transferring in from other clinical services. In optimizing an educational approach, covering physicians who may not be primarily in house or hospital employees must be included, as do other services that provide care to critically ill patients hospital-wide such as rapid response teams. If changes are required after implementation to tailor an alert system to an individual hospital processes and culture, ongoing education may be a critical component of care improvement. Early warning systems can only contribute to improved patient safety insofar as they facilitate providers making correct, timely management decisions. If systematic deficiencies are noted in provider responses to particular clinical scenarios, educational interventions are required.

Hospital Practices and Culture

Maternal alerts need to be audited to determine whether the system is working correctly. Routine clinical practices may be identified that contribute to alerts being missed. For example, if an alert system relies on nurses or a nursing assistant entering vital signs into an EMR application that automatically generates an alert, and vital signs are not entered until the end of a shift, alerts will be less likely to be communicated on a timely basis. Similarly, if the provider responsible for responding to alerts has concurrent clinical responsibilities, and there is no backup plan for patient evaluation when there is a concurrent clinical emergency (on the labor floor, for example), delays in care may occur that put patients at increased risk. Early warning systems may improve patient safety not only by improving timely communication and patient evaluation, but also by calling attention to systematic issues that may be improved by quality assessment processes. Specific clinical scenarios that may be useful in auditing maternal alerts include:

• Timely treatment of persistent severe range hypertension.

• Diagnosis of severe anemia requiring transfusion prior to overt symptomatology, hemodynamic instability, or both.

• Timely response to sepsis.

• Diagnosis of pulmonary embolism, heart failure, and other major cardiopulmonary processes prior to overt symptomatology, hemodynamic instability, or both.

Leadership Support

Departmental and administrative leadership support is critical to successful implementation of early warning systems. Many centers will have numerous time-intensive bureaucratic processes to be navigated for an alert system to be implemented. Very often there will be other administrative and clinical initiatives competing for resources and support. Hospital leadership can expedite and facilitate maternal warning implementation by identifying key personnel to help navigate the process and by prioritizing an alert system for approval and implementation. Because both implementation and auditing of early warning systems are time-intensive and require dedicated resources on an ongoing basis, allocation of protected time for both nursing and physicians tasked with responsibility for ongoing evaluation of an alert system may be necessary.

Alignment with Other Quality Improvement Initiatives

Maternal early warning systems align with other major initiatives that are currently being widely disseminated including efforts to improve management of obstetric hemorrhage and severe hypertension.[14] [20] Such systems may facilitate identification and treatment of patients with these conditions; similarly, vital sign abnormalities compatible with severe hemorrhage, such as tachycardia and hypotension, may be representative of other life-threatening conditions such as VTE and sepsis. Maternal alerts represent an opportunity to create an “umbrella” system under which to incorporate quality improvement initiatives such as adoption of the hemorrhage “bundle”[20] and hypertension treatment algorithms.[14] For example, the MEWT tool creates pathways for diagnosis and management for conditions such sepsis, cardiopulmonary processes, hypertension in pregnancy, and obstetrical hemorrhage based on vital sign triggers.[13]

Information Technology and Electronic Medical Records

Evaluating EMR capabilities should be among the first steps in implementing a maternal alert system as incorporating changes into an EMR to facilitate alert warnings and notifications is a process that may take months or even years. Specific attributes of EMR systems that may facilitate early warning systems responses include:

• Automated provider alerts.

• Decision aids for diagnostic evaluations.

• Data collection on process measures.

• Alert-specific documentation tools.

Obstacles to making changes to EMRs to facilitate early warning systems may be technical, administrative, or both. It is recommended to involve clinical informaticists as early as possible in the process. Because there may be a long queue of EMR initiatives hospital-wide, administrative and leadership support may be critical for successful upgrades and alterations. An EMR tailored to support a maternal alert system may offer many benefits including timely auditing of alerts, automated alert notification for providers, special documentation templates, decision support tools, and other features. While EMR features may enhance alert system capabilities, maternal early warning implementation should not be delayed based on EMR factors. Because of the timeline involved with EMR alterations, it may be necessary to initiate an alert system first, and wait for EMR optimizations later. Collaboration within and across hospital systems using similar EMR platforms and alert criteria should be encouraged to reduce redundancy in EMR implementation efforts. Early warning systems and scores have been used by other specialties including pediatrics,[21] [22] general medical and surgical admission populations,[23] [24] [25] and medical[26] and surgical specialties;[27] if an alert system has been implemented for another population within a hospital, relevant EMR features may be readily transferable to the obstetric population. Even if specific vital sign parameters must be tailored to maternal physiology, from an institutional perspective, implementation can be leveraged by using existing resources for IT configuration, implementation, training, and auditing by aligning the effort with other institutional early warning systems. Caution should be exercised in use of nonmaternal-based sepsis screens from the general population as they do not perform well in patients due to higher baseline heart rates, lower systolic, diastolic, and mean arterial blood pressure, and higher baseline white blood cell counts; sepsis guidelines and criteria for the general population are rapidly evolving and management of sepsis among obstetric patients is an area of emerging research.

Discussion

Maternal early warning systems represent a promising strategy for reducing risk and identifying patients with potential impending critical illness secondary to a broad range of conditions. Data from the CDC's PMSS demonstrated cardiovascular and other medical noncardiovascular disease to be the most common and second-most common causes of maternal mortality, respectively, from 2011 to 2013, accounting overall for 30% of maternal deaths.[28] While targeted approaches may provide effective means of reducing maternal risk from VTE, obstetric hemorrhage, and hypertension,[14] [15] [20] a more heterogeneous group of conditions less amenable to risk reduction via targeted strategies may account for an increasing proportion of maternal mortality.

While early warning systems may meaningfully improve maternal safety on a hospital-wide basis by promoting appropriate situational awareness, they are less uniformly implementable than interventions such as expanding VTE prophylaxis or implementing a hypertension treatment protocol. Even though similar alert criteria may be used by different centers, effective escalation policies will necessarily be different and individually tailored on a hospital basis. Along these lines, future research on maternal alerts can be dichotomized conceptually into (1) assessing test characteristics of alert criteria, and (2) implementing science in integrating early warning systems into practice. Test characteristics and implementation are certainly related; for a maternal early warning system to contribute to improved health outcomes, it must (1) identify patients at risk for critical illness who benefit from timely intervention, and (2) limit false positive alerts which may preclude the alert system from being able to be sustained within the practice culture of a hospital. Data from the MEWT tool suggests that outcomes may be improved when alerts are kept to a low false positive rate, and the tool incorporates guidance on escalation (“pathways”).

This session came to the following conclusions regarding maternal early warnings:

• Successful maternal alert system implementation requires multidisciplinary buy-in and coordination.

• An ongoing education program that addresses staff turnover and that addresses educational needs and problems detected via alert audits is necessary.

• Leadership support may be a critical catalyst for success.

• Effectively implementing and managing an alert system is a significant time commitment that may require dedicated resources.

• An optimal alert system will be tailored to the needs, resources, and capabilities of an individual hospital and will act as an “umbrella” incorporating other protocols designed to reduce maternal risk.

• Integration with IT may be important in optimizing an alert system; however, because of the timeline that EMR modifications may involve, clinical informaticists should be involved from the outset, and alert implementation should not be deferred indefinitely secondary to IT considerations. Based on the comments of both presenters at the session and attendees, obstetrics is in the early stages of implementation of these processes relative to other medical specialties.

Finally, while research on early warning systems is evolving and has demonstrated important positive findings, future work is needed to validate optimal alert parameters as well as further knowledge on how to best implement systems across a wide range of clinical settings including triage, emergency department, and outpatient settings.

Financial Disclosure

None.

• References

• 1 Centers for Disease Control and Prevention Pregnancy Mortality Surveillance System. Available at: https://www.cdc.gov/reproductivehealth/maternalinfanthealth/pmss.html . Accessed May 23, 2017
• 2 Khan KS, Wojdyla D, Say L, Gülmezoglu AM, Van Look PF. WHO analysis of causes of maternal death: a systematic review. Lancet 2006; 367 (9516): 1066-1074
• 3 Callaghan WM, Creanga AA, Kuklina EV. Severe maternal morbidity among delivery and postpartum hospitalizations in the United States. Obstet Gynecol 2012; 120 (05) 1029-1036
• 4 Berg CJ, Callaghan WM, Syverson C, Henderson Z. Pregnancy-related mortality in the United States, 1998 to 2005. Obstet Gynecol 2010; 116 (06) 1302-1309
• 5 Creanga AA, Berg CJ, Syverson C, Seed K, Bruce FC, Callaghan WM. Pregnancy-related mortality in the United States, 2006-2010. Obstet Gynecol 2015; 125 (01) 5-12
• 6 Centers for Disease Control and Prevent. Severe maternal morbidity in the United States. Available at: https://www.cdc.gov/reproductivehealth/maternalinfanthealth/severematernalmorbidity.html . Accessed March 2018
• 7 Cantwell R, Clutton-Brock T, Cooper G. , et al. Saving Mothers' Lives: reviewing maternal deaths to make motherhood safer: 2006-2008. The Eighth Report of the Confidential Enquiries into Maternal Deaths in the United Kingdom. BJOG 2011; 118 (Suppl. 01) 1-203
• 8 The California Pregnancy-Associated Mortality Review. Report from 2002 and 2003 Maternal Death Reviews. Sacramento: California Department of Public Health, Maternal Child and Adolescent Health Division; 2011
• 9 Clark SL. Strategies for reducing maternal mortality. Semin Perinatol 2012; 36 (01) 42-47
• 10 Mhyre JM, DʼOria R, Hameed AB. , et al. The maternal early warning criteria: a proposal from the national partnership for maternal safety. Obstet Gynecol 2014; 124 (04) 782-786
• 11 Knight M, Kenyon S, Brocklehurst P, Neilson J, Shakespeare J, Kurinczuk JJ. (Eds.) on behalf of MBRRACEUK. Saving Lives, Improving Mothers' Care - Lessons learned to inform future maternity care from the UK and Ireland Confidential Enquiries into Maternal Deaths and Morbidity 2009 -12. Oxford: National Perinatal Epidemiology Unit, University of Oxford; 2014
• 12 Singh S, McGlennan A, England A, Simons R. A validation study of the CEMACH recommended modified early obstetric warning system (MEOWS). Anaesthesia 2012; 67 (01) 12-18
• 13 Shields LE, Wiesner S, Klein C, Pelletreau B, Hedriana HL. Use of Maternal Early Warning Trigger tool reduces maternal morbidity. Am J Obstet Gynecol 2016; 214 (04) 527.e1-527.e6
• 14 Clark SL, Christmas JT, Frye DR, Meyers JA, Perlin JB. Maternal mortality in the United States: predictability and the impact of protocols on fatal postcesarean pulmonary embolism and hypertension-related intracranial hemorrhage. Am J Obstet Gynecol 2014; 211 (01) 32.e1-32.e9
• 15 DʼAlton ME, Friedman AM, Smiley RM. , et al. National Partnership for Maternal Safety: consensus bundle on venous thromboembolism. Obstet Gynecol 2016; 128 (04) 688-698
• 16 The Joint Commission Sentinel Event Alert. Medical device alarm safety in hospitals. The Joint Commission. April 8, 2013. . Available at: https://www.jointcommission.org/assets/1/6/SEA_50_alarms_4_26_16.pdf . Accessed March 2018
• 17 ECRI Institute releases top 10 health technology hazards report for 2014. Available at: https://www.ecri.org/Press/Pages/2014_Top_Ten_Hazards.aspx . Accessed December 1, 2014
• 18 Singh A, Guleria K, Vaid NB, Jain S. Evaluation of maternal early obstetric warning system (MEOWS chart) as a predictor of obstetric morbidity: a prospective observational study. Eur J Obstet Gynecol Reprod Biol 2016; 207: 11-17
• 19 Hedriana HL, Wiesner S, Downs BG, Pelletreau B, Shields LE. Baseline assessment of a hospital-specific early warning trigger system for reducing maternal morbidity. Int J Gynaecol Obstet 2016; 132 (03) 337-341
• 20 Main EK, Goffman D, Scavone BM. , et al; National Partnership for Maternal Safety; Council on Patient Safety in Women's Health Care. National Partnership for Maternal Safety: consensus bundle on obstetric hemorrhage. Obstet Gynecol 2015; 126 (01) 155-162
• 21 Seiger N, Maconochie I, Oostenbrink R, Moll HA. Validity of different pediatric early warning scores in the emergency department. Pediatrics 2013; 132 (04) e841-e850
• 22 Gold DL, Mihalov LK, Cohen DM. Evaluating the Pediatric Early Warning Score (PEWS) system for admitted patients in the pediatric emergency department. Acad Emerg Med 2014; 21 (11) 1249-1256
• 23 Smith ME, Chiovaro JC, O'Neil M. , et al. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc 2014; 11 (09) 1454-1465
• 24 Subbe CP, Davies RG, Williams E, Rutherford P, Gemmell L. Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. Anaesthesia 2003; 58 (08) 797-802
• 25 Groarke JD, Gallagher J, Stack J. , et al. Use of an admission early warning score to predict patient morbidity and mortality and treatment success. Emerg Med J 2008; 25 (12) 803-806
• 26 Bokhari SW, Munir T, Memon S, Byrne JL, Russell NH, Beed M. Impact of critical care reconfiguration and track-and-trigger outreach team intervention on outcomes of haematology patients requiring intensive care admission. Ann Hematol 2010; 89 (05) 505-512
• 27 Patel MS, Jones MA, Jiggins M, Williams SC. Does the use of a “track and trigger” warning system reduce mortality in trauma patients?. Injury 2011; 42 (12) 1455-1459
• 28 Kuklina EV, Whiteman MK, Hillis SD. , et al. An enhanced method for identifying obstetric deliveries: implications for estimating maternal morbidity. Matern Child Health J 2008; 12 (04) 469-477