Validation of a Scoring System to Identify Women with Near-Miss Maternal Morbidity

Abstract

Objective To validate a five-factor scoring system that identifies parturients who experience near-miss morbidity.

Study Design and Setting This study was conducted in an urban, tertiary care hospital over a 2-year period. A narrative case summary was prepared for women with high potential for significant obstetric morbidity. The summary was then reviewed by three physicians, and the extent of morbidity was assigned based on subjective assessment. The same cases were then scored using the proposed five-factor scoring system previously described by Geller et al. Test characteristics of the scoring system were assessed.

Results Eight hundred fifteen cases with a high potential for significant morbidity were identified. Subjective review and the scoring system classified 4.5% and 4.2% as near-miss morbidity, respectively, with the scoring system having a corresponding sensitivity of 81.1% (95% confidence interval 64.8 to 92.0%) and a specificity of 99.5% (95% confidence interval 98.7 to 99.9%).

Conclusion The scoring system produced similar results to those obtained at its initial development and demonstrated acceptable sensitivity and specificity for identifying near-miss morbidity.

Keywords

near-miss morbidity - maternal mortality - scoring system

Little progress has been made in the effort to lower the rate of maternal mortality in the United States over several decades.[1] Indeed, recent data suggest that this rate may actually be rising.[2] This lack of progress is further demonstrated by the inability to narrow the racial disparities that exists between white, black, and Hispanic populations with regard to maternal mortality.[1]

These data cumulatively serve to emphasize the need to improve perinatal care. However, despite the unacceptable epidemiological trends of maternal mortality rate in the United States, the absolute number of cases remains quite low and is infrequent at any one institution. Because the number of women who die as a result of pregnancy is so low, evaluation of maternal mortality cases as a tool to improve perinatal health has a limited utility. It is difficult to discern patterns of care that need refinement with such a small number of applicable cases. As maternal morbidity is orders of magnitude more common than maternal mortality, expanding the focus of case review to include maternal morbidity would provide a larger population of interest.[3] The broadening of the focus to maternal morbidity has the potential to give insight into methods to reduce maternal mortality, lessen racial disparities in adverse obstetric outcomes, and identify preventable factors that lead to poor maternal outcome. Before morbidity can be effectively studied, however, a standard metric needs to be developed. The metric must identify women with morbidity that is significant enough to warrant study and be plausibly related to maternal mortality.

In 2004¸ Geller et al devised a scoring system that identified significant morbidity. After conceptualizing morbidity as a continuum (i.e., from minor to severe), these investigators defined “near-miss” morbidity as the most severe morbidity that occurs prior to, but does not result in, death.[4] In this framework, “near-miss” does not refer to a woman who nearly misses having a morbid event, but who nearly misses a mortal event. Geller's system uses multiple well-defined variables to identify women with near-miss morbidity and differentiates them from other women with morbidity that is not as severe. This system can then be used to identify women from perinatal databases without having to rely on individual chart review and abstraction. Although Geller et al demonstrated a reasonable sensitivity and specificity for their scoring system, this system has not been evaluated in settings other than the originally studied institution, and its generalizability remains uncertain. The objective of our study was to further assess and validate the Geller scoring system for the identification of near-miss maternal morbidity.

Materials and Methods

This was a retrospective cohort study of women who delivered at a high-volume, urban, tertiary care center over a 2-year period (2001 to 2002). This period of time was chosen as it corresponded to the time during which patients for the Geller et al study were also selected. Similarly, other methods for identification of patients were identical to those described by Geller et al.[4] After receiving institutional review board approval, women with a high potential for obstetric morbidity were identified using a perinatal and billing database that included clinical information with searchable diagnoses and all deliveries that occurred over the selected 2-year time period at the institution of interest. Identification was based on the presence of comorbidities, adverse outcomes, or particular procedures. A detailed list of parameters that were used to identify cases is outlined in [Table 1]. The medical records of these parturients were obtained, and a narrative summary of their hospital course was prepared by a single trained research assistant. This summary was then reviewed by three physicians. Each physician, based on his or her impression of the patient's overall clinical course, assigned the case a given degree of morbidity: none, minor, severe, or near-miss. The final morbidity designation was based on the consensus opinion of the three physicians; when the three physicians did not agree on the degree of morbidity, the final level of morbidity was assigned by majority rule. For example, if two of the reviewers coded a given case as near-miss morbidity and one of the reviewers coded that same case as severe morbidity, then the case was identified for final analyses as near-miss morbidity. Because no gold standard exists to assign the presence or the degree of maternal morbidity, the final physician-derived designation was considered the gold standard to which the Geller score was compared.

Table 1 Obstetric Parameters Used for Identification of Cases[4]

Diseases/Conditions	Morbid Events	Procedures/Interventions
Severe preeclampsia/eclampsia	Hemorrhage >1500 mL	Transfusion
Embolism	Wound dehiscence	ICU admission
Infection	Organ system failure	Extended Intubation
Ectopic/molar pregnancy	Abnormal vital signs/laboratories	Surgical Intervention
Cardiac disease	Abruptio	Return to the operating room
Cerebral vascular accident	Seizures	Readmission to the hospital
Accreta/increta/percreta	Stroke	Multiple medical interventions
ITP/TTP	Pulmonary edema	Hysterectomy
	DIC	Prolonged hospital stay
	ARDS

ARDS, acute respiratory distress syndrome; DIC, disseminated intravascular coagulopathy; ICU, intensive care unit; ITP, idiopathic thrombocytopenic purpura; TTP, thrombotic thrombocytopenic purpura.

All cases were then scored utilizing the weighted five-factor scoring system developed by Geller et al ([Table 2]).[4] In this system, each case was assigned a total score based on the presence or absence of the five factors. Each chart was abstracted for the occurrence of the relevant events. When a given factor was present, the case was given the appropriate number of points, and a total score was then obtained by summing all of the accorded points. Any woman who received a score of greater than or equal to 8 was identified as having had near-miss morbidity. Sensitivity, specificity, as well as positive and negative predictive values for the identification of near-miss morbidity were calculated for a Geller score of 8 or greater and for each component contributing to the score.

Table 2 Components of the Scoring System Proposed by Geller et al[4]

	Points
Organ system failure (≥1 system)[a]	5
ICU admission	4
Transfusion >3 U PRBC	3
Intubation >12 h	2
Unanticipated surgical intervention	1

ICU, intensive care unit; PRBC, packed red blood cells.

^a Central nervous system (coma, intracranial hemorrhage, blindness); cardiovascular (cardiac arrest or parenteral treatment of hypotension); pulmonary (acute respiratory distress syndrome, respiratory arrest); hematologic (disseminated intravascular coagulation, platelets <50,000/mL); renal (creatinine >2.0 mg/dL, dialysis); gastrointestinal (liver failure, colostomy).

Results

During the period of study, 17,275 women delivered at Northwestern Memorial Hospital. Eight hundred fifteen cases were identified from the initial database search as being at high risk for obstetric morbidity. There were no maternal deaths. Each rater's determination with regard to frequency of the different levels of morbidity among this cohort is presented in [Table 3]. The final physician designation, based on review of the overall narrative, determined that 130 (15.9%) women had severe morbidity and 37 (4.5%) women had near-miss morbidity. Thus, the overall prevalence of near-miss morbidity is estimated to be 0.2% (37/17,275). Concordance of opinion among the three raters in determining whether a woman had near-miss morbidity was high, with agreement in 93% of cases. The frequency of near-miss morbidity among the selected population was similar to that reported by Geller et al in their population (6.5%).[4]

Table 3 Morbidity Based on Narrative Review Stratified by Rater (n = 816)

	Rater 1	Rater 2	Rater 3
No or minor morbidity	74.6	83.6	77.8
Severe morbidity	19.7	10.9	18.8
Near-miss morbidity	5.6	5.5	3.4

All data presented as percents.

With regard to the components present in the Geller et al scoring system,[4] 13.5% of women had an unanticipated surgical intervention, 3.4% had an extended intubation, 4.4% had a transfusion greater than 3 U, 7.6% were admitted to the intensive care unit (ICU), and 4.2% had at least one organ system fail. These events translated into a scoring distribution as follows: 89.7% of women had scores from 0 to 2, 6.1% had scores from 3 to7, and 4.2% had scores of at least 8. This last group of women, therefore, was predicted by the scoring system to have had experienced near-miss morbidity.

Of these 34 women identified by the Geller scoring system as having near-miss morbidity, 30 (88.2%) actually had experienced near-miss morbidity as determined by the “gold standard” of the full narrative. Conversely, seven women who had near-miss morbidity were not identified using the scoring system. Thus, the scoring system had a sensitivity of 81.1% (95% confidence interval [CI] 64.8 to 92.0%) and a specificity of 99.5% (95% CI 98.7 to 99.9%) for the identification of near-miss morbidity. Positive and negative predictive values were 88.2% and 99.1%, respectively. [Table 4] demonstrates the sensitivity and specificity of each individual component in the Geller et al scoring system for identifying near-miss morbidity. The sensitivity and specificity of the five-factor scoring system were significantly greater than provided by each of the components independently.

Table 4 Sensitivity and Specificity of Individual Events for Identifying Near-Miss Morbidity

	Sensitivity	Specificity
Organ system failure	42.1	97.7
ICU admission	78.9	95.9
Transfusion >3 U PRBC	63.2	98.8
Intubation >12 h	42.1	97.7
Unanticipated surgical intervention	73.7	89.5

ICU, intensive care unit; PRBC, packed red blood cells.

Discussion

In this study, we have demonstrated that the scoring system proposed by Geller et al to identify women who have experienced near-miss morbidity retains its high sensitivity and specificity when used in a population other than the one in which it was initially developed.[4] The values for sensitivity and specificity (81.1% and 99.5%) that we obtained were similar to those (100% and 93.9%) obtained in their analysis. Other aspects of the results, such as the distribution of different types of morbidity and the fact that the test characteristics were better for the five-factor scoring system than for any individual contributing factor, were also similar between studies. This analysis gives further support to the validity of the scoring system as a method by which women can easily be identified as having had very significant, or “near-miss,” morbidity.

There is a great need to determine a uniform measure of significant maternal morbidity. This need is underscored by the rising maternal death rate in the United States as a whole. Yet, maternal death, although clearly important to track on a national level, occurs so uncommonly that it cannot be used in an individual institution for quality of care surveillance. Similarly, because a death is so uncommon, it is difficult to use to discern care processes and systems that are associated with recurrent adverse outcomes. Other individual outcomes may be used to indicate marked maternal morbidity. However, there is no uniform agreement about which is best to use, and each has its own potential limitations. For example, although ICU admission is often used as a marker of significant morbidity, the variability among institutions regarding criteria for ICU admission may limit its use as a useful comparator.[5] [6] Furthermore, as our study reveals, individual events, such as ICU admission or extended intubation, are not as useful as a multiple-factor scoring system in identifying which women actually experienced the greatest morbidity, as judged by clinicians based on a full care narrative.

The scoring system proposed by Geller et al, and externally validated in this study, offers several potential advantages.[4] First, it provides a uniform approach to the identification of women with obstetric morbidity that could be used by different institutions as a common metric to compare outcomes. Second, it identifies an outcome that is significant, but still frequent enough, that individual institutions can explore patterns of care and risk factors that may be associated with the adverse outcome. Third, the scoring system can be used to identify cases from perinatal or administrative databases minimizing the need for full institutionwide chart review.

Limitations of the study also should be noted. This study was performed at a single center, and accordingly the results may not be replicable at other institutions. Although our center was different in multiple aspects from the one in which the scoring system was originally generated, both are urban tertiary care centers with residency programs, and the ability of the scoring system to work equally as well in rural, community, or nonteaching hospitals is not certain.

Nevertheless, this study has replicated the findings of Geller et al[4] and has shown that the scoring system can be used to identify “near-miss” maternal morbidity. Further evaluation may reveal its applicability at additional institutions, as well as its ability to serve as a useful indicator of maternal outcomes and obstetric care.

Notes

Research for this project was done while the author (Whitney B. You, M.D., M.P.H.) was a National Research Service Award postdoctoral fellow at the Institute for Healthcare Studies under an institutional award from the Agency for Healthcare Research and Quality (T-32 HS 000078–11).

A related abstract was presented as a poster at the 2009 Society of Maternal Fetal Medicine meeting in San Diego, California, and the 2009 Academy of Health Annual Meeting in Chicago, Illinois.

• References

• 1 Chang J, Elam-Evans LD, Berg CJ , et al. Pregnancy-related mortality surveillance—United States, 1991–1999. MMWR Surveill Summ 2003; 52: 1-8
• 2 Hogan MC, Foreman KJ, Naghavi M , et al. Maternal mortality for 181 countries, 1980-2008: a systematic analysis of progress towards Millennium Development Goal 5. Lancet 2010; 375: 1609-1623
• 3 Callaghan WM, Mackay AP, Berg CJ. Identification of severe maternal morbidity during delivery hospitalizations, United States, 1991–2003. Am J Obstet Gynecol 2008; 199: 133 , e1–e8
• 4 Geller SE, Rosenberg D, Cox S, Brown M, Simonson L, Kilpatrick S. A scoring system identified near-miss maternal morbidity during pregnancy. J Clin Epidemiol 2004; 57: 716-720
• 5 Bouvier-Colle MH. Maternal intensive care and near-miss mortality in obstetrics. Br J Obstet Gynaecol 1999; 106: 1229-1234
• 6 Baskett TF, Sternadel J. Maternal intensive care and near-miss mortality in obstetrics. Br J Obstet Gynaecol 1998; 105: 981-984

Address for correspondence and reprint requests

Whitney B. You, MD, MPH

34800 Bob Wilson Drive

Building 3, Suite 100

San Diego, CA 92134   

Email: [e-mail]