A Study of the Diagnostic Accuracy of an Existing Multivariable Test to Predict Shoulder Dystocia

Abstract

Objectives To evaluate the diagnostic accuracy of a multivariable prediction model, the Shoulder Screen (Perigen, Inc.), and compare it with the American College of Obstetricians and Gynecologists (ACOG) guidelines to prevent harm from shoulder dystocia.

Study Design The model was applied to two groups of 199 patients each who delivered during a 4-year period. One group experienced shoulder dystocia and the other group delivered without shoulder dystocia. The model's accuracy was analyzed. The performance of the model was compared with the ACOG guideline.

Results The sensitivity, specificity, positive, and negative predictive values of the model were 23.1, 99.5, 97.9, and 56.4%, respectively. The sensitivity of the ACOG guideline was 10.1%. The false-positive rate of the model was 0.5%. The accuracy of the model was 61.3%.

Conclusion A multivariable prediction model can predict shoulder dystocia and is more accurate than ACOG guidelines.

Keywords

shoulder dystocia - brachial plexus injury - prediction - medicolegal

Shoulder dystocia occurrences during vaginal delivery result in ∼5,000 new permanent brachial plexus palsy cases annually in the United States. Up to 63,000 adults are afflicted with permanent loss of upper extremity function.[1] Malpractice claims alleging permanent brachial plexus palsy was frequent and second only to birth asphyxia in total dollars paid for obstetrics claims.[2]

Shoulder dystocia occurs in 0.2 to 3% of deliveries.[3] Multiple studies cited in the American College of Obstetricians and Gynecologists (ACOG) Neonatal Brachial Plexus Palsy Task Force publication of the ACOG substantiate that ∼10% of shoulder dystocia deliveries result in brachial plexus palsy, accounting for 1.5 per 1,000 deliveries.[4] Permanent brachial plexus palsy affects ∼15% of these neonates.[4]

The medicolegal costs of claims involving permanent brachial plexus and other permanent injury can exceed $1 million per claim. Over a 5-year period, one of the largest professional liability insurers in the United States paid more than $10 million for 14 brachial plexus palsy claims, an average of nearly $750,000 in malpractice payments and attorneys' fees per claim. These costs represent only the claims for which the insurance company paid for an injured child. During that same 5-year period, 45 unproven claims for shoulder dystocia complications also cost millions of additional dollars in just attorney and staff expenses (MagMutual Insurance Company, Atlanta, GA).

The ACOG guidelines for prevention and management of shoulder dystocia state that shoulder dystocia cannot be predicted or prevented accurately, although there are several known risk factors.[3] Nevertheless, those guidelines provide that cesarean delivery should be considered in three scenarios to prevent shoulder dystocia: when estimated fetal weight >5,000 g or when estimated fetal weight >4,500 g in a diabetic gravida or with a history of shoulder dystocia in a previous delivery. In essence then, these recommendations do constitute a prediction that the expected probability of shoulder dystocia is sufficiently high under certain circumstances to justify elective cesarean delivery. Other than the ACOG guidance, the prevailing position in obstetrics practice is that despite knowing the risk factors for shoulder dystocia, one cannot determine accurately enough when it will occur to justify avoiding vaginal delivery.[5] [6] [7] [8] Importantly, none of these references examined a risk assessment based on combinations of maternal and fetal size and shape, or sought to determine if those with persistent injury were also unpredictable.

In an effort to address this problem, Dyachenko et al[9] and Hamilton et al[10] reported that the probability of experiencing shoulder dystocia with neonatal injury (brachial plexus palsy, fracture, or encephalopathy) could be estimated for patients after 36 weeks' gestation, with multivariable logistic regression modeling, utilizing the advances in statistical modeling that were used in screening for trisomy 21. Their formula computed the risk of shoulder dystocia with persistent brachial plexus palsy specifically, using the variables, maternal height and body mass index (BMI), birth weight, and birth weight percentile. The final iteration of this model computes the risks of shoulder dystocia with and without temporary and permanent brachial plexus injury and is currently available for purchase in a web-based software application called the Shoulder Screen (PeriGen, Inc., Cary, NC).

We undertook this cross-sectional diagnostic test accuracy study to assess the diagnostic accuracy of the Shoulder Screen in our patient population. The primary objective was to evaluate how well this multivariable prediction model distinguished cases of known shoulder dystocia from vaginal deliveries without shoulder dystocia. Other objectives were to compare the performance of the model with the intervention criteria based on birth weight promulgated by ACOG and to assess whether the model could potentially decrease malpractice costs.

Methods

New Hanover Regional Medical Center is a tertiary hospital in North Carolina with a residency program in obstetrics and gynecology where ∼4,000 deliveries occur annually. The definition of shoulder dystocia used in clinical practice at this institution is a delivery requiring use of specialized maneuvers, beyond gentle downward traction of the head, to deliver the shoulders. All patients who experienced shoulder dystocia during vaginal delivery for the period January 1, 2010, to December 31, 2013, during which more than 17,000 deliveries occurred, were identified by querying the International Classification of Diseases-9 codes for the diagnosis of shoulder dystocia. These patients were designated as Group SD.

The patient group that did not experience shoulder dystocia was selected by choosing every ninth patient from randomly selected medical record numbers of all patients from the same population who had normal vaginal deliveries without shoulder dystocia during the same time frame. Every ninth patient was chosen to comply with the Institutional Review Board limitations placed on the number of patients whose medical records could be abstracted. These patients were designated as Group NVD. Gravity, parity, maternal height, maternal weight, BMI, gestational age at delivery, birth weight, birth weight percentile, and Apgar scores were recorded for each birth. The medical records of all infants delivered with a shoulder dystocia were analyzed to ascertain the presence of brachial plexus injury, fracture of the clavicle or humerus (from X-rays), neonatal encephalopathy, cardiopulmonary resuscitation (CPR) or use of positive pressure ventilation in the delivery suite, or 5-minute Apgar score <6.

The variables required for the Shoulder Screen to calculate risk of shoulder dystocia are parity, maternal height, maternal BMI, birth weight, and birth weight percentile. The equation produces a number between 0 and 1, called the risk score. Each number correlates with a specific risk of shoulder dystocia, as determined by Dyachenko et al[9] and Hamilton et al.[10] The likelihood of shoulder dystocia with and without persistent injury is directly related to the size of the risk score. Hamilton et al[10] showed that risk scores of >0.4 were present in 54.8% of patients with shoulder dystocia and persistent brachial plexus palsy, 23.1% of patients with temporary brachial plexus palsy, 16.5% of patients with uncomplicated shoulder dystocia, and in 2.5% of women who delivered vaginally without shoulder dystocia.

For our study, a Shoulder Screen risk score for shoulder dystocia elevated enough to justify offering a cesarean section in clinical practice was defined as a score ≥0.4. This score was based on the above-described reports of the prediction model's detection rates and false-positive rates at higher and lower scores.[9] [10]

We employed statistical measures under the STARD 2015 guidelines for reporting diagnostic accuracy studies to accomplish our primary objective to evaluate how well this multivariable prediction model distinguished cases of known shoulder dystocia from vaginal deliveries without shoulder dystocia. These measures included sensitivity and specificity, false-positive rate, positive and negative likelihood ratios, and the receiver operating characteristic curve. We also examined test performance in a subgroup of patients whose shoulder dystocia was complicated by infants experiencing fracture, brachial plexus injury, encephalopathy, need for CPR or positive pressure ventilation in the delivery suite, or 5-minute Apgar score <6. Categorical variables were compared using chi-square test or Fisher's exact test, as appropriate. Continuous variables were assessed for normality using the D'Agostino and Pearson's test and Kolmogorov–Smirnov's test. Variables that were not normally distributed were compared with the Mann–Whitney's test and normally distributed variables were compared using the Student's t-test. All tests were two tailed and a probability value of <0.05 was significant.

Results

There were 201 patients (199 with complete data) in Group SD and 199 patients in Group NVD. Demographically, median values for Group SD revealed statistically significant more advanced gestational age at delivery, higher maternal weight, greater BMI, and higher Shoulder Screen test results than Group NVD patients. Also, the rates of nulliparity, birth weight more than 4,500 g, and birth weight over the 90th percentile were higher in Group SD. These results are presented in [Table 1]. No infant in either group weighed more than 5,000 g.

Table 1 Maternal and fetal characteristics

	Group SD n = 199 Median (IQR)	Group NVD n = 199 Median (IQR)	p-Value
Gestational age	39.7 (39.0–40.6)	39.1 (38.4–39.7)	<0.0001
Maternal height	1.63 (1.57–1.68)	1.63 (1.57–1.68)	0.15
Maternal weight	84.8 (74.8–98.4)	75.8 (67.6–84.8)	<0.0001
Maternal BMI	31.9 (28.3–37.1)	29.2 (25.7–32.7)	<0.0001
Birth weight	3940 (3600–4218)	3260 (3005–3600)	<0.0001
Birth weight percentile	0.82 (0.58–0.93)	0.37 (0.16–0.61)	<0.0001
Shoulder Screen score	0.14 (0.03–0.039)	0.01(0.002–0.03)	<0.001
	n (%)	n (%)
Nulliparity	91 (45.7)	33 (16.6)	<0.0001
Birth weight > 4,500 g	20 (10.1)	0 (0.0)	<0.0001
Birth weight > 0.90	65 (32.7)	23 (11.5)	<0.0001

Abbreviations: BMI, body mass index; IQR, interquartile range.

[Table 2] shows the sensitivity (23.1%), specificity (99.5%), false-positive rate (0.5%), positive and negative predictive values (97.9 and 56.4%), positive and negative likelihood ratios (46 and 0.77), and accuracy (61.3%) of the Shoulder Screen test in detecting all the Group SD patients and distinguishing them from the Group NVD patients. The false-positive rate represents potentially unnecessary cesarean deliveries that could result from the Shoulder Screen.

Table 2 Standard performance measures of the Shoulder Screen test (a positive test = test score >0.4)

Shoulder Screen test	Group SD	Group NVD
Positive	46	1
	True positive (a)	False positive (b)
Negative	153	198
	False negative (c)	True negative (d)
Sensitivity	a/(a + c)	23.1%
Specificity	d/(b + d)	99.5%
False-positive rate	1-specificity	0.5%
Positive likelihood ratio	Sensitivity/(1 − specificity)	46
Negative likelihood ratio	(1/sensitivity)/specificity	0.77
Accuracy	(a + d)/(a + b + c + d)	61.3%

Since our study population had no neonates with birth weight more than 5,000 g, we compared the Shoulder Screen with the more stringent ACOG criteria described earlier for diabetic patients for detecting patients at risk of shoulder dystocia, estimated fetal weight of ≥4,500 g. These results are presented in [Table 3]. Sensitivity was significantly higher with the Shoulder Screen test compared with the 4,500 g criterion (23.1 vs. 10.1%, p = 0.0007) in detecting shoulder dystocia. Since the majority of patients with shoulder dystocia had infants weighed less than 4,500 g, the Shoulder Screen test detected 23.1% and the ACOG criterion detected only 10.1% of all patients with shoulder dystocia.

Table 3 Standard performance measures using BW only (positive = BW >4,500 g)

	Group SD	Group NVD
Positive (BW >4,500 g)	20	0
	True positive (a)	False positive (b)
Negative (BW <4,500 g)	179	199
	False negative (c)	True negative (d)
Sensitivity	a/(a + c)	10.1%
Specificity	d/(b + d)	100%
False-positive rate	1 − specificity	0.0%
Positive likelihood ratio	Sensitivity/(1 − specificity)	NA
Negative likelihood ratio	(1 − sensitivity)/specificity	0.90
Accuracy	(a + b)/(a + b + c + d)	55.0%

Abbreviations: BW, birth weight; NA, not available.

Some of the infants born to Group SD patients had neonatal complications. A total of 24/199 (12%) infants in this group had one or more of the following complications: humerus fracture, brachial plexus injury, encephalopathy, requirement for CPR or positive pressure ventilation, or a 5-minute Apgar ≤6. Brachial plexus injury was present in four infants and was persistent at the time of discharge in one infant. The most common injury was radiographically demonstrated humerus fracture in 8 (4%). The Shoulder Screen at our selected risk score of >0.4 identified 33.3% of the infants with complications. The ACOG birth weight criterion detected none of them. [Table 4] shows the detection rate for these patients using the Shoulder Screen compared with detection using the ACOG.

Table 4 Detection rates using Shoulder Screen Test versus birth weight criterion in complicated and uncomplicated shoulder dystocia

	Shoulder Screen score >0.4		Birth weight >4,500 g
	n	n (%)	n (%)	p-Value
SD with complication	24	8 (33.3)	0 (0)	0.0039
SD without complication	175	38 (21.7)	20 (11.4)	0.0104

The receiver operating characteristic curve in [Fig. 1] shows the sensitivity in Group SD and false-positive rates in Group NVD for each possible Shoulder Screen test result that could be chosen as the threshold for clinical intervention. The location of our selected risk score of >0.4 is shown in red. The area under the curve is 0.86.

Fig. 1 Receiver operating characteristic curve for the Shoulder Screen multivariable test. The red circle indicates the performance of the test at a risk score result of 0.4.

Discussion

The findings of this study suggest that some shoulder dystocias may be predicted, and their complications, thereby, possibly prevented. Even though the Shoulder Screen test, utilizing a score of >0.4 is not perfect, it did detect one-third of the potentially seriously injured infants in our study. It performed better than the ACOG intervention based on birth weight exceeding 4,500 g, especially since all infants with complicated shoulder dystocia in our study weighed less than 4,500 g. Furthermore, with its application to only diabetic patients, the ACOG clinical guideline would have even lower detection rates than shown in [Table 4].

The strength of this study is that the patients were drawn from a population consisting equally of private, commercially insured patients and poor, uninsured and Medicaid insured patients.

A weakness of this study is that we were unable to show detection rates for persistent brachial plexus palsy. Even though our study population consisted of more than 17,000 deliveries, only one case of persistent nerve injury occurred. Analysis of ∼100,000 deliveries would be required to show the detection rate for permanent brachial plexus palsy. To overcome this problem, Dyachenko et al[9] and Hamilton et al[10] were able to use 221 malpractice claim cases involving shoulder dystocia with persistent brachial plexus injury in their development of the Shoulder Screen.

Other researchers have reported attempts to predict shoulder dystocia with multivariate statistical techniques, ultrasound to identify the infant with macrosomia or with disproportionately large bisacromial or trunk diameter, and empiric risk scores based on the number and type of risk factors. These methods have yielded mixed results.[7] [11] [12] [13] Deaver and Cohen devised a test to predict risk of shoulder dystocia utilizing statistical modeling and reported that a risk score of 0.72 would prevent 36% of brachial plexus palsy and result in 14 unnecessary cesarean deliveries for each brachial plexus palsy that was prevented.[14] However, none of these models approached the problem as an anatomical, biomechanical misfit, and interaction between maternal and fetal size.[10] None had sufficient numbers to examine shoulder dystocia with permanent brachial plexus palsy separately.

Criticisms of the Shoulder Screen test could include: (1) it is too cumbersome for the busy practitioner to use; (2) it results in an unacceptable increase in the elective cesarean delivery rate; and (3) ultrasound estimated fetal weight has to be used instead of the actual birth weight in the clinical application of the model.

To gather the four (birth weight percentile is calculated in the model) clinical variables used to compute the Shoulder Screen test score from a patient's medical record, access the Web site, enter the data, and retrieve the score should be completed with minimal effort and time and can be done by an assistant.

Concern about an increase in the cesarean rate is unfounded with the results of our study. Selection of a Shoulder Screen test score of >0.4 as the point at which cesarean delivery is offered would potentially increase the cesarean delivery rate by less than 1%.

The evidence concerning the accuracy of estimated fetal weight by ultrasound in estimating actual birth weight is conflicting. An accurate assessment of fetal birth weight is paramount in deciding which patients are at high risk of brachial plexus palsy.[7] [15] [16] [17] [18] [19] [20] [21] Like the Shoulder Screen, the ACOG guidelines on management of fetal macrosomia are derived from studies utilizing birth weight, but the clinical application of the guidelines utilizes estimated fetal weight. The ACOG practice bulletin advises that the accuracy of estimated fetal weight using ultrasound biometry is no better than clinical palpation maneuvers since ultrasound software formulas have a 13% error for infant >4,500 g.[19] However, Dudley reported the mean deviation exceeded 14% of birth weight in only 5% of ultrasound weight estimates. At 4,500 g, the overestimation never reached 15% and the underestimation of weight never reached 20%.[21] Furthermore, Alsulyman et al found an absolute error of 13% for birth weight ≥ 4,500 g and 8% for birth weight <4,000 g.[22] The Shoulder Screen maintains its detection rate accuracy at plus or minus 16.3% of actual birth weight.[9]

To evaluate the prospective clinical use of the Shoulder Screen, Daly et al published a report of their experience utilizing the Shoulder Screen test in 8,767 deliveries at two sites.[23] They also chose 0.4 as the high-risk score at which cesarean delivery was offered. Estimated fetal weight by ultrasound was used as birth weight in the calculations. Patients with risk scores of <0.4 planned vaginal delivery. Over the course of the study, the incidence of shoulder dystocia fell, the rate of inductions of labor for macrosomia decreased, while the primary cesarean delivery rate did not increase. The population studied was too small to detect whether the occurrence of brachial plexus palsy decreased.

The Shoulder Screen test is able to predict more cases of shoulder dystocia by lowering the threshold risk score at which to offer cesarean section, but the false-positive rate increases accordingly. Each practice and institution using the test chooses whatever threshold score is prudent for its institutional goals, considering the advantages of higher detection rates of the adverse outcome and the disadvantages with the associated false positives that are potentially unnecessary cesarean deliveries.

The search for a method to make patients and clinicians safer from the medical and economic harm from the serious injury of shoulder dystocia with brachial plexus palsy is paramount. As a result malpractice costs associated with brachial plexus palsy would decrease.

This study demonstrates that the Shoulder Screen test accurately detects a portion of patients at increased risk for shoulder dystocia deliveries and deliveries with permanent injury, utilizing ultrasound estimated fetal weight rather than birth weight with a minimal increase in the primary cesarean delivery rate.

Conflict of Interest

None.

Acknowledgments

The authors would like to thank Emily Hamilton, MD, McGill University, Montreal, Canada and PeriGen, Cary, NC for data analysis, and Bethany Barrow, MD, Lovelace Medical Group-Las Estancias, Albuquerque, NM and Jennifer Salter, DO, St. Petersburg General Hospital, St. Petersburg, FL, for assistance in data gathering.

• References

• 1 Chauhan SP, Blackwell SB, Ananth CV. Neonatal brachial plexus palsy: incidence, prevalence, and temporal trends. Semin Perinatol 2014; 38 (04) 210-218
• 2 CRICO 2010 Annual Benchmarking Report: Malpractice Risks in Obstetrics. Available at: https://www.rmf.harvard.edu/Malpractice-Data/Annual-Benchmark-Reports/Risks-in-Obstetrics/ . Retrieved February 9, 2018
• 3 American College of Obstetricians and Gynecologists. Shoulder Dystocia. Practice Bulletin 178. Washington, DC: American College of Obstetricians and Gynecologists; 2017
• 4 American College of Obstetricians and Gynecologists. Task Force on Neonatal Brachial Plexus Palsy. Washington, DC: American College of Obstetricians and Gynecologists; 2014
• 5 Foad SL, Mehlman CT, Ying J. The epidemiology of neonatal brachial plexus palsy in the United States. J Bone Joint Surg Am 2008; 90 (06) 1258-1264
• 6 Ouzounian JG, Korst LM, Miller DA, Lee RH. Brachial plexus palsy and shoulder dystocia: obstetric risk factors remain elusive. Am J Perinatol 2013; 30 (04) 303-307
• 7 Parantainen J, Palomäki O, Talola N, Uotila J. Clinical and sonographic risk factors and complications of shoulder dystocia - a case-control study with parity and gestational age matched controls. Eur J Obstet Gynecol Reprod Biol 2014; 177: 110-114
• 8 Zuarez-Easton S, Zafran N, Garmi G, Nachum Z, Salim R. Are there modifiable risk factors that may predict the occurrence of brachial plexus injury?. J Perinatol 2015; 35 (05) 349-352
• 9 Dyachenko A, Ciampi A, Fahey J, Mighty H, Oppenheimer L, Hamilton EF. Prediction of risk for shoulder dystocia with neonatal injury. Am J Obstet Gynecol 2006; 195 (06) 1544-1549
• 10 Hamilton EF, Ciampi A, Dyachenko A, Lerner HM, Miner L, Sandmire HF. Is shoulder dystocia with brachial plexus injury preventable?. Fetal Matern Med Rev 2008; 19 (04) 293-310
• 11 Hill MG, Cohen WR. Shoulder dystocia: prediction and management. Womens Health (Lond) 2016; 12 (02) 251-261
• 12 Belfort MA, Dildy GA, Saade GR, Suarez V, Clark SL. Prediction of shoulder dystocia using multivariate analysis. Am J Perinatol 2007; 24 (01) 5-10
• 13 Cohen B, Penning S, Major C, Ansley D, Porto M, Garite T. Sonographic prediction of shoulder dystocia in infants of diabetic mothers. Obstet Gynecol 1996; 88 (01) 10-13
• 14 Deaver JE, Cohen WR. An approach to the prediction of neonatal Erb palsy. J Perinat Med 2009; 37 (02) 150-155
• 15 Zhang X, Decker A, Platt RW, Kramer MS. How big is too big? The perinatal consequences of fetal macrosomia. Am J Obstet Gynecol 2008; 198 (05) 517.e1-517.e6
• 16 Dajani NK, Magann EF. Complications of shoulder dystocia. Semin Perinatol 2014; 38 (04) 201-204
• 17 Hehir MP, Mchugh AF, Maguire PJ, Mahony R. Extreme macrosomia--obstetric outcomes and complications in birthweights >5000 g. Aust N Z J Obstet Gynaecol 2015; 55 (01) 42-46
• 18 Iffy L, Varadi V, Papp Z. Epidemiologic aspects of shoulder dystocia-related neurological birth injuries. Arch Gynecol Obstet 2015; 291 (04) 769-777
• 19 American College of Obstetricians and Gynecologists. Macrosomia. Practice Bulletin 173. Washington, DC: American College of Obstetricians and Gynecologists; 2016
• 20 Rouse DJ, Owen J, Goldenberg RL, Cliver SP. The effectiveness and costs of elective cesarean delivery for fetal macrosomia diagnosed by ultrasound. JAMA 1996; 276 (18) 1480-1486
• 21 Dudley NJ. A systematic review of the ultrasound estimation of fetal weight. Ultrasound Obstet Gynecol 2005; 25 (01) 80-89
• 22 Alsulyman OM, Ouzounian JG, Kjos SL. The accuracy of intrapartum ultrasonongraphic fetal weight estimation. Am J Obstet Gynecol 1997; 177: 503-506
• 23 Daly MV, Bender C, Townsend KE, Hamilton EF. Outcomes associated with a structured prenatal counseling program for shoulder dystocia with brachial plexus injury. Am J Obstet Gynecol 2012; 207 (02) 123.e1-123.e5