Psychometric Evaluation of the Modified Neonatal Resuscitation Program Adherence Assessment Tool When Utilized for In Situ Simulation and Telesimulation Scenarios

Abstract

Objective

Neonatal resuscitation is a high acuity, low occurrence event that requires precision, teamwork, and rapid decision-making. Simulation using Neonatal Resuscitation Program (NRP) guidelines allows teams to build competency and preparedness, improving neonatal outcomes. Accurate, reliable performance evaluation during simulation is essential to the provision of meaningful feedback. Evidence supporting the reliability and validity of existing tools when used in telesimulation (TS) is limited. We sought to evaluate the psychometric properties of the modified NRP (mNRP) tool when assessing interprofessional team performance in onsite and TS environments.

Study Design

We employed a methodological design to conduct secondary analyses of data from a larger study using simulation to enhance resuscitation skills as assessed by the mNRP. Item-level data from 96 simulations were subjected to Classical Test Theory-based item analyses, including evaluation of item difficulty, discrimination, and item-to-total correlation; reliability; and exploratory factor analysis (EFA). Interrater reliability (IRR) between novice and expert raters was assessed for a subset of cases.

Results

Item difficulties reflected a desirable mix of difficulty in endorsement, suggesting that items capture a range of guideline adherence. Most item discrimination (range: 0.05–0.81) and corrected item-total correlation (range: 0.005–0.68) values had moderate-to-strong, positive correlations with total scores, indicating discriminative ability. EFA yielded three and four components for the onsite and TS groups, respectively. Cronbach's α was 0.76 (onsite) or 0.78 (TS). Lowest versus highest mNRP quartiles differed significantly in both formats (p < 0.001 for each), supporting construct validity. IRR ranged from 0.5 to 0.9, supporting moderate to good agreement between novice and expert raters.

Conclusion

Findings provide evidence supporting the reliability and validity of the mNRP tool when applied in both the in situ and TS settings. IRR was acceptable for expert and novice evaluators. This analysis provides additional validity evidence for the mNRP when used to evaluate interprofessional team performance in both onsite and TS formats.

Key Points

• Neonatal resuscitation is a high-acuity, low-occurrence event.

• Many rural clinicians feel underprepared for high acuity, low occurrence events.

• Simulation-based training can improve preparedness.

• Performance is assessed using the mNRP tool.

• Psychometric features support the tool for use in onsite and TS formats.

Keywords

simulation training - resuscitation education - psychometrics - rural health - interprofessional team training - neonatal resuscitation

Contributors' Statement

A.Z., M.M., T.D.S., L.M., L.S., and A.C. conceived, designed the study, and obtained research funding to support the work described here. A.Z., M.M., L.M., L.S., T.P., S.S., and S.M. supervised the conduct of the study and data collection. L.S. and T.P. managed the data, including quality control. T.D.S. and A.C. provided statistical advice on the study design and analyzed the data. T.D.S., A.C., M.M., A.Z., and L.M. interpreted the data. S.M., T.D.S., and L.M. generated the first draft of the manuscript, and all authors contributed substantially to its revision. All authors take responsibility for the manuscript as a whole.

Note

Abstracts summarizing the work reported here were presented at the Pediatric Academic Societies Meeting in Honolulu, Hawaii, United States, in April 2025 and at the Costas Lambrew Research Retreat in Portland, Maine, United States, in May 2025. An abstract has been submitted for consideration for presentation at the International Meeting on Simulation in Healthcare to be held in San Antonio, Texas, United States, in January 2026.

Publikationsverlauf

Eingereicht: 08. September 2025

Angenommen: 13. Oktober 2025

Accepted Manuscript online:
14. Oktober 2025

Artikel online veröffentlicht:
30. Oktober 2025

• References

• 1 Garvey AA, Dempsey EM. Simulation in neonatal resuscitation. Front Pediatr 2020; 8: 59
• 2 Hand I. Neonatal resuscitation in low volume hospital settings. Children (Basel) 2022; 9 (05) 607
• 3 Lindhard MS, Thim S, Laursen HS, Schram AW, Paltved C, Henriksen TB. Simulation-based neonatal resuscitation team training: a systematic review. Pediatrics 2021; 147 (04) e2020042010
• 4 Huang J, Tang Y, Tang J. et al. Educational efficacy of high-fidelity simulation in neonatal resuscitation training: a systematic review and meta-analysis. BMC Med Educ 2019; 19 (01) 323
• 5 Dempsey E, Pammi M, Ryan AC, Barrington KJ. Standardised formal resuscitation training programmes for reducing mortality and morbidity in newborn infants. Cochrane Database Syst Rev 2015; 2015 (09) CD009106
• 6 Evans CL, Bazant E, Atukunda I. et al. Peer-assisted learning after onsite, low-dose, high-frequency training and practice on simulators to prevent and treat postpartum hemorrhage and neonatal asphyxia: a pragmatic trial in 12 districts in Uganda. PLoS One 2018; 13 (12) e0207909
• 7 Jukkala AM, Henly SJ. Provider readiness for neonatal resuscitation in rural hospitals. J Obstet Gynecol Neonatal Nurs 2009; 38 (04) 443-452
• 8 Craig A, Cutler A, Kerecman J. et al. Association of low hospital birth volume and adverse short-term outcomes for neonates treated with therapeutic hypothermia in rural states. Res Sq 2024
• 9 Melendi M, Zanno AE, Holmes JA. et al. Development and evaluation of a rural longitudinal neonatal resuscitation program telesimulation program (MOOSE: Maine Ongoing Outreach Simulation Education). Am J Perinatol 2025; 42 (06) 796-805
• 10 Carbine DN, Finer NN, Knodel E, Rich W. Video recording as a means of evaluating neonatal resuscitation performance. Pediatrics 2000; 106 (04) 654-658
• 11 van der Heide PA, van Toledo-Eppinga L, van der Heide M, van der Lee JH. Assessment of neonatal resuscitation skills: a reliable and valid scoring system. Resuscitation 2006; 71 (02) 212-221
• 12 Weiner GM. Ed. Textbook of Neonatal Resuscitation. 8th ed.. Itasca, IL: American Academy of Pediatrics; 2021
• 13 Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ 2003; 37 (09) 830-837
• 14 Zanno A, Melendi M, Chipman M. et al. A novel telesimulation program, Maine Ongoing Outreach Simulation Education (MOOSE), evaluates reproducibility in NRP adherence at additional rural hospitals. Simul Healthc 2024; 19 (01) e1-e51
• 15 Zanno A, Melendi M, Cutler A. et al. Simulation-Based outreach program improves rural hospitals' team confidence in neonatal resuscitation. Cureus 2022; 14 (09) e28670
• 16 Nunnally JC. Psychometric theory. New York, NY: McGraw-Hill; 1978
• 17 Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15 (02) 155-163
• 18 Cristobal E, Flavián C, Guinaliu M. Perceived e-service quality (PeSQ) measurement validation and effects on consumer satisfaction and web site loyalty. Manag Serv Qual 2007; 17: 317-340
• 19 Crocker L, Algina J. Introduction to classical and modern test theory. Mason, OH: Cengage Learning; 2008
• 20 Kaiser HF. The application of electronic computers to factor analysis. Educ Psychol Meas 1960; 20: 141-151
• 21 Zwick WR, Velicer WF. Factors influencing four rules for determining the number of components to retain. Multivariate Behav Res 1982; 17 (02) 253-269
• 22 Burns N, Grove SK. The concepts of measurement. In: Burns N, Grove SK. eds. The practice of nursing research: Appraisal, synthesis, and generation of evidence. 6th ed.. St. Louis, MO: Elsevier Saunders; 2009: 371-394
• 23 Gelbart B, Hiscock R, Barfield C. Assessment of neonatal resuscitation performance using video recording in a perinatal centre. J Paediatr Child Health 2010; 46 (7–8): 378-383
• 24 Cattell RB. The scree test for the number of factors. Multivariate Behav Res 1966; 1 (02) 245-276