Appl Clin Inform 2024; 15(02): 295-305
DOI: 10.1055/s-0044-1785688
Research Article

Identifying Barriers to The Implementation of Communicating Narrative Concerns Entered by Registered Nurses, An Early Warning System SmartApp

Mollie Hobensack
1   Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, New York City, New York, United States
Jennifer Withall
2   Department of Biomedical Informatics, Columbia University, New York City, New York, United States
Brian Douthit
3   Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee, United States
Kenrick Cato
4   Department of Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
Patricia Dykes
5   Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States
Sandy Cho
6   Department of Clinical Informatics, Newton-Wellesley Hospital, Newton, Massachusetts, United States
Graham Lowenthal
7   Brigham and Women's Hospital, Boston, Massachusetts, United States
Catherine Ivory
8   Vanderbilt University Medical Center, Nashville, Tennessee, United States
Po-Yin Yen
9   Washington University School of Medicine in St. Louis, St. Louis, MO, United States
Sarah Rossetti
2   Department of Biomedical Informatics, Columbia University, New York City, New York, United States
› Author Affiliations
Funding This project is supported by the National Institute of Nursing Research (1R01NR016941-01, T32NR007969), American Nurses Foundation (ANF): Reimagining Nursing Initiative, and Jonas Scholarship.


Background Nurses are at the frontline of detecting patient deterioration. We developed Communicating Narrative Concerns Entered by Registered Nurses (CONCERN), an early warning system for clinical deterioration that generates a risk prediction score utilizing nursing data. CONCERN was implemented as a randomized clinical trial at two health systems in the Northeastern United States. Following the implementation of CONCERN, our team sought to develop the CONCERN Implementation Toolkit to enable other hospital systems to adopt CONCERN.

Objective The aim of this study was to identify the optimal resources needed to implement CONCERN and package these resources into the CONCERN Implementation Toolkit to enable the spread of CONCERN to other hospital sites.

Methods To accomplish this aim, we conducted qualitative interviews with nurses, prescribing providers, and information technology experts in two health systems. We recruited participants from July 2022 to January 2023. We conducted thematic analysis guided by the Donabedian model. Based on the results of the thematic analysis, we updated the α version of the CONCERN Implementation Toolkit.

Results There was a total of 32 participants included in our study. In total, 12 themes were identified, with four themes mapping to each domain in Donabedian's model (i.e., structure, process, and outcome). Eight new resources were added to the CONCERN Implementation Toolkit.

Conclusions This study validated the α version of the CONCERN Implementation Toolkit. Future studies will focus on returning the results of the Toolkit to the hospital sites to validate the β version of the CONCERN Implementation Toolkit. As the development of early warning systems continues to increase and clinician workflows evolve, the results of this study will provide considerations for research teams interested in implementing early warning systems in the acute care setting.

Human Subject Protections

Consent was provided by all participants included in the study. Participant names and videos were not recorded during the interviews; only audio was recorded. The interviewer (M.H.) cleaned the qualitative transcripts prior to further analysis. All data were stored on a protective drive that was password protected. All recordings and transcripts will be destroyed following publication. Approval from the Institutional Review Boards at both study sites was provided.

Publication History

Received: 02 October 2023

Accepted: 06 February 2024

Article published online:
17 April 2024

© 2024. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Morgan RW, Fitzgerald JC, Weiss SL, Nadkarni VM, Sutton RM, Berg RA. Sepsis-associated in-hospital cardiac arrest: epidemiology, pathophysiology, and potential therapies. J Crit Care 2017; 40: 128-135
  • 2 Song MJ, Lee DS, Choi YY. et al. Incidence of preventable cardiopulmonary arrest in a mature part-time rapid response system: a prospective cohort study. PLoS One 2022; 17 (02) e0264272
  • 3 Rhee C, Jones TM, Hamad Y. et al; Centers for Disease Control and Prevention (CDC) Prevention Epicenters Program. Prevalence, underlying causes, and preventability of sepsis-associated mortality in US acute care hospitals. JAMA Netw Open 2019; 2 (02) e187571-e187571
  • 4 Schnock KO, Kang MJ, Rossetti SC. et al. Identifying nursing documentation patterns associated with patient deterioration and recovery from deterioration in critical and acute care settings. Int J Med Inform 2021; 153: 104525
  • 5 Rossetti SC, Dykes PC, Knaplund C. et al. The Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) clinical decision support early warning system: protocol for a cluster randomized pragmatic clinical trial. JMIR Res Protoc 2021; 10 (12) e30238
  • 6 Davis MM, Howk S, Spurlock M, McGinnis PB, Cohen DJ, Fagnan LJ. A qualitative study of clinic and community member perspectives on intervention toolkits: “Unless the toolkit is used it won't help solve the problem”. BMC Health Serv Res 2017; 17 (01) 497
  • 7 CONCERN Study. Columbia Department of Biomedical Informatics. Published 2023. Accessed December 5, 2023 at:
  • 8 Kang MJ, Rossetti SC, Knaplund C. et al. Nursing documentation variation across different medical facilities within an integrated healthcare system. Comput Inform Nurs 2021; 39 (12) 845-850
  • 9 Implementation Toolkit Logic Model CONCERN. American Nurses Foundation. Accessed December 5, 2023 At:
  • 10 Thoele K, Ferren M, Moffat L, Keen A, Newhouse R. Development and use of a toolkit to facilitate implementation of an evidence-based intervention: a descriptive case study. Implement Sci Commun 2020; 1 (01) 86
  • 11 Fall TIPS. A Patient-Centered Fall Prevention Toolkit. Agency for Healthcare Research and Quality. Published 2021. Accessed December 5, 2023 at:
  • 12 Dykes PC, Khasnabish S, Burns Z. et al. Development and validation of a fall prevention efficiency scale. J Patient Saf 2022; 18 (02) 94-101
  • 13 Murray E, May C, Mair F. Development and formative evaluation of the e-Health Implementation Toolkit (e-HIT). BMC Med Inform Decis Mak 2010; 10 (01) 61
  • 14 Saunders B, Sim J, Kingstone T. et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant 2018; 52 (04) 1893-1907
  • 15 Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005; 83 (04) 691-729
  • 16 Moorman LP. Principles for real-world implementation of bedside predictive analytics monitoring. Appl Clin Inform 2021; 12 (04) 888-896
  • 17 Kurtzman ET, Ghazal LV, Girouard S. et al. Nursing workforce challenges in the postpandemic world. J Nurs Regul 2022; 13 (02) 49-60
  • 18 McDermott C. Reimagining the preceptor role. Nurs Adm Q 2023; 47 (03) 227-233
  • 19 Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: a tool to enhance trustworthiness or merely a nod to validation?. Qual Health Res 2016; 26 (13) 1802-1811