Subscribe to RSS
DOI: 10.1055/a-2594-3770
“I worry we'll blow right by it:” Barriers to Uptake of the STRATIFY-CDS for Acute Heart Failure
Funding This work was supported by the National Heart Lung and Blood Institute (grant no.: R01 HL157596). Dr. Christensen was supported by an institutional training grant from the National Institute of General Medical Science (grant no.: T32 GM135094).

Abstract
Background
Clinical decision support (CDS) tools in electronic health records (EHRs) often face low uptake due to limited usability, workflow integration, and other implementation issues. We recently designed and implemented the STRATIFY-CDS tool, which calculates a validated risk-prediction model and recommends disposition for emergency department (ED) patients with acute heart failure. Despite applying human-centered design and implementation science strategies, initial utilization in the first 3 months of the STRATIFY-CDS tool was just 3%.
Objective
To identify usability issues and contextual barriers to uptake of STRATIFY-CDS tool among ED clinicians.
Methods
We performed an exploratory qualitative and simulation study with ED clinicians at Vanderbilt University Medical Center who had used the STRATIFY-CDS tool at least once. Semi-structured interviews with interactive simulation (summative usability) were conducted via videoconference. Two authors performed thematic analysis informed by the Technology Acceptance Model.
Results
Of 13 invited ED clinicians, 10 participated (7 attending and 3 resident physicians) with 1 to 11 prior tool uses. Although the main user interface had high perceived usability, participants struggled to find the launch button. The perceived utility was low-to-moderate and varied based on whether the recommendation matched the participant's clinical gestalt. When there was mismatch, perceived utility was lower, and participants needed more information about the risk model and supporting evidence, which were not readily available. Despite educational implementation strategies and ED leadership approval, there was not a strong social norm to use the tool.
Conclusion
Although the main user interface had high usability, poor visibility of the launch button coupled with low familiarity with the underlying evidence and lack of a social norm impaired uptake of the STRATIFY-CDS tool. Future work on CDS design should test novel non-interruptive launch mechanisms and evaluate training with simulation as an implementation strategy to bolster initial confidence and excitement around the CDS.
Keywords
clinical decision support - user-centered design - electronic health record - heart failure - risk adjustmentProtection of Human and Animal Subjects
The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed and approved by the Vanderbilt University Medical Center Institutional Review Board. Informed consent was obtained from all participants.
Publication History
Received: 31 January 2025
Accepted: 24 April 2025
Article published online:
05 September 2025
© 2025. Thieme. All rights reserved.
Georg Thieme Verlag KG
Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany
-
References
- 1 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
- 2 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3 (01) 17
- 3 Chaparro JD, Beus JM, Dziorny AC. et al. Clinical decision support stewardship: best practices and techniques to monitor and improve interruptive alerts. Appl Clin Inform 2022; 13 (03) 560-568
- 4 McDaniel RB, Burlison JD, Baker DK. et al. Alert dwell time: introduction of a measure to evaluate interruptive clinical decision support alerts. J Am Med Inform Assoc 2016; 23 (e1): e138-e141
- 5 Chisholm CD, Weaver CS, Whenmouth L, Giles B. A task analysis of emergency physician activities in academic and community settings. Ann Emerg Med 2011; 58 (02) 117-122
- 6 Patterson BW, Pulia MS, Ravi S. et al. Scope and influence of electronic health record-integrated clinical decision support in the emergency department: a systematic review. Ann Emerg Med 2019; 74 (02) 285-296
- 7 Lewis JR. Usability: lessons learned … and yet to be learned. Int J Hum Comput Interact 2014; 30 (09) 663-684
- 8 Salwei ME, Hoonakker P, Carayon P, Wiegmann D, Pulia M, Patterson BW. Usability of a human factors-based clinical decision support in the emergency department: lessons learned for design and implementation. Hum Factors 2024; 66 (03) 647-657
- 9 Kawamoto K, McDonald CJ. Designing, conducting, and reporting clinical decision support studies: recommendations and call to action. Ann Intern Med 2020; 172 (11, suppl): S101-S109
- 10 Campbell R. The five “rights” of clinical decision support. J AHIMA 2013; 84 (10) 42-47 , quiz 48
- 11 Semler MW, Weavind L, Hooper MH. et al. An electronic tool for the evaluation and treatment of sepsis in the ICU: a randomized controlled trial. Crit Care Med 2015; 43 (08) 1595-1602
- 12 Chan SL, Lee JW, Ong MEH. et al. Implementation of prediction models in the emergency department from an implementation science perspective—determinants, outcomes, and real-world impact: a scoping review. Ann Emerg Med 2023; 82 (01) 22-36
- 13 Kilsdonk E, Peute LW, Jaspers MWM. Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. Int J Med Inform 2017; 98: 56-64
- 14 Novak LL, Anders S, Unertl KM, France DJ, Weinger MB. Improving the effectiveness of health information technology: the case for situational analytics. Appl Clin Inform 2019; 10 (04) 771-776
- 15 Carayon P, Wetterneck TB, Rivera-Rodriguez AJ. et al. Human factors systems approach to healthcare quality and patient safety. Appl Ergon 2014; 45 (01) 14-25
- 16 Proctor E, Silmere H, Raghavan R. et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38 (02) 65-76
- 17 Watson J, Hutyra CA, Clancy SM. et al. Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?. JAMIA Open 2020; 3 (02) 167-172
- 18 Trinkley KE, Wright G, Allen LA. et al. Sustained effect of clinical decision support for heart failure: a natural experiment using implementation science. Appl Clin Inform 2023; 14 (05) 822-832
- 19 Salwei ME, Carayon P, Wiegmann D, Pulia MS, Patterson BW, Hoonakker PLT. Usability barriers and facilitators of a human factors engineering-based clinical decision support technology for diagnosing pulmonary embolism. Int J Med Inform 2021; 158: 104657
- 20 Ward MJ, Chavis B, Banerjee R, Katz S, Anders S. User-centered design in pediatric acute care settings antimicrobial stewardship. Appl Clin Inform 2021; 12 (01) 34-40
- 21 Reese TJ, Mixon AS, Matheny ME. et al. Using intervention mapping to design and implement a multicomponent intervention to improve antibiotic and NSAID prescribing. Transl Behav Med 2023; 13 (12) 928-943
- 22 Kurmani S, Squire I. Acute heart failure: definition, classification and epidemiology. Curr Heart Fail Rep 2017; 14 (05) 385-392
- 23 Sax DR, Mark DG, Rana JS. et al. Current emergency department disposition of patients with acute heart failure: an opportunity for improvement. J Card Fail 2022; 28 (10) 1545-1559
- 24 Storrow AB, Jenkins CA, Self WH. et al. The burden of acute heart failure on U.S. emergency departments. JACC Heart Fail 2014; 2 (03) 269-277
- 25 Heidenreich PA, Bozkurt B, Aguilar D. et al; ACC/AHA Joint Committee Members. 2022 AHA/ACC/HFSA guideline for the management of heart failure: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation 2022; 145 (18) e895-e1032
- 26 Sax DR, Mark DG, Hsia RY, Tan TC, Tabada GH, Go AS. Short-term outcomes and factors associated with adverse events among adults discharged from the emergency department after treatment for acute heart failure. Circ Heart Fail 2017; 10 (12) e004144
- 27 Collins SP, Pang PS, Fonarow GC, Yancy CW, Bonow RO, Gheorghiade M. Is hospital admission for heart failure really necessary?: the role of the emergency department and observation unit in preventing hospitalization and rehospitalization. J Am Coll Cardiol 2013; 61 (02) 121-126
- 28 Collins SP, Lindsell CJ, Naftilan AJ. et al. Low-risk acute heart failure patients: external validation of the Society of Chest Pain Center's recommendations. Crit Pathw Cardiol 2009; 8 (03) 99-103
- 29 Collins SP, Lindsell CJ, Jenkins CA. et al. Risk stratification in acute heart failure: rationale and design of the STRATIFY and DECIDE studies. Am Heart J 2012; 164 (06) 825-834
- 30 Sax DR, Mark DG, Huang J. et al. Use of machine learning to develop a risk-stratification tool for emergency department patients with acute heart failure. Ann Emerg Med 2021; 77 (02) 237-248
- 31 Collins SP, Jenkins CA, Harrell Jr FE. et al. Identification of emergency department patients with acute heart failure at low risk for 30-day adverse events: the STRATIFY decision tool. JACC Heart Fail 2015; 3 (10) 737-747
- 32 Miller K, Mosby D, Capan M. et al. Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. J Am Med Inform Assoc 2018; 25 (05) 585-592
- 33 Xu J, Anders S, Pruttianan A. et al. Human performance measures for the evaluation of process control human-system interfaces in high-fidelity simulations. Appl Ergon 2018; 73: 151-165
- 34 Anders S, Albert R, Miller A. et al. Evaluation of an integrated graphical display to promote acute change detection in ICU patients. Int J Med Inform 2012; 81 (12) 842-851
- 35 Thayer JG, Franklin A, Miller JM, Grundmeier RW, Rogith D, Wright A. A scoping review of rule-based clinical decision support malfunctions. J Am Med Inform Assoc 2024; 31 (10) 2405-2413
- 36 McCoy AB, Russo EM, Johnson KB. et al. Clinician collaboration to improve clinical decision support: the Clickbusters initiative. J Am Med Inform Assoc 2022; 29 (06) 1050-1059
- 37 Mixon AS, Goggins K, Bell SP. et al. Preparedness for hospital discharge and prediction of readmission. J Hosp Med 2016; 11 (09) 603-609
- 38 Stolldorf DP, Ridner SH, Vogus TJ. et al. Implementation strategies in the context of medication reconciliation: a qualitative study. Implement Sci Commun 2021; 2 (01) 63
- 39 Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci 2022; 17 (01) 75
- 40 Shneiderman B, Plaisant C. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 4th ed.. Pearson/Addison Wesley; 2004
- 41 Nielsen J. Jakob Nielsen's Ten Usability Heuristics. Nielsen Norman Group. 2020 . Accessed January 11, 2023 at: https://www.nngroup.com/articles/ten-usability-heuristics/
- 42 Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J Am Med Inform Assoc 2016; 23 (05) 899-908
- 43 Waltz TJ, Powell BJ, Fernández ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci 2019; 14 (01) 42
- 44 Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: a comparison of two theoretical models. Manage Sci 1989; 35 (08) 982-1003
- 45 Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010; 43 (01) 159-172
- 46 Holden RJ, Carayon P, Gurses AP. et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics 2013; 56 (11) 1669-1686
- 47 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74
- 48 Novak LL, Holden RJ, Anders SH, Hong JY, Karsh BT. Using a sociotechnical framework to understand adaptations in health IT implementation. Int J Med Inform 2013; 82 (12) e331-e344
- 49 Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag 2009; 23 (04) 38-45
- 50 Casey SD, Reed ME, LeMaster C. et al. Physicians' perceptions of clinical decision support to treat patients with heart failure in the ED. JAMA Netw Open 2023; 6 (11) e2344393
- 51 Brooke J. SUS: A quick and dirty usability scale. Usability Eval Ind; 1995: 189
- 52 Harris PA, Taylor R, Minor BL. et al; REDCap Consortium. The REDCap consortium: building an international community of software platform partners. J Biomed Inform 2019; 95: 103208
- 53 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
- 54 Ramanadhan S, Revette AC, Lee RM, Aveling EL. Pragmatic approaches to analyzing qualitative data for implementation science: an introduction. Implement Sci Commun 2021; 2 (01) 70
- 55 Corbin J, Strauss A. Strategies for qualitative analysis. In: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications; 2015: 88-101
- 56 Mahler SA, Riley RF, Hiestand BC. et al. The HEART Pathway randomized trial: identifying emergency department patients with acute chest pain for early discharge. Circ Cardiovasc Qual Outcomes 2015; 8 (02) 195-203
- 57 Ahmad T, Desai NR, Yamamoto Y. et al. Alerting clinicians to 1-year mortality risk in patients hospitalized with heart failure: the REVEAL-HF randomized clinical trial. JAMA Cardiol 2022; 7 (09) 905-912
- 58 Wright A, Sittig DF, Ash JS. et al. Lessons learned from implementing service-oriented clinical decision support at four sites: a qualitative study. Int J Med Inform 2015; 84 (11) 901-911
- 59 Sax DR, Mark DG, Rana JS. et al. Pilot trial of an electronic decision support to improve care for emergency department patients with acute heart failure. ESC Heart Fail 2024; 11 (06) 4432-4436
- 60 Sax DR, Sturmer LR, Mark DG, Rana JS, Reed ME. Barriers and opportunities regarding implementation of a machine learning-based acute heart failure risk stratification tool in the emergency department. Diagnostics (Basel) 2022; 12 (10) 2463
- 61 Osheroff J. Improving Outcomes with Clinical Decision Support. HIMSS Publishing; 2012
- 62 Salwei ME, Carayon P, Hoonakker PLT. et al. Workflow integration analysis of a human factors-based clinical decision support in the emergency department. Appl Ergon 2021; 97: 103498
- 63 Carr JR, Jones BE, Collingridge DS. et al. Deploying an electronic clinical decision support tool for diagnosis and treatment of pneumonia into rural and critical access hospitals: utilization, effect on processes of care, and clinician satisfaction. J Rural Health 2022; 38 (01) 262-269
- 64 Williams DJ, Nian H, Suresh S. et al. Prognostic clinical decision support for pneumonia in the emergency department: a randomized trial. J Hosp Med 2024; 19 (09) 802-811
- 65 Schwartz JM, George M, Rossetti SC. et al. Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study. JMIR Hum Factors 2022; 9 (02) e33960
- 66 Henry KE, Kornfield R, Sridharan A. et al. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system. NPJ Digit Med 2022; 5 (01) 97
- 67 Smulowitz PB, Burke RC, Ostrovsky D, Novack V, Isbell L, Landon BE. Attitudes toward risk among emergency physicians and advanced practice clinicians in Massachusetts. J Am Coll Emerg Physicians Open 2021; 2 (05) e12573
- 68 Adams KT, Howe JL, Fong A. et al. An analysis of patient safety incident reports associated with electronic health record interoperability. Appl Clin Inform 2017; 8 (02) 593-602
- 69 Endsley MR, Cooke N, McNeese N, Bisantz A, Militello L, Roth E. Special issue on human-AI teaming and special issue on AI in healthcare. J Cogn Eng Decis Mak 2022; 16 (04) 179-181
- 70 Smith MW, Brown C, Virani SS. et al. Incorporating guideline adherence and practice implementation issues into the design of decision support for beta-blocker titration for heart failure. Appl Clin Inform 2018; 9 (02) 478-489
- 71 Weinger MB, Wiklund ME, Gardner-Bonneau DJ. eds. Handbook of Human Factors in Medical Device Design. CRC Press; 2010
- 72 Shahrezaei A, Sohani M, Taherkhani S, Zarghami SY. The impact of surgical simulation and training technologies on general surgery education. BMC Med Educ 2024; 24 (01) 1297
- 73 Sarma A, Normand SLT, Hatfield LA, Matheny ME, Resnic FS. The learning curve for medical devices: experience with vascular closure devices in Massachusetts. J Am Coll Cardiol 2013; 61 (10) E1502