Operationalizing AI in Stroke Alerts: Balancing Sensitivity and Specificity in Predicting Acute Cerebrovascular Disease

Abstract

Acute stroke alerts are frequently triggered by conditions unrelated to cerebrovascular disease, resulting in false positives that burden clinical teams and contribute to diagnostic ambiguity. At a large academic center, we developed ScanNER v2, a machine learning (ML) model based on large-language models (LLMs) and structured clinical data to predict the presence of acute cerebrovascular disease (ACD) in approximately 16,000 stroke alerts occurring over 10 years with an area under the receiver-operating curve and F1 score of 0.72 and overall positive predictive value of 0.68. In this perspective, we outline a practical framework for operationalizing this model within hospital-based stroke systems. We first describe our health-system experience developing and validating an AI-enabled pipeline, named “ScanNER 2,” then take the point of view of two implementation angles (high sensitivity and high specificity), outlining the operational and clinical tradeoffs for each approach. We also highlight challenges related to implementation, clinical governance, workflow integration, and equity, emphasizing guardrails required for responsible deployment. As stroke centers increasingly adopt AI-assisted tools, this type of thought experiment is essential to ensure that such ML-based innovations effectively enhance the core mission of delivering timely, high-quality acute stroke care.

Keywords

machine learning - LLM - large language model - implementation - acute stroke - predictive modeling - diagnostic acceleration

Publication History

Received: 20 November 2025

Accepted: 09 December 2025

Accepted Manuscript online:
11 December 2025

Article published online:
23 December 2025

• References

• 1 Powers WJ, Rabinstein AA, Ackerson T. et al. Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2019; 50 (12) e344-e418 Erratum in: Stroke. 2019;50(12):e440-e441
• 2 Kim T, Jeong HY, Suh GJ. Clinical differences between stroke and stroke, mimics in code stroke patients. J Korean Med Sci 2022; 37 (07) e54
• 3 Aroor S, Singh R, Goldstein LB. BE-FAST (Balance, Eyes, Face, Arm, Speech, Time): reducing the proportion of strokes missed using the FAST mnemonic. Stroke 2017; 48 (02) 479-481
• 4 Cumbler E, Simpson J. Code stroke: multicenter experience with in-hospital stroke alerts. J Hosp Med 2015; 10 (03) 179-183
• 5 Erekat AN, Stein LDelman BN. et al. Natural language processing to automate cerebrovascular event identification in stroke alerts. Stroke Vasc Intervent Neurol 2025; 5 (06) e001846
• 6 Long B, Koyfman A. Clinical mimics: an emergency medicine–focused review of stroke mimics. J Emerg Med 2017; 52 (02) 176-183
• 7 Appukutty AJ, Skolarus LE, Springer MV, Meurer WJ, Burke JF. Increasing false positive diagnoses may lead to overestimation of stroke incidence, particularly in the young: a cross-sectional study. BMC Neurol 2021; 21 (01) 152
• 8 Erekat AN, Downes MH, Stein LK. et al. Predicting acute cerebrovascular events in stroke alerts using large-language models and structured data, medArxiv.org. 2025 . Preprint
• 9 Trevethan R. Sensitivity, specificity, and predictive values: foundations and pitfalls. Front Public Health 2017; 5: 307
• 10 Sung SF, Hung LC, Hu YH. Developing a stroke alert trigger for clinical decision support at emergency triage using machine learning. Int J Med Inform 2021; 152: 104505
• 11 Abedi V, Misra D, Chaudhary D. et al. Machine learning–based prediction of stroke in emergency departments. Ther Adv Neurol Disord 2024;17:17562864241239108
• 12 Maxim LD, Niebo R, Utell MJ. Screening tests: a review with examples. Inhal Toxicol 2014; 26 (13) 811-828
• 13 Saver JL, Fonarow GC, Smith EE. et al. Time to treatment with intravenous tissue plasminogen activator and outcome from acute ischemic stroke. JAMA 2013; 309 (23) 2480-2488
• 14 Wang F, Beecy A. Implementing AI models in clinical workflows: a roadmap. BMJ Evid Based Med 2025; 30 (05) 285-287
• 15 Kwong JCC, Erdman L, Khondker A. et al. The silent trial - the bridge between bench-to-bedside clinical AI applications. Front Digit Health 2022;4:929508
• 16 Cai T, Ni H, Yu M. et al. DeepStroke: an efficient stroke screening framework for emergency rooms with multimodal adversarial deep learning. Med Image Anal 2022; 80: 102522
• 17 Albrecht M, Shanks D, Shah T. et al. Enhancing clinical documentation with ambient artificial intelligence: a quality improvement survey assessing clinician perspectives on work burden, burnout, and job satisfaction. JAMIA Open 2025; 8 (01) ooaf013
• 18 Rosenthal J, Beecy A, Sabuncu MR. Rethinking clinical trials for medical AI with dynamic deployments of adaptive systems. NPJ Digit Med 2025; 8 (01) 252