Barriers to Implementing an Artificial Intelligence Model for Unplanned Readmissions

Abstract

Background Electronic health record (EHR) vendors now offer “off-the-shelf” artificial intelligence (AI) models to client organizations. Our health system faced difficulties in promoting end-user utilization of a new AI model for predicting readmissions embedded in the EHR.

Objectives The aim is to conduct a case study centered on identifying barriers to uptake/utilization.

Methods A qualitative study was conducted using interviews with stakeholders. The interviews were used to identify relevant stakeholders, understand current workflows, identify implementation barriers, and formulate future strategies.

Results We discovered substantial variation in existing workflows around readmissions. Some stakeholders did not perform any formal readmissions risk assessment. Others accustomed to using existing risk scores such as LACE+ had concerns about transitioning to a new model. Some stakeholders had existing workflows in place that could accommodate the new model, but they were not previously aware that the new model was in production. Concerns expressed by end-users included: whether the model's predictors were relevant to their work, need for adoption of additional workflow processes, need for training and change management, and potential for unintended consequences (e.g., increased health care resource utilization due to potentially over-referring discharged patients to home health services).

Conclusion AI models for risk stratification, even if “off-the-shelf” by design, are unlikely to be “plug-and-play” in health care settings. Seeking out key stakeholders and defining clear use cases early in the implementation process can better facilitate utilization of these models.

Keywords

readmissions - case management - predictive models - predictive analytics - electronic health records - artificial intelligence - machine learning - health system - clinical informatics

Protection of Human and Animal Subjects

The study was performed in compliance with the Declaration of Helsinki and was reviewed by the UCSD Institutional Review Board, which declared the study as a quality improvement protocol and certified that the study did not qualify as human subjects research according to the Code of Federal Regulations, Title 45, part 46 and UCSD Standard Operating Policies and Procedures.

Funding

This study was supported by the National Institutes of Health/National Library of Medicine (grant T15LM011271). The funding organization had no role in the design or conduct of the study.

Authors' Contributions

S.L.B., J.B., and A.M.S. conceived and designed the study. S.L.B. and J.B. conducted interviews and data collection. S.L.B., J.B., and A.M.S. participated in data analysis and interpretation. S.L.B. and J.B. drafted the manuscript. All authors provided critical review of the manuscript for important intellectual content and approved the final version of the manuscript.

Publication History

Received: 14 April 2020

Accepted: 30 June 2020

Article published online:
19 September 2020

• References

• 1 Parikh RB, Kakad M, Bates DW. Integrating predictive analytics into high-value care: the dawn of precision delivery. JAMA 2016; 315 (07) 651-652
• 2 Harris AH. Path from predictive analytics to improved patient outcomes: a framework to guide use, implementation, and evaluation of accurate surgical predictive models. Ann Surg 2017; 265 (03) 461-463
• 3 Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood) 2014; 33 (07) 1148-1154
• 4 He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019; 25 (01) 30-36
• 5 Fountaine T, McCarthy B, Saleh T. Building the AI-powered organization. Harvard Business Review. 2019 . Available at: https://hbr.org/2019/07/building-the-ai-powered-organization . Accessed April 13, 2020
• 6 Joynt KE, Jha AK. A path forward on Medicare readmissions. N Engl J Med 2013; 368 (13) 1175-1177
• 7 Institute of Medicine (U.S.), Rewarding Provider Performance: Aligning Incentives in Medicare. Washington, DC: National Academies Press; 2007
• 8 Zuckerman RB, Sheingold SH, Orav EJ, Ruhter J, Epstein AM. Readmissions, observation, and the hospital readmissions reduction program. N Engl J Med 2016; 374 (16) 1543-1551
• 9 Nuckols TK, Keeler E, Morton S. , et al. Economic evaluation of quality improvement interventions designed to prevent hospital readmission: a systematic review and meta-analysis. JAMA Intern Med 2017; 177 (07) 975-985
• 10 Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med 2009; 360 (14) 1418-1428
• 11 Ibrahim AM, Koester C, Al-Akchar M. , et al. HOSPITAL Score, LACE Index and LACE+ Index as predictors of 30-day readmission in patients with heart failure. BMJ Evid Based Med 2019; DOI: 0.1136/bmjebm-2019-111271.
• 12 Caplan IF, Sullivan PZ, Kung D. , et al. LACE+ index as predictor of 30-day readmission in brain tumor population. World Neurosurg 2019; 127: e443-e448
• 13 Caplan IF, Zadnik Sullivan P, Glauser G. , et al. The LACE+ index fails to predict 30-90 day readmission for supratentorial craniotomy patients: a retrospective series of 238 surgical procedures. Clin Neurol Neurosurg 2019; 182: 79-83
• 14 Ettyreddy AR, Kao WTK, Roland LT, Rich JT, Chi JJ. Utility of the LACE scoring system in predicting readmission following tracheotomy and laryngectomy. Ear Nose Throat J 2019; 98 (04) 220-222
• 15 Shaw J, Rudzicz F, Jamieson T, Goldfarb A. Artificial intelligence and the implementation challenge. J Med Internet Res 2019; 21 (07) e13659
• 16 Greenhalgh T, Wherton J, Papoutsi C. , et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017; 19 (11) e367
• 17 Rogers EM. Diffusion of Innovations. 5th ed. New York, NY: Free Press; 2003
• 18 Benda NC, Das LT, Abramson EL. , et al. “How did you get to this number?” Stakeholder needs for implementing predictive analytics: a pre-implementation qualitative study. J Am Med Inform Assoc 2020; 27 (05) 709-716
• 19 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74
• 20 Sussillo D, Barak O. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput 2013; 25 (03) 626-649