Download PDF
Open Access
CC BY 4.0 · Yearb Med Inform 2024; 33(01): 103-114
DOI: 10.1055/s-0044-1800729
Section 3: Clinical Information Systems
Survey

Bridging the Gap: Challenges and Strategies for the Implementation of Artificial Intelligence-based Clinical Decision Support Systems in Clinical Practice

Authors

  • Niels Peek

    1   The Healthcare Improvement Studies Institute (THIS Institute), Department of Public Health and Primary Care, University of Cambridge. Cambridge, UK

  • Daniel Capurro

    2   Centre for the Digital Transformation of Health, University of Melbourne & The Royal Melbourne Hospital. Melbourne, Australia

  • Vlada Rozova

    3   Centre for the Digital Transformation of Health, University of Melbourne. Melbourne, Australia

  • Sabine N van der Veer

    4   Centre for Health Informatics, Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester. Manchester, UK
Further Information
Also available at
  Permissions and Reprints
Preview

Summary

Objectives: Despite the surge in development of artificial intelligence (AI) algorithms to support clinical decision-making, few of these algorithms are used in practice. We reviewed recent literature on clinical deployment of AI-based clinical decision support systems (AI-CDSS), and assessed the maturity of AI-CDSS implementation research. We also aimed to compare and contrast implementation of rule-based CDSS with implementation of AI-CDSS, and to give recommendations for future research in this area.

Methods: We searched PubMed and Scopus for publications in 2022 and 2023 that focused on AI and/or CDSS, health care, and implementation research, and extracted: clinical setting; clinical task; translational research phase; study design; participants; implementation theory, model or framework used; and key findings.

Results: We selected and described a total of 31 recent papers addressing implementation of AI-CDSS in clinical practice, categorised into four groups: (i) Implementation theories, frameworks, and models (4 papers); (ii) Stakeholder perspectives (22 papers); (iii) Implementation feasibility (three papers); and (iv) Technical infrastructure (2 papers). Stakeholders saw potential benefits of AI-CDSS, but emphasized the need for a strong evidence base and indicated that systems should fit into clinical workflows. There were clear similarities with rule-based CDSS, but also differences with respect to trust and transparency, knowledge, intellectual property, and regulation.

Conclusions: The field of AI-CDSS implementation research is still in its infancy. It can be strengthened by grounding studies in established theories, models and frameworks from implementation science, focusing on the perspectives of stakeholder groups other than healthcare professionals, conducting more real-world implementation feasibility studies, and through development of reusable technical infrastructure that facilitates rapid deployment of AI-CDSS in clinical practice.

Keywords

Artificial Intelligence - Clinical Decision Support Systems - Implementation Science - Review


Publication History

Article published online:
08 April 2025

© 2024. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, and reproduction so long as the original work is properly cited. (https://creativecommons.org/licenses/by/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Arshi B, Wynants L, Rijnhart E, Reeve K, Cowley LE, Smits LJ. What proportion of clinical prediction models make it to clinical practice? Protocol for a two-track follow-up study of prediction model development publications. BMJ Open. 2023;13(5):e073174. DOI: 10.1136/bmjopen-2023-073174.
    PubMed
  • 2 Svedberg P, Reed J, Nilsen P, Barlow J, Macrae C, Nygren J. Toward Successful Implementation of Artificial Intelligence in Health Care Practice: Protocol for a Research Program. JMIR Res Protoc. 2022;11(3):e34920. DOI: 10.2196/34920.
    PubMed
  • 3 Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal. BMJ. 2020;369:1–22. DOI: 10.1136/bmj.m1328.
    PubMed
  • 4 Terenziani P, Raviola P, Bruschi O, Torchio M, Marzuoli M, Molino G. Representing knowledge levels in clinical guidelines. Proceedings of the Joint European Conference on Artificial Intelligence in Medicine and Medical Decision Making. Lecture Notes in Artificial Intelligence. 1999. p. 254–60. DOI: 10.1007/3-540-48720-4_28.
    PubMed
  • 5 Mann S, Berdahl CT, Baker L, Girosi F. Artificial intelligence applications used in the clinical response to COVID-19: A scoping review. PLOS Digit Heal. 2022;1(10):e0000132. DOI: 10.1371/journal.pdig.0000132.
    PubMed
  • 6 Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:50. DOI: 10.1186/1748-5908-7-50.
    PubMed
  • 7 Morris ZS, wooding S, Grant J. The answer is 17 years, what is the question: Understanding time lags in translational research. J R Soc Med. 2011;104(12):510–520. DOI: 10.1258/jrsm.2011.110180.
    PubMed
  • 8 Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006;1(1):1–3. DOI: 10.1186/1748-5908-1-1.
    PubMed
  • 9 Magrabi F, Ammenwerth E, McNair JB, De Keizer NF, Hyppönen H, Nykänen P, et al. Artificial Intelligence in clinical decision support: Challenges for evaluating AI and practical implications. Yearb Med Inform. 2019;28(1):128–134. DOI: 10.1055/s-0039-1677903.
    PubMed
  • 10 Shortliffe EH, Sepúlveda MJ. Clinical decision support in the era of Artificial Intelligence. JAMA. 2018;320(21):2199–2200. DOI: 10.1001/jama.2018.17163.
    PubMed
  • 11 Papadopoulos P, Soflano M, Chaudy Y, Adejo W, Connolly TM. A systematic review of technologies and standards used in the development of rule-based clinical decision support systems. Health Technol 2022;12:713–727. DOI: 10.1007/s12553-022-00672-9.
    PubMed
  • 12 Miller A, Moon B, Anders S, Walden R, Brown S, Montella D. Integrating computerized clinical decision support systems into clinical work: A meta-synthesis of qualitative research. Int J Med Inform. 2015;84(12):1009–1018. DOI: 10.1016/j.ijmedinf.2015.09.005.
    PubMed
  • 13 Liberati EG, Ruggiero F, Galuppo L, Gorli M, González-Lorenzo M, Maraldi M, et al. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci. 2017;12(1):1–13. DOI: 10.1186/s13012-017-0644-2.
    PubMed
  • 14 Meunier PY, Raynaud C, Guimaraes E, Gueyffier F, Letrilliart L. Barriers and facilitators to the use of clinical decision support systems in primary care: A mixed-methods systematic review. Ann Fam Med. 2023;21(1):57–69. DOI: 10.1370/afm.2908.
    PubMed
  • 15 Hogg HDJ, Al-Zubaidy M, Keane PA, Hughes G, Beyer FR, Maniatopoulos G. Evaluating the translation of implementation science to clinical artificial intelligence: a bibliometric study of qualitative research. Front Heal Serv. 2023;3:1161822. DOI: 10.3389/frhs.2023.1161822.
    PubMed
  • 16 Fujimori R, Liu K, Soeno S, Naraba H, Ogura K, Hara K, et al. Acceptance, barriers, and facilitators to implementing artificial intelligence–based decision support systems in emergency departments: Quantitative and qualitative evaluation. JMIR Form Res. 2022;6(6):e36501. DOI: 10.2196/36501.
    PubMed
  • 17 Neher M, Petersson L, Nygren JM, Svedberg P, Larsson I, Nilsen P. Innovation in healthcare: Leadership perceptions about the innovation characteristics of artificial intelligence—a qualitative interview study with healthcare leaders in Sweden. Implement Sci Commun. 2023;4(1):1–13. DOI: 10.1186/s43058-023-00458-8.
    PubMed
  • 18 Romero-Brufau S, Wyatt KD, Boyum P, Mickelson M, Moore M, Cognetta-Rieke C. A lesson in implementation: A pre-post study of providers' experience with artificial intelligence-based clinical decision support. Int J Med Inform. 2020;137:104072. DOI: 10.1016/j.ijmedinf.2019.104072.
    PubMed
  • 19 Van der Vegt AH, Scott IA, Dermawan K, Schnetler RJ, Kalke VR, Lane PJ. Implementation frameworks for end-to-end clinical AI: Derivation of the SALIENT framework. J Am Med Informatics Assoc. 2023;30(9):1503–1515. DOI: 10.1093/jamia/ocad088.
    PubMed
  • 20 Van der Vegt AH, Scott IA, Dermawan K, Schnetler RJ, Kalke VR, Lane PJ. Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework. J Am Med Informatics Assoc. 2023;30(7):1349–1361. DOI: 10.1093/jamia/ocad075.
    PubMed
  • 21 Hogg HDJ, Al-Zubaidy M, Talks J, Denniston AK, Kelly CJ, Malawana J, et al. Stakeholder perspectives of clinical artificial intelligence implementation: Systematic review of qualitative evidence. J Med Internet Res. 2023;25:e39742. DOI: 10.2196/39742.
    PubMed
  • 22 Lambert SI, Madi M, Sopka S, Lenes A, Stange H, Buszello CP, et al. An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals. npj Digit Med. 2023;6(1):1–14. DOI: 10.1038/s41746-023-00852-5.
    PubMed
  • 23 Abell B, Naicker S, Rodwell D, Donovan T, Tariq A, Baysari M, et al. Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review. Implement Sci. 2023;18(1):1–20. DOI: 10.1186/s13012-023-01287-y.
    PubMed
  • 24 Brereton TA, Malik MM, Lifson M, Greenwood JD, Peterson KJ, Overgaard SM. The role of artificial intelligence model documentation in translational science: Scoping review. Interact J Med Res. 2023;12:e45903. DOI: 10.2196/45903.
    PubMed
  • 25 Eltawil FA, Atalla M, Boulos E, Amirabadi A, Tyrrell PN. Analyzing barriers and enablers for the acceptance of artificial intelligence innovations into radiology practice: A scoping review. Tomography. 2023;9(4):1443–1455. DOI: 10.3390/tomography9040115.
    PubMed
  • 26 Fernando M, Abell B, Tyack Z, Donovan T, McPhail SM, Naicker S. Using theories, models, and frameworks to inform implementation cycles of computerized clinical decision support systems in tertiary health care settings: Scoping review. J Med Internet Res. 2023;25:e45163. DOI: 10.2196/45163.
    PubMed
  • 27 Gama F, Tyskbo D, Nygren J, Barlow J, Reed J, Svedberg P. Implementation frameworks for artificial intelligence translation into health care practice: Scoping review. J Med Internet Res. 2022;24(1):e32215. DOI: 10.2196/32215.
    PubMed
  • 28 King H, Wright J, Treanor D, Williams B, Randell R. What works where and how for uptake and impact of artificial intelligence in pathology: Review of theories for a realist evaluation. J Med Internet Res. 2023;25:e38039. DOI: 10.2196/38039.
    PubMed
  • 29 Assadi A, Laussen PC, Goodwin AJ, Goodfellow S, Dixon W, Greer RW, Jegatheeswaran A, et al. An integration engineering framework for machine learning in healthcare. Front Digit Health. 2022;4:932411. DOI: 10.3389/fdgth.2022.932411.
    PubMed
  • 30 Gannon F. The steps from translatable to translational research. EMBO Rep. 2014;15(11):1107–1108. DOI: 10.15252/embr.201439587.
    PubMed
  • 31 Birken SA, Powell BJ, Shea CM, Haines ER, Alexis Kirk M, Leeman J, et al. Criteria for selecting implementation science theories and frameworks: Results from an international survey. Implement Sci. 2017;12(1):1–9. DOI: 10.1186/s13012-017-0656-y.
    PubMed
  • 32 Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):1–16. DOI: 10.1186/s13012-022-01245-0.
    PubMed
  • 33 Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13(3):319–340. DOI: 10.2307/249008.
    PubMed
  • 34 Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance and information technology: Toward a unified view. MIS Quarterly. 2003;27(3):425–478. DOI: 10.2307/30036540.
    PubMed
  • 35 Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR, et al. Designing medical informatics research and library--resource projects to increase what is learned. J Am Med Infomatics Assoc. 1994;1(1):28–33. DOI: 10.1136/jamia.1994.95236134.
    PubMed
  • 36 Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. BMC Med. 2015;13(1):1–10. DOI: 10.1186/s12916-014-0241-z.
    PubMed
  • 37 Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma JB, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021;11(7):1–7. DOI: 10.1136/bmjopen-2020-048008.
    PubMed
  • 38 Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022;377:e070904. DOI: 10.1136/bmj-2022-070904.
    PubMed
  • 39 Liu X, Rivera SC, Moher D, Calvert MJ, Denniston AK. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: The CONSORT-AI Extension. BMJ. 2020;370:m3164. DOI: 10.1136/bmj.m3164.
    PubMed
  • 40 Strohm L, Hehakaya C, Ranschaert ER, Boon WPC, Moors EHM. Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur Radiol. 2020;30(10):5525–5532. DOI: 10.1007/s00330-020-06946-y.
    PubMed
  • 41 Terry AL, Kueper JK, Beleno R, Brown JB, Cejic S, Dang J, et al. Is primary health care ready for artificial intelligence? What do primary health care stakeholders say? BMC Med Inform Decis Mak. 2022;22(1):1–11. DOI: 10.1186/s12911-022-01984-6.
    PubMed
  • 42 Van der Meijden SL, de Hond AAH, Thoral PJ, Steyerberg EW, Kant IMJ, Cinà G, et al. Intensive care unit physicians' perspectives on artificial intelligence–based clinical decision support tools: Preimplementation survey study. JMIR Hum Factors. 2023;10:1–12. DOI: 10.2196/39114.
    PubMed
  • 43 Wang SM, Hogg HDJ, Sangvai D, Patel MR, Weissler EH, Kellogg KC, et al. Development and integration of machine learning algorithm to identify peripheral arterial disease: Multistakeholder qualitative study. JMIR Form Res. 2023;7(1):1–15. DOI: 10.2196/43963.
    PubMed
  • 44 Weinert L, Müller J, Svensson L, Heinze O. Perspective of information technology decision makers on factors influencing adoption and implementation of artificial intelligence technologies in 40 German hospitals: Descriptive analysis. JMIR Med Informatics. 2022;10(6):e34678. DOI: 10.2196/34678.
    PubMed
  • 45 Bergquist M, Rolandsson B, Gryska E, Laesser M, Hoefling N, Heckemann R, et al. Trust and stakeholder perspectives on the implementation of AI tools in clinical radiology. Eur Radiol. 2023;34(1):338–347. DOI: 10.1007/s00330-023-09967-5.
    PubMed
  • 46 Fazakarley CA, Breen M, Leeson P, Thompson B, Williamson V. Experiences of using artificial intelligence in healthcare: A qualitative study of UK clinician and key stakeholder perspectives. BMJ Open. 2023;13(12):1–8. DOI: 10.1136/bmjopen-2023-076950.
    PubMed
  • 47 Hesso I, Kayyali R, Dolton DR, Joo K, Zacharias L, Charalambous A, et al. Cancer care at the time of the fourth industrial revolution: an insight to healthcare professionals' perspectives on cancer care and artificial intelligence. Radiat Oncol. 2023;18(1):1–16. DOI: 10.1186/s13014-023-02351-z.
    PubMed
  • 48 Ho V, Johnson CB, Ghanzouri I, Amal S, Asch S, Ross E. Physician- and patient-elicited barriers and facilitators to implementation of a machine learning–based screening tool for peripheral arterial disease: Preimplementation study with physician and patient stakeholders. JMIR Cardio. 2023;7(1):1–10. DOI: 10.2196/44732.
    PubMed
  • 49 Manetti S, Cumetti M, De Benedictis A, Lettieri E. Adoption of novel biomarker test parameters with machine learning-based algorithms for the early detection of sepsis in hospital practice. J Nurs Manag. 2022;30(8):3754–3764. DOI: 10.1111/jonm.13807.
    PubMed
  • 50 Petersson L, Larsson I, Nygren JM, Nilsen P, Neher M, Reed JE, et al. Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden. BMC Health Serv Res. 2022;22(1):1–16. DOI: 10.1186/s12913-022-08215-8.
    PubMed
  • 51 Pumplun L, Fecho M, Wahl N, Peters F, Buxmann P. Adoption of machine learning systems for medical diagnostics in clinics: Qualitative interview study. J Med Internet Res. 2021;23(10):e29301. DOI: 10.2196/29301.
    PubMed
  • 52 Redrup Hill E, Mitchell C, Brigden T, Hall A. Ethical and legal considerations influencing human involvement in the implementation of artificial intelligence in a clinical pathway: A multi-stakeholder perspective. Front Digit Health. 2023;5:1139210. DOI: 10.3389/fdgth.2023.1139210.
    PubMed
  • 53 Beauchamp TL, Childress JF. The Principles of Biomedical Ethics, 5th edn. New York: Oxford University Press, 2001. ISBN 10: 0195143310.
    PubMed
  • 54 Schepart A, Burton A, Durkin L, Fuller A, Charap E, Bhambri R, et al. Artificial intelligence–enabled tools in cardiovascular medicine: A survey of current use, perceptions, and challenges. Cardiovasc Digit Heal J. 2023;4(3):101–110. DOI: 10.1016/j.cvdhj.2023.04.003.
    PubMed
  • 55 Petitgand C, Motulsky A, Denis JL, Régis C. Investigating the barriers to physician adoption of an artificial intelligence-based decision support system in emergency care: An interpretative qualitative study. Stud Health Technol Inform. 2020;270:1001–1005. DOI: 10.3233/SHTI200312.
    PubMed
  • 56 Smak Gregoor AM, Sangers TE, Eekhof JA, Howe S, Revelman J, Litjens RJ, et al. Artificial intelligence in mobile health for skin cancer diagnostics at home (AIM HIGH): a pilot feasibility study. eClinicalMedicine. 2023;60:102019. DOI: 10.1016/j.eclinm.2023.102019.
    PubMed
  • 57 Tseng TW, Su CF, Lai F. Fast Healthcare Interoperability Resources for inpatient deterioration detection with time-Series vital signs: Design and implementation study. JMIR Med Informatics. 2022;10(10):1–12. DOI: 10.2196/42429.
    PubMed
  • 58 Afshar M, Adelaine S, Resnik F, Mundt MP, Long J, Leaf M, et al. Deployment of real-time natural language processing and deep learning clinical decision support in the electronic health record: Pipeline implementation for an opioid misuse screener in hospitalized adults. JMIR Med Informatics. 2023;11:e44977. DOI: 10.2196/44977.
    PubMed
  • 59 Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44–56. DOI: 10.1038/s41591-018-0300-7.
    PubMed
  • 60 Coiera E. The last mile: Where artificial intelligence meets reality. J Med Internet Res. 2019;21(11):1–4. DOI: 10.2196/16323.
    PubMed
  • 61 Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19(11):e367. DOI: 10.2196/jmir.8775.
    PubMed
  • 62 Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform. 2008;77(6):386–398. DOI: 10.1016/j.ijmedinf.2007.08.011.
    PubMed
  • 63 Khairat S, Marc D, Crosby W, Al Sanousi A. Reasons for physicians not adopting clinical decision support systems: Critical analysis. JMIR Med Informatics. 2018; 6(2):e24. DOI: 10.2196/medinform.8912.
    PubMed
  • 64 May C, Finch T. Implementing, embedding, and integrating practices: An outline of normalization process theory. Sociology. 2009;43(3):535–554. DOI: 10.1177/0038038509103208.
    PubMed
  • 65 Schwartz JM, George M, Rossetti SC, Dykes PC, Minshall SR, Lucas E, et al. Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: Qualitative descriptive study. JMIR Hum Factors. 2022;9(2):1–15. DOI: 10.2196/33960.
    PubMed
  • 66 Madsen M, Gregor S. Measuring human-computer trust. Proceedings of the Eleventh Australasian Conference on Information Systems,. 2000. https://api.semanticscholar.org/CorpusID:18821611 (accessed 10th June 2024).
    PubMed
  • 67 Epic Systems. seismometer. https://github.com/epic-open-source/seismometer (accessed 29th May 2024).
    PubMed