Download PDF
Appl Clin Inform 2023; 14(02): 374-391
DOI: 10.1055/a-2035-5342
Research Article

Implementation Fidelity of Chatbot Screening for Social Needs: Acceptability, Feasibility, Appropriateness

Raina Langevin
1   Department of Human Centered Design and Engineering, University of Washington, Seattle, Washington, United States
,
Andrew B. L. Berry
2   Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States
,
Jinyang Zhang
3   Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, Washington, United States
,
Callan E. Fockele
4   Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States
,
Layla Anderson
4   Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States
,
Dennis Hsieh
5   Department of Emergency Medicine, Harbor-University of California Los Angeles Medical Center, Torrance, California, United States
,
Andrea Hartzler
6   Biomedical Informatics and Medical Education, University of Washington, Seattle, Washington, United States
,
Herbert C. Duber
4   Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States
7   Office of Health and Science, Washington State Department of Health, Seattle, Washington, United States
,
Gary Hsieh
1   Department of Human Centered Design and Engineering, University of Washington, Seattle, Washington, United States
› Author Affiliations Funding This project was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under award number: UL1 TR002319. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
› Further Information
Also available at
    Permissions and Reprints

Abstract

Objectives Patient and provider-facing screening tools for social determinants of health have been explored in a variety of contexts; however, effective screening and resource referral remain challenging, and less is known about how patients perceive chatbots as potential social needs screening tools. We investigated patient perceptions of a chatbot for social needs screening using three implementation outcome measures: acceptability, feasibility, and appropriateness.

Methods We implemented a chatbot for social needs screening at one large public hospital emergency department (ED) and used concurrent triangulation to assess perceptions of the chatbot use for screening. A total of 350 ED visitors completed the social needs screening and rated the chatbot on implementation outcome measures, and 22 participants engaged in follow-up phone interviews.

Results The screened participants ranged in age from 18 to 90 years old and were diverse in race/ethnicity, education, and insurance status. Participants (n = 350) rated the chatbot as an acceptable, feasible, and appropriate way of screening. Through interviews (n = 22), participants explained that the chatbot was a responsive, private, easy to use, efficient, and comfortable channel to report social needs in the ED, but wanted more information on data use and more support in accessing resources.

Conclusion In this study, we deployed a chatbot for social needs screening in a real-world context and found patients perceived the chatbot to be an acceptable, feasible, and appropriate modality for social needs screening. Findings suggest that chatbots are a promising modality for social needs screening and can successfully engage a large, diverse patient population in the ED. This is significant, as it suggests that chatbots could facilitate a screening process that ultimately connects patients to care for social needs, improving health and well-being for members of vulnerable patient populations.

Keywords

patients - chatbot interaction - implementation - digital - social determinants of health

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. Study procedures were approved by the University of Washington Institutional Review Board (IRB) and received a waiver of written consent. In the chatbot screening, participants read a short introduction to the study and were asked if they consent to participating by clicking “Okay, let's start” in order to proceed.


Supplementary Material



Publication History

Received: 02 September 2022

Accepted: 14 February 2023

Accepted Manuscript online:
14 February 2023

Article published online:
17 May 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Marmot M. Social determinants of health inequalities. Lancet 2005; 365 (9464): 1099-1104
    CrossrefPubMedGoogle Scholar
  • 2 Gottlieb LM, Wing H, Adler NE. A systematic review of interventions on patients' social and economic needs. Am J Prev Med 2017; 53 (05) 719-729
    CrossrefPubMedGoogle Scholar
  • 3 Adler NE, Glymour MM, Fielding J. Addressing social determinants of health and health inequalities. JAMA 2016; 316 (16) 1641-1642
    CrossrefPubMedGoogle Scholar
  • 4 Marmot M, Friel S, Bell R, Houweling TA, Taylor S. ; Commission on Social Determinants of Health. Closing the gap in a generation: health equity through action on the social determinants of health. Lancet 2008; 372 (9650): 1661-1669
    CrossrefPubMedGoogle Scholar
  • 5 Sulo S, Feldstein J, Partridge J, Schwander B, Sriram K, Summerfelt WT. Budget impact of a comprehensive nutrition-focused quality improvement program for malnourished hospitalized patients. Am Health Drug Benefits 2017; 10 (05) 262-270
    PubMedGoogle Scholar
  • 6 Castrucci B, Auerbach J. Meeting individual social needs falls short of addressing social determinants of health. Health Affairs Blog 2019 Accessed March 2, 2023 at: https://www.healthaffairs.org/do/10.1377/forefront.20190115.234942/
    PubMedGoogle Scholar
  • 7 Malecha PW, Williams JH, Kunzler NM, Goldfrank LR, Alter HJ, Doran KM. Material needs of emergency department patients: a systematic review. Acad Emerg Med 2018; 25 (03) 330-359
    CrossrefPubMedGoogle Scholar
  • 8 Gordon JA. The hospital emergency department as a social welfare institution. Ann Emerg Med 1999; 33 (03) 321-325
    CrossrefPubMedGoogle Scholar
  • 9 Hsu C, Cruz S, Placzek H. et al. Patient perspectives on addressing social needs in primary care using a screening and resource referral intervention. J Gen Intern Med 2020; 35 (02) 481-489
    CrossrefPubMedGoogle Scholar
  • 10 Tong ST, Liaw WR, Kashiri PL. et al. Clinician experiences with screening for social needs in primary care. J Am Board Fam Med 2018; 31 (03) 351-363
    CrossrefPubMedGoogle Scholar
  • 11 Persaud S. Addressing social determinants of health through advocacy. Nurs Adm Q 2018; 42 (02) 123-128
    CrossrefPubMedGoogle Scholar
  • 12 Bleacher H, Lyon C, Mims L, Cebuhar K, Begum A. The feasibility of screening for social determinants of health: seven lessons learned. Fam Pract Manag 2019; 26 (05) 13-19
    PubMedGoogle Scholar
  • 13 Zulman DM, Maciejewski ML, Grubber JM. et al. Patient-reported social and behavioral determinants of health and estimated risk of hospitalization in high-risk veterans affairs patients. JAMA Netw Open 2020; 3 (10) e2021457
    CrossrefPubMedGoogle Scholar
  • 14 Power-Hays A, Li S, Mensah A, Sobota A. Universal screening for social determinants of health in pediatric sickle cell disease: a quality-improvement initiative. Pediatr Blood Cancer 2020; 67 (01) e28006
    CrossrefPubMedGoogle Scholar
  • 15 Chang C, Ceci C, Uberoi M, Waselewski M, Chang T. Youth perspectives on their medical team's role in screening for and addressing social determinants of health. J Adolesc Health 2022; 70 (06) 928-933
    CrossrefPubMedGoogle Scholar
  • 16 Berger-Jenkins E, Monk C, DʼOnfro K. et al. Screening for both child behavior and social determinants of health in pediatric primary care. J Dev Behav Pediatr 2019; 40 (06) 415-424
    CrossrefPubMedGoogle Scholar
  • 17 Katz-Wise SL, Reisner SL, White Hughto JM, Budge SL. Self-reported changes in attractions and social determinants of mental health in transgender adults. Arch Sex Behav 2017; 46 (05) 1425-1439
    CrossrefPubMedGoogle Scholar
  • 18 Gottlieb L, Hessler D, Long D, Amaya A, Adler N. A randomized trial on screening for social determinants of health: the iScreen study. Pediatrics 2014; 134 (06) e1611-e1618
    CrossrefPubMedGoogle Scholar
  • 19 Tai-Seale M, Downing NL, Jones VG. et al. Technology-enabled consumer engagement: promising practices at four health care delivery organizations. Health Aff (Millwood) 2019; 38 (03) 383-390
    CrossrefPubMedGoogle Scholar
  • 20 Gold R, Bunce A, Cowburn S. et al. Adoption of social determinants of health EHR tools by community health centers. Ann Fam Med 2018; 16 (05) 399-407
    CrossrefPubMedGoogle Scholar
  • 21 Ancker JS, Barrón Y, Rockoff ML. et al. Use of an electronic patient portal among disadvantaged populations. J Gen Intern Med 2011; 26 (10) 1117-1123
    CrossrefPubMedGoogle Scholar
  • 22 Singh P, Jonnalagadda P, Morgan E, Fareed N. Outpatient portal use in prenatal care: differential use by race, risk, and area social determinants of health. J Am Med Inform Assoc 2022; 29 (02) 364-371
    CrossrefPubMedGoogle Scholar
  • 23 Anthony DL, Campos-Castillo C, Lim PS. Who isn't using patient portals and why? Evidence and implications from a national sample of US adults. Health Aff (Millwood) 2018; 37 (12) 1948-1954
    CrossrefPubMedGoogle Scholar
  • 24 Wu AW, Weston CM, Ibe CA. et al. The Baltimore community-based organizations neighborhood network: enhancing capacity together (CONNECT) cluster RCT. Am J Prev Med 2019; 57 (02) e31-e41
    CrossrefPubMedGoogle Scholar
  • 25 Capp R, Misky GJ, Lindrooth RC. et al. Coordination program reduced acute care use and increased primary care visits among frequent emergency care users. Health Aff (Millwood) 2017; 36 (10) 1705-1711
    CrossrefPubMedGoogle Scholar
  • 26 Kaufman S, Ali N, DeFiglio V, Craig K, Brenner J. Early efforts to target and enroll high-risk diabetic patients into urban community-based programs. Health Promot Pract 2014; 15 (2, Suppl): 62S-70S
    CrossrefPubMedGoogle Scholar
  • 27 Lin MP, Blanchfield BB, Kakoza RM. et al. ED-based care coordination reduces costs for frequent ED users. Am J Manag Care 2017; 23 (12) 762-766
    PubMedGoogle Scholar
  • 28 Wilcox D, McCauley PS, Delaney C, Molony SL. Evaluation of a hospital: community partnership to reduce 30-day readmissions. Prof Case Manag 2018; 23 (06) 327-341
    CrossrefPubMedGoogle Scholar
  • 29 Laranjo L, Dunn AG, Tong HL. et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc 2018; 25 (09) 1248-1258
    CrossrefPubMedGoogle Scholar
  • 30 Montenegro JL, da Costa CA, da Rosa Righi R. Survey of conversational agents in health. Expert Syst Appl 2019; 129: 56-67
    CrossrefPubMedGoogle Scholar
  • 31 Xiao Z, Zhou MX, Liao QV. et al. Tell me about yourself: using an AI-powered chatbot to conduct conversational surveys with open-ended questions. ACM Trans Comput Hum Interact (TOCHI) 2020; 27 (03) 1-37
    CrossrefPubMedGoogle Scholar
  • 32 Kocielnik R, Agapie E, Argyle A. et al. HarborBot: a chatbot for social needs screening. AMIA Annu Symp Proc 2019; 2019: 552-561
    PubMedGoogle Scholar
  • 33 Kocielnik R, Langevin R, George JS. , et al. Can I talk to you about your social needs? Understanding preference for conversational user interface in health. Paper presented at: CUI 2021: 3rd Conference on Conversational User Interfaces, Bilbao, Spain; July 27–29, 2021:1–10
    PubMed
  • 34 Lucas GM, Gratch J, King A, Morency LP. It's only a computer: virtual humans increase willingness to disclose. Comput Human Behav 2014; 37: 94-100
    CrossrefPubMedGoogle Scholar
  • 35 Auriacombe M, Moriceau S, Serre F. et al. Development and validation of a virtual agent to screen tobacco and alcohol use disorders. Drug Alcohol Depend 2018; 193: 1-6
    CrossrefPubMedGoogle Scholar
  • 36 Philip P, Micoulaud-Franchi JA, Sagaspe P. et al. Virtual human as a new diagnostic tool, a proof of concept study in the field of major depressive disorders. Sci Rep 2017; 7 (01) 42656
    CrossrefPubMedGoogle Scholar
  • 37 Lucas GM, Rizzo A, Gratch J. et al. Reporting mental health symptoms: breaking down barriers to care with virtual human interviewers. Front Robot AI 2017; 4: 51
    CrossrefPubMedGoogle Scholar
  • 38 Stowell E, Lyson MC, Saksono H. , et al. Designing and evaluating mHealth interventions for vulnerable populations: a systematic review. Paper presented at: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, Canada; April 21–26, 2018:1–17
    PubMed
  • 39 Cornelius JB, St Lawrence JS, Howard JC. et al. Adolescents' perceptions of a mobile cell phone text messaging-enhanced intervention and development of a mobile cell phone-based HIV prevention intervention. J Spec Pediatr Nurs 2012; 17 (01) 61-69
    CrossrefPubMedGoogle Scholar
  • 40 Kocaballi AB, Sezgin E, Clark L. et al. Design and evaluation challenges of conversational agents in health care and well-being: selective review study. J Med Internet Res 2022; 24 (11) e38525
    CrossrefPubMedGoogle Scholar
  • 41 Tudor Car L, Dhinagaran DA, Kyaw BM. et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res 2020; 22 (08) e17158
    CrossrefPubMedGoogle Scholar
  • 42 Milne-Ives M, de Cock C, Lim E. et al. The effectiveness of artificial intelligence conversational agents in health care: systematic review. J Med Internet Res 2020; 22 (10) e20346
    CrossrefPubMedGoogle Scholar
  • 43 Kamita T, Ito T, Matsumoto A, Munakata T, Inoue T. A chatbot system for mental healthcare based on SAT counseling method. Mob Inf Syst 2019; 2019: 1-11
    PubMedGoogle Scholar
  • 44 Ly KH, Ly AM, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv 2017; 10: 39-46
    CrossrefPubMedGoogle Scholar
  • 45 Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health 2017; 4 (02) e19
    CrossrefPubMedGoogle Scholar
  • 46 Inkster B, Sarda S, Subramanian V. An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR Mhealth Uhealth 2018; 6 (11) e12106
    CrossrefPubMedGoogle Scholar
  • 47 Griol D, Callejas Z. Mobile conversational agents for context-aware care applications. Cognit Comput 2016; 8 (02) 336-356
    CrossrefPubMedGoogle Scholar
  • 48 Galescu L, Allen J, Ferguson G, Quinn J, Swift M. . Speech recognition in a dialog system for patient health monitoring. Paper presented at: 2009 IEEE International Conference on Bioinformatics and Biomedicine Workshop, Washington, DC, United States; November 1–4, 2009:302–307 IEEE.
    PubMed
  • 49 Rhee H, Allen J, Mammen J, Swift M. Mobile phone-based asthma self-management aid for adolescents (mASMAA): a feasibility study. Patient Prefer Adherence 2014; 8: 63-72
    CrossrefPubMedGoogle Scholar
  • 50 Vita S, Marocco R, Pozzetto I. et al. The'doctor apollo'chatbot: a digital health tool to improve engagement of people living with HIV. J Int AIDS Soc 2018; 21 (Suppl 8): e25187
    PubMedGoogle Scholar
  • 51 Denecke K, Hochreutener SL, Pöpel A, May R. Self-anamnesis with a conversational user interface: concept and usability study. Methods Inf Med 2018; 57 (5-06): 243-252
    Article in Thieme ConnectPubMedGoogle Scholar
  • 52 Ni L, Lu C, Liu N, Liu J. . Mandy: towards a smart primary care chatbot application. In: International Symposium on Knowledge and Systems Sciences, Bangkok, Thailand; Singapore: Springer; November 17–19, 2017:38–52
    PubMed
  • 53 Razzaki S, Baker A, Perov Y. et al. A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis arXiv preprint arXiv 2018;1806:10698. 2018 June 27
    PubMed
  • 54 Ghosh S, Bhatia S, Bhatia A. Quro: facilitating user symptom check using a personalised chatbot-oriented dialogue system. Stud Health Technol Inform 2018; 252: 51-56
    PubMedGoogle Scholar
  • 55 Wilson N, MacDonald EJ, Mansoor OD, Morgan J. In bed with Siri and Google Assistant: a comparison of sexual health advice. BMJ 2017; 359: j5635
    CrossrefPubMedGoogle Scholar
  • 56 Wang H, Zhang Q, Ip M, Lau JT. Social media–based conversational agents for health management and interventions. Computer 2018; 51 (08) 26-33
    CrossrefPubMedGoogle Scholar
  • 57 Elmasri D, Maeder A. . A conversational agent for an online mental health intervention. In: International Conference on Brain Informatics, Cham,: Springer; October 13, 2016: 243–251
    PubMed
  • 58 Chaix B, Bibault JE, Pienkowski A. et al. When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot. JMIR Cancer 2019; 5 (01) e12856
    CrossrefPubMedGoogle Scholar
  • 59 Fan X, Chao D, Zhang Z, Wang D, Li X, Tian F. Utilization of self-diagnosis health chatbots in real-world settings: case study. J Med Internet Res 2021; 23 (01) e19928
    CrossrefPubMedGoogle Scholar
  • 60 Weiner BJ, Lewis CC, Stanick C. et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017; 12 (01) 108
    CrossrefPubMedGoogle Scholar
  • 61 Creswell JW, Creswell JD. . Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage Publications; 2017
    PubMed
  • 62 Tanabe P, Gimbel R, Yarnold PR, Kyriacou DN, Adams JG. Reliability and validity of scores on The Emergency Severity Index version 3. Acad Emerg Med 2004; 11 (01) 59-65
    CrossrefPubMedGoogle Scholar
  • 63 Centers for Medicare and Medicaid Services. . The accountable health communities health-related social needs screening tool. AHC Screening Tool. 2019. Accessed March 15, 2022 at: https://innovation.cms.gov/files/worksheets/ahcm-screeningtool.pdf
    PubMed
  • 64 Benefits Eligibility Screening Tool (BEST). . ( n.d. ). Accessed December 9, 2021 at: https://www.disabilitybenefitscenter.org/glossary/benefits-eligibility-screening-tool
    PubMed
  • 65 Johnson S, Liu P, Campa D. , et al. Los Angeles County Health Agency: Social & Behavioral Determinants of Health Screening Guide [White Paper]. 2019
    PubMed
  • 66 Harris PA. Research Electronic Data Capture (REDCap) - planning, collecting and managing data for clinical and translational research. BMC Bioinformatics 2012; 13 (Suppl 12): A15
    CrossrefPubMedGoogle Scholar
  • 67 Emerald City Resource Guide [Internet].. Real Change; 2021. Accessed December 2022 at: https://www.realchangenews.org/emerald-city-resource-guide
    PubMed
  • 68 Washington 2-1-1 [Internet]. Washington 211; 2022. Accessed December 12, 2022 at: https://wa211.org/
    PubMed
  • 69 Morris NS, MacLean CD, Chew LD, Littenberg B. The Single Item Literacy Screener: evaluation of a brief instrument to identify limited reading ability. BMC Fam Pract 2006; 7 (01) 21
    CrossrefPubMedGoogle Scholar
  • 70 Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15 (09) 1277-1288
    CrossrefPubMedGoogle Scholar
  • 71 Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4 (01) 50
    CrossrefPubMedGoogle Scholar
  • 72 Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are enough?. Qual Health Res 2017; 27 (04) 591-608
    CrossrefPubMedGoogle Scholar
  • 73 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-112
    CrossrefPubMedGoogle Scholar
  • 74 Lybarger K, Ostendorf M, Yetisgen M. Annotating social determinants of health using active learning, and characterizing determinants using neural event extraction. J Biomed Inform 2021; 113: 103631
    CrossrefPubMedGoogle Scholar
  • 75 LaForge K, Gold R, Cottrell E. et al. How 6 organizations developed tools and processes for social determinants of health screening in primary care: an overview. J Ambul Care Manage 2018; 41 (01) 2-14
    CrossrefPubMedGoogle Scholar
  • 76 Buitron de la Vega P, Losi S, Sprague Martinez L. et al. Implementing an EHR-based screening and referral system to address social determinants of health in primary care. Med Care 2019; 57 (Suppl 6, Suppl 2): S133-S139
    CrossrefPubMedGoogle Scholar
  • 77 Drake C, Batchelder H, Lian T. et al. Implementation of social needs screening in primary care: a qualitative study using the health equity implementation framework. BMC Health Serv Res 2021; 21 (01) 975
    CrossrefPubMedGoogle Scholar
  • 78 Wallace AS, Luther B, Guo JW, Wang CY, Sisler S, Wong B. Implementing a social determinants screening and referral infrastructure during routine emergency department visits, Utah, 2017–2018. Prev Chronic Dis 2020; 17: E45
    CrossrefPubMedGoogle Scholar
  • 79 Hassan A, Blood EA, Pikcilingis A. et al. Youths' health-related social problems: concerns often overlooked during the medical visit. J Adolesc Health 2013; 53 (02) 265-271
    CrossrefPubMedGoogle Scholar
  • 80 Sundar KR. Universal screening for social needs in a primary care clinic: a quality improvement approach using the your current life situation survey. Perm J 2018; 22: 18-089
    CrossrefPubMedGoogle Scholar
  • 81 Butler ED, Morgan AU, Kangovi S. Screening for unmet social needs: Patient engagement or alienation?. NEJM Catal 2020; 1 (04) 10
    PubMedGoogle Scholar