Appl Clin Inform 2023; 14(02): 374-391
DOI: 10.1055/a-2035-5342
Research Article

Implementation Fidelity of Chatbot Screening for Social Needs: Acceptability, Feasibility, Appropriateness

Raina Langevin
1   Department of Human Centered Design and Engineering, University of Washington, Seattle, Washington, United States
Andrew B. L. Berry
2   Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States
Jinyang Zhang
3   Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, Washington, United States
Callan E. Fockele
4   Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States
Layla Anderson
4   Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States
Dennis Hsieh
5   Department of Emergency Medicine, Harbor-University of California Los Angeles Medical Center, Torrance, California, United States
Andrea Hartzler
6   Biomedical Informatics and Medical Education, University of Washington, Seattle, Washington, United States
Herbert C. Duber
4   Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States
7   Office of Health and Science, Washington State Department of Health, Seattle, Washington, United States
Gary Hsieh
1   Department of Human Centered Design and Engineering, University of Washington, Seattle, Washington, United States
› Author Affiliations
Funding This project was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under award number: UL1 TR002319. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.


Objectives Patient and provider-facing screening tools for social determinants of health have been explored in a variety of contexts; however, effective screening and resource referral remain challenging, and less is known about how patients perceive chatbots as potential social needs screening tools. We investigated patient perceptions of a chatbot for social needs screening using three implementation outcome measures: acceptability, feasibility, and appropriateness.

Methods We implemented a chatbot for social needs screening at one large public hospital emergency department (ED) and used concurrent triangulation to assess perceptions of the chatbot use for screening. A total of 350 ED visitors completed the social needs screening and rated the chatbot on implementation outcome measures, and 22 participants engaged in follow-up phone interviews.

Results The screened participants ranged in age from 18 to 90 years old and were diverse in race/ethnicity, education, and insurance status. Participants (n = 350) rated the chatbot as an acceptable, feasible, and appropriate way of screening. Through interviews (n = 22), participants explained that the chatbot was a responsive, private, easy to use, efficient, and comfortable channel to report social needs in the ED, but wanted more information on data use and more support in accessing resources.

Conclusion In this study, we deployed a chatbot for social needs screening in a real-world context and found patients perceived the chatbot to be an acceptable, feasible, and appropriate modality for social needs screening. Findings suggest that chatbots are a promising modality for social needs screening and can successfully engage a large, diverse patient population in the ED. This is significant, as it suggests that chatbots could facilitate a screening process that ultimately connects patients to care for social needs, improving health and well-being for members of vulnerable patient populations.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. Study procedures were approved by the University of Washington Institutional Review Board (IRB) and received a waiver of written consent. In the chatbot screening, participants read a short introduction to the study and were asked if they consent to participating by clicking “Okay, let's start” in order to proceed.

Supplementary Material

Publication History

Received: 02 September 2022

Accepted: 14 February 2023

Accepted Manuscript online:
14 February 2023

Article published online:
17 May 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Marmot M. Social determinants of health inequalities. Lancet 2005; 365 (9464): 1099-1104
  • 2 Gottlieb LM, Wing H, Adler NE. A systematic review of interventions on patients' social and economic needs. Am J Prev Med 2017; 53 (05) 719-729
  • 3 Adler NE, Glymour MM, Fielding J. Addressing social determinants of health and health inequalities. JAMA 2016; 316 (16) 1641-1642
  • 4 Marmot M, Friel S, Bell R, Houweling TA, Taylor S. ; Commission on Social Determinants of Health. Closing the gap in a generation: health equity through action on the social determinants of health. Lancet 2008; 372 (9650): 1661-1669
  • 5 Sulo S, Feldstein J, Partridge J, Schwander B, Sriram K, Summerfelt WT. Budget impact of a comprehensive nutrition-focused quality improvement program for malnourished hospitalized patients. Am Health Drug Benefits 2017; 10 (05) 262-270
  • 6 Castrucci B, Auerbach J. Meeting individual social needs falls short of addressing social determinants of health. Health Affairs Blog 2019 Accessed March 2, 2023 at:
  • 7 Malecha PW, Williams JH, Kunzler NM, Goldfrank LR, Alter HJ, Doran KM. Material needs of emergency department patients: a systematic review. Acad Emerg Med 2018; 25 (03) 330-359
  • 8 Gordon JA. The hospital emergency department as a social welfare institution. Ann Emerg Med 1999; 33 (03) 321-325
  • 9 Hsu C, Cruz S, Placzek H. et al. Patient perspectives on addressing social needs in primary care using a screening and resource referral intervention. J Gen Intern Med 2020; 35 (02) 481-489
  • 10 Tong ST, Liaw WR, Kashiri PL. et al. Clinician experiences with screening for social needs in primary care. J Am Board Fam Med 2018; 31 (03) 351-363
  • 11 Persaud S. Addressing social determinants of health through advocacy. Nurs Adm Q 2018; 42 (02) 123-128
  • 12 Bleacher H, Lyon C, Mims L, Cebuhar K, Begum A. The feasibility of screening for social determinants of health: seven lessons learned. Fam Pract Manag 2019; 26 (05) 13-19
  • 13 Zulman DM, Maciejewski ML, Grubber JM. et al. Patient-reported social and behavioral determinants of health and estimated risk of hospitalization in high-risk veterans affairs patients. JAMA Netw Open 2020; 3 (10) e2021457
  • 14 Power-Hays A, Li S, Mensah A, Sobota A. Universal screening for social determinants of health in pediatric sickle cell disease: a quality-improvement initiative. Pediatr Blood Cancer 2020; 67 (01) e28006
  • 15 Chang C, Ceci C, Uberoi M, Waselewski M, Chang T. Youth perspectives on their medical team's role in screening for and addressing social determinants of health. J Adolesc Health 2022; 70 (06) 928-933
  • 16 Berger-Jenkins E, Monk C, DʼOnfro K. et al. Screening for both child behavior and social determinants of health in pediatric primary care. J Dev Behav Pediatr 2019; 40 (06) 415-424
  • 17 Katz-Wise SL, Reisner SL, White Hughto JM, Budge SL. Self-reported changes in attractions and social determinants of mental health in transgender adults. Arch Sex Behav 2017; 46 (05) 1425-1439
  • 18 Gottlieb L, Hessler D, Long D, Amaya A, Adler N. A randomized trial on screening for social determinants of health: the iScreen study. Pediatrics 2014; 134 (06) e1611-e1618
  • 19 Tai-Seale M, Downing NL, Jones VG. et al. Technology-enabled consumer engagement: promising practices at four health care delivery organizations. Health Aff (Millwood) 2019; 38 (03) 383-390
  • 20 Gold R, Bunce A, Cowburn S. et al. Adoption of social determinants of health EHR tools by community health centers. Ann Fam Med 2018; 16 (05) 399-407
  • 21 Ancker JS, Barrón Y, Rockoff ML. et al. Use of an electronic patient portal among disadvantaged populations. J Gen Intern Med 2011; 26 (10) 1117-1123
  • 22 Singh P, Jonnalagadda P, Morgan E, Fareed N. Outpatient portal use in prenatal care: differential use by race, risk, and area social determinants of health. J Am Med Inform Assoc 2022; 29 (02) 364-371
  • 23 Anthony DL, Campos-Castillo C, Lim PS. Who isn't using patient portals and why? Evidence and implications from a national sample of US adults. Health Aff (Millwood) 2018; 37 (12) 1948-1954
  • 24 Wu AW, Weston CM, Ibe CA. et al. The Baltimore community-based organizations neighborhood network: enhancing capacity together (CONNECT) cluster RCT. Am J Prev Med 2019; 57 (02) e31-e41
  • 25 Capp R, Misky GJ, Lindrooth RC. et al. Coordination program reduced acute care use and increased primary care visits among frequent emergency care users. Health Aff (Millwood) 2017; 36 (10) 1705-1711
  • 26 Kaufman S, Ali N, DeFiglio V, Craig K, Brenner J. Early efforts to target and enroll high-risk diabetic patients into urban community-based programs. Health Promot Pract 2014; 15 (2, Suppl): 62S-70S
  • 27 Lin MP, Blanchfield BB, Kakoza RM. et al. ED-based care coordination reduces costs for frequent ED users. Am J Manag Care 2017; 23 (12) 762-766
  • 28 Wilcox D, McCauley PS, Delaney C, Molony SL. Evaluation of a hospital: community partnership to reduce 30-day readmissions. Prof Case Manag 2018; 23 (06) 327-341
  • 29 Laranjo L, Dunn AG, Tong HL. et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc 2018; 25 (09) 1248-1258
  • 30 Montenegro JL, da Costa CA, da Rosa Righi R. Survey of conversational agents in health. Expert Syst Appl 2019; 129: 56-67
  • 31 Xiao Z, Zhou MX, Liao QV. et al. Tell me about yourself: using an AI-powered chatbot to conduct conversational surveys with open-ended questions. ACM Trans Comput Hum Interact (TOCHI) 2020; 27 (03) 1-37
  • 32 Kocielnik R, Agapie E, Argyle A. et al. HarborBot: a chatbot for social needs screening. AMIA Annu Symp Proc 2019; 2019: 552-561
  • 33 Kocielnik R, Langevin R, George JS. , et al. Can I talk to you about your social needs? Understanding preference for conversational user interface in health. Paper presented at: CUI 2021: 3rd Conference on Conversational User Interfaces, Bilbao, Spain; July 27–29, 2021:1–10
  • 34 Lucas GM, Gratch J, King A, Morency LP. It's only a computer: virtual humans increase willingness to disclose. Comput Human Behav 2014; 37: 94-100
  • 35 Auriacombe M, Moriceau S, Serre F. et al. Development and validation of a virtual agent to screen tobacco and alcohol use disorders. Drug Alcohol Depend 2018; 193: 1-6
  • 36 Philip P, Micoulaud-Franchi JA, Sagaspe P. et al. Virtual human as a new diagnostic tool, a proof of concept study in the field of major depressive disorders. Sci Rep 2017; 7 (01) 42656
  • 37 Lucas GM, Rizzo A, Gratch J. et al. Reporting mental health symptoms: breaking down barriers to care with virtual human interviewers. Front Robot AI 2017; 4: 51
  • 38 Stowell E, Lyson MC, Saksono H. , et al. Designing and evaluating mHealth interventions for vulnerable populations: a systematic review. Paper presented at: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, Canada; April 21–26, 2018:1–17
  • 39 Cornelius JB, St Lawrence JS, Howard JC. et al. Adolescents' perceptions of a mobile cell phone text messaging-enhanced intervention and development of a mobile cell phone-based HIV prevention intervention. J Spec Pediatr Nurs 2012; 17 (01) 61-69
  • 40 Kocaballi AB, Sezgin E, Clark L. et al. Design and evaluation challenges of conversational agents in health care and well-being: selective review study. J Med Internet Res 2022; 24 (11) e38525
  • 41 Tudor Car L, Dhinagaran DA, Kyaw BM. et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res 2020; 22 (08) e17158
  • 42 Milne-Ives M, de Cock C, Lim E. et al. The effectiveness of artificial intelligence conversational agents in health care: systematic review. J Med Internet Res 2020; 22 (10) e20346
  • 43 Kamita T, Ito T, Matsumoto A, Munakata T, Inoue T. A chatbot system for mental healthcare based on SAT counseling method. Mob Inf Syst 2019; 2019: 1-11
  • 44 Ly KH, Ly AM, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv 2017; 10: 39-46
  • 45 Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health 2017; 4 (02) e19
  • 46 Inkster B, Sarda S, Subramanian V. An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR Mhealth Uhealth 2018; 6 (11) e12106
  • 47 Griol D, Callejas Z. Mobile conversational agents for context-aware care applications. Cognit Comput 2016; 8 (02) 336-356
  • 48 Galescu L, Allen J, Ferguson G, Quinn J, Swift M. . Speech recognition in a dialog system for patient health monitoring. Paper presented at: 2009 IEEE International Conference on Bioinformatics and Biomedicine Workshop, Washington, DC, United States; November 1–4, 2009:302–307 IEEE.
  • 49 Rhee H, Allen J, Mammen J, Swift M. Mobile phone-based asthma self-management aid for adolescents (mASMAA): a feasibility study. Patient Prefer Adherence 2014; 8: 63-72
  • 50 Vita S, Marocco R, Pozzetto I. et al. The'doctor apollo'chatbot: a digital health tool to improve engagement of people living with HIV. J Int AIDS Soc 2018; 21 (Suppl 8): e25187
  • 51 Denecke K, Hochreutener SL, Pöpel A, May R. Self-anamnesis with a conversational user interface: concept and usability study. Methods Inf Med 2018; 57 (5-06): 243-252
  • 52 Ni L, Lu C, Liu N, Liu J. . Mandy: towards a smart primary care chatbot application. In: International Symposium on Knowledge and Systems Sciences, Bangkok, Thailand; Singapore: Springer; November 17–19, 2017:38–52
  • 53 Razzaki S, Baker A, Perov Y. et al. A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis arXiv preprint arXiv 2018;1806:10698. 2018 June 27
  • 54 Ghosh S, Bhatia S, Bhatia A. Quro: facilitating user symptom check using a personalised chatbot-oriented dialogue system. Stud Health Technol Inform 2018; 252: 51-56
  • 55 Wilson N, MacDonald EJ, Mansoor OD, Morgan J. In bed with Siri and Google Assistant: a comparison of sexual health advice. BMJ 2017; 359: j5635
  • 56 Wang H, Zhang Q, Ip M, Lau JT. Social media–based conversational agents for health management and interventions. Computer 2018; 51 (08) 26-33
  • 57 Elmasri D, Maeder A. . A conversational agent for an online mental health intervention. In: International Conference on Brain Informatics, Cham,: Springer; October 13, 2016: 243–251
  • 58 Chaix B, Bibault JE, Pienkowski A. et al. When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot. JMIR Cancer 2019; 5 (01) e12856
  • 59 Fan X, Chao D, Zhang Z, Wang D, Li X, Tian F. Utilization of self-diagnosis health chatbots in real-world settings: case study. J Med Internet Res 2021; 23 (01) e19928
  • 60 Weiner BJ, Lewis CC, Stanick C. et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017; 12 (01) 108
  • 61 Creswell JW, Creswell JD. . Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage Publications; 2017
  • 62 Tanabe P, Gimbel R, Yarnold PR, Kyriacou DN, Adams JG. Reliability and validity of scores on The Emergency Severity Index version 3. Acad Emerg Med 2004; 11 (01) 59-65
  • 63 Centers for Medicare and Medicaid Services. . The accountable health communities health-related social needs screening tool. AHC Screening Tool. 2019. Accessed March 15, 2022 at:
  • 64 Benefits Eligibility Screening Tool (BEST). . ( n.d. ). Accessed December 9, 2021 at:
  • 65 Johnson S, Liu P, Campa D. , et al. Los Angeles County Health Agency: Social & Behavioral Determinants of Health Screening Guide [White Paper]. 2019
  • 66 Harris PA. Research Electronic Data Capture (REDCap) - planning, collecting and managing data for clinical and translational research. BMC Bioinformatics 2012; 13 (Suppl 12): A15
  • 67 Emerald City Resource Guide [Internet].. Real Change; 2021. Accessed December 2022 at:
  • 68 Washington 2-1-1 [Internet]. Washington 211; 2022. Accessed December 12, 2022 at:
  • 69 Morris NS, MacLean CD, Chew LD, Littenberg B. The Single Item Literacy Screener: evaluation of a brief instrument to identify limited reading ability. BMC Fam Pract 2006; 7 (01) 21
  • 70 Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15 (09) 1277-1288
  • 71 Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4 (01) 50
  • 72 Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are enough?. Qual Health Res 2017; 27 (04) 591-608
  • 73 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-112
  • 74 Lybarger K, Ostendorf M, Yetisgen M. Annotating social determinants of health using active learning, and characterizing determinants using neural event extraction. J Biomed Inform 2021; 113: 103631
  • 75 LaForge K, Gold R, Cottrell E. et al. How 6 organizations developed tools and processes for social determinants of health screening in primary care: an overview. J Ambul Care Manage 2018; 41 (01) 2-14
  • 76 Buitron de la Vega P, Losi S, Sprague Martinez L. et al. Implementing an EHR-based screening and referral system to address social determinants of health in primary care. Med Care 2019; 57 (Suppl 6, Suppl 2): S133-S139
  • 77 Drake C, Batchelder H, Lian T. et al. Implementation of social needs screening in primary care: a qualitative study using the health equity implementation framework. BMC Health Serv Res 2021; 21 (01) 975
  • 78 Wallace AS, Luther B, Guo JW, Wang CY, Sisler S, Wong B. Implementing a social determinants screening and referral infrastructure during routine emergency department visits, Utah, 2017–2018. Prev Chronic Dis 2020; 17: E45
  • 79 Hassan A, Blood EA, Pikcilingis A. et al. Youths' health-related social problems: concerns often overlooked during the medical visit. J Adolesc Health 2013; 53 (02) 265-271
  • 80 Sundar KR. Universal screening for social needs in a primary care clinic: a quality improvement approach using the your current life situation survey. Perm J 2018; 22: 18-089
  • 81 Butler ED, Morgan AU, Kangovi S. Screening for unmet social needs: Patient engagement or alienation?. NEJM Catal 2020; 1 (04) 10