Keywords
EHRs and systems - human–computer interaction - clinical decision support - natural language processing - clinical data management
Background and Significance
Background and Significance
Electronic health records (EHRs) are essential to patient care and serve as a data repository and communication tool. EHRs usually display data by type, presenting similar data like medications, notes, or laboratory results together. This data segmentation forces providers with clinical questions to perform extensive, time-consuming searches to gather the required data elements.[1]
Voice assistants have been used for a variety of tasks[2] including medication adherence,[3] data collection,[4] and companionship in adults.[5] While many clinicians use voice technology for nonclinical purposes, a minority also uses it in the clinical domain.[6] A review of existing voice assistant systems revealed limited development in the context of EHRs, specifically designed to address the unique needs and challenges of health care providers.[2] We developed a voice-mediated EHR assistant known as the Vanderbilt EHR Voice Assistant (VEVA) that summarizes EHR information and allows contextually aware ordering in preparation for a clinical encounter. VEVA's voice interface allows searching and summarizing health record data, which may improve workflows and reduce provider burnout.[7]
[8] We evaluated the usability and acceptability of VEVA by practicing physicians in guided interactions in an outpatient setting.
Vanderbilt Electronic Health Records Voice Assistant Overview
VEVA accepts voice inputs (e.g., “VEVA, give me the last A1C for Mr. Smith”) as imperative or interrogative queries, translates voice into text, and then uses the natural language processing (NLP) engine to map the text to executable EHR commands. VEVA executes user queries via Fast Health Interoperability Resources (FHIR) and other application programming interfaces. VEVA's business logic synthesizes relevant results and returns them to the user as voice replies, text, and/or figures as appropriate (Demo [Appendix A Video 1]).
Appendix A Video 1 VEVA brief demo. VEVA, Vanderbilt Electronic Health Record Voice Assistant.
We engineered VEVA as a responsive web application for mobile devices. The user interface is comprised of a JavaScript application using the Angular framework[9] and is integrated with Vanderbilt's EHR using Substitutable Medical Applications and Reusable Technologies on FHIR resources,[10] which provide user authentication and patient context to VEVA. RESTful services[11] built in Java provide business logic and store data to an Oracle database. The third-party Nuance Florence software[12] serves as the automatic speech recognition NLP engine leveraging Nuance's subject matter expertise in medical speech recognition.[13] The technology choices for VEVA were based on a comprehensive review of the current best practices and emerging trends in the domain (VEVA Schematic [Appendix B]).[2]
[14]
Objective
The effect of EHR vice assistants like VEVA on the searching effectiveness and efficiency is not known. However, as an important first step, it is essential to understand VEVA's usability and acceptability among providers. For this study, we describe a usability and acceptability evaluation of VEVA by physicians practicing in an outpatient setting.
Methods
We recruited physicians and nurse practitioners from the Vanderbilt Pediatric Endocrine Division. Outpatient pediatric endocrinology was selected as the area of focus for this study due to the specialized nature of pediatric diabetes management, which lent itself to effective query and summarization of discrete clinical findings with the voice assistant in preparation for a clinical encounter. Providers were notified about the study and its aims during a presentation at the Weekly Pediatric Endocrine Lecture Series. Providers were consented through an institutional review board-approved process.
Providers engaged in six guided interactions with VEVA: medical summary, A1C, weight, blood pressure, health maintenance, and laboratory alert. Users phrased queries in their own words. After interacting with VEVA, users provided feedback focusing on its usability.
Provider interviews regarding their VEVA interactions were audio-recorded and coded by at least two of the seven-member research team. Our qualitative analysis followed a systematic approach, including code generation, thematic analysis, and intercoder adjudication. Identified themes were compared and discrepancies resolved through consensus discussions.
Following the interview, each provider completed a System Usability Scale (SUS) Assessment[15] and rated VEVA's effectiveness using a 5-point Likert scale ranging from strongly disagree (1) to strongly agree (5).
Results
Fourteen providers (mean age: 39, range: 29–65), including 10 physicians and four nurse practitioners, participated in the VEVA usability assessment. The 14 providers deemed the VEVA prototype highly usable (mean SUS score 81, scores of greater than 68 are considered above average across other information systems such as EHRs). The highest rated SUS item was “I thought the system was easy to use,” with an average score of 4.5 [0.52 SD]. The lowest rated item on the SUS was “I found the various functions in this system were well integrated,” with an average score of 3.79 [0.70 SD]. Nine out of the 14 providers (64%) indicated willingness to use the VEVA prototype in its current form assuming continued improvement iterations of the platform. Qualitative results from VEVA system primary interactions are shown in [Table 1].
Table 1
Key qualitative themes identified from provider Vanderbilt Electronic Health Record Voice Assistant interviews
|
Summary item
|
Structure of summary item
|
User perception of usefulness: number of users
|
User perception of length: number of users
|
|
Number words
|
Duration (s)
|
Latency (s)
|
Positive sentiment
|
Neutral sentiment
|
Negative
sentiment
|
Too long
|
Appropriate
|
Too short
|
|
Overview
|
102
|
37
|
4
|
11
|
3
|
0
|
2
|
10
|
2
|
|
Suggested modifications (per number of users): exclude glucose calculation derived from A1C (9); add medication and routes of administration (8); Add option for bullets (7); add method of insulin administration, availability of continuous glucose monitoring school absence (6); exclude language preference (5); exclude next scheduled visit date (3); exclude duration of diabetes (2); add major medical problems (2)
|
|
A1C
|
40
|
15
|
0.4
|
10
|
4
|
0
|
1
|
10
|
3
|
|
Suggested modifications (per number of users): omit previous A1C comparison (7); add incorporation of normative reference ranges with the graphical visualization of the A1C (2); add colorization to improve the visualization (2)
|
|
Weight
|
27
|
9
|
0.4
|
13
|
0
|
1
|
3
|
11
|
0
|
|
Suggested modifications (per number of users): add weight summary as growth chart (5); add weight percentiles (5); add Inclusion of height and growth information (4)
|
|
Blood pressure (BP)
|
40
|
21
|
0.4
|
10
|
3
|
1
|
8
|
6
|
0
|
|
Suggested modifications (per number of users): add BP percentiles (8); add interpretation of BP change as concerning/not concerning (5); add bars on graph to indicate normal values (2); tabular format of data (2)
|
|
Health maintenance
|
50–95
|
21–41
|
0.4
|
13
|
1
|
0
|
3
|
11
|
0
|
|
Suggested modifications (per number of users): add interactive ordering component (3); add normal laboratories (3); add overdue laboratories (2)
|
|
Decision support
(TSH alert)
|
8
|
3
|
—
|
13
|
1
|
0
|
–
|
–
|
–
|
|
Suggested modifications (per number of users): alert tone prior to decision support (2)
|
Abbreviation: TSH = Thyroid Stimulating Hormone
Discussion
We developed a novel web-based voice assistant for EHR interaction capable of receiving verbal commands, collating requested information, and presenting it to the user, thus eliminating the user's need to search for disparate EHR data. VEVA's SUS scores were better than benchmark scores across other EHR information systems, suggesting that providers perceived VEVA as usable. Most providers agreed to use VEVA in its current state in their clinical practice, whereas others suggested simple improvements.
To date, voice is used in health care predominately in one of three domains: “Voice for (1) documentation, (2) commands, and (3) interactive response and navigation for patients.”[2] Speech recognition for EHR documentation is associated with significantly lower SUS scores, most likely as a function of the effort required to correct transcription mistakes, which was the main reason to abandon speech documentation for 70% of users in a 2010 study.[16] We used VEVA for a new domain—summarization—which avoided the semantic complexity of documentation and led to higher acceptance as indicated by the SUS. Our work suggests that voice could be exploited to address the challenge of “foraging for EHR information.”[17]
VEVA's translation of text to speech was occasionally inexact and occasionally experienced intermittent latency. For example, VEVA would pronounce the Roman numeral I in “Type I Diabetes” as the letter “i” instead of the number “one” resulting in the expression “Type i diabetes” instead of “Type one diabetes.” This highlights the need for expanded prosodic and pronunciation training tailored to medical vocabulary and terminology. While text is typically read silently, voice assistant tools that now speak aloud clinical content must account for these context-specific vocalization considerations, which is a newer paradigm. The finding that VEVA mispronounced or misunderstood some requests while still being rated very usable reveals a discrepancy in the user experience. It suggests the concept of a flexibility threshold, where users may tolerate some degree of error if the technology otherwise proves useful at addressing other workflow needs. Further research on user expectations and that threshold of usability would provide valuable insights given the high threshold of accuracy expected for EHR interactions. Users suggested enhancing verbal laboratory test ordering processes, which could offer a more intuitive alternative to the traditional multistep methods with keyboard and mouse. Further, while voice commands are an integral feature of VEVA, additional integration of visual aids like tables and graphs could provide a more comprehensive data interpretation platform. The utility of VEVA's summaries was acknowledged, with suggestions to integrate it automatically into clinical notes. Feedback varied on the length of the prose of information delivery, with some providers favoring detailed explanations, whereas others preferred brevity. User feedback underscored the potential for user-centric refinements that could accommodate diverse user preferences and allow users more control over their experience.
Limitations
Our study has several limitations, including its single-site focus and its specialty-specific intervention, which might affect its replicability across broader domains. Environments with high EHR utilization and information needs are most poised to benefit, whereas complex workflows reliant on paper or data not readily available in the EHR may present adoption barriers. The use of VEVA was explored in a smaller sample, which was suitable for qualitative analysis but limited quantitative approaches. Although this study focused on individual provider use in preparation for clinic encounters, exploring VEVA interactions with patients merits future research.
Conclusion
We developed a voice assistant tool for our pediatric endocrinology clinic with a SUS score in a highly usable range. Our prototype elicited noteworthy requests for improvements and additional features that enhanced our understanding of the expectations surrounding human–computer interactions with EHR voice assistant tools. With expanded usability testing, we can determine if VEVA integrates successfully into routine outpatient clinical workflows and potentially assess the future opportunities for incorporation with patient portal systems. The advent of advanced large language models, which were not available at the time of VEVA's initial design, now present new compelling opportunities for augmenting conversational agent capabilities. Applying this technology thoughtfully to VEVA iterations could open new possibilities for voice assistant architecture and improved natural language interactions. Overall, our findings highlight promising directions both for refining VEVA locally and advancing EHR voice assistants more broadly.
Clinical Relevance Statement
Clinical Relevance Statement
Voice-mediated EHR voice assistants like VEVA that search and summarize health record data have the potential to improve workflows and serve as a useful tool in health care encounters as well as reduce provider burnout.
Multiple-Choice Questions
Multiple-Choice Questions
-
Which of the following is true about the VEVA usability assessment?
-
The VEVA prototype was deemed highly usable, with a mean SUS score of 60
-
All 14 providers who participated in the VEVA usability assessment were MDs
-
Providers unanimously agreed that VEVA's voice component sounded natural and high quality
-
A total of 64% of providers indicated that they would be willing to use the VEVA prototype in its current form assuming continued improvement iterations of the platform
Correct Answer: The correct answer is option d. According to the text, 9 out of the 14 providers (64%) indicated that they would be willing to use the VEVA prototype in its current form assuming continued improvement iterations of the platform. Therefore, option d is correct. Option a is incorrect because the mean SUS score was 81, not 60. Option b is incorrect because 3 of the providers were nurse practitioners, not all of them were MDs. Option c is incorrect because some providers commented that the VEVA voice component sounded “unnatural” and “stunted” in quality.
-
What is the primary purpose of the VEVA?
-
To replace traditional medical consultations for patients
-
To provide companionship to health care providers
-
To assist health care providers with searching and summarizing EHR data
-
To conduct automated laboratory tests for patients
Correct Answer: The correct answer is option c. The primary purpose of the VEVA is to assist health care providers with searching and summarizing EHR data. It aims to improve workflows and reduce provider burnout by providing a voice interface for querying and summarizing health record information.
