Keywords
EHRs and systems - human–computer interaction - clinical decision support - natural
language processing - clinical data management
Background and Significance
Background and Significance
Electronic health records (EHRs) are essential to patient care and serve as a data
repository and communication tool. EHRs usually display data by type, presenting similar
data like medications, notes, or laboratory results together. This data segmentation
forces providers with clinical questions to perform extensive, time-consuming searches
to gather the required data elements.[1]
Voice assistants have been used for a variety of tasks[2] including medication adherence,[3] data collection,[4] and companionship in adults.[5] While many clinicians use voice technology for nonclinical purposes, a minority
also uses it in the clinical domain.[6] A review of existing voice assistant systems revealed limited development in the
context of EHRs, specifically designed to address the unique needs and challenges
of health care providers.[2] We developed a voice-mediated EHR assistant known as the Vanderbilt EHR Voice Assistant
(VEVA) that summarizes EHR information and allows contextually aware ordering in preparation
for a clinical encounter. VEVA's voice interface allows searching and summarizing
health record data, which may improve workflows and reduce provider burnout.[7]
[8] We evaluated the usability and acceptability of VEVA by practicing physicians in
guided interactions in an outpatient setting.
Vanderbilt Electronic Health Records Voice Assistant Overview
VEVA accepts voice inputs (e.g., “VEVA, give me the last A1C for Mr. Smith”) as imperative
or interrogative queries, translates voice into text, and then uses the natural language
processing (NLP) engine to map the text to executable EHR commands. VEVA executes
user queries via Fast Health Interoperability Resources (FHIR) and other application
programming interfaces. VEVA's business logic synthesizes relevant results and returns
them to the user as voice replies, text, and/or figures as appropriate (Demo [Appendix A Video 1]).
Appendix A Video 1 VEVA brief demo. VEVA, Vanderbilt Electronic Health Record Voice Assistant.
We engineered VEVA as a responsive web application for mobile devices. The user interface
is comprised of a JavaScript application using the Angular framework[9] and is integrated with Vanderbilt's EHR using Substitutable Medical Applications
and Reusable Technologies on FHIR resources,[10] which provide user authentication and patient context to VEVA. RESTful services[11] built in Java provide business logic and store data to an Oracle database. The third-party
Nuance Florence software[12] serves as the automatic speech recognition NLP engine leveraging Nuance's subject
matter expertise in medical speech recognition.[13] The technology choices for VEVA were based on a comprehensive review of the current
best practices and emerging trends in the domain (VEVA Schematic [Appendix B]).[2]
[14]
Objective
The effect of EHR vice assistants like VEVA on the searching effectiveness and efficiency
is not known. However, as an important first step, it is essential to understand VEVA's
usability and acceptability among providers. For this study, we describe a usability
and acceptability evaluation of VEVA by physicians practicing in an outpatient setting.
Methods
We recruited physicians and nurse practitioners from the Vanderbilt Pediatric Endocrine
Division. Outpatient pediatric endocrinology was selected as the area of focus for
this study due to the specialized nature of pediatric diabetes management, which lent
itself to effective query and summarization of discrete clinical findings with the
voice assistant in preparation for a clinical encounter. Providers were notified about
the study and its aims during a presentation at the Weekly Pediatric Endocrine Lecture
Series. Providers were consented through an institutional review board-approved process.
Providers engaged in six guided interactions with VEVA: medical summary, A1C, weight,
blood pressure, health maintenance, and laboratory alert. Users phrased queries in
their own words. After interacting with VEVA, users provided feedback focusing on
its usability.
Provider interviews regarding their VEVA interactions were audio-recorded and coded by at least two of
the seven-member research team. Our qualitative analysis followed a systematic approach,
including code generation, thematic analysis, and intercoder adjudication. Identified
themes were compared and discrepancies resolved through consensus discussions.
Following the interview, each provider completed a System Usability Scale (SUS) Assessment[15] and rated VEVA's effectiveness using a 5-point Likert scale ranging from strongly
disagree (1) to strongly agree (5).
Results
Fourteen providers (mean age: 39, range: 29–65), including 10 physicians and four
nurse practitioners, participated in the VEVA usability assessment. The 14 providers
deemed the VEVA prototype highly usable (mean SUS score 81, scores of greater than
68 are considered above average across other information systems such as EHRs). The
highest rated SUS item was “I thought the system was easy to use,” with an average
score of 4.5 [0.52 SD]. The lowest rated item on the SUS was “I found the various
functions in this system were well integrated,” with an average score of 3.79 [0.70
SD]. Nine out of the 14 providers (64%) indicated willingness to use the VEVA prototype
in its current form assuming continued improvement iterations of the platform. Qualitative
results from VEVA system primary interactions are shown in [Table 1].
Table 1
Key qualitative themes identified from provider Vanderbilt Electronic Health Record
Voice Assistant interviews
|
Summary item
|
Structure of summary item
|
User perception of usefulness: number of users
|
User perception of length: number of users
|
|
Number words
|
Duration (s)
|
Latency (s)
|
Positive sentiment
|
Neutral sentiment
|
Negative
sentiment
|
Too long
|
Appropriate
|
Too short
|
|
Overview
|
102
|
37
|
4
|
11
|
3
|
0
|
2
|
10
|
2
|
|
Suggested modifications (per number of users): exclude glucose calculation derived from A1C (9); add medication and routes of administration
(8); Add option for bullets (7); add method of insulin administration, availability
of continuous glucose monitoring school absence (6); exclude language preference (5);
exclude next scheduled visit date (3); exclude duration of diabetes (2); add major
medical problems (2)
|
|
A1C
|
40
|
15
|
0.4
|
10
|
4
|
0
|
1
|
10
|
3
|
|
Suggested modifications (per number of users): omit previous A1C comparison (7); add incorporation of normative reference ranges
with the graphical visualization of the A1C (2); add colorization to improve the visualization
(2)
|
|
Weight
|
27
|
9
|
0.4
|
13
|
0
|
1
|
3
|
11
|
0
|
|
Suggested modifications (per number of users): add weight summary as growth chart (5); add weight percentiles (5); add Inclusion
of height and growth information (4)
|
|
Blood pressure (BP)
|
40
|
21
|
0.4
|
10
|
3
|
1
|
8
|
6
|
0
|
|
Suggested modifications (per number of users): add BP percentiles (8); add interpretation of BP change as concerning/not concerning
(5); add bars on graph to indicate normal values (2); tabular format of data (2)
|
|
Health maintenance
|
50–95
|
21–41
|
0.4
|
13
|
1
|
0
|
3
|
11
|
0
|
|
Suggested modifications (per number of users): add interactive ordering component (3); add normal laboratories (3); add overdue
laboratories (2)
|
|
Decision support
(TSH alert)
|
8
|
3
|
—
|
13
|
1
|
0
|
–
|
–
|
–
|
|
Suggested modifications (per number of users): alert tone prior to decision support (2)
|
Abbreviation: TSH = Thyroid Stimulating Hormone
Discussion
We developed a novel web-based voice assistant for EHR interaction capable of receiving
verbal commands, collating requested information, and presenting it to the user, thus
eliminating the user's need to search for disparate EHR data. VEVA's SUS scores were
better than benchmark scores across other EHR information systems, suggesting that
providers perceived VEVA as usable. Most providers agreed to use VEVA in its current
state in their clinical practice, whereas others suggested simple improvements.
To date, voice is used in health care predominately in one of three domains: “Voice
for (1) documentation, (2) commands, and (3) interactive response and navigation for
patients.”[2] Speech recognition for EHR documentation is associated with significantly lower
SUS scores, most likely as a function of the effort required to correct transcription
mistakes, which was the main reason to abandon speech documentation for 70% of users
in a 2010 study.[16] We used VEVA for a new domain—summarization—which avoided the semantic complexity
of documentation and led to higher acceptance as indicated by the SUS. Our work suggests
that voice could be exploited to address the challenge of “foraging for EHR information.”[17]
VEVA's translation of text to speech was occasionally inexact and occasionally experienced
intermittent latency. For example, VEVA would pronounce the Roman numeral I in “Type
I Diabetes” as the letter “i” instead of the number “one” resulting in the expression
“Type i diabetes” instead of “Type one diabetes.” This highlights the need for expanded
prosodic and pronunciation training tailored to medical vocabulary and terminology.
While text is typically read silently, voice assistant tools that now speak aloud
clinical content must account for these context-specific vocalization considerations,
which is a newer paradigm. The finding that VEVA mispronounced or misunderstood some
requests while still being rated very usable reveals a discrepancy in the user experience.
It suggests the concept of a flexibility threshold, where users may tolerate some
degree of error if the technology otherwise proves useful at addressing other workflow
needs. Further research on user expectations and that threshold of usability would
provide valuable insights given the high threshold of accuracy expected for EHR interactions.
Users suggested enhancing verbal laboratory test ordering processes, which could offer
a more intuitive alternative to the traditional multistep methods with keyboard and
mouse. Further, while voice commands are an integral feature of VEVA, additional integration
of visual aids like tables and graphs could provide a more comprehensive data interpretation
platform. The utility of VEVA's summaries was acknowledged, with suggestions to integrate
it automatically into clinical notes. Feedback varied on the length of the prose of
information delivery, with some providers favoring detailed explanations, whereas
others preferred brevity. User feedback underscored the potential for user-centric
refinements that could accommodate diverse user preferences and allow users more control
over their experience.
Limitations
Our study has several limitations, including its single-site focus and its specialty-specific
intervention, which might affect its replicability across broader domains. Environments
with high EHR utilization and information needs are most poised to benefit, whereas
complex workflows reliant on paper or data not readily available in the EHR may present
adoption barriers. The use of VEVA was explored in a smaller sample, which was suitable
for qualitative analysis but limited quantitative approaches. Although this study
focused on individual provider use in preparation for clinic encounters, exploring
VEVA interactions with patients merits future research.
Conclusion
We developed a voice assistant tool for our pediatric endocrinology clinic with a
SUS score in a highly usable range. Our prototype elicited noteworthy requests for
improvements and additional features that enhanced our understanding of the expectations
surrounding human–computer interactions with EHR voice assistant tools. With expanded
usability testing, we can determine if VEVA integrates successfully into routine outpatient
clinical workflows and potentially assess the future opportunities for incorporation
with patient portal systems. The advent of advanced large language models, which were
not available at the time of VEVA's initial design, now present new compelling opportunities
for augmenting conversational agent capabilities. Applying this technology thoughtfully
to VEVA iterations could open new possibilities for voice assistant architecture and
improved natural language interactions. Overall, our findings highlight promising
directions both for refining VEVA locally and advancing EHR voice assistants more
broadly.
Clinical Relevance Statement
Clinical Relevance Statement
Voice-mediated EHR voice assistants like VEVA that search and summarize health record
data have the potential to improve workflows and serve as a useful tool in health
care encounters as well as reduce provider burnout.
Multiple-Choice Questions
Multiple-Choice Questions
-
Which of the following is true about the VEVA usability assessment?
-
The VEVA prototype was deemed highly usable, with a mean SUS score of 60
-
All 14 providers who participated in the VEVA usability assessment were MDs
-
Providers unanimously agreed that VEVA's voice component sounded natural and high
quality
-
A total of 64% of providers indicated that they would be willing to use the VEVA prototype
in its current form assuming continued improvement iterations of the platform
Correct Answer: The correct answer is option d. According to the text, 9 out of the 14 providers
(64%) indicated that they would be willing to use the VEVA prototype in its current
form assuming continued improvement iterations of the platform. Therefore, option
d is correct. Option a is incorrect because the mean SUS score was 81, not 60. Option
b is incorrect because 3 of the providers were nurse practitioners, not all of them
were MDs. Option c is incorrect because some providers commented that the VEVA voice
component sounded “unnatural” and “stunted” in quality.
-
What is the primary purpose of the VEVA?
-
To replace traditional medical consultations for patients
-
To provide companionship to health care providers
-
To assist health care providers with searching and summarizing EHR data
-
To conduct automated laboratory tests for patients
Correct Answer: The correct answer is option c. The primary purpose of the VEVA is to assist health
care providers with searching and summarizing EHR data. It aims to improve workflows
and reduce provider burnout by providing a voice interface for querying and summarizing
health record information.
