Keywords
patient-reported outcome measures - multilingualism - mobile health
Introduction
Health data are nowadays typically collected and stored electronically. The amount
of these data is increasing exponentially, thus producing new opportunities for clinical
research.[1] Electronic health records (EHRs) improve health care efficiency and safety,[2]
[3] as well as enable rapid methods for the secondary use of clinical records. Unfortunately,
the collaboration and sharing of information between outpatient clinics and hospitals
is still difficult due to legislation, security concerns, incompatibility of the systems,
and lack of standards in use.
The medical informatics community is conducting several initiatives to solve the lack
of interoperability between electronic health information systems. One of the most
widely accepted solutions is the use of standards that enable the classification of
elements in a way that several systems are compliant with, including transport protocols
and security layers to preserve data integrity and privacy. The adoption of these
standards is an ongoing task and not always straightforward. The large variety of
standards and a lack of global decisions on which to use (especially regarding transport
protocols and formats) produce a lack of consensus and interoperability problems.
Patient-reported outcomes (PROs) are defined by the Food and Drug Administration of
the United States as “any report of the status of a patient's health condition that
comes directly from the patient, without interpretation of the patient's response
by a clinician or anyone else.”[4] PROs have become a valuable tool for improving patient diagnosis and treatment,
as well as the secondary use of health data for clinical research, although their
selection needs to be handled carefully.[5] The validation of PROs and their translations, defined by Acquadro et al as “the
ability to express and investigate equivalent concepts across all language versions,”
ensures their correctness and enables their use in clinical routine and research practice.[6]
Language barriers affect quality of care around the world, hampering comprehension
and adherence as well as patient and provider satisfaction.[7] With increasing migration in Europe,[8] physicians often need to treat patients who do not speak the local language(s),
which hampers patient diagnosis and treatment. A solution could be the engagement
of translators, but they are not always available or even requested.[9] Translated and validated PROs can be used to gather patient information regardless
of the local language.
Electronic systems and mobile devices provide vital opportunities for the collection
of PROs. Electronic PRO (ePRO) systems can lead to more accurate and complete data,
improved protocol compliance, avoidance of secondary data entry errors, easier implementation
of skip patterns, less administrative burden, high respondent acceptance, reduced
sample size requirements, and potential cost savings.[10] In addition, ePROs could be used to collect patient data in any language and the
results be visualized in the local language. This would support patient diagnosis
for international patients when a translator is not an option and enable the easy
analysis of clinical data in international clinical studies. So far, the majority
of ePROs have been single purpose: they were developed for a certain study, and thus
they are generally neither multilingual, multicenter, scalable, nor interoperable.
There is therefore a need for an ePRO that allows for the collection of patient data
in several centers across different countries and in several languages, being able
to export the data in different formats and standards. Furthermore, the system should
be evaluated and accepted by both clinicians and patients.
The Institute of Medical Informatics (IMI) of the University of Münster started in
2010 the development of the ePRO MoPat,[11] which was solely implemented for use at a single center (the Center for Chronic
Pruritus at the University Hospital of Münster) in a single language (German). MoPat
was newly developed (MoPat2) in 2014 and is being routinely updated.[12] MoPat2 is a Web-based ePRO with a user-friendly interface that enables patients
to answer medical questionnaires. Patient answers are encapsulated into encrypted
messages, either in Health Level-7 or Operational Data Model (ODM) format, and securely
transferred to EHRs and/or research databases.[13] MoPat2 is a generic ePRO that includes several functions such as the import of questionnaires
from ODM files, creation of new questionnaires and surveys, generation of user statistics,
and a score calculator based on questionnaire responses.
Objectives
The aim of this research is to report the development of a multicenter, multilingual,
and interoperable ePRO and the evaluation of its user satisfaction in an international
setting.
Methods
Development: Process Steps and Technologies
The development process started with the requirements acquisition and design of the
software architecture. In this initial process step, existing solutions were explored
and their reuse was analyzed.
The second step involved the development of the system. MoPat2 was developed using
the Java 8 programming language, utilizing the frameworks jQuery Mobile 1.4.5, Spring
4, and Maven 3. The resulting functionality will be shown in detail in the “Results”
section. Its development was completed with automatic and semiautomatic tests using
jUnit and the Selenium plugin for Mozilla Firefox.
Evaluation
The newly developed functionality was evaluated by the European Network on Assessment
of Severity and Burden of Pruritus (PruNet), an expert group funded by the European
Academy of Dermatology and Venereology (EADV). The aim of the said expert group is
to harmonize the assessment of itch in routine dermatological care in Europe[14] and establish a consensus among several experts on which PROs best serve this purpose.
The selected PROs[14] were then translated and validated in several languages following their selection.
The validation of these PROs required a multilingual functionality that allows patients
from different countries to answer the same set of questions in their native language.
For a better and easier analysis of the results, the data collected had to be stored
in a central system and remain exportable in a format compliant with statistic tools.
Organizational Setting
The PruNet study was led by the Center for Chronic Pruritus and the IMI at the University
Hospital of Münster. The multilingual ePRO MoPat2 was evaluated in eight centers in
eight different countries: the Policlinico A. Gemelli, Catholic University of Sacred
Heart, Rome, Italy; the Medical University of Graz, Austria; the University Hospital
of Brest, France; the Wroclaw Medical University, Poland; the Adnan Menderes University,
Aydın, Turkey; the Kantonsspital Aarau, Switzerland; the Hospital Sant Pau, Barcelona,
Spain; and the University Hospital of Münster, Germany.
Data Collection
Data were collected in 1-month periods, from July 2015 until March 2016, in each clinic.
During this time, all dermatological patients who visited these clinical institutions
and clinicians working at the clinics involved in the PruNet project were asked to
complete an electronic survey including a user satisfaction questionnaire using MoPat2.
The PRO validation for the PruNet study[14] included two MoPat2 patient surveys: the first survey was completed by patients
following their arrival at the clinical institution, and the second survey after their
visit with the physician. The first survey contained 67 questions and included the
PROs requiring validation and a MoPat2 user satisfaction questionnaire.
Two questionnaires were created by the IMI team to assess the user satisfaction of
MoPat2, with one addressing clinicians and the other patients. The clinician questionnaire
contains seven Boolean questions (yes/no), one multiple-choice question with five
possible answers, and four free-text questions. In contrast, the patient questionnaire
contains eight Boolean, two multiple-choice, and four free-text questions. Although
MoPat2 can be used in every tablet personal computer (PC) running on Android v4.0,
iOS v7.0, or Windows Mobile v8.0 or higher; only iPad 2 tablet devices running on
iOS v9.0 were used in this study for the purpose of data collection. As no instructions
are provided to the users regarding the use of MoPat2, the satisfaction questionnaires
contain questions concerning the general usability of the iPad. MoPat2 was not referred
to, since users are unaware of its name and/or role.
The questions in the satisfaction questionnaires are meant to assess the user satisfaction
of MoPat2 (questions 1, 2, 8, and 9 in the patients' questionnaire; 1, 2, 3, and 8
in the clinicians') and the acceptance of the MoPat2 PROs (questions 3–7 in the patients'
questionnaire and 4–7 in the clinicians'). The analysis will be focused on the former.
The user satisfaction questionnaires have been included as an additional file in the
[Supplementary Appendix A].
Data Analysis
A descriptive and explorative analysis of the results was performed using R and the
corrected contingency coefficient K*∈[0,1] between patient responses and age, gender, and language of the respondents
calculated.
Results
Development of ePRO Multilingual Functions
Multilingual capabilities for the administration and user functions of MoPat2 have
been developed. Since it was previously only possible to store a single question text,
the database required enhancement. The single column storing the corresponding text
was changed to a multivalue attribute that stores the combined language code according
to International Organization for Standardization (ISO) 639, the specific country
code according to ISO 3166, and the bidirectionally translated text for the defined
language. The administration interface was extended to allow the addition of all languages
available in Java 8 Locale class. To improve the simplicity of the user interface,
the input fields for single languages have been made collapsible and are marked with
colors to indicate whether they have been included.
This functionality allows users to create questionnaires and surveys in all known
languages. The graphical user interface (GUI) was translated in seven different languages:
English, French, German, Italian, Polish, Spanish, and Turkish.
The workflow for conducting a survey using MoPat2 ([Fig. 1]) is as follows: (1) The clinician selects a language for the GUI and logs into MoPat2.
(2) The clinician enters patient's case number, pseudonym, or study identification
number. (3) The clinician selects a questionnaire bundle (the set of questionnaires
that conforms the electronic survey in MoPat2) and the patient's language. If the
questionnaire bundle is in a language not yet supported by the general GUI, the survey
(answers and questions) will be in the patient's language and the general GUI (buttons
next, previous, etc.) will remain in the currently displayed language. (4) The patient
receives the tablet PC and completes the survey, handing it back to the clinician
once the survey has been completed.
Fig. 1 Process steps to be followed to perform a multilingual survey using MoPat2.
As a Web-service, MoPat2 is available through the Internet. A Web-clip with MoPat2's
URL was installed in the devices used for surveying the patients. Once a patient completes
a survey, responses are securely sent via a Secure Sockets Layer to x4T: an electronic
data capture system located in Münster.[15] x4T was also used for subject recruitment and survey completeness tracking. All
values received in x4T apart from free-text fields were displayed in the home language.
User Satisfaction
The user satisfaction evaluation included the clinicians involved in the use of MoPat2
and the patients visiting the clinic that were recruited for the study and signed
the agreement form.
Clinician User Satisfaction
Twenty-eight clinicians completed the user satisfaction survey. The MoPat2-relevant
results are as follows:
-
Ninety-three percent of clinicians (26 out of 28) think that patient data collection
with MoPat2 is acceptable.
-
Ninety-six percent of clinicians (27 out of 28) think that MoPat2 is an appropriate
tool for recording pruritus in patients.
-
On a scale from 1 (“very good”) to 5 (“very bad”), the participating clinicians rated
the usefulness of the iPad system (MoPat2) with an average score of 2 (“good”).
Patient User Satisfaction
A total of 495 patients participated in the PruNet study and 468 (94.5%) completed
the user satisfaction questionnaire. Participants completed the survey including the
satisfaction questionnaire in 18 minutes and 2 seconds in average with a median value
of 9 minutes and 20 seconds.
The age distribution by survey language selected in MoPat2 is depicted in [Fig. 2]. The participants who completed the survey in Turkish are, on average, slightly
younger (45 years) than the general patient population (52 years), whereas Italian-speaking
participants were found to be slightly older (58 years).
Fig. 2 Distribution of population by language and age: tr_TR (Turkish-Turkey), fr_FR (French-France),
pl_PL (Polish-Poland), de_DE (German-Germany), es_ES (Spanish-Spain) and it_IT (Italian-Italy).
One patient in Spain preferred to complete the survey in English (en_GB). The dashed
line represents the inclusion criterion of a minimum age of 18 years. The solid blue
lines represent the median and quartiles of their collective ages. The dashed line
represents the inclusion criterion of a minimum age of 18 years. The solid blue lines
represent the median and quartiles of their collective ages.
The results of the patient user satisfaction questionnaire are shown in [Table 1].
Table 1
Patient responses to the user satisfaction questionnaire
|
Question
|
True
|
False
|
NA
|
K* Gender[a]
|
K* Language[a]
|
|
1. Do you feel you have been given sufficient information about the purpose of the
iPad survey?
|
452 (96.6%)
|
14 (3%)
|
2 (0.4%)
|
0.00
|
0.31
|
|
2. Did you get on well using the iPad to complete these questionnaires?[b]
|
439 (93.9%)
|
27 (5.8%)
|
2 (0.3%)
|
0.04
|
0.33
|
|
2a. I generally don't get on well with iPads/tablets[c]
|
20 (74%)
|
7 (26%)
|
–
|
0.34
|
0.27
|
|
2a. The display/the font is too small[c]
|
4 (15%)
|
23 (85%)
|
–
|
0.16
|
0.31
|
|
2a. I had problems selecting the answer/number on the display and had to make frequent
corrections[c]
|
7 (26%)
|
20 (74%)
|
–
|
0.15
|
0.50
|
|
2a. I find it hard to enter data into the iPad because of my manual/visual limitations
(e.g., osteoarthritis of the hands, rheumatism, eye disease, or similar)[c]
|
9 (33%)
|
18 (67%)
|
–
|
0.37
|
0.26
|
|
2b. Other (free text)[c]
|
0 (0%)
|
27 (100%)
|
–
|
|
|
|
3. Did you find the individual questions easy to understand?
|
451 (96.4%)
|
14 (3%)
|
3 (0.6%)
|
0.00
|
0.26
|
|
4. Did you find it helpful to have to answer the questions in the order they were
given?
|
437 (93.4%)
|
27 (5.8%)
|
4 (0.8%)
|
0.17
|
0.17
|
|
5. Were there any questions where you had difficulty in choosing an answer?
|
152 (32.5%)
|
314 (67.1%)
|
2 (0.4%)
|
0.13
|
0.24
|
|
6. Did the survey on the iPad seem too long?
|
74 (15.9%)
|
392 (83.8%)
|
2 (0.3%)
|
0.03
|
0.25
|
|
7. Would you be willing to use the iPad if it was used for routine surveys in an outpatient
clinic?[b]
|
416 (88.9%)
|
50 (10.7%)
|
2 (0.4%)
|
0.01
|
0.27
|
|
7a. Is answering questions on the iPad too much effort for you?[c]
|
21 (42%)
|
29 (58%)
|
–
|
0.42
|
0.82
|
|
7a. Do you have difficulty using an iPad?[c]
|
21 (42%)
|
29 (58%)
|
–
|
0.25
|
0.65
|
|
7b. Other (free text)[c]
|
11 (22%)
|
39 (78%)
|
–
|
0.05
|
0.74
|
|
8. Do you feel that the iPad is a suitable tool for collecting data on itching?
|
420 (89.8%)
|
44 (9.5%)
|
4 (0.9%)
|
0.14
|
0.32
|
|
8a. Why not? (free text)[c]
|
27 (61%)
|
17 (39%)
|
–
|
|
|
|
9. Did anything else come to mind? (free text)
|
130 (27.8%)
|
338 (72.3%)
|
–
|
|
|
a The corrected contingency coefficient K*∈[0,1] in the last two columns is used as a measure of association where higher values
mean higher disassociation.
b For subquestions note that in question 2, there is no “false,” i.e., no data in subquestions
for language tags “en_GB,” “es_ES,” and “pl_PL,” and in question 7, none for “en_GB.”
c The questions 2a, 2b, 7a, 7b, and 8a were shown only when the questions 2, 7, and/or
8 were answered with a “No,” respectively.
Twenty-seven patients expressed a hindered ability to use the iPad to complete the
questionnaire ([Table 1]), being “I generally don't get on well with iPads” the most common reason (20 out
of 27).
Fifty patients reported they would be unwilling to use the survey app (MoPat2) in
an outpatient clinic ([Table 1]). These patients were then asked for the reason why in a multiple-choice format
and free-text subquestion. The effort involved with using an iPad and the difficulties
in using one were the most common reasons given (both with 21 out of 50 responses).
Two of the 50 refused to answer when asked to provide a reason for their skepticism.
Five participants had concerns regarding both the effort needed to answer questions
and the difficulties using an iPad. Another 16 participants specified the effort needed
to answer questions and the same number reported difficulties using an iPad as their
only concern. Eleven selected “other reasons.” An analysis of the free-text responses
was performed using Google Translate for the unknown languages and did not reveal
any issues regarding the MoPat2.
The responses to the patient user satisfaction survey were also correlated with the
language and gender of the patients. The highest disassociations were found on the
correlation with language groups, especially on the answers to the subquestion 7 “Is
answering questions on the iPad too much effort for you?” (K* = 0.84) and “Do you have difficulty using an iPad?” (K* = 0.65) ([Table 1]). Those associations were, in most cases, overrepresentations of participants who
answered the questionnaires in Turkish (and in a few cases those speaking Italian).
The results for question 7 were also compared with the age of the respondents ([Fig. 3]). A difference of 9 years was observed among the median age of the positive and
negative respondents. The median age of the group not willing to further use MoPat2
was 61 years, while the median age of the remaining participants was 52.
Fig. 3 Responses to the question “Would you be willing to use the iPad if it were used for
routine surveys in an outpatient clinic?” correlated with the age of the respondents.
The form of associations was analyzed via the inspection of (extended) mosaic plots,
but is not detailed in this section as the authors did not consider them significant.
The complete analysis can be found in the [Supplementary Appendix B].
Discussion
Multilingual and multicenter capabilities were implemented for the ePRO MoPat2. The
system was successfully evaluated by 28 clinicians and 468 patients in 8 European
clinical centers and represents the first report on the use and evaluation of an electronic
system for the collection of multilingual PROs.
The majority of the interviewed clinicians (96%) stated that the iPad is a valid tool
for measuring pruritus and reported a high user satisfaction of the survey system
(overall score of 2.0, “good”) despite being unable to perceive the benefits of using
MoPat2 for their clinical practice, as they did not have access to the patient responses
(the data collected was solely used for the validation of the PRO translations). This
might have negatively affected the system's overall score. Among the patients, the
responses were positive when asked if they got on well using the iPad and most of
them would be willing to further use the system on an outpatient clinic. These findings
suggest a high user satisfaction for MoPat2.
The patients unwilling to use MoPat2 for completing surveys in an outpatient care
center are, in average, 9 years older than the ones willing to use it ([Fig. 3]). Other studies on this specific issue found similar results,[16] although trends could be evolving toward more acceptance of mobile-based systems
by the elderly,[17] meaning that a future evaluation of the system could result in a higher MoPat2 acceptance
by the elderly and the gap between the age of the users willing to use the system
could be reduced.
To avoid fatigue and study “drop outs” due to too many questions in the survey[18] as well as reducing to the minimum the completion time of the survey before the
clinical examination, the number of questions in the survey was limited to the PruNet
study-oriented PROs and a single user satisfaction questionnaire. The use of standard
tools such as the System Usability Scale[19] could enhance the scalability of the study, but would provide less specific information
about the system and the iPad for the collection of PROs.
When an ePRO is needed, it must be decided whether a new one has to be implemented,
as there are already several systems available and some offer functions similar to
those of MoPat2.[20]
[21]
[22]
[23] LimeSurvey[24] offers an extensive spectrum of possibilities for multilingual online questionnaires
but does not include EHR linkage capabilities and data protection may be an issue.
REDCap is one of the best-known data capture systems, but is mainly clinical research-oriented
as it does not include a connection with the EHR.[25] Other ePROs such as PatientViewpoint and Patient Care View are able to export the
data to an EHR, but in a single format and language.[26] The Computer-based Health Evaluation Software[27] is a multilingual ePRO that can be used for data collection in multicenter clinical
studies. The two most important mobile device operative systems also have their own
initiatives to support the electronic collection of PROs: ResearchKit (Apple)[28] and ResearchStack (Android)[29] provide a framework for the easy creation of ePROs. Other initiatives such as C3-PRO
try to establish a framework for integration of ePROs.[30] However, none of these ePROs has the functional capabilities and versatility of
MoPat2, especially with regard to its multilingualism and interoperability.
Since the use of MoPat2 for data collection within the PruNet, its development and
use has continued and its general GUI has been translated into the following languages
(apart from the seven previously reported): Arabic, Dari, Farsi, Hindi, Kurdish, Dutch,
Norwegian, Portuguese, Russian, Albanese, and Swedish.
MoPat2 is currently only usable at the clinical site. Remote capabilities should be
implemented so that MoPat2 can be used by patients outside of a clinical setting.
A future evaluation of these features should include a standard measurement tool for
results scalability, as well as possible differences when using the system at home
or at the clinical setting.
Conclusions
The successful implementation of multilingual functions for an ePRO and its use in
a multicountry setting for the collection of anonymous data for a clinical study have
been demonstrated. MoPat2 can now be used for the collection of clinical routine and
study data in an international environment. The user satisfaction evaluation of the
system reveals a general acceptance of the system by both clinicians and patients
from several countries. Future work should include the possibility to use MoPat2 outside
of a clinical setting.
