CC BY-NC-ND 4.0 · Indian J Radiol Imaging 2023; 33(01): 036-045
DOI: 10.1055/s-0042-1758198
Original Article

Mobile Technology in Medicine: Development and Validation of an Adapted System Usability Scale (SUS) Questionnaire and Modified Technology Acceptance Model (TAM) to Evaluate User Experience and Acceptability of a Mobile Application in MRI Safety Screening

1   Faculty of Medicine Universiti Sultan Zainal Abidin, Kuala Terengganu, Terengganu, Malaysia
,
1   Faculty of Medicine Universiti Sultan Zainal Abidin, Kuala Terengganu, Terengganu, Malaysia
,
Myat Moe Thwe Aung
1   Faculty of Medicine Universiti Sultan Zainal Abidin, Kuala Terengganu, Terengganu, Malaysia
,
Azizah Ab Ghani
1   Faculty of Medicine Universiti Sultan Zainal Abidin, Kuala Terengganu, Terengganu, Malaysia
,
Husbani Mohd Amin Rebuan
1   Faculty of Medicine Universiti Sultan Zainal Abidin, Kuala Terengganu, Terengganu, Malaysia
› Author Affiliations
Funding This study was funded by Universiti Sultan Zainal Abidin for Graduate On Time (GOT) grant scheme with project code: UniSZA/2019/GOT/02.
 

Abstract

Background Magnetic resonance imaging (MRI) safety screening is a crucial procedure for patient preparation before entering into MRI room. Many hospitals in Malaysia are still using the MRI safety checklist printed form. Besides, clinicians will not get a definite conclusion about whether the patient is contraindicated for MRI or not. Hence, we have created one mobile application named MagnetoSafe to overcome this issue. The application will provide an instant decision on whether the patient has no contraindication, relative contraindication, or absolute contraindicated for MRI. We need to check for acceptability and user experience for any newly created mobile application.

Objective This study was designed to check the validity of the adapted Technology Acceptance Model (TAM) and System Usability Scale (SUS) Questionnaire.

Method The validity and reliability of the questionnaire were investigated. Subsequently, 52 fully completed responses were collected.

Results Face and content validity of the questionnaires are considered acceptable with only minor changes to Item 10 of SUS. The Cronbach's alpha for the SUS questionnaire (10 questions) is −0.49, which is not acceptable. The Cronbach's alpha for TAM questionnaire (3 domains; 14 questions) is acceptable, which is 0.910 for perceived usefulness, 0.843 for perceived ease of use, and 0.915 for intention to use.

Conclusion Face validity of the adapted SUS and modified TAM questionnaires is acceptable with only minor changes to Item 10 in SUS. Content validity with experts is good. However, the reliability of the SUS questionnaire is not acceptable and therefore adapted SUS will not be used for assessing user experience. The reliability of the modified TAM questionnaire with the original three-factor structure is considered acceptable and can be used to evaluate the user's acceptability of MagnetoSafe.


#

Introduction

In clinical practice, magnetic resonance imaging (MRI) safety screening is a crucial procedure for patient preparation before entering into MRI room. Most of the hospitals in Malaysia are still using the MRI safety checklist printed form. This method is not environment friendly as many papers have to be used. Clinicians need to interview the patient and ask for any MRI contraindication. Most of our MRI screening forms do not have a complete list of MRI contraindications. Besides, after filling out the form, clinicians will not get a definite conclusion whether the patient is contraindicated for MRI or not. Hence, with the advancement of mobile technology and application, one mobile application named MagnetoSafe was created. Inside this application, the complete list of ferromagnetic objects was integrated into it, and at the end, the application will provide an instant decision regarding MRI safety.[1] Clinicians can know whether the patient is contraindicated or not before booking an appointment for MRI scanning. Therefore, it can be used for the early detection of MRI unsafe situations and to avoid delay in patient care by finding an alternative imaging modality for the patient's clinical condition in such situations.

The acceptability and usability of this application need to be tested. Based on the literature review, System Usability Scale (SUS) questionnaire is an excellent tool for the broad measurement of the user experience of a newly developed system.[2] This questionnaire consists of 10 questions that are easy to administer and can be used on a small sample size with reliable results.[2] [3] For this study, we adapted the SUS questionnaire by changing some wordings. Moreover, the perceived usability, ease of use, and intention of use of the application also need to be investigated. Therefore, Technology Acceptance Model (TAM) questionnaire was used as it encompasses these components. TAM questionnaire is a widely used tool to help understand and explain user behavior in a technology system.[4] [5] Many researchers have used this questionnaire to test new technology, and the results are reliable.[6] In this study, we have done minor modifications to TAM questionnaires.


#

Literature Review

Adapted Technology Acceptance Model

TAM is an information system theory, a robust and precise model to understand information technology usage. In 1989, Fred Davis developed and published this model in his dissertation.[7] Because of the growing demand for technology at that time, this model spread to one of the most cited models in the context of technology diffusion. Many models predict the diffusion of a system. However, the TAM is the only model that emphasizes information system.[5] According to F. Davis, there are two important goals in this model,[6] to understand the technology acceptance processes and to increase the theoretical insights for the design, which provide a basis for user acceptance testing methodology and evaluate the proposed system before their actual implementation. The acceptance of technological innovation is derived from one's attitude towards using the system, which, in turn, is determined by two beliefs, that is, perceived usefulness (PU) and perceived ease of use (PEU). The measurement scales for PU and PEU in the original TAM model are shown in [Table 1]. PU is defined as a “prospective user's subjective probability that using a specific application system will increase his or her job performance within an organisational context.” The questions on PU are shown as [Table 2]. PEU is the “degree to which the prospective user expects the target system to be free of effort.” The questions on PEU are shown in [Table 3]. In addition to this, F. Davis also states that the system usage is determined by behavioral intention to use (BI). BI is determined by the “person's attitude towards using a system (A) and perceived usefulness (U).”[6] A meta-analysis study further proved these changes, which showed that behavioral intention to use (IU) the system was found to be affected directly by the user's perception of usefulness and ease of use.[8] By definition, behavioral IU measures how likely a user employs the system.[9] Kim et al created the questions to assess the behavioral intention[10] ([Table 4]). The key concept framework of the original TAM is shown in [Fig. 1]. In summary, multiple modifications and adjustments have been made to the TAM model and the questionnaire. In 1996, the final version of TAM was found in a paper by Venkatesh and Davis 1996[11] [12] ([Fig. 2]).

Zoom Image
Fig. 1 Original Technology Acceptance Model proposed by F. Davis.[7]
Zoom Image
Fig. 2 Final version of Technology Acceptance Model.[12]
Table 1

Measurement scales for PU and PEU in the original Technology Acceptance Model (TAM) model by F. Davis[7] [13]

Perceived usefulness (PU)

Perceived ease of use (PEU)

Innovation technology

-Work more quickly

-Job performance

-Increase productivity

-Effectiveness

-Makes job easier

-Useful

-Easy to learn

-Clear and understandable

-Easy to become skillful

-easy to use

-Controllable

-Flexible

Lab experiment with email and graphics

Table 2

Six items scale for perceived usefulness worded toward CHART-MASTER[7]

Item no.

Candidate item for psychometric measures for perceived usefulness

1

Using CHART-MASTER in my job would enable me to accomplish tasks more quickly

2

Using CHART-MASTER would improve my job performance

3

Using CHART-MASTER in my job would increase my productivity

4

Using CHART-MASTER would enhance my effectiveness on the job

5

Using CHART-MASTER would make it easier to do my job

6

I would find CHART-MASTER useful in my job

Table 3

Six items scale for perceived ease of use worded towards CHART-MASTER[7]

Item no.

Candidate item for psychometric measures for perceived ease of use

1

Learning to operate CHART-MASTER would be easy for me

2

I would find it easy to get CHART-MASTER to do what I want it to do

3

My interaction with CHART-MASTER would be clear and understandable

4

I would find CHART-MASTER to be flexible to interact with

5

It would be easy for me to become skillful at using CHART-MASTER. Likely

6

I would find CHART-MASTER easy to use

Table 4

Behavioral intention by Kim et al[10]

Item no.

Candidate item for psychometric measures for behavioral intention to use

1.

I will consider using this service

2.

I plan to use this service.

3

I will continue to use this service.

4.

I will inform other of the goodness of this service

In this study, we used a modified version of TAM questionnaire, which consists of PU and PEU as the independent variables and IU as the sole dependent variable.

We also incorporated the adapted theoretical framework by Pande et al,[14] which assesses both the usability and acceptability component by combining SUS and TAM questionnaires. This is because user experience from SUS questionnaire did play an important role in determining the system's acceptability, which subsequently affects the behavioral IU ([Fig. 3]).

Zoom Image
Fig. 3 An adapted framework from Pande et al comprising System Usability Scale and Technology Acceptance Model.[14]

#

System Usability Scale Questionnaire

The SUS questionnaire was originally created by John Brooke at Digital Equipment Corporation in United Kingdom in 1986.[3] This questionnaire measures three important aspects. The first aspect of the effectiveness of using this technology measures users' ability to achieve their objectives. The second aspect is efficiency, in which how much user's effort and user's resources are expended in achieving those objectives. The third aspect is satisfaction, whereby how satisfactory the user's experience is.

This questionnaire allows us to evaluate various products and services, including hardware, software, mobile devices, websites, and applications. With the advent of technology, SUS has become a widely used industry standard, with references in over 1300 articles and publications.[15] There are a few benefits of using this questionnaire; (i) it is a very easy scale to administer to participants, (ii) it can be used on small sample sizes, and (iii) it can effectively differentiate between usable and unusable systems.[2] [3] It consists of 10 items with five response options for respondents, from Strongly agree to Strongly disagree[3] ([Fig. 4]). There are two tones in this questionnaire whereby the odd-numbered questions are all considered positive tone, and even-numbered questions are all considered negative tone. Based on the Likert scale, if participants choose “Strongly Disagree,” a minimum of 1 point will be given. Whereas if participants choose “Strongly Agree,” a maximum of 5 points will be given.[10]

Zoom Image
Fig. 4 Original question of System Usability Scale digital equipment corporation, 1986.[3]

After that, we calculate the SUS score by using the formula SUS score = (X + Y) × 2.5 [ X = Sum of the points for all odd-numbered questions – 5] [ Y = 25–Sum of the points for all even-numbered questions]. This formula will give a SUS acceptability score out of 100.

To elaborate further, 5 marks for “Strongly agree” answer in positive statements (if all are “strongly agree” answers for all 5 positive statements (odd number question), then the total marks will be 25, and X value = 25-5 =20.

However, 1 mark for “Strongly disagree” answer in negative statements (if all are “strongly disagree” answers for all 5 negative statements (even number question), then the total marks will be 5, and Y value = 25-5 = 20.

Based on the above example, SUS acceptability score is (20 + 20) × 2.5 = 100.

According to the articles, a SUS score above a 68 would be considered as good acceptability and a score below 68 is considered below average.[16] [Fig. 5] shows the interpretation of SUS scoring.[17]

Zoom Image
Fig. 5 Standard version of the System Usability Scale and the interpretation.[15]

#
#

Methods

The study was carried out in the Faculty of Medicine, Universiti Sultan Zainal Abidin (UniSZA), Malaysia, with 52 clinicians as respondents. Approval from the Education Ministry of Malaysia and UniSZA Human Research Ethics Committee was taken. The original questionnaires, that is, TAM and SUS Questionnaire were cited. Minor modification and adaptation of the questionnaires were performed. For example, the word “system” in SUS was replaced by “app.” On the other hand, the wordings in TAM questionnaire are unchanged except the “system” is replaced by the “MagnetoSafe apps.” Nevertheless, the scoring system of these two questionnaires remains unchanged.

First, face validity was done with five subjects. After that, participants were selected based on the inclusion criteria in which the participant must have the experience using a manual MRI safety checklist form before. Participants installed the mobile application on their phones and started using it. Five case scenarios that simulated the actual clinical situation were given to them to key in. For those participants without knowledge of using medical applications, an enumerator will explain the mobile user interface to them personally. Finally, the adapted TAM and SUS questionnaires were given to them. Data were collected from May to August 2021. Data from the questionnaires were entered into SPSS v. 22.

Mobile Application

MagnetoSafe is an Android-based mobile application created to complement the MRI screening checklist form. This application can be installed on all Android-based mobile phones. For those without Android-based mobile phones, we will provide an Android-based tablet. Due to budget limitations, we unable to create the software in iOS.

Users were required to key in their username and password on the homepage. Then, they will be navigated to the second page, where they have to key in the patient's information and clinical details. Participant needs to select the suitable item from the list of dropdown menu. They need to scroll until find the item which fit the patient.

Once they have hit the submit button, the mobile application will show the result of whether the patient is safe for an MRI procedure or not. [Fig. 6] demonstrates the mobile user interface. At least five simulated case scenarios were given to the participant ([Fig. 7]).

Zoom Image
Fig. 6 Mobile user interface of MagnetoSafe.
Zoom Image
Fig. 7 Example of case scenario.

#

Questionnaires

The questionnaire used in this study is divided into three sections: the first section on background information, including the demographic data; the second section on the SUS questionnaire component; and the third section on the TAM model questionnaire.

The questionnaire is shown in [Figs. 8] to [10].

Zoom Image
Fig. 8 Questionnaire on background information.
Zoom Image
Fig. 9 Adapted System Usability Scale questionnaire.
Zoom Image
Fig. 10 Modified Technology Acceptance Model questionnaire.

#
#

Result and Discussion

Demographic Data

The demographic characteristics and background information of participants were analyzed. Out of 52 participants, 23 (44.2%) are female and 29 (55.8%) are male. Ten participants (19%) are in the age group of 25 to 34, 18 (34.6%) in the age group of 35 to 44, 17 (32.7%) in the age group of 45 to 54, and 7 (3.5%) in 55 to 64. For educational level, 39 participants (75%) had completed a master's degree, 10 participants (19%) had a bachelor's degree, and 3 participants (6%) had completed a PhD. Background information showed 14 participants (27%) have experience using mobile applications for medical purposes and 38 participants (73%) have no experience. Among the 38 participants who have no experience using a mobile application for medical purposes, the majority are 45 to 54 years old. The experience status of the participants in using the nonmedical or medical mobile application was presented in [Table 5].

Table 5

The experience status of participant in using mobile application (n =52)

Items

Yes

No

Do you have experience in using any mobile application for nonmedical purpose?

47 (90.4%)

5 (9.6%)

Do you have experience in using any mobile application for medical purposes?

14 (26.9%)

38 (73.1%)


#

Validity of the Questionnaire

First of all, the contents of the draft questionnaire were checked by two radiologists and an expert in information technology. Then, the face validity was carried out with five participants comprising two medical officers, one house officer, and two specialists from a tertiary hospital. In item number 10 of the SUS questionnaire, “How do you rate your skill as a technology-savvy person?,” one participant was confused about the word “technology savvy,” and thus the definition of technology-savvy was added below that question. Two subjects had difficulty understanding the word “cumbersome” in SUS point 8, and this word was replaced by “awkward.” This is because we find other SUS questionnaire articles using this word as well.[17] Face validity on the modified TAM questionnaire was unremarkable.

Public health experts were asked to check the contents. Then the first draft of the questionnaire was distributed to four lecturers from UniSZA Medical Faculty to check the face validity of the questionnaire. After reshaping the questionnaire, the pretest study was conducted among 30 office staff and medical lectures from UniSZA Medical Faculty using Google Form to evaluate the construct validity and reliability of the questionnaire. As a result of the pretest study, the items in the questionnaire were revised and edited based on the respondents' comments and the public health experts' opinions


#

Reliability of Adapted Technology Acceptance Model Questionnaire

From the reliability statistics, the Cronbach's alpha for ten items in adapted SUS is −0.049. [Table 6] shows the item-total statistics. According to Tavakol and Dennick, Cronbach's alpha is acceptable if the value is between 0.70 and 0.80, and good if the level is above 0.80.[18] This means that these items are unrelated to each other, and this adapted questionnaire is unreliable. One possible explanation is item number 2, “I found the application unnecessarily complex,” and item number 3, “I thought the application was easy to use,” are related to each other and give the same meaning.

Table 6

SPSS output for item-total statistics of SUS

Scale mean if item deleted

Scale variance if item deleted

Corrected item-total correlation

Cronbach's alpha if item deleted

1. I think that I would like to use the MagnetoSafe application frequently

27.29

8.327

0.255

−0.256[a]

2. I found the application unnecessarily complex

29.69

12.217

−0.372

0.169

3. I thought the application was easy to use

27.21

9.856

−0.002

−0.052[a]

4. I think that I would need the support of a technical person to be able to use this system

28.06

8.291

−0.004

−0.059[a]

5. I found the various functions in this application were well integrated

27.29

9.699

0.086

−0.102[a]

6. I thought there was too much inconsistency in this application

29.63

10.119

−0.046

−0.022[a]

7. I would imagine that most people would learn to use this application very quickly

27.23

9.828

−0.020

–.039[a]

8. I found the application very awkward to use

29.71

10.994

−0.185

0.064

9. I felt very confident using the application

27.21

10.366

−0.068

−0.011[a]

10. I needed to learn a lot of things before I could get going with this application

28.62

7.026

0.179

−0.303[a]

Abbreviation: SUS, System Usability Scale.


a The value is negative due to a negative average covariance among items. This violates reliability model assumptions. You may want to check item codings.



#

Reliability of Adapted Technology Acceptance Model Questionnaire

The adapted TAM questionnaire consists of 14 items, including PEU, PU, and IU domains. The Cronbach's alpha for 14 items is 0.926.

From the reliability statistics, the Cronbach's alpha for seven items in the PU domain is 0.910. This means that if we repeatedly measure the questionnaire response to the same participant, this will give us 91% reliability that is good.

The Cronbach's alpha for PEU is 0.843, whereas for IU, it is 0.915 ([Table 7]). Therefore, the reliability of all the respective domains in the adapted TAM questionnaire is good.

Table 7

Reliability statistics for adapted TAM questionnaire

Domains in TAM questionnaire

No of items

Cronbach's alpha

PU

7

0.910

PEU

4

0.843

IU

3

0.915

Abbreviations: IU, intention to use; PEU, perceived ease of use; PU, perceived usefulness; TAM, Technology Acceptance Model.



#

Reliability on the Combination of Adapted SUS and Adapted TAM Questionnaire

However, if the two sets of questionnaires are combined, the Cronbach's alpha of the 24 items is 0.797, which is acceptable. Despite this result, SUS cannot be used because of negative reliability.


#
#

Conclusion

In summary, the face validity of the adapted SUS and modified TAM questionnaires is acceptable with only minor changes to Item 10 in SUS. Content validity with experts is good. However, the reliability of SUS questionnaire is not acceptable and therefore adapted SUS will not be used for assessing user experience. The reliability of the modified TAM questionnaire with the original three-factor structure is considered acceptable and can be used to assess the user's acceptability of MagnetoSafe.


#
#

Conflict of Interest

None declared.

Acknowledgement

The authors would also like to extend our great appreciation to Faculty of Medicine, Ethics Committee of Universiti Sultan Zainal Abidin, and all research participants.


Address for correspondence

Norhasiza Mat Jusoh, MB, BCh, and BAO
Faculty of Medicine, Universiti Sultan Zainal Abidin
20400, Kuala Terengganu
Malaysia   

Publication History

Article published online:
07 December 2022

© 2022. Indian Radiological Association. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Thieme Medical and Scientific Publishers Pvt. Ltd.
A-12, 2nd Floor, Sector 2, Noida-201301 UP, India


Zoom Image
Fig. 1 Original Technology Acceptance Model proposed by F. Davis.[7]
Zoom Image
Fig. 2 Final version of Technology Acceptance Model.[12]
Zoom Image
Fig. 3 An adapted framework from Pande et al comprising System Usability Scale and Technology Acceptance Model.[14]
Zoom Image
Fig. 4 Original question of System Usability Scale digital equipment corporation, 1986.[3]
Zoom Image
Fig. 5 Standard version of the System Usability Scale and the interpretation.[15]
Zoom Image
Fig. 6 Mobile user interface of MagnetoSafe.
Zoom Image
Fig. 7 Example of case scenario.
Zoom Image
Fig. 8 Questionnaire on background information.
Zoom Image
Fig. 9 Adapted System Usability Scale questionnaire.
Zoom Image
Fig. 10 Modified Technology Acceptance Model questionnaire.