J Wrist Surg 2019; 08(05): 388-394
DOI: 10.1055/s-0039-1688948
Scientific Article
Thieme Medical Publishers 333 Seventh Avenue, New York, NY 10001, USA.

Face Validity and Content Validity of a Game for Distal Radius Fracture Rehabilitation

1  Department of Surgery, Academic Medical Center, Amsterdam UMC, AMC, University of Amsterdam, Amsterdam, the Netherlands
,
2  Department of Surgery, Amsterdam UMC, AMC, University of Amsterdam, Amsterdam, the Netherlands
,
3  Department of Plastic, Reconstructive and Hand Surgery, Amsterdam UMC, AMC, University of Amsterdam, Amsterdam, the Netherlands
,
4  Department of Surgery, Onze Lieve Vrouwe Gasthuis, Amsterdam, the Netherlands
,
5  Department of Surgery, Amsterdam Gastroenterology and Metabolism Research Institute, Amsterdam UMC, AMC, University of Amsterdam, Amsterdam, the Netherlands
› Author Affiliations
Further Information

Address for correspondence

Marlies P. Schijven, MD, PhD, MHSc
Department of Surgery, Amsterdam Gastroenterology and Metabolism Research Institute
Amsterdam UMC, Location AMC
University of Amsterdam   

Publication History

17 November 2018

09 April 2019

Publication Date:
28 May 2019 (online)

 

Abstract

Background Patients recovering from a variety of wrist injuries are frequently advised to exercise to regain lost wrist and hand function. Treatment regimens to regain motion in the wrist are highly variable, and adherence to exercise protocols is known to be low. A serious game ReValidate! incorporating standardized exercise regimens was developed to motivate patients. In this study, the game is evaluated regarding its face validity and content validity.

Methods In this cross-sectional study, a mixed group of “users” (n = 53) including patients currently recovering from wrist injury, and a mixed group of “experts” (n = 46) including professionals advising patients on therapy regimen after wrist injury played at least one complete level of the serious game. Players evaluated the game by means of a structured questionnaire regarding its content, clinical applicability, and user experience. Questions were answered on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree).

Results All groups valued the game as being able to support wrist rehabilitation and being of use to patients recovering from a distal radius fracture (users: median 4, P25–P75 3–4 vs. experts: median 4, P25–P75 3.50–5; p = not significant). The types of exercises performed during the game were considered to be both realistic and complete compared with regular physiotherapy exercises (users: median 4, P25–P75 3–4 vs. experts: median 4, P25–P75 3–5, p = not significant).

Conclusions The ReValidate! serious game can be regarded as a valid tool for patients to regain their wrist function after injury.

Level of evidence This is a Level II study.


#

Distal radius fractures are associated with a considerable impact on health care–related and socioeconomic costs and show an increasing incidence worldwide.[1] [2] [3] This worldwide incidence is estimated to be around 50 patients per 10,000 person-years.[4] [5] Of the total health care–related and socioeconomic costs for hand and wrist injuries, including distal radius fractures, 75% is caused by loss of productivity during the recovery period.[5] [6] [7] [8]

To accelerate rehabilitation after distal radius fractures and improve functional outcomes, patients are either referred for physiotherapy or prescribed self-monitored home-based exercise programs.[7] Treatment guidelines for distal radius fractures state that this choice is up to the surgeon's and patient's individual preferences,[9] leading to arbitrary rehabilitation regimens differing by region, hospital, and even individual surgeon. Furthermore, it is estimated that only 35% of patients fully execute physiotherapy regimens as prescribed.[10] This is largely influenced by practical constraints such as time, costs, and travel distances, as well as difficulty in remembering exercise protocols.[11] This problem is currently underappreciated and inadequately researched.

Serious games and wearable technology may be regarded as promising interventions for home-exercise programs.[12] [13] Serious games are defined as fun and challenging digital applications that have the specific purpose of teaching their users practical skills or knowledge.[14] Their application in rehabilitation from chronic neurologic and musculoskeletal conditions has been widely researched.[15] [16] [17] [18] [19] Recent systematic reviews conclude that serious games are potentially effective with regard to improving function and monitoring outcome, although good quality validity research is often lacking.[13] [18] [19] Moreover, the application of recently developed wearable motion sensors (wearable technology) could improve real-time monitoring of patients in home-based rehabilitation settings.[12] [13] Despite this hypothesized effect of “wearable-controlled” serious games, reports remain sparse.[13]

The serious game ReValidate! (MotekForceLink and Virtual Play, Amsterdam, the Netherlands) has been developed to improve home-based exercise programs following distal radius fractures.

ReValidate! is played on a smartphone or tablet computer, which is likely to decrease the threshold of exercise. Before being adopted as a medical rehabilitation platform, it is vital to research the validity of the game. Face validity and content validity are the first steps in a validation program.[20] These steps are essential, as users and health care providers are unlikely to adopt a new treatment tool if it is unclear whether its usefulness is perceived satisfactory (face validity) and its content is correct (content validity).[21] The aim of this study is to test the beliefs of potential users and medical professionals to determine face validity and content validity of the ReValidate! serious game for wrist rehabilitation.

Methods

Participants

A total of 99 participants were included and analyzed in two groups: the user group including patients and their relatives as volunteers, and the expert group consisting of medical professionals who potentially prescribe treatment regimens to patients recovering from wrist fractures. Consecutive patients and their relatives who visited the outpatient Department of Trauma Surgery or the outpatient Department of Plastic, Reconstructive and Hand Surgery of the Academic Medical Center, Amsterdam, the Netherlands, were asked to participate. Patients who were currently suffering or had previously suffered from any type of wrist injury were approached by their surgeon. Medical professionals working at the outpatient departments of the Academic Medical Center or at the Hand and Wrist Physiotherapy Center in Amsterdam were approached by a researcher to participate in the study. After providing oral informed consent, all participants were instructed to play ReValidate! for 15 minutes, completing at least one level of the game. Participants then filled out a structured questionnaire. The study protocol was approved by the local medical ethical committee (registration number MEC-AMC 17.015).


#

Serious Game

The ReValidate! game (developed by MotekForceLink and Virtual Play, Amsterdam, the Netherlands) is built to be played on a mobile device (iPhone or iPad, Apple Inc., Cupertino, CA) and is controlled by the Myo gesture control armband (Thalmic Labs, Kitchener, Ontario, Canada) or two Valedo sensors (Hocoma AG, Zurich, Switzerland), respectively ([Fig. 1]). By placing one sensor around the hand and the second sensor or mobile device around the proximal forearm, the game can be controlled by isolated wrist movements (please see the video available at https://youtu.be/2YDpdC0HKH8 [The game can be played on various mobile platforms and incorporates a personal rehabilitation program. Patients are stimulated to keep playing the game by challenging levels of increasing difficulty, collectibles, and bonus points]).

Zoom Image
Fig. 1 (A) The game is played on a smartphone and controlled by the Myo gesture control armband (Thalmic Labs, Kitchener, Ontario, Canada). (B) Two Valedo sensors (Hocoma AG, Zurich, Switzerland) are placed around the hand and arm to control the game played on a tablet computer.

One level is performed three to five times per day. Within each level, three avatars must complete an underwater course, all having their own unique movement that coincides with the human wrist motion ([Fig. 2]). The game provides visual support on how to execute wrist exercises correctly and simultaneously provides a fun and challenging exercise environment. It consists of six phases of increasing difficulty, each split into seven different levels. The program is directed on the patient practicing one level per day for a total of 6 weeks. Before a level starts, the player performs a guided “warm-up” session ([Fig. 3]). The game measures the player's function and provides a personalized target function per day (e.g., his/her personal maximum range of motion). This ensures that the game stays challenging while remaining playable for all patients and provides a safety measure to prevent too much strain on the recovering wrist. The levels need to be completed within a time cap to obtain a maximum score, and bonus points can be obtained by picking up collectibles along the course.

Zoom Image
Fig. 2 Each wrist movement has its own level and avatar in the game. (A) the angler fish is controlled by pronation and supination. (B) The shark is controlled by flexion and extension. (C) The penguin is controlled by radial and ulnar deviation.
Zoom Image
Fig. 3 A guided warm-up session is performed before a level starts. The dark blue hand on the left shows the movement to be made by the light-blue hand, representing the motions as made by the patient. (A) Pronation and supination. (B) Flexion and extension. (C) Radial deviation and ulnar deviation.

#

Questionnaire and Statistical Analysis

After completing the game, all participants filled out a structured questionnaire. The questionnaire contained seven items on demographic characteristics. Statements on perceived usefulness of the game, its content, user experience, and applicability in real life were used to determine participants' beliefs about the game. Statements were scored on a 5-point Likert scale, with 1 representing “completely disagree,” 3 representing “neutral,” and 5 representing “completely agree.” The usefulness of the game was evaluated by six statements on the perceived value, the level of support, and whether the game could be a possible replacement of physiotherapy. User experience was evaluated in nine statements on the perceived amusement, motivation during play, and feelings of fun or boredom. Applicability was evaluated in eight statements, determining the expected frequency of playing the game and whether participants would recommend the game to patients.

The content of the game was compared with regular physiotherapy exercises in six statements. Participants in the user group who were unfamiliar with wrist physiotherapy were instructed to answer the questions according to their beliefs and expectations about participating in such an exercise program.

Four open questions were used to evaluate points for improvement, exercises to be added to or left out of the game, and other comments. Differences between user groups were assessed using nonparametric tests, in which p-values of < 0.05 were considered statistically significant.


#
#

Results

Population

The user group (n = 53) consisted of 39 patients, currently or previously suffering from wrist injuries, as well as 14 healthy volunteers. The expert group (n = 46) consisted of nine trauma surgeons, 10 surgical residents, 10 physiotherapists, and 17 other medical personnel ([Table 1]).

Table 1

Demographics

Users

Experts

Group size

53

46

Age

Mean (range)

48 (18–79)

34 (22–65)

Sex

M

18 (33.9%)

22 (47.8%)

F

35 (66.1%)

24 (52.2%)

Function

Patients with a wrist injury (%)

39 (73.5%)

N/A

 Fracture (%)

18 (33.9%)

 Other wrist injury (%)

21 (39.6%)

Healthy volunteers (%)

14 (26.4%)

Function

Trauma/hand/wrist surgeon (%)

N/A

9 (19.6%)

Surgical resident (%)

10 (21.7%)

Physiotherapist (%)

10 (21.7%)

Other (%)

17 (37%)

Videogaming frequency

(More than) weekly (%)

8 (14.2%)

12 (26%)

Monthly (%)

19 (35.8%)

12 (26%)

Never (%)

26 (49.1%)

22 (47.8%)

Abbreviations: F, female; M, male; N/A, not applicable.



#

Perceived Usefulness

The majority of the statements concerning perceived usefulness were answered in a positive manner, with a median of >4 (agree). Both users and experts were neutral about the game being a possible replacement for physiotherapy (median = 3, “neutral”). There were no statistically significant differences between the groups ([Table 2]).

Table 2

Perceived usefulness of the game as a treatment tool

Perceived usefulness

Potential users

Medical experts

p-Value[a]

Median

P25

P75

Median

P25

P72

Valuable exercise

4.00

4.00

5.00

4.00

4.00

5.00

NS

Useful for me

4.00

3.00

4.00

4.00

3.00

4.00

NS

Useful for all patients

4.00

4.00

4.00

4.00

3.00

5.00

NS

Supportive for wrist rehabilitation

4.00

3.00

4.00

4.00

3.50

5.00

NS

Valuable addition to treatment

4.00

3.25

4.00

4.00

4.00

5.00

NS

Possible replacement of physiotherapy

3.00

2.00

3.00

3.00

2.00

4.00

NS

a Mann–Whitney U test.



#

Content

Both groups regarded the exercises in the game to be similar to regular physiotherapy. They considered the game to offer a complete training program and a correct number of exercises ([Table 3]). Both the users and the experts believed the game did not contain ineffective exercises overall, yet the expert group felt more strongly about this statement (median = 2, P25–P75 = 1–2) than the user group (median = 2, P25–P75 = 2–2, respectively; p = 0.007).

Table 3

Content of the game as exercise treatment

Content

Potential users

Medical experts

p-Value[a]

Median

P25

P75

Median

P25

P72

Game matches physiotherapy exercises

4.00

3.00

4.00

4.00

3.00

5.00

NS

Game exercises follow the protocol

3.00

2.50

4.00

3.00

3.00

4.00

NS

Sufficient number of exercises

4.00

3.00

4.00

4.00

3.00

4.00

NS

Too many exercises in the game

2.00

2.00

2.00

2.00

2.00

2.00

NS

Too few exercises in the game

2.00

2.00

3.00

2.00

2.00

3.00

NS

Ineffective exercises in the game

2.00

2.00

2.00

2.00

1.00

2.00

0.007

Abbreviation: NS, nonsignificant.


a Mann–Whitney U test.



#

User Experience

Both groups considered the game to be fun, challenging, and motivating ([Table 4]). Neither group found the game frustrating, childish, or boring. Except for a difference between the groups in finding the game too simple, no statistically significant differences were found. The expert group was “neutral” when answering the question if the game was too simple yet showed a larger variation (P25–P75 = 2–4) compared with the user group (median = 3, P25–P75 = 2–3; p = 0.021; [Table 4]).

Table 4

User experience

User experience

Potential users

Medical experts

p-Value[a]

Median

P25

P75

Median

P25

P72

Fun

4.00

4.00

5.00

4.00

4.00

4.00

NS

Amusing

4.00

3.00

4.00

4.00

4.00

4.00

NS

Frustrating

2.00

2.00

2.00

2.00

2.00

2.00

NS

Challenging

4.00

3.00

4.00

3.00

3.00

4.00

NS

Childish

2.00

2.00

3.00

2.00

2.00

3.00

NS

Boring

2.00

2.00

3.00

2.00

2.00

3.00

NS

Too simple

3.00

2.00

3.00

3.00

2.00

4.00

0.021

Addictive

3.00

2.00

3.00

2.00

2.00

3.00

NS

Motivating

4.00

4.00

4.00

4.00

4.00

5.00

NS

Abbreviation: NS, nonsignificant.


a Mann–Whitney U test.



#

Applicability

Both groups agreed with the statements on whether they would enjoy playing the game frequently and enjoy exercising with the game ([Table 5]). Both groups considered the game to be suitable for all ages. The user group less strongly believed they would recommend the game to others (median 4, P25–P75 3–4 vs. 4, P25–P75 4–5; p = 0.001), or that they would exercise more with the game than with physiotherapy (median = 3.50, P25–P75 3–4 vs. 4, P25–P75 3–5; p = 0.018). The users neither agreed nor disagreed on whether they would only play the game if this was obligatory (median = 3, P25–P75 2.50–4), whereas the experts disagreed with this statement (median = 2, P25–P75 2–3; p = 0.000; [Table 5]).

Table 5

Applicability

Applicability

Potential users

Medical experts

p-Value[a]

Median

P25

P75

Median

P25

P72

Enjoy playing the game more often

3.50

3.00

4.00

4.00

3.00

4.00

NS

Enjoy exercising with the game

4.00

3.00

4.00

4.00

3.00

4.00

NS

Exercise more with the game than physiotherapy

3.50

3.00

4.00

4.00

3.00

5.00

0.018

Would only play if obligatory

3.00

2.50

4.00

2.00

2.00

3.00

0.000

Recommend others to play the game

4.00

3.00

4.00

4.00

4.00

5.00

0.001

Fit for adults

4.00

4.00

4.00

4.00

4.00

5.00

NS

Fit for children

4.00

4.00

5.00

5.00

4.00

5.00

NS

Fit for the elderly

4.00

3.00

4.00

3.00

3.00

4.00

NS

Abbreviation: NS, nonsignificant.


a Mann–Whitney U test.



#
#

Discussion

The worldwide market for medical serious games is rapidly expanding.[22] Serious games have already shown their advantages in health education, motivation, and potentially increasing treatment adherence.[23] [24] [25] Innovative solutions for the empowerment of patients must be welcomed if they can reduce the health care–related and socioeconomic burden of distal radius fractures. This study shows that ReValidate! has the potential to become a valid treatment tool in rehabilitation after distal radius fractures. Both the user and expert groups found the game to be a valuable support tool for wrist rehabilitation, providing a fun and challenging method of performing wrist exercises.

Beneficial effects of serious games in neurorehabilitation and musculoskeletal rehabilitation have been reported in systematic reviews.[13] [18] [19] Although studies indicate that serious games may be promising tools in rehabilitation, it is worrisome that validity tests of most games are not reported.[13] It must be ensured that important theoretical constructs such as face validity and construct validity of games have been tested and found to be well embedded in games before they are used as a prescribed treatment tool. When validation guidelines are followed, no opportunities are missed with regard to incorporating user insights during the design process of medical serious games. The framework developed by Graafland et al[20] describes a stepwise validation model specifically built for use in designing serious games to be used in health care.

Most games directed at rehabilitation use consumer-based, stationary game consoles such as the Nintendo Wii.[15] [26] [27] [28] ReValidate! uses the smartphone as a platform, which has an unprecedented penetration worldwide and is easily accessed by users. Serious games developed as a smartphone application can easily be downloaded from app stores. This enables patients to exercise whenever and wherever they deem fit.

In the current era, shared decision-making and making patients the owner of their own health care data are major topics.[29] [30] Patients want to be involved in decision-making and want to be in control of whether they contact their health care provider, and if so, when and how they do this. ReValidate! combines serious gaming in mobile application format with consumer-based, readily available wearable technology as game controller. This provides patients with an easily accessible rehabilitation modality, enabling them to receive personalized feedback from the game and remote monitoring without the need for extra visits to the hospital or physiotherapist.[12] [13]

Also, in the aging population, in which the number of distal radius fractures—and their subsequent costs—are on the rise, these considerations are widespread.[31] [32] [33] [34] Moreover, the elderly population is adopting personal mobile technology.[35] Serious games designed for mobile platforms may thus provide a powerful tool in facilitating personalized rehabilitation in this particular and often homebound population. Next, it may be of equal value for the younger population, facilitating an earlier return to the workplace. Future research should include a cost-effectiveness analysis to prove this value.

Limitations

While the patients included in the user group had experienced a previous or current recovery from a wrist injury, not everyone had received physiotherapy guidance during their rehabilitation. The participants in the user group who had never experienced a wrist injury lacked any experience on this topic. This may influence the answers given by these participants regarding the content validity of the game and the potential value of the game in a standardized wrist rehabilitation protocol. Though the lack of users' experience with standardized wrist rehabilitation could introduce bias, it is the experts' opinions that are needed to show content validity. Furthermore, all participants were told the game was designed specifically to be of use in wrist fracture rehabilitation, whereas not all participants recovering from a wrist injury had necessarily suffered a fracture.

Next, bias could have been introduced by the voluntary nature of the study. Asking people to participate willingly may have led to a selection bias. The Pygmalion effect, being the positive influence on the questionnaire's outcomes caused by the enthusiasm of the researcher, could have also been a source of bias. To minimize this effect, the researcher was not affiliated to the game developer, and questionnaires were filled out anonymously.

To minimize bias, different definitions for the same construct were given in the questionnaire (e.g., “the game is useful for me” and “the game is a valuable exercise for my wrist”), as well as conflicting statements (e.g., “the game is fun” vs. “the game is boring”). Although this type of bias remains difficult to eradicate, outcomes show that similar constructs were evaluated similarly, whereas conflicting constructs were evaluated oppositely.


#
#

Conclusions

Many e-health innovations do not follow a stepwise approach in development and validation nor do they include actual user groups during this process. One may risk flaws in design as well as maladaptation in health care if validation trajectories, including user group assessments, are not well evaluated and embedded throughout innovation development. A first step in developing a serious game for rehabilitation is to scrutinize the content on actual believability for the proposed user as well as for the proposed prescriber of the innovation.

This trial shows that users and medical experts believe this serious game to be a useful and fun experience in wrist fracture rehabilitation. Although this game is fun, easy-to-use, portable, patient-tailored in gameplay while providing patient-specific functional outcome measurements, it has the potential to be a very useful exercise tool in future rehabilitation. Future research will focus on whether ReValidate! improves clinical outcomes compared with standard rehabilitation protocols.


#
#

Conflict of Interest

None declared.

Note

This research was performed at the Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands.



Address for correspondence

Marlies P. Schijven, MD, PhD, MHSc
Department of Surgery, Amsterdam Gastroenterology and Metabolism Research Institute
Amsterdam UMC, Location AMC
University of Amsterdam   


Zoom Image
Fig. 1 (A) The game is played on a smartphone and controlled by the Myo gesture control armband (Thalmic Labs, Kitchener, Ontario, Canada). (B) Two Valedo sensors (Hocoma AG, Zurich, Switzerland) are placed around the hand and arm to control the game played on a tablet computer.
Zoom Image
Fig. 2 Each wrist movement has its own level and avatar in the game. (A) the angler fish is controlled by pronation and supination. (B) The shark is controlled by flexion and extension. (C) The penguin is controlled by radial and ulnar deviation.
Zoom Image
Fig. 3 A guided warm-up session is performed before a level starts. The dark blue hand on the left shows the movement to be made by the light-blue hand, representing the motions as made by the patient. (A) Pronation and supination. (B) Flexion and extension. (C) Radial deviation and ulnar deviation.