Reliability and Validity of a Mobile Device Application for Use in Sports-Related Concussion Balance Assessment

Keywords

concussion - balance assessment - accelerometry - mobile devices - posture

Concussions continue to be a primary concern for sports medicine clinicians. Sport-related concussions are a major contributor to the number of traumatic brain injuries, accounting for an estimated 1.6 to 3.8 million injuries each year in the United States alone.[1] Due to a complicated underlying pathophysiological process, concussions are subtle and produce a wide array of signs, including cognitive, somatic, and sensorimotor symptoms.[2] The diverse symptomatology makes clinical assessment and management of concussions challenging for sports medicine clinicians.

Consensus statements regarding the evaluation and care of individuals after a suspected concussion repeatedly emphasize the importance of balance assessment during a multicomponent evaluation.[2] [3] Balance assessments allow clinicians to determine a person's ability to integrate somatosensory, visual, and vestibular information to maintain an upright posture. Failure of a person to maintain balance following a concussion may be indicative of sensorimotor alterations.[4] Balance assessment also provides information for estimating prognosis and allowing clinicians to predict the extent of expected recovery.[5]

The most common method for assessing balance in potentially concussed athletes is the Balance Error Scoring System (BESS).[2] The BESS is a free assessment, requires little to no special equipment, and can be done on the sidelines of sporting events. The BESS relies on thoroughly trained observers to determine the number of balance errors a person made during static standing trials.[4] Balance errors determined by the trained observers include stepping out of place, removing one's hands from hips, and so on. Although this approach is cost-effective and portable, the assessment has mixed evidence supporting its use.[6] [7] Specifically, studies have questioned the sensitivity of the BESS,[8] as well as the intra- and interrater reliability of the assessment.[9] The BESS has only been found useful within the initial 48 hours following injury,[10] making long-term tracking of balance recovery difficult when using only the BESS.

The gold standard method for assessing balance in healthy and injured people is to measure changes in body sway during static stances while standing on a force platform.[11] Force platforms are sensitive, reliable, and objective tools to measure balance.[12] Center of pressure (COP) sway variables are calculated from measured ground reaction forces acquired during standing balance. The COP sway variables give information regarding an individual's ability to control their center of mass (COM) and maintain stable balance and are widely used to assess balance in healthy and pathological populations.[13] [14] Although force platform technology provides this precise method for assessing balance, the platforms are costly and require proper installation, maintenance, and skilled interpretation of the collected data. The platforms also have limited portability, making assessment difficult outside of clinics or laboratories. Due to these constraints, force platform balance assessment is often not possible in many clinical and athletic settings. Accelerometers have been evaluated as potential alternatives to force platform measurement, as body-worn accelerometers provide a relatively more affordable and portable method for assessing postural control.[15] [16] Accelerometer technology is promising and is available in most mobile devices, making accelerometer-based balance assessment available to clinicians without the need for extra equipment.

Mobile devices may serve as alternatives for use in objective balance assessment when force platforms and accelerometer systems are not feasible because many mobile devices contain triaxial accelerometers accessible to downloadable applications (apps) created for clinical use. SWAY (Sway Medical LLC, Tulsa, OK) is one such app developed for concussion balance assessment. SWAY works with iOS products (Apple Inc., Cupertino, CA) and uses the acceleration time series collected during static stances to quantify balance. Previous research evaluated the mechanical accuracy of SWAY results of healthy adults recorded during single-leg stance compared with force platform measurement.[17] However, no other stances typically used in clinical settings for assessing individuals with a suspected concussion (i.e., two-foot stance, tandem stance) have not been studied. In addition, the clinical validity and reliability of the SWAY app have not been evaluated, which is essential prior to widespread adoption of this technology in sports medicine environments.

The purpose of this study was to evaluate the ability of SWAY, a mobile device app used to access the device's triaxial accelerometer, to quantify balance in healthy individuals. The first aim of this study was to determine if accelerometry-based measures of balance produced by SWAY are as reliable as force platform obtained COP sway variables under the same testing conditions. The second aim of the study was to assess the validity of the accelerometry-based measures by determining the relation of accelerometry data collected by SWAY and force platform COP sway variables when collected concurrently. We hypothesized SWAY would demonstrate similar test–retest reliability to force platform COP sway variables, and that SWAY scores and force platform COP sway variables would demonstrate good correlation coefficients.[18]

Methods

Participants

Twenty-seven healthy volunteers participated in the study (12 men, 15 women; age: 29.7 ± 10.9 years; height: 170.1 ± 10.5 cm; weight: 72.1 ± 16.6 kg). Individuals were excluded if they reported a known orthopaedic, musculoskeletal, or neurologic injury in the prior 6 months. One participant was unable to complete all balance stances independently and was excluded, bringing the total to 26 participants. All participants reported they had not consumed substances or medications prior to testing that could have affected their ability to maintain stability. All participants provided written informed consent prior to participation in accordance with requirements of the Institutional Review Board for this study.

Materials

SWAY was downloaded and installed on a single mobile device that was used throughout all testing (Apple iPod Touch 5th Gen, iOS Version 7.1, Apple Inc.). SWAY is an FDA-approved app for detecting changes in postural control using the integrated accelerometers of Apple iOS mobile devices. The app instructs users through a series of balance stances, replicating the stances used in the BESS.[6] The SWAY app collects data at 10 Hz during each 10-second test period. SWAY provides a score at the end of each trial, calculated by summing acceleration changes over time (jerk; m/s³) occurring in each 10 second testing period, compiled across three planes of movement, and normalized to a 0 to 100 scale. An AMTI force platform (AMTI 1000, Advanced Mechanical Technology Inc., Watertown, MA) embedded in the floor was used to simultaneously record ground reaction forces at 100 Hz. Force platform COP sway variables were calculated by a custom MATLAB program (MATLAB version R2013b, MathWorks Inc., Natick, MA). The calculated COP variables include sway area (total area enclosed by the edge of the COP path created while standing) representing the outermost limits of movement, root mean square (RMS) distance (RMS of the distance from the mean COP) representing displacement away from mean COP, and mean velocity (the average velocity of the COP).[19]

Procedures

Demographic information was collected prior to balance testing. Participants then performed a series of static balance stances while standing on the force platform and holding the mobile device in an upright position against their chest ([Fig. 1]). Participants remained in each balance stance for 10 seconds and were instructed to maintain a steady balance to the best of their ability. Each test sequence included three stances: feet together, tandem with the dominant foot forward, and a single-leg standing on the dominant foot ([Fig. 1]). Participants repeated this stance sequence four times: twice with eyes open and then twice with eyes closed. All four tests of single-leg stance were, however, completed with eyes open due to frequent postural corrections causing participants to step off the force platform and thus invalidate data collection during the development of the test protocol. At the end of each stance sequence, participants rested in a chair for 1 minute before the next stance sequence was initiated. Testing sessions lasted approximately 15 minutes, including rest breaks.

Fig. 1 (A) Scheme depicting subject holding mobile device during test sequence. (B) Stances used during test sequence.

Data Processing

All force platform data were resampled to 20 Hz and truncated to include only the middle 7 seconds of data to control for any imprecision in simultaneous initiation of data collection between the mobile device and the force platform.

The SWAY app uses a proprietary algorithm to calculate a SWAY score ranging from 0 to 100, with higher scores indicating better balance control. Mechanical validity of the triaxial accelerometers housed in the mobile devices has been described previously.[20] [21] SWAY scores were calculated for each of the 12 balance trials and were used in the analysis. A quality check of these data included omitting force platform and corresponding SWAY data where the subject failed to complete the trial for a balance condition successfully. These included trials where participants stepped off the force platform, hopped to recover loss of balance, or instances of a toe touch by the nonsupporting foot during single-leg stance.

Statistical Analysis

Statistical comparisons were performed using SPSS (Version 22, SPSS Inc., Chicago, IL). Test–retest reliability was assessed using an intraclass correlation coefficient (ICC 3,1) calculation, and the p-value and 95% confidence intervals for each ICC were determined. A Pearson product–moment correlation was used to assess the concurrent validity between SWAY and COP variables. The p-value and r-value for each comparison were determined. An α level of 0.05 was set a priori.

Results

SWAY and force platform results are listed in [Table 1]. Participants produced the largest amount of postural sway during the Tandem stance/eyes closed condition as measured by both SWAY scores and COP sway variables. SWAY identified feet together stance/eyes closed as the condition producing the least amount of postural sway in the participants, whereas COP sway area was lowest during feet together/eyes open stance.

Table 1 Averaged trial results of SWAY scores and COP measures across stances and conditions

Stance, condition	Mean (SD)
	SWAY	Area	RMS	Velocity
Feet together, EO	99.18 (1.31)	25.33 (14.84)	6.14 (1.98)	13.75 (3.00)
Feet together, EC	99.34 (0.91)	40.03 (24.28)	6.83 (2.26)	18.84 (5.89)
Tandem stance, EO	98.76 (1.60)	52.22 (32.44)	6.72 (2.64)	27.35 (7.45)
Tandem stance, EC	96.01 (3.32)	133.18 (69.83)	9.80 (2.87)	50.14 (15.13)
Single-leg stance, EO	97.13 (2.70)	108.59 (54.49)	8.74 (2.28)	39.87 (9.01)

Abbreviations: COP, center of pressure; EC, eyes closed; EO, eyes open; RMS, root mean square; SD, standard deviation; SWAY, SWAY score.

Note: Units for COP variables: mm² (area); mm (RMS); mm/s (velocity). SWAY score is an arbitrary unit.

Reliability

Test–retest reliability of SWAY and force platform COP sway variables are presented in [Table 2]. Generally, SWAY produced similar ICC values to those of sway area, RMS distance, and mean velocity. The ICC value produced by the SWAY scores for tandem stance/eyes open (ICC = 0.206) was relatively low in comparison to COP sway area, RMS, and mean velocity for the same test condition (ICC = 0.595–0.654). SWAY scores for feet together/eyes open, tandem stance/eyes closed, and single-leg stance/eyes open produced higher ICC values than all COP variables in each stance condition.

Table 2 Test–retest reliability coefficients of SWAY and COP sway variables

Stance, condition	ICC values (95% CI bounds)
	SWAY	Area	RMS	Velocity
Feet together, EO	0.410[a] (0.03–0.69)	0.270(–0.13 to 0.60)	0.187(–0.21 to 0.54)	0.348[a](–0.05 to 0.65)
Feet together, EC	0.451[a] (0.08–0.71)	0.776[b] (0.56–0.90)	0.625[b] (0.31–0.82)	0.785[b] (0.57–0.90)
Tandem stance, EO	0.206(–0.20 to 0.55)	0.627[b] (0.32–0.82)	0.654[b] (0.36–0.83)	0.595[b] (0.27–0.80)
Tandem stance, EC	0.566[a] (–0.18 to 0.80)	0.406[a] (–0.06 to 0.73)	0.508[a] (0.10–0.77)	0.407[a] (–0.03 to 0.71)
Single-leg stance, EO	0.359[a] (–0.06 to 0.67)	0.243(–0.19 to 0.60)	0.083 (–0.34 to 0.48)	0.312(–0.12 to 0.64)

Abbreviations: COP, center of pressure; CI, confidence interval; EC, eyes closed; EO, eyes open; ICC, intraclass correlation coefficient; RMS, root mean square; SWAY, SWAY score.

^a ICC values were significant at p < 0.05.

^b ICC values were significant at p < 0.001.

Validity

[Table 3] summarizes the Pearson product–moment correlations between SWAY and the force platform COP variables. The sway area, RMS, and mean velocity showed significant correlations with the SWAY scores during tandem stance/eyes open (r = –0.430 to –0.493). SWAY scores were also significantly correlated with mean velocity during single-leg stance/eyes open (r = –0.486). No significant correlations were found between SWAY scores and force platform COP variables during the feet together stance, regardless of visual condition.

Table 3 Correlation of SWAY scores with COP sway variables across test conditions

Stance, condition	Area	RMS	Velocity
Feet together, EO	–0.342	–0.361	–0.245
Feet together, EC	–0.218	–0.200	–0.181
Tandem, EO	–0.433[a]	–0.493[a]	–0.430[a]
Tandem, EC	–0.319	–0.394	–0.353
Single-leg stance, EO	–0.417	–0.420	–0.486[a]

Abbreviations: COP, center of pressure; EC, eyes closed; EO, eyes open; RMS, root mean square.

^a p-Values were significant at p < 0.05.

Discussion

The present study sought to evaluate the reliability and validity of SWAY, a software app for iOS mobile devices that uses the device's built-in accelerometer to quantify balance control. We hypothesized that SWAY would demonstrate similar test–retest reliability to force platform COP sway variables. This hypothesis was supported, although ICC values remained relatively low for both methods. We also hypothesized that SWAY and COP sway variables would demonstrate good correlation coefficients across all stance conditions. This hypothesis was partially supported. Correlation coefficients between SWAY scores during tandem stance/eyes open and single-leg stance/eyes open showed a significant correlation with force platform COP variables. The SWAY scores from other combinations of stance and visual conditions did not display a significant association with coinciding COP variables.

Comparisons between SWAY scores and the COP variables produced similar ICC values, indicating that the two methods produced comparable test–retest reliability results. A surprising outcome was the relatively low ICC values produced by SWAY scores and the COP variables. This may indicate that repeated measures taken in quick succession can lead to measurement inaccuracies. The low values for both methods also may be attributable to low variability between recruited participants. The study population consisted entirely of healthy individuals who were well within their capacity for balance during the tests. The standard deviations for each stance condition were relatively narrow, indicating low variability between participants and thus limited the magnitude of ICC calculations.[22] This also may explain why the test–retest ICC values for the force platform COP variables were much lower than previously reported in similar populations.[23] Our finding, however, that SWAY is comparable in reliability to the gold standard force platform COP variables illustrates that SWAY could be useful in objective monitoring of changes in a person's static balance over time following a concussion. This often is necessary for use in clinical settings when repeated assessments are conducted to monitor improvements or regressions in balance control over time.

This study also documented significant correlations between SWAY scores and force platform COP variables used to characterize balance, although the correlations produced only moderate associations.[18] The lack of strong associations for the remaining stances and conditions may be due to the devices measuring different aspects of balance.[15] The mobile device was held by each subject at the chest, capturing accelerations relatively close to each participant's approximate COM. By contrast, COP sway variables are approximations of the COM sway based on ground reaction force measurements captured at the floor. Measuring balance control near the COM may be more representative of postural control ability and responses to fluctuations in body sway. Conversely, displacement of COP measured by force platforms represent neuromuscular responses necessary to control torque at the ankle[24] rather than only COP sway path.[25] Measurement of balance at the approximate COM allows clinicians and researchers to directly investigate the influence of sensory systems on postural sway without the compounding influence of the neuromuscular response necessary to activate ankle musculature.

The significance of association between SWAY and COP variables during tandem stance is an important finding for clinicians assessing individuals after a suspected concussion. Evaluations incorporating narrow stances increase the sensitivity and specificity of clinical balance assessments,[26] allowing for accurate measurement of postural sway changes over time. This may be helpful to clinicians who are interested in tracking the recovery of balance over time. While tandem stance was the only valid stance in this study of healthy individuals, populations with balance deficits may produce more variability with other stances. Future research should evaluate individuals with balance instability as they complete the SWAY assessment protocol.

Recently, several methods have been proposed to improve the reliability and validity of the BESS protocol, including using Wii Balance Boards,[27] portable force platform systems,[28] and accelerometers.[29] While most of these methods improve the reliability and validity of the BESS, these approaches add costly equipment to the free assessment, add data processing time for interpretation of results, and limit the portability of the assessment. The SWAY app may be a worthwhile alternative to consider without the need for additional equipment or a dedicated space for administration. Future research should evaluate alternative methods for SWAY administration that may eliminate variability between tests, such as strapping the mobile device to eliminate any accelerations detected from unintended hand movements.

To our knowledge, this is the first study evaluating the reliability and validity of a mobile device app intended for use as an assessment of balance in athletes with a suspected concussion. SWAY is an innovative app allowing sports medicine clinicians the ability to assess balance objectively, quickly, and efficiently. Perhaps equally important, the SWAY app eliminates the need for specialized and costly equipment, as well as the extensive postprocessing of data necessary with force platforms and other accelerometers. As Mancini and Horak state, “clinical practice needs automatic algorithms for quantifying balance control during tasks, normative values, composite scores, and user-friendly interfaces so tests can be accomplished quickly…”[30] SWAY has the potential to address these needs without the necessity of specialized equipment. Ultimately, balance assessment is just one tool available to clinicians assessing injured athletes. Pairing balance assessment with other multidimensional tools is necessary when evaluating athletes with a suspected concussion.

Although the results of the present study are promising, our study had limitations addressable by future research. First, the study population comprised healthy individuals without injuries that could impact balance. While appropriate for a study focused on determining reliability and validity of the technology, this limits generalizability of results intended for diagnostic purposes. Second, reliability of measures collected sequentially over several days may be more representative of how SWAY would be used in clinical settings, but was not feasible with the design of this study. Finally, participants were required to hold the device to their sternum. In addition to increasing the challenge to maintain balance while assuming an uncommon stance, any extraneous hand movements may have produced unintended accelerations. Using a harness to hold the mobile device against the person's trunk was not done in this study, as our intent was to conduct testing with SWAY exactly according to the app's instructions for use. Future research should investigate the clinical utility of the app in athletic populations as well as determine the diagnostic utility of the app when compared with clinical and sideline balance measures such as the BESS.

Conclusions

SWAY, a software app for iOS mobile devices, demonstrated both reliability and validity while testing healthy individuals across static stances. Based on our findings, SWAY scores during tandem stance/eyes open produced the strongest association when compared with force platform COP variables. Although some correlations were low between SWAY and force platform measures of balance, SWAY demonstrated a similar pattern in reliability testing observed with COP variables. Despite being a promising tool for clinical evaluation of balance ability after a concussion, further research must investigate the use of SWAY as a measure of balance in athletic populations prior to widespread implementation and use.

• References

• 1 Langlois JA, Rutland-Brown W, Wald MM. The epidemiology and impact of traumatic brain injury: a brief overview. J Head Trauma Rehabil 2006; 21 (05) 375-378
• 2 Harmon KG, Drezner J, Gammons M. , et al; American Medical Society for Sports Medicine. American Medical Society for Sports Medicine position statement: concussion in sport. Clin J Sport Med 2013; 23 (01) 1-18
• 3 McCrory P, Meeuwisse W, Aubry M. , et al; Kathryn Schneider, PT, PhD, Charles H. Tator, MD, PHD. Consensus statement on concussion in sport--the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Clin J Sport Med 2013; 23 (02) 89-117
• 4 Guskiewicz KM. Balance assessment in the management of sport-related concussion. Clin Sports Med 2011; 30 (01) 89-102 , ix
• 5 Yang CC, Hua MS, Tu YK, Huang SJ. Early clinical characteristics of patients with persistent post-concussion symptoms: a prospective study. Brain Inj 2009; 23 (04) 299-306
• 6 Bell DR, Guskiewicz KM, Clark MA, Padua DA. Systematic review of the balance error scoring system. Sports Health 2011; 3 (03) 287-295
• 7 Mulligan IJ, Boland MA, McIlhenny CV. The balance error scoring system learned response among young adults. Sports Health 2013; 5 (01) 22-26
• 8 Giza CC, Kutcher JS, Ashwal S. , et al. Summary of evidence-based guideline update: evaluation and management of concussion in sports: report of the Guideline Development Subcommittee of the American Academy of Neurology. Neurology 2013; 80 (24) 2250-2257
• 9 Hunt TN, Ferrara MS, Bornstein RA, Baumgartner TA. The reliability of the modified Balance Error Scoring System. Clin J Sport Med 2009; 19 (06) 471-475
• 10 McCrea M, Guskiewicz KM, Marshall SW. , et al. Acute effects and recovery time following concussion in collegiate football players: the NCAA Concussion Study. JAMA 2003; 290 (019) 2556-2563
• 11 Guskiewicz KM, Perrin DH, Gansneder BM. Effect of mild head injury on postural stability in athletes. J Athl Train 1996; 31 (04) 300-306
• 12 Visser JE, Carpenter MG, van der Kooij H, Bloem BR. The clinical utility of posturography. Clin Neurophysiol 2008; 119 (011) 2424-2436
• 13 Gray VL, Ivanova TD, Garland SJ. Reliability of center of pressure measures within and between sessions in individuals post-stroke and healthy controls. Gait Posture 2014; 40 (01) 198-203
• 14 Lafond D, Corriveau H, Hébert R, Prince F. Intrasession reliability of center of pressure measures of postural steadiness in healthy elderly people. Arch Phys Med Rehabil 2004; 85 (06) 896-901
• 15 Moe-Nilssen R, Helbostad JL. Trunk accelerometry as a measure of balance control during quiet standing. Gait Posture 2002; 16 (01) 60-68
• 16 Whitney SL, Roche JL, Marchetti GF. , et al. A comparison of accelerometry and center of pressure measures during computerized dynamic posturography: a measure of balance. Gait Posture 2011; 33 (04) 594-599
• 17 Patterson JA, Amick RZ, Thummar T, Rogers ME. Validation of measures from the smartphone sway balance application: a pilot study. Int J Sports Phys Ther 2014; 9 (02) 135-139
• 18 Portney LG, Watkins MP. Correlation. In: Cohen M. , ed. Foundations of Clinical Research: Applications to Practice. 3rd ed. Upper Saddle River, NJ: Pearson Education; 2009: 523-538
• 19 Prieto TE, Myklebust JB, Hoffmann RG, Lovett EG, Myklebust BM. Measures of postural steadiness: differences between healthy young and elderly adults. IEEE Trans Biomed Eng 1996; 43 (09) 956-966
• 20 Amick RZ, Patterson JA, Jorgensen MJ. Sensitivity of tri-axial accelerometers within mobile consumer electronic devices: a pilot study. Int J Appl Sci Technol 2013; 3: 97-100
• 21 Khoo Chee Han C, Shanmugam RA, Choon Siew Kit D. Accuracy, consistency, and reproducibility of the triaxial accelerometer in the ipod touch: a pilot study. JMIR Mhealth Uhealth 2014; 2 (04) e39
• 22 Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res 2005; 19 (01) 231-240
• 23 Meshkati Z, Namazizadeh M, Salavati M, Mazaheri M. Reliability of force-platform measures of postural sway and expertise-related differences. J Sport Rehabil 2011; 20 (04) 442-456
• 24 Winter D. Human balance and posture control during standing and walking. Gait Posture 1995; 3: 193-214
• 25 Heebner NR, Akins JS, Lephart SM, Sell TC. Reliability and validity of an accelerometry based measure of static and dynamic postural stability in healthy and active individuals. Gait Posture 2015; 41 (02) 535-539
• 26 Lehmann JF, Boswell S, Price R. , et al. Quantitative evaluation of sway as an indicator of functional balance in post-traumatic brain injury. Arch Phys Med Rehabil 1990; 71 (12) 955-962
• 27 Chang JO, Levy SS, Seay SW, Goble DJ. An alternative to the balance error scoring system: using a low-cost balance board to improve the validity/reliability of sports-related concussion balance testing. Clin J Sport Med 2014; 24 (03) 256-262
• 28 Alsalaheen BA, Haines J, Yorke A, Stockdale K, Broglio SP. Reliability and concurrent validity of instrumented balance error scoring system using a portable force plate system. Phys Sportsmed 2015; 43 (03) 221-226
• 29 King LA, Horak FB, Mancini M. , et al. Instrumenting the balance error scoring system for use with patients reporting persistent balance problems after mild traumatic brain injury. Arch Phys Med Rehabil 2014; 95 (02) 353-359
• 30 Mancini M, Horak FB. The relevance of clinical balance assessment tools to differentiate balance deficits. Eur J Phys Rehabil Med 2010; 46 (02) 239-248