J Am Acad Audiol 2020; 31(08): 599-612
DOI: 10.1055/s-0040-1717066
Research Article

Test-Retest Reliability of Ecological Momentary Assessment in Audiology Research

Yu-Hsiang Wu
1   Department of Communication Sciences and Disorders, The University of Iowa, Iowa City, Iowa
Elizabeth Stangl
1   Department of Communication Sciences and Disorders, The University of Iowa, Iowa City, Iowa
Octav Chipara
2   Department of Computer Science, The University of Iowa, Iowa City, Iowa
Xuyang Zhang
1   Department of Communication Sciences and Disorders, The University of Iowa, Iowa City, Iowa
› Author Affiliations
Funding The present research was supported by NIH/NIDCD R03DC012551 and R01DC015997 and NSF SCH 1838830.


Background Ecological momentary assessment (EMA) is a methodology involving repeated surveys to collect in situ data that describe respondents' current or recent experiences and related contexts in their natural environments. Audiology literature investigating the test-retest reliability of EMA is scarce.

Purpose This article examines the test-retest reliability of EMA in measuring the characteristics of listening contexts and listening experiences.

Research Design An observational study.

Study Sample Fifty-one older adults with hearing loss.

Data Collection and Analysis The study was part of a larger study that examined the effect of hearing aid technologies. The larger study had four trial conditions and outcome was measured using a smartphone-based EMA system. After completing the four trial conditions, participants repeated one of the conditions to examine the EMA test-retest reliability. The EMA surveys contained questions that assessed listening context characteristics including talker familiarity, talker location, and noise location, as well as listening experiences including speech understanding, listening effort, loudness satisfaction, and hearing aid satisfaction. The data from multiple EMA surveys collected by each participant were aggregated in each of the test and retest conditions. Test-retest correlation on the aggregated data was then calculated for each EMA survey question to determine the reliability of EMA.

Results At the group level, listening context characteristics and listening experience did not change between the test and retest conditions. The test-retest correlation varied across the EMA questions, with the highest being the questions that assessed talker location (median r = 1.0), reverberation (r = 0.89), and speech understanding (r = 0.85), and the lowest being the items that quantified noise location (median r = 0.63), talker familiarity (r = 0.46), listening effort (r = 0.61), loudness satisfaction (r = 0.60), and hearing aid satisfaction (r = 0.61).

Conclusion Several EMA questions yielded appropriate test-retest reliability results. The lower test-retest correlations for some EMA survey questions were likely due to fewer surveys completed by participants and poorly designed questions. Therefore, the present study stresses the importance of using validated questions in EMA. With sufficient numbers of surveys completed by respondents and with appropriately designed survey questions, EMA could have reasonable test-retest reliability in audiology research.

Presentation at Meetings

Portions of this paper were presented at the annual meeting of the American Auditory Society, March, 2017, Scottsdale, Arizona, USA.

Publication History

Received: 23 September 2019

Accepted: 10 February 2020

Article published online:
06 November 2020

© 2020. American Academy of Audiology. This article is published by Thieme.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

  • References

  • 1 Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008; 4: 1-32
  • 2 Preminger JE, Cunningham DR. Case-study analysis of various field study measures. J Am Acad Audiol 2003; 14 (01) 39-55
  • 3 Walden BE, Surr RK, Cord MT, Dyrlund O. Predicting hearing aid microphone preference in everyday listening. J Am Acad Audiol 2004; 15 (05) 365-396
  • 4 Wu YH, Bentler RA. Impact of visual cues on directional benefit and preference: Part II--field tests. Ear Hear 2010; 31 (01) 35-46
  • 5 Wu YH, Bentler RA. Do older adults have social lifestyles that place fewer demands on hearing?. J Am Acad Audiol 2012; 23 (09) 697-711
  • 6 Bentler R, Wu YH, Kettel J, Hurtig R. Digital noise reduction: outcomes from laboratory and field studies. Int J Audiol 2008; 47 (08) 447-460
  • 7 Galvez G, Turbin MB, Thielman EJ, Istvan JA, Andrews JA, Henry JA. Feasibility of ecological momentary assessment of hearing difficulties encountered by hearing aid users. Ear Hear 2012; 33 (04) 497-507
  • 8 Timmer BHB, Hickson L, Launer S. Do hearing aids address real-world hearing difficulties for adults with mild hearing impairment? Results from a pilot study using ecological momentary assessment. Trends Hear 2018; 22: 2331216518783608
  • 9 Wu YH, Stangl E, Chipara O, Hasan SS, DeVries S, Oleson J. Efficacy and effectiveness of advanced hearing aid directional and noise reduction technologies for older adults with mild to moderate hearing loss. Ear Hear 2019; 40 (04) 805-822
  • 10 Wu YH, Stangl E, Zhang X, Bentler RA. Construct validity of the ecological momentary assessment in audiology research. J Am Acad Audiol 2015; 26 (10) 872-884
  • 11 Timmer BHB, Hickson L, Launer S. Ecological momentary assessment: feasibility, construct validity, and future applications. Am J Audiol 2017; 26 (3S): 436-442
  • 12 Weinstein BE, Spitzer JB, Ventry IM. Test-retest reliability of the Hearing Handicap Inventory for the Elderly. Ear Hear 1986; 7 (05) 295-299
  • 13 Cox RM, Alexander GC. The abbreviated profile of hearing aid benefit. Ear Hear 1995; 16 (02) 176-186
  • 14 Cox RM, Alexander GC. Measuring satisfaction with amplification in daily life: the SADL scale. Ear Hear 1999; 20 (04) 306-320
  • 15 Kramer SE, Goverts ST, Dreschler WA, Boymans M, Festen JM. International Outcome Inventory for Hearing Aids (IOI-HA): results from The Netherlands. Int J Audiol 2002; 41 (01) 36-41
  • 16 Singh G, Kathleen Pichora-Fuller M. Older adults’ performance on the speech, spatial, and qualities of hearing scale (SSQ): test-retest reliability and a comparison of interview and self-administration methods. Int J Audiol 2010; 49 (10) 733-740
  • 17 Kline P. The Handbook of Psychological Testing. 2nd ed.. London: Routledge; 2000
  • 18 Nunnally JC, Bernstein IH. Psychometric Theory. 3rd ed.. New York: McGraw-Hill; 1994
  • 19 Hektner JM, Schmidt JA, Csikszentmihalyi M. Psychometrics of ESM data. In: Experience Sampling Method: Measuring the Quality of Everyday Life. Thousand Oaks, CA: Sage; 2007: 103-121
  • 20 Csikszentmihalyi M, Larson R. Validity and reliability of the experience sampling method. In: Csikszentmihalyi M. ed. Flow and the Foundations of Positive Psychology. Dordrecht, The Netherlands: Springer; 2014: 35-54
  • 21 Keidser G, Dillon H, Flax M, Ching T, Brewer S. The NAL-NL2 prescription procedure. Audiology Res 2011; 1 (01) e24
  • 22 Hasan SS, Lai F, Chipara O, Wu YH. AudioSense: Enabling real-time evaluation of hearing aid technology in-situ. Paper presented at the 26th IEEE International Symposium on Computer-Based Medical Systems, Porto, Portugal: 2013
  • 23 Stone AA, Broderick JE, Schwartz JE, Shiffman S, Litcher-Kelly L, Calvanese P. Intensive momentary reporting of pain with an electronic diary: reactivity, compliance, and patient satisfaction. Pain 2003; 104 (1-2): 343-351
  • 24 Wu YH, Stangl E, Chipara O, Hasan SS, Welhaven A, Oleson J. Characteristics of real-world signal-to-noise ratios and speech listening situations of older adults with mild to moderate hearing loss. Ear Hear 2018; 39 (02) 293-304
  • 25 Palmer CV, Mueller GH, Moriarty M. Profile of aided loudness: a validation procedure. Hear J 1999; 52: 34-36
  • 26 Ventry IM, Weinstein BE. The hearing handicap inventory for the elderly: a new tool. Ear Hear 1982; 3 (03) 128-134
  • 27 Gatehouse S, Noble W. The Speech, Spatial and Qualities of Hearing Scale (SSQ). Int J Audiol 2004; 43 (02) 85-99
  • 28 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1 (8476): 307-310
  • 29 Humes LE, Rogers SE, Main AK, Kinney DL. The acoustic environments in which older adults wear their hearing aids: insights from datalogging sound environment classification. Am J Audiol 2018; 27 (04) 594-603
  • 30 Klein KE, Wu YH, Stangl E, Bentler RA. Using a digital language processor to quantify the auditory environment and the effect of hearing aids for adults with hearing loss. J Am Acad Audiol 2018; 29 (04) 279-291
  • 31 Mares ML, Woodard EH. In search of the older audience: adult age differences in television viewing. J Broadcast Electron Media 2006; 50: 595-614
  • 32 Humes LE, Halling D, Coughlin M. Reliability and stability of various hearing-aid outcome measures in a group of elderly hearing-aid wearers. J Speech Hear Res 1996; 39 (05) 923-935
  • 33 Moore TM, Picou EM. A potential bias in subjective ratings of mental effort. J Speech Lang Hear Res 2018; 61 (09) 2405-2421
  • 34 Faiers G, McCarthy P. Study explores how paying affects hearing aid users' satisfaction. Hear J 2004; 57: 25-32
  • 35 Gatehouse S. Glasgow hearing aid benefit profile: derivation and validation of a client-centered outcome measure for hearing aid services. J Am Acad Audiol 1999; 10: 103
  • 36 Cox RM, Alexander GC. The International Outcome Inventory for Hearing Aids (IOI-HA): psychometric properties of the English version. Int J Audiol 2002; 41 (01) 30-35
  • 37 Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 2008; 11 (02) 322-333