Download PDF
J Am Acad Audiol 2020; 31(08): 599-612
DOI: 10.1055/s-0040-1717066
Research Article

Test-Retest Reliability of Ecological Momentary Assessment in Audiology Research

Yu-Hsiang Wu
1   Department of Communication Sciences and Disorders, The University of Iowa, Iowa City, Iowa
,
Elizabeth Stangl
1   Department of Communication Sciences and Disorders, The University of Iowa, Iowa City, Iowa
,
Octav Chipara
2   Department of Computer Science, The University of Iowa, Iowa City, Iowa
,
Xuyang Zhang
1   Department of Communication Sciences and Disorders, The University of Iowa, Iowa City, Iowa
› Author Affiliations Funding The present research was supported by NIH/NIDCD R03DC012551 and R01DC015997 and NSF SCH 1838830.
› Further Information
   Permissions and Reprints

Abstract

Background Ecological momentary assessment (EMA) is a methodology involving repeated surveys to collect in situ data that describe respondents' current or recent experiences and related contexts in their natural environments. Audiology literature investigating the test-retest reliability of EMA is scarce.

Purpose This article examines the test-retest reliability of EMA in measuring the characteristics of listening contexts and listening experiences.

Research Design An observational study.

Study Sample Fifty-one older adults with hearing loss.

Data Collection and Analysis The study was part of a larger study that examined the effect of hearing aid technologies. The larger study had four trial conditions and outcome was measured using a smartphone-based EMA system. After completing the four trial conditions, participants repeated one of the conditions to examine the EMA test-retest reliability. The EMA surveys contained questions that assessed listening context characteristics including talker familiarity, talker location, and noise location, as well as listening experiences including speech understanding, listening effort, loudness satisfaction, and hearing aid satisfaction. The data from multiple EMA surveys collected by each participant were aggregated in each of the test and retest conditions. Test-retest correlation on the aggregated data was then calculated for each EMA survey question to determine the reliability of EMA.

Results At the group level, listening context characteristics and listening experience did not change between the test and retest conditions. The test-retest correlation varied across the EMA questions, with the highest being the questions that assessed talker location (median r = 1.0), reverberation (r = 0.89), and speech understanding (r = 0.85), and the lowest being the items that quantified noise location (median r = 0.63), talker familiarity (r = 0.46), listening effort (r = 0.61), loudness satisfaction (r = 0.60), and hearing aid satisfaction (r = 0.61).

Conclusion Several EMA questions yielded appropriate test-retest reliability results. The lower test-retest correlations for some EMA survey questions were likely due to fewer surveys completed by participants and poorly designed questions. Therefore, the present study stresses the importance of using validated questions in EMA. With sufficient numbers of surveys completed by respondents and with appropriately designed survey questions, EMA could have reasonable test-retest reliability in audiology research.

Keywords

ecological momentary assessment - hearing aid - outcome

Presentation at Meetings

Portions of this paper were presented at the annual meeting of the American Auditory Society, March, 2017, Scottsdale, Arizona, USA.




Publication History

Received: 23 September 2019

Accepted: 10 February 2020

Article published online:
06 November 2020

© 2020. American Academy of Audiology. This article is published by Thieme.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

 

  • References

  • 1 Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008; 4: 1-32
    CrossrefPubMedGoogle Scholar
  • 2 Preminger JE, Cunningham DR. Case-study analysis of various field study measures. J Am Acad Audiol 2003; 14 (01) 39-55
    Article in Thieme ConnectPubMedGoogle Scholar
  • 3 Walden BE, Surr RK, Cord MT, Dyrlund O. Predicting hearing aid microphone preference in everyday listening. J Am Acad Audiol 2004; 15 (05) 365-396
    Article in Thieme ConnectPubMedGoogle Scholar
  • 4 Wu YH, Bentler RA. Impact of visual cues on directional benefit and preference: Part II--field tests. Ear Hear 2010; 31 (01) 35-46
    CrossrefPubMedGoogle Scholar
  • 5 Wu YH, Bentler RA. Do older adults have social lifestyles that place fewer demands on hearing?. J Am Acad Audiol 2012; 23 (09) 697-711
    Article in Thieme ConnectPubMedGoogle Scholar
  • 6 Bentler R, Wu YH, Kettel J, Hurtig R. Digital noise reduction: outcomes from laboratory and field studies. Int J Audiol 2008; 47 (08) 447-460
    CrossrefPubMedGoogle Scholar
  • 7 Galvez G, Turbin MB, Thielman EJ, Istvan JA, Andrews JA, Henry JA. Feasibility of ecological momentary assessment of hearing difficulties encountered by hearing aid users. Ear Hear 2012; 33 (04) 497-507
    CrossrefPubMedGoogle Scholar
  • 8 Timmer BHB, Hickson L, Launer S. Do hearing aids address real-world hearing difficulties for adults with mild hearing impairment? Results from a pilot study using ecological momentary assessment. Trends Hear 2018; 22: 2331216518783608
    CrossrefPubMedGoogle Scholar
  • 9 Wu YH, Stangl E, Chipara O, Hasan SS, DeVries S, Oleson J. Efficacy and effectiveness of advanced hearing aid directional and noise reduction technologies for older adults with mild to moderate hearing loss. Ear Hear 2019; 40 (04) 805-822
    CrossrefPubMedGoogle Scholar
  • 10 Wu YH, Stangl E, Zhang X, Bentler RA. Construct validity of the ecological momentary assessment in audiology research. J Am Acad Audiol 2015; 26 (10) 872-884
    Article in Thieme ConnectPubMedGoogle Scholar
  • 11 Timmer BHB, Hickson L, Launer S. Ecological momentary assessment: feasibility, construct validity, and future applications. Am J Audiol 2017; 26 (3S): 436-442
    CrossrefPubMedGoogle Scholar
  • 12 Weinstein BE, Spitzer JB, Ventry IM. Test-retest reliability of the Hearing Handicap Inventory for the Elderly. Ear Hear 1986; 7 (05) 295-299
    CrossrefPubMedGoogle Scholar
  • 13 Cox RM, Alexander GC. The abbreviated profile of hearing aid benefit. Ear Hear 1995; 16 (02) 176-186
    CrossrefPubMedGoogle Scholar
  • 14 Cox RM, Alexander GC. Measuring satisfaction with amplification in daily life: the SADL scale. Ear Hear 1999; 20 (04) 306-320
    CrossrefPubMedGoogle Scholar
  • 15 Kramer SE, Goverts ST, Dreschler WA, Boymans M, Festen JM. International Outcome Inventory for Hearing Aids (IOI-HA): results from The Netherlands. Int J Audiol 2002; 41 (01) 36-41
    CrossrefPubMedGoogle Scholar
  • 16 Singh G, Kathleen Pichora-Fuller M. Older adults’ performance on the speech, spatial, and qualities of hearing scale (SSQ): test-retest reliability and a comparison of interview and self-administration methods. Int J Audiol 2010; 49 (10) 733-740
    CrossrefPubMedGoogle Scholar
  • 17 Kline P. The Handbook of Psychological Testing. 2nd ed.. London: Routledge; 2000
    Google Scholar
  • 18 Nunnally JC, Bernstein IH. Psychometric Theory. 3rd ed.. New York: McGraw-Hill; 1994
    Google Scholar
  • 19 Hektner JM, Schmidt JA, Csikszentmihalyi M. Psychometrics of ESM data. In: Experience Sampling Method: Measuring the Quality of Everyday Life. Thousand Oaks, CA: Sage; 2007: 103-121
    Google Scholar
  • 20 Csikszentmihalyi M, Larson R. Validity and reliability of the experience sampling method. In: Csikszentmihalyi M. ed. Flow and the Foundations of Positive Psychology. Dordrecht, The Netherlands: Springer; 2014: 35-54
    Google Scholar
  • 21 Keidser G, Dillon H, Flax M, Ching T, Brewer S. The NAL-NL2 prescription procedure. Audiology Res 2011; 1 (01) e24
    CrossrefPubMedGoogle Scholar
  • 22 Hasan SS, Lai F, Chipara O, Wu YH. AudioSense: Enabling real-time evaluation of hearing aid technology in-situ. Paper presented at the 26th IEEE International Symposium on Computer-Based Medical Systems, Porto, Portugal: 2013
    PubMedGoogle Scholar
  • 23 Stone AA, Broderick JE, Schwartz JE, Shiffman S, Litcher-Kelly L, Calvanese P. Intensive momentary reporting of pain with an electronic diary: reactivity, compliance, and patient satisfaction. Pain 2003; 104 (1-2): 343-351
    CrossrefPubMedGoogle Scholar
  • 24 Wu YH, Stangl E, Chipara O, Hasan SS, Welhaven A, Oleson J. Characteristics of real-world signal-to-noise ratios and speech listening situations of older adults with mild to moderate hearing loss. Ear Hear 2018; 39 (02) 293-304
    CrossrefPubMedGoogle Scholar
  • 25 Palmer CV, Mueller GH, Moriarty M. Profile of aided loudness: a validation procedure. Hear J 1999; 52: 34-36
    CrossrefGoogle Scholar
  • 26 Ventry IM, Weinstein BE. The hearing handicap inventory for the elderly: a new tool. Ear Hear 1982; 3 (03) 128-134
    CrossrefPubMedGoogle Scholar
  • 27 Gatehouse S, Noble W. The Speech, Spatial and Qualities of Hearing Scale (SSQ). Int J Audiol 2004; 43 (02) 85-99
    CrossrefPubMedGoogle Scholar
  • 28 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1 (8476): 307-310
    CrossrefPubMedGoogle Scholar
  • 29 Humes LE, Rogers SE, Main AK, Kinney DL. The acoustic environments in which older adults wear their hearing aids: insights from datalogging sound environment classification. Am J Audiol 2018; 27 (04) 594-603
    CrossrefPubMedGoogle Scholar
  • 30 Klein KE, Wu YH, Stangl E, Bentler RA. Using a digital language processor to quantify the auditory environment and the effect of hearing aids for adults with hearing loss. J Am Acad Audiol 2018; 29 (04) 279-291
    Article in Thieme ConnectPubMedGoogle Scholar
  • 31 Mares ML, Woodard EH. In search of the older audience: adult age differences in television viewing. J Broadcast Electron Media 2006; 50: 595-614
    CrossrefGoogle Scholar
  • 32 Humes LE, Halling D, Coughlin M. Reliability and stability of various hearing-aid outcome measures in a group of elderly hearing-aid wearers. J Speech Hear Res 1996; 39 (05) 923-935
    CrossrefPubMedGoogle Scholar
  • 33 Moore TM, Picou EM. A potential bias in subjective ratings of mental effort. J Speech Lang Hear Res 2018; 61 (09) 2405-2421
    CrossrefPubMedGoogle Scholar
  • 34 Faiers G, McCarthy P. Study explores how paying affects hearing aid users' satisfaction. Hear J 2004; 57: 25-32
    CrossrefGoogle Scholar
  • 35 Gatehouse S. Glasgow hearing aid benefit profile: derivation and validation of a client-centered outcome measure for hearing aid services. J Am Acad Audiol 1999; 10: 103
    Google Scholar
  • 36 Cox RM, Alexander GC. The International Outcome Inventory for Hearing Aids (IOI-HA): psychometric properties of the English version. Int J Audiol 2002; 41 (01) 30-35
    CrossrefPubMedGoogle Scholar
  • 37 Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 2008; 11 (02) 322-333
    CrossrefPubMedGoogle Scholar