Download PDF
Semin Hear 2023; 44(02): 140-154
DOI: 10.1055/s-0043-1767667
Review Article

Test–Retest Reliability and the Role of Task Instructions when Measuring Listening Effort Using a Verbal Response Time Paradigm

Samantha J. Gustafson
1   Department of Communication Sciences and Disorders, University of Utah, Salt Lake City, Utah
,
Crystal Ortiz
1   Department of Communication Sciences and Disorders, University of Utah, Salt Lake City, Utah
,
Loren Nelson
1   Department of Communication Sciences and Disorders, University of Utah, Salt Lake City, Utah
› Author Affiliations Funding Work reported here was funded by a Research Grant in Hearing and Balance from the American Academy of Audiology Foundation (to S.J.G.). This research was also supported by the Office of Undergraduate Research through the Undergraduate Research Opportunity Program (to C.O.) at the University of Utah and by the National Center for Advancing Translational Sciences of the National Institutes of Health (UL1TR002538).
› Further Information
    Permissions and Reprints

Abstract

Listening amidst competing noise taxes one's limited cognitive resources, leading to increased listening effort. As such, there is interest in incorporating a reliable test of listening effort into the clinical test battery. One clinically promising method for measuring listening effort is verbal response time (VRT) because it can be obtained using already-established clinical tasks. In order for widespread implementation of the VRT paradigm, a better understanding of the psychometric properties is needed. The purpose of this work was to improve the understanding of the reliability and sensitivity of the VRT listening task. Using within-subject study designs, we completed a pilot study to evaluate the test–retest reliability (Study 1) and the effects of task instructions and listening condition (Study 2). Preliminary results show that the VRT paradigm enjoys good to excellent test–retest reliability and that neither task instructions nor listening condition meaningfully influence VRT once measurement error is accounted for. Future studies should account for measurement error when considering statistically significant versus meaningful effects of experimental parameters when using listening effort tasks.

Keywords

listening effort - task instructions - verbal response time - test–retest reliability


Publication History

Article published online:
04 April 2023

© 2023. Thieme. All rights reserved.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

 

  • References

  • 1 Mattys SL, Davis MH, Bradlow AR, Scott SK. Speech recognition in adverse conditions: a review. Lang Cogn Process 2012; 27 (7–8): 953-978
    CrossrefPubMedGoogle Scholar
  • 2 Pals C, Sarampalis A, van Rijn H, Başkent D. Validation of a simple response-time measure of listening effort. J Acoust Soc Am 2015; 138 (03) EL187-EL192
    CrossrefPubMedGoogle Scholar
  • 3 Peelle JE. Listening effort: how the cognitive consequences of acoustic challenge are reflected in brain and behavior. Ear Hear 2018; 39 (02) 204-214
    CrossrefPubMedGoogle Scholar
  • 4 Pichora-Fuller MK, Kramer SE, Eckert MA. et al. Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL). Ear Hear 2016; 37 (Suppl. 01) 5S-27S
    CrossrefPubMedGoogle Scholar
  • 5 Alhanbali S, Dawes P, Lloyd S, Munro KJ. Hearing handicap and speech recognition correlate with self-reported listening effort and fatigue. Ear Hear 2018; 39 (03) 470-474
    CrossrefPubMedGoogle Scholar
  • 6 Hughes SE, Hutchings HA, Rapport FL, McMahon CM, Boisvert I. Social connectedness and perceived listening effort in adult cochlear implant users: a grounded theory to establish content validity for a new patient-reported outcome measure. Ear Hear 2018; 39 (05) 922-934
    CrossrefPubMedGoogle Scholar
  • 7 Rosemann S, Thiel CM. Neuroanatomical changes associated with age-related hearing loss and listening effort. Brain Struct Funct 2020; 225 (09) 2689-2700
    CrossrefPubMedGoogle Scholar
  • 8 Hétu R, Jones L, Getty L. The impact of acquired hearing impairment on intimate relationships: implications for rehabilitation. Audiology 1993; 32 (06) 363-381
    CrossrefPubMedGoogle Scholar
  • 9 Kramer SE, Kapteyn TS, Houtgast T. Occupational performance: comparing normally-hearing and hearing-impaired employees using the Amsterdam Checklist for Hearing and Work. Int J Audiol 2006; 45 (09) 503-512
    CrossrefPubMedGoogle Scholar
  • 10 Hornsby BWY. The effects of hearing aid use on listening effort and mental fatigue associated with sustained speech processing demands. Ear Hear 2013; 34 (05) 523-534
    CrossrefPubMedGoogle Scholar
  • 11 Hick CB, Tharpe AM. Listening effort and fatigue in school-age children with and without hearing loss. J Speech Lang Hear Res 2002; 45 (03) 573-584
    CrossrefPubMedGoogle Scholar
  • 12 Johnson J, Xu J, Cox R, Pendergraft P. A comparison of two methods for measuring listening effort as part of an audiologic test battery. Am J Audiol 2015; 24 (03) 419-431
    CrossrefPubMedGoogle Scholar
  • 13 Meister H, Rählmann S, Lemke U, Besser J. Verbal response times as a potential indicator of cognitive load during conventional speech audiometry with matrix sentences. Trends Hear 2018; 22: 2331216518793255
    CrossrefPubMedGoogle Scholar
  • 14 Wu YH, Stangl E, Zhang X, Perkins J, Eilers E. Psychometric functions of dual-task paradigms for measuring listening effort. Ear Hear 2016; 37 (06) 660-670
    CrossrefPubMedGoogle Scholar
  • 15 McGarrigle R, Munro KJ, Dawes P. et al. Listening effort and fatigue: what exactly are we measuring? A British Society of Audiology Cognition in Hearing Special Interest Group ‘white paper’. Int J Audiol 2014; 53 (07) 433-440
    CrossrefPubMedGoogle Scholar
  • 16 Alhanbali S, Dawes P, Millman RE, Munro KJ. Measures of listening effort are multidimensional. Ear Hear 2019; 40 (05) 1084-1097
    CrossrefPubMedGoogle Scholar
  • 17 Ohlenforst B, Zekveld AA, Jansma EP. et al. Effects of hearing impairment and hearing aid amplification on listening effort: a systematic review. Ear Hear 2017; 38 (03) 267-281
    CrossrefPubMedGoogle Scholar
  • 18 Strand JF, Brown VA, Merchant MB, Brown HE, Smith J. Measuring listening effort: convergent validity, sensitivity, and links with cognitive and personality measures. J Speech Lang Hear Res 2018; 61 (06) 1463-1486
    CrossrefPubMedGoogle Scholar
  • 19 Piquado T, Benichov JI, Brownell H, Wingfield A. The hidden effect of hearing acuity on speech recall, and compensatory effects of self-paced listening. Int J Audiol 2012; 51 (08) 576-583
    CrossrefPubMedGoogle Scholar
  • 20 Rönnberg J, Lunner T, Zekveld A. et al. The Ease of Language Understanding (ELU) model: theoretical, empirical, and clinical advances. Front Syst Neurosci 2013; 7: 31
    CrossrefPubMedGoogle Scholar
  • 21 Oates PA, Kurtzberg D, Stapells DR. Effects of sensorineural hearing loss on cortical event-related potential and behavioral measures of speech-sound processing. Ear Hear 2002; 23 (05) 399-415
    CrossrefPubMedGoogle Scholar
  • 22 Korczak PA, Kurtzberg D, Stapells DR. Effects of sensorineural hearing loss and personal hearing AIDS on cortical event-related potential and behavioral measures of speech-sound processing. Ear Hear 2005; 26 (02) 165-185
    CrossrefPubMedGoogle Scholar
  • 23 Gillis M, Decruy L, Vanthornhout J, Francart T. Hearing loss is associated with delayed neural responses to continuous speech. Eur J Neurosci 2022; 55 (06) 1671-1690
    CrossrefPubMedGoogle Scholar
  • 24 Nada NM, Kolkaila EA, Gabr TA, El-Mahallawi TH. Speech auditory brainstem response audiometry in adults with sensorineural hearing loss. Egyptian J Ear, Nose, Throat Allied Sci 2016; 17 (02) 87-94
    CrossrefPubMedGoogle Scholar
  • 25 White BE, Langdon C. The cortical organization of listening effort: new insight from functional near-infrared spectroscopy. Neuroimage 2021; 240: 118324
    CrossrefPubMedGoogle Scholar
  • 26 Gatehouse S, Gordon J. Response times to speech stimuli as measures of benefit from amplification. Br J Audiol 1990; 24 (01) 63-68
    CrossrefPubMedGoogle Scholar
  • 27 Brennan MA, Lewis D, McCreery R, Kopun J, Alexander JM. Listening effort and speech recognition with frequency compression amplification for children and adults with hearing loss. J Am Acad Audiol 2017; 28 (09) 823-837
    Article in Thieme ConnectPubMedGoogle Scholar
  • 28 Gustafson S, McCreery R, Hoover B, Kopun JG, Stelmachowicz P. Listening effort and perceived clarity for normal-hearing children with the use of digital noise reduction. Ear Hear 2014; 35 (02) 183-194
    CrossrefPubMedGoogle Scholar
  • 29 Holube I. Speech intelligibility in fluctuating maskers. Proc Int Symp Audit Audiol Res 2011; 3: 57-64
    PubMedGoogle Scholar
  • 30 Lewis D, Schmid K, O'Leary S, Spalding J, Heinrichs-Graham E, High R. Effects of noise on speech recognition and listening effort in children with normal hearing and children with mild bilateral or unilateral hearing loss. J Speech Lang Hear Res 2016; 59 (05) 1218-1232
    CrossrefPubMedGoogle Scholar
  • 31 McGarrigle R, Gustafson SJ, Hornsby BWY, Bess FH. Behavioral measures of listening effort in school-age children: examining the effects of signal-to-noise ratio, hearing loss, and amplification. Ear Hear 2019; 40 (02) 381-392
    CrossrefPubMedGoogle Scholar
  • 32 Oosthuizen I, Picou EM, Pottas L, Myburgh HC, Swanepoel W. Listening effort in native and nonnative English-speaking children using low linguistic single- and dual-task paradigms. J Speech Lang Hear Res 2020; 63 (06) 1979-1989
    CrossrefPubMedGoogle Scholar
  • 33 Oosthuizen I, Picou EM, Pottas L, Myburgh HC, Swanepoel W. Listening effort in school-age children with normal hearing compared to children with limited useable hearing unilaterally. Am J Audiol 2021; 30 (02) 309-324
    CrossrefPubMedGoogle Scholar
  • 34 Visentin C, Valzolgher C, Pellegatti M, Potente P, Pavani F, Prodi N. A comparison of simultaneously-obtained measures of listening effort: pupil dilation, verbal response time and self-rating. Int J Audiol 2022; 61 (07) 561-573
    CrossrefPubMedGoogle Scholar
  • 35 Houben R, van Doorn-Bierman M, Dreschler WA. Using response time to speech as a measure for listening effort. Int J Audiol 2013; 52 (11) 753-761
    CrossrefPubMedGoogle Scholar
  • 36 Gustafson SJ, Nelson L, Silcox JW. Effect of auditory distractors on speech recognition and listening effort. Ear and Hearing. Accessed March 21, 2023 https://journals.lww.com/ear-hearing/pages/articleviewer.aspx?year=9900&issue=00000&article=00129&type=Fulltext
    PubMedGoogle Scholar
  • 37 Gagné JP, Besser J, Lemke U. Behavioral assessment of listening effort using a dual-task paradigm: a review. Trends Hear 2017; 21: 2331216516687287
    PubMedGoogle Scholar
  • 38 McCreery RW, Stelmachowicz PG. The effects of limited bandwidth and noise on verbal processing time and word recall in normal-hearing children. Ear Hear 2013; 34 (05) 585-591
    CrossrefPubMedGoogle Scholar
  • 39 Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15 (02) 155-163
    CrossrefPubMedGoogle Scholar
  • 40 Le Prell CG. Current issues in clinical and translational research in the hearing sciences, audiology, and otolaryngology. In: Translational Research in Audiology, Neurotology, and the Hearing Sciences. Springer; 2016: 19-50
    Google Scholar
  • 41 Batterham AM, George KP. Reliability in evidence-based clinical practice: a primer for allied health professionals. Phys Ther Sport 2000; 1 (02) 54-62
    CrossrefPubMedGoogle Scholar
  • 42 Crocker L, Algina J. Introduction to Classical and Modern Test Theory. ERIC; 1986
    Google Scholar
  • 43 Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res 2005; 19 (01) 231-240
    PubMedGoogle Scholar
  • 44 Bruton A, Conway JH, Holgate ST. Reliability: what is it, and how is it measured?. Physiotherapy 2000; 86 (02) 94-99
    CrossrefPubMedGoogle Scholar
  • 45 Krueger M, Schulte M, Brand T, Holube I. Development of an adaptive scaling method for subjective listening effort. J Acoust Soc Am 2017; 141 (06) 4680-4693
    CrossrefPubMedGoogle Scholar
  • 46 Picou EM, Ricketts TA. The relationship between speech recognition, behavioural listening effort, and subjective ratings. Int J Audiol 2018; 57 (06) 457-467
    CrossrefPubMedGoogle Scholar
  • 47 Reinten I, De Ronde-Brons I, Houben R, Dreschler W. Measuring the influence of noise reduction on listening effort in hearing-impaired listeners using response times to an arithmetic task in noise. Trends Hear 2021;25:23312165211014437
    PubMedGoogle Scholar
  • 48 Devocht EMJ, Janssen AML, Chalupper J, Stokroos RJ, George ELJ. The benefits of bimodal aiding on extended dimensions of speech perception: intelligibility, listening effort, and sound quality. Trends Hear 2017; 21: 2331216517727900
    CrossrefPubMedGoogle Scholar
  • 49 Picou EM, Ricketts TA. The effect of changing the secondary task in dual-task paradigms for measuring listening effort. Ear Hear 2014; 35 (06) 611-622
    CrossrefPubMedGoogle Scholar
  • 50 Neagu MB, Dau T, Hyvärinen P, Bækgaard P, Lunner T, Wendt D. Investigating pupillometry as a reliable measure of individual's listening effort. Proceedings of the International Symposium on Auditory and Audiological Research 2019; 7: 365-372
    PubMedGoogle Scholar
  • 51 Jones M, Dobson A, O'Brian S. A graphical method for assessing agreement with the mean between multiple observers using continuous measures. Int J Epidemiol 2011; 40 (05) 1308-1313
    CrossrefPubMedGoogle Scholar
  • 52 Degeest S, Corthals P, Keppler H. A Dutch version of a dual-task paradigm for measuring listening effort: a pilot study regarding its short-term test-retest reliability. B-ENT 2021; 17 (03) 135-144
    CrossrefPubMedGoogle Scholar
  • 53 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Int J Nurs Stud 2010; 47 (08) 931-936
    CrossrefPubMedGoogle Scholar
  • 54 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1 (8476): 307-310
    CrossrefPubMedGoogle Scholar
  • 55 Bland JM, Altman DG. Comparing methods of measurement: why plotting difference against standard method is misleading. Lancet 1995; 346 (8982): 1085-1087
    CrossrefPubMedGoogle Scholar
  • 56 Beninato M, Portney LG. Applying concepts of responsiveness to patient management in neurologic physical therapy. J Neurol Phys Ther 2011; 35 (02) 75-81
    CrossrefPubMedGoogle Scholar
  • 57 Portney LG, Watkins MP. Foundations of Clinical Research: Applications to Practice. Vol 892. Pearson/Prentice Hall Upper Saddle River, NJ; 2009
    Google Scholar
  • 58 Beckerman H, Roebroeck ME, Lankhorst GJ, Becher JG, Bezemer PD, Verbeek ALM. Smallest real difference, a link between reproducibility and responsiveness. Qual Life Res 2001; 10 (07) 571-578
    CrossrefPubMedGoogle Scholar
  • 59 Donoghue D, Stokes EK. Physiotherapy Research and Older People (PROP) group. How much change is true change? The minimum detectable change of the Berg Balance Scale in elderly people. J Rehabil Med 2009; 41 (05) 343-346
    CrossrefPubMedGoogle Scholar
  • 60 Choi S, Lotto A, Lewis D, Hoover B, Stelmachowicz P. Attentional modulation of word recognition by children in a dual-task paradigm. J Speech Lang Hear Res 2008; 51 (04) 1042-1054
    CrossrefPubMedGoogle Scholar
  • 61 Woodworth RS. Accuracy of voluntary movement. Psychological Review 1899; 3: 1-106
    PubMedGoogle Scholar
  • 62 Heitz RP. The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Front Neurosci 2014; 8: 150
    CrossrefPubMedGoogle Scholar
  • 63 Adler LA, Kessler RC, Spencer T. Adult ADHD Self-Report Scale-v1. 1 (ASRS-v1. 1) Symptom Checklist. New York, NY: World Health Organization; 2003
    Google Scholar
  • 64 Bonino AY, Malley AR. Measuring open-set, word recognition in school-aged children: corpus of monosyllabic target words and speech maskers. J Acoust Soc Am 2019; 146 (04) EL393-EL398
    CrossrefPubMedGoogle Scholar
  • 65 Nagaraj NK, Magimairaj BM, Schwartz S. Auditory distraction in school-age children relative to individual differences in working memory capacity. Atten Percept Psychophys 2020; 82 (07) 3581-3593
    CrossrefPubMedGoogle Scholar
  • 66 Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 1994; 6 (04) 284
    Google Scholar
  • 67 Hopkins WG. Measures of reliability in sports medicine and science. Sports Med 2000; 30 (01) 1-15
    CrossrefPubMedGoogle Scholar
  • 68 Ohlenforst B, Zekveld AA, Lunner T. et al. Impact of stimulus-related factors and hearing impairment on listening effort as indicated by pupil dilation. Hear Res 2017; 351: 68-79
    CrossrefPubMedGoogle Scholar
  • 69 Oosthuizen I, Picou EM, Pottas L, Myburgh HC, Swanepoel W. Listening effort in school-aged children with limited useable hearing unilaterally: examining the effects of a personal, digital remote microphone system and a contralateral routing of signal system. Trends Hear 2021; 25: 2331216520984700
    PubMedGoogle Scholar
  • 70 Watson PF, Petrie A. Method agreement analysis: a review of correct methodology. Theriogenology 2010; 73 (09) 1167-1179
    CrossrefPubMedGoogle Scholar