Ecological validity of the five digit test and the oral trails test

ABSTRACT

Tests evaluating the attentional-executive system are widely used in clinical practice. However, proximity of an objective cognitive test with real-world situations (ecological validity) is not frequently investigated. The present study evaluate the association between measures of the Five Digit Test (FDT) and the Oral Trails Test (OTT) with self-reported cognitive failures in everyday life as measured by the Cognitive Failures Questionnaire (CFQ). Brazilian adults from 18-to-65 years old voluntarily performed the FDT and OTT tests and reported the frequency of cognitive failures in their everyday life through the CFQ. After controlling for the age effect, the measures of controlled attentional processes were associated with cognitive failures, yet the cognitive flexibility of both FDT and OTT accounted for by the majority of variance in most aspects of the CFQ factors. The FDT and the OTT measures were predictive of real-world problems such as cognitive failures in everyday activities/situations.

RESUMO

Testes que avaliam o sistema atencional-executivo são amplamente adotados no contexto clínico. Contudo, a proximidade de uma medida cognitiva objetiva com situações reais (validade ecológica) são pouco investigadas. O presente estudo objetiva avaliar a associação entre o Teste dos Cinco Dígitos (FDT) e o Oral Trails Test (OTT) com falhas cognitivas do dia a dia relatadas no Questionário de Falhas Cognitivas (CFQ). Sessenta adultos brasileiros, de 18 a 65 e cinco anos, realizaram o FDT e o OTT e reportaram a frequência das falhas cognitivas por meio do CFQ. Controlando o efeito da idade, as medidas de processos atencionais controlados foram associadas com as falhas cognitivas, sendo que o componente de flexibilidade cognitiva do FDT e do OTT explicou a maior parte da variância das falhas. O FDT e o OTT foram preditores de problemas reais do dia a dia, como as falhas cognitivas.

neuropsychology - attention - psychometrics - cognition - stroop test

neuropsicologia - atenção - psicometria - cognição - teste de stroop

Despite of intense development in the last decades neuropsychologists still have a limited number of adapted, validated and standardized neuropsychological tests to use in clinical and research contexts where patients have low formal education[1]. An important aspect to be evaluated by neuropsychologists are the components of the attentional system, which involves automatic and controlled processes. The attentional processes influence different ecological contexts, such as reading, writing, arithmetic and are strongly related to executive functions, a set of cognitive skills responsible for controlling and handling goal-oriented behavior[2]. Tests evaluating attention are generally used in research and clinical contexts, yet the examinee’s sociodemographic and educational background constantly represents an important barrier for test selection, application and interpretation[3]. Recently, a new set of attention’s tests seems to overcome some of these limitations in our context, but some aspects related to its properties must be investigated[4],[5].

Petersen and Posner[2] suggested a human attention system subdivided into three functionally independent networks. The alerting network allows maintenance of a vigilant and alert state, the orienting network is responsible for the movement of attention through space to attend to sensory events, and the executive control network allows for the monitoring and resolution of conflict between expectation, stimulus, and response. The three attentional networks have a modular nature, yet strongly interconnected[6]. The automatic processes (alerting and orienting) have great conceptual overlap with the definition of processing speed, a cognitive property associated with the time spent in different tasks execution frequently found in psychometric studies[7]. The controlled ones are superposed with components of the executive function, particularly the attentional components of the inhibitory control system[8].

One of the most used methods to evaluate attentional processes is the interference paradigm, where an automatic attentional routine must be inhibited favoring a controlled response which occurs in the classic Stroop Color-Word Test (SCWT)[9]. However, the interference effect of the SCWT could be inaccurate for specific populations such as preschoolers, illiterates or even individuals with low formal schooling since reading efficiency moderates the time of the task execution[10]. The Five Digit Test (FDT) was design to overcome some of these limitations requiring to test-takers to read or count Arabic numerals (1–5) or count stars[11]. Four parts compose the FDT: reading, counting, choosing (selective attention), and switching (alternate attention)[11]. The Oral Trails Test (OTT)[12] is another alternative to the SCWT combined with elements from the Trail Making Test[13]. The OTT interference effect is expected from shapes and colors of four fruits: banana (yellow), apple (red), grape (purple) and orange (orange). Additionally to the interference effect, the OTT has a visual search component and demands the knowledge of the ascending numerical order of Arabic digits from 1-to-20. Five parts compose the OTT: semantic search, spatial search, selective attention, and interchange (alternating attention). The OTT also presents low literacy requirements.

Beyond its psychometric properties, the practical application of a test is extremely important in Neuropsychology. An important and often neglected aspect of neuropsychological testing is the ecological validity. This property refer to which extent the score on a neuropsychological test relates to patient’s behavior in everyday life[14]. Although most of the neuropsychological tests may be useful for of the clinical profile in psychiatric or neurological disorders, several tests have limited potential to predict ecological outcomes[15]. There are several evidences of the attentional system importance on everyday learning and behavior[2], but the extension of the association of specific tests with everyday outcomes is of extreme relevance for the validation of new tests measuring automatic and controlled processes.

The Cognitive Failures Questionnaire (CFQ)[16] is a reliable measurement of self-reported problems in perceptual, memorial and motoric functions in daily life and have an association with attentional performance[17]. Considering the importance of the proximity of an objective cognitive test with real-world situations (ecological validity) for its practical implications, we sought to investigate the association between the FDT and the OTT measures (automatic and controlled processes) with self-reported cognitive failures as measured by the CFQ.

METHODS

Participants

Sixty Brazilian adults with ages between 18 and 65 years old were enrolled in this study. The inclusion criteria was age over 18 years, volunteer interest in participation and written consent for participation. The exclusion criteria was severe sensory or motors deficits, which could affect participants’ performance in the proposed tasks, history of psychotropic substances abuse or related history of neurological disease and/or severe psychiatry disorders, which could affect the global cognitive status (e.g. stroke, epilepsy, multiple learning disorders, developmental disorders, uncontrolled systemic diseases). This study was approved by the Faculdade de Ciências Médicas de Minas Gerais ethics committee board (process 745.264) and is in accordance to Helsiki declaration. The subjects gave written consent for participation.

Neuropsychological and psychological assessment

Five Digit Test (FDT) ¹¹ :The FDT is a numeric-Stroop paradigm applying four steps. Parts 1 and 2 involve automatic processes such as reading (the Arabic algorisms 1,2,3,4 and 5) and counting (quantities from one-to-five). The Part 3 (choosing) involves interference control since an automatic numerical transcoding (i.e., naming -transform a number from the Arabic digital format to the oral verbal format) has to be inhibited in favor of a controlled one (i.e., count Arabic digits that not represent the set cardinality) (e.g., stimulus =”1,1,1” and response =”three”). Part 4 (shifting) involves a set-shift from rules of Part 1 to Part 3 and vice-versa depending on an explicit marker. Executive scores are calculates for inhibition (choosing – reading) and flexibility (shifting – reading). The used measure to evaluate participants’ performance was the time spent to complete the tasks in each part. Faster times indicates a better performance. An example of the test stimuli is show in [Figure 1].

Figure 1Five Digit Test. Parts 1 and 2 (Reading and Counting) are related to simple processing speed and automatic attentional processes. Parts 3 and 4 (Choosing and Shifting) involve inhibition and cognitive flexibility and are related to executive functions and controlled attentional processes. Inhibition and Flexibility scores are computed subtracting the Reading time from Choosing and Shifting times, respectively. A disproportionate slowness in the Choosing/Inhibition and Shifting/Flexibility when compared to normative parameters may be indicative of deficits in executive functions.

Oral Trails Test (OTT) ¹² :The test has 5 Parts. In Part 1 (naming), the participant is required to name the number and the fruit in each one of a series of 20 squares printed in one page. All stimulus are shown in a canonical way in the first step. In Part 2 (spatial search), the stimulus are shown in a random numerical order and participants are asked to name the number and the fruit in each one of the 20 squares, but in its ascending numerical order. Parts 3 (choosing) and 4 (executive search) consist in naming the number and the fruit that have the color (but not the shape) of the stimulus in each square with the stimulus in Part 4 disposed in a random order. In the Part 5 (shifting), participants are instructed to perform the same as Part 3 (no visual search required), but depending on an explicit marker they had to change the rule and name the shape (not the color) of the fruit of the stimulus. The used measure to evaluate participants’ performance was the time spent to complete the task in each part. Faster times indicates a better performance. An example of the test stimuli is show in [Figure 2].

Figure 2Oral Trails Test. Part 1 (Naming) involves the naming of ordered stimuli. Part 2 (spatial search) involves naming and visual search of non-ordered stimuli. These two steps are related do simple processing speed and attention. Parts 3 (choosing), 4 (executive search), and 5 (shifting) involve processes related to inhibition and cognitive flexibility, which are measures of controlled attentional processes and executive functions. A disproportionate slowness in time execution of choosing and shifting when compared to reading, and of executive search when compared to visual search, based on normative parameters, may be indicative of deficits in executive functions.

Cognitive Failures Questionnaire (CFQ) ¹⁶ :CFQ is an instrument of self-reported everyday cognitive failures frequency. The CFQ consists of 25 items. Participants are asked to report the frequency of incidents such as ‘Do you read something and find you haven’t been thinking about it and must read it again?’ in the past 6 months on a 5-point Likert-type scale. Following the recommendations of Rast and colleagues[18], the CFQ was split into three factors: Forgetfulness, Distractibility, and False Triggering, in addition to the total score. The first factor is associated with memory complaints, the second with inattentive behavior, and the last one may be seen as resulting from loss of activation in attentional resources. Higher scores are representative of more frequent cognitive failures on daily life. The translation and preliminary study of validity was sent for appreciation to a scientific journal.

Statistical analysis

After data distribution assessment and descriptive statistics the association between the cognitive measures and the CFQ was investigated through partial correlations controlling the effect of age (a factor previously associated with cognitive performance[7],[8] and cognitive failures[18]). The prior analysis had an exploratory nature and aimed to identify candidate variables for the following analysis. Measures associated to CFQ at p < 0.10 were added in stepwise regression models containing CFQ as the dependent measure and the associated variables as predictors. The stepwise strategy was adopted to avoid colinearity. The significance value for regression models was established at p < 0.05. Procedures were performed in the SPSS 20.0.

RESULTS

[Table 1] presents the demographic profile of the sample. Participants were predominantly young adults with ages between 18 and 26 years (67%). [Table 2] presents the participants’ scores (mean and standard deviation) in the neuropsychological and psychological measures.

Table 1 Demographic characteristics of the sample (n = 60).

Age
18 – 23	34%
23 – 26	33%
27+	33%
Gender
Male	32%
Female	68%
Highest academic degree
Primary	5%
Secondary/technical	58%
College	37%
Mental disorder or neurological disease
No	80%
Yes	20%
Use of medication
No	73%
Yes	27%
Socioeconomic class
A	22%
B1	15%
B2	32%
C1	13%
C2	15%
D-E	3%
Occupation
Studying	32%
Studying and working	30%
Not studying nor working	6%
Working	32%

Table 2 Participants’ scores in the Cognitive Failures Questionnaire and performance in the attentional tests: FDT and OTT.

	Mean	SD
Cognitive failures	44.35	14.91
Cognitive failures (forgetfulness)	15.68	5.03
Cognitive failures (distractibility)	14.63	5.25
Cognitive failures (false triggering)	12.37	5.43
FDT – Reading (time)	22.67	6.62
FDT - Counting (time)	25.43	5.77
FDT – Inhibition (time)	15.97	7.57
FDT – Flexibility (time)	24.82	10.12
OTT – Naming (time)	20.08	4.39
OTT – Visual search (time)	45.52	10.8
OTT – Choosing (time)	29.77	10.43
OTT – Executive search (time)	54.08	13.83
OTT – Shifting (time)	34.65	11.40

FDT: Five Digit Test; OTT: Oral Trails Test; SD: standard deviation.

[Table 3] presents the coefficients of the partial correlation (controlled by age) between the attentional scores measured by the FDT and the OTT and the CFQ factors. Statistical significant correlations between the FDT and the CFQ occurred more consistently in the flexibility measure (step 4) with moderate correlations with CFQ forgetfulness, False Triggering and total score. For the OTT measures, we observed the same pattern with significant correlations between the measures of controlled attention (choosing, executive-search and shifting) and the CFQ factors, except for the association of inhibition with cognitive failures by false alarm. This preliminary analysis showed the controlled/executive attentional processes to be more related to cognitive failures than the automatic attentional process in this sample.

Table 3 Coefficients of partial correlation (controlled by age) between the FDT and OTT scores and the Cognitive Failures Questionnaire scores.

Cognitive tests	Cognitive failures questionnaire
	Total	Forgetfulness	Distractibility	False triggering
FDT – Reading (time)	0.161	0.138	0.047	0.231*
FDT - Counting (time)	0.238*	0.194	0.175	0.232*
FDT – Inhibition (time)	0.180	0.254*	0.173	0.099
FDT – Flexibility (time)	0.314**	0.352***	0.196	0.302**
OTT – Naming (time)	0.153	0.122	0.207	0.100
OTT – Visual search (time)	0.000	0.038	-0.049	0.008
OTT – Choosing (time)	0.233*	0.219*	0.263**	0.177
OTT – Executive search (time)	0.265**	0.231*	0.218*	0.263**
OTT – Shifting (time)	0.312**	0.249*	0.343***	0.261**

FDT: Five Digit Test; OTT: Oral Trails Test. *p < 0.1, **p < 0.05, ***p < 0.01.

In [Table 4] is presented data from the stepwise multiple linear regression model, analyzing the influence of age and the cognitive variables on CFQ factors. Age and FDT flexibility accounted for by 26% of the CFQ total score variance, and for by 19% of the False Triggering factor variance. Age and OTT shifting represented 18% of the variance on cognitive failures by forgetfulness and 25% of the variance on cognitive failures by Distractibility. Results suggest that the cognitive flexibility measured both for the FDT and the OTT was a significant predictor of self-reported cognitive failures in general.

Table 4 Stepwise multiple linear regression model analyzing the contribution of age and neuropsychological tests for cognitive failures.

Outcome	F	p	R2	Predictors	Beta	P
CFQ total score	10.25	< 0.001	26%	Age	0.543	< 0.001
				FDT – flexibility	0.305	0.016
CFQ forgetfulness	7.66	0.001	18%	Age	0.451	< 0.001
				OTT – shifting	0.359	0.006
CFQ distractibility	10.03	< 0.001	25%	Age	0.679	< 0.001
				OTT – shifting	0.404	0.008
CFQ false triggering	7.78	0.001	19%	Age	0.481	< 0.001
				FDT – flexibility	0.303	0.020

FDT: Five Digit Test; OTT: Oral Trails Test; CFQ: Cognitive Failures Questionnaire.

DISCUSSION

In this study, controlled aspects of the cognitive attentional system, measured by the Stroop paradigms FDT and OTT, were associated with the frequency of cognitive failures in everyday life. The cognitive flexibility of both tests explained a medium portion of the variance in most of the measures of the CFQ, even controlling for the age effect. Weaker correlations were observed between the FDT flexibility measure, the Distractibility factor of the CFQ and between the OTT shifting, and the Forgetfulness and False Triggering factors of the CFQ.

Bradbent and Colleagues[16] explain that cognitive failures differ from failures on capacity, which means that cognitive failures are a lapse and a momentary change on normal flux of certain function. However, significant correlations are reported between cognitive failures and performance on ecological measures of attention and memory[19]. Then, cognitive traits may influence cognitive failures in everyday life, yet those cognitive failures not always will be a reflection of the cognitive performance since they are transient, as seen in other studies[20],[21]. This is in accordance with our results, where moderate correlations between the flexibility component and the cognitive failures would reflect an association and not a perfect superposition of this cognitive domain with a more complex behavioral concept. The lack of correlation between the FDT flexibility and the Distractibility factor was unexpected since attentional resources would influence this kind of cognitive failures. It could be possible that transient factors such as the level of daily demands are more relevant for cognitive failures related to Distractibility. For example, people after a significant change in life style, as retirement, may report less distracted behavior[22],[23].

Most important for our purposes, measures of the FDT and the OTT were predictive of real-world problems such as cognitive failures in everyday activities/situations. Human capacity to selectively attend, choose something and focusing on that or suppressing attention of a stimuli is enabled for by attentional control and is driven by the intrinsic properties of a stimuli[2]. Then, the design of a test is of utmost importance for an interference control effect (attentional inhibition). The FDT and OTT, however, seems to go beyond interference control demanding from subjects to perform a dual-task version of the previous parameters of the tests. The flexibility component of both tests was the most predictive of cognitive failures. It has been described that cognitive flexibility is a more complex executive function building on inhibitory and working memory demands[8]. Cognitive flexibility permit us to adjust performance to changed demands. The impact of these executive functions increase with task complexity and daily life demands[8] which help us to understand why lower performance on FDT and OTT was associated with higher levels of cognitive failures. Cognitive failures are expected to increase in a context of ascending amount of information, which intensify the necessity for the identification of predictors of errors that occurs due to slips of memory and lapses of attention. People with high CFQ scores usually bump into things, forget names or lose task goals during task execution and so are more likely to fail during the pursuit of task goals or even to suffer more accidents[24]. Then, the assessment of attentional control and other executive measures could be helpful in contexts of high daily demands with increased risk for errors due to cognitive failures.

This work is limited by its power of generalization since we investigated only healthy adults and only one ecological measure (i.e., cognitive failures) was used as an associated outcome. Further studies should investigate the validity of the FDT and OTT in a larger sample. The effect of FDT and OTT measures on outcomes such as cognitive failures should also be investigated in clinical populations since alterations in the attentional-executive system if often present[25].

Our results suggest that FDT and OTT showed evidences of ecological validity, using as parameter everyday cognitive failures reported by the participants. Ecological validity is an important aspect to be verified on neuropsychological tests, since in many situations they are used to evaluate prognosis or to predict patients’ behavior.

Conflict of interest:

There is no conflict of interest to declare.

Support:

Support: Psychology and Neuropsychology Experimentation Laboratoy (Labep_neuropsi) from Faculdade de Ciências Médicas de Minas Gerais – FCMMG and funded by Fundação Educacional Lucas Machado – FELUMA. INCT-MM Research Grants (FAPEMIG: CBB-APQ-00075-09/CNPq 573646/2008-2).

• References

• 1 Vasconcelos LG, Brucki SMD, Bueno OFA. Cognitive and functional dementia assessment tools: review of Brazilian literature. Dement Neuropsychol. 2007;1(1):18-23.
• 2 Petersen SE, Posner MI. The attention system of the human brain: 20 years after. Annu Rev Neurosci. 2012;35(1):73-89. doi:10.1146/annurev-neuro-062111-150525
• 3 Santos EB, Tudesco Ide S, Caboclo LO, Yacubian EM.Low educational level effects on the performance of healthy adults on a Neuropsychological Protocol suggested by the Commission on Neuropsychology of the Liga Brasileira de Epilepsia. Arq Neuropsiquiatr. 2011;69(5):778-84. doi:10.1590/S0004-282X2011000600011
• 4 Paula JJ, Querino EH, Oliveira TD, Sedó M, Malloy-Diniz LF. Transcultural issues on the assessment of executive functions and processing speed in older adults with low formal education: usefulness of the five digits test in the assessment of dementia. Geriatr Gerontol Int. 2015;15(3):388-9. doi:10.1111/ggi.12364
• 5 Lin GH, Lu Y, Wu CT, Chiu EC, Huang SL, Hsueh IP, Hsieh CL. Psychometric properties of the Five-Digit Test in patients with stroke. Disabil Rehabil. 2015. [Epub ahead of print]. doi:10.3109/09638288.2015.1031288
• 6 Fan J, McCandliss BD, Sommer T, Raz A, Posner MI. Testing the efficiency and independence of attentional networks. J Cogn Neurosci. 2002;14(3):340-7. doi:10.1162/089892902317361886
• 7 Salthouse TA. Aging and measures of processing speed. Biol Psychol. 2000;54(1-3):35-54. doi:10.1016/S0301-0511(00)00052-1
• 8 Diamond A. Executive functions. Annu Rev Psychol. 2013;64(1):135-68. doi:10.1146/annurev-psych-113011-14375
• 9 Alvarez JA, Emory E. Executive function and the frontal lobes: a meta-analytic review. Neuropsychol Rev. 2006;16(1):17-42. doi:10.1007/s11065-006-9002-x
• 10 Cox, CS, Chee E, Chase GA, Baumgardner TL, Schuerholz LJ, Reader MJ, Denckla MB. Reading proficiency affects the construct validity of the Stroop test interference score. Clin Neuropsychol. 1997;11(2):105-10. doi:10.1080/13854049708407039
• 11 Sedó MA. FDT: test de los cinco dígitos. Madrid: TEA; 2007.
• 12 Sedó MA. Trazados orales: um test neurológico multicultural com bajos requerimientos académicos. Int J Psychol Res. 2008;1(1):20-6.
• 13 Reitan RM. Validity of the trail making test as an indicator of organic brain damage. Percept Mot Skills. 1958;8(3):271-6. doi:10.2466/pms.1958.8.3.271
• 14 Chaytor N, Schmitter-Edgecombe M. The ecological validity of neuropsychological tests: a review of the literature on everyday cognitive skills. Neuropsychol Rev. 2003;13(4):181-97. doi:10.1023/B:NERV.1501849483.91468.fb
• 15 Spooner DM, Pachana NA Ecological validity in neuropsychological assessment: a case for greater consideration in research with neurologically intact populations. Arch Clin Neuropsychol. 2006;21(4):327-37. doi:10.1016/j.acn.2006.04.004
• 16 Broadbent DE, Cooper PF, FitzGerald P, Parkes KR. The cognitive failures questionnaire (CFQ) and its correlates. Br J Clin Psychol. 1982;21(1):1-16. doi:10.1111/j.2044-8260.1982.tb01421.x
• 17 Smilek D, Carriere JS, Cheyne JA. Failures of sustained attention in life, lab, and brain: ecological validity of the SART. Neuropsychologia. 2010;48(9):2564-70. doi:10.1016/j.neuropsychologia.2010.05.002
• 18 Rast P, Zimprich D, Van Boxtel M, Jolles J. Factor structure and measurement invariance of the cognitive failures questionnaire across the adult life span. Assessment. 2009;16(2):145-58. doi:10.1177/1073191108324440
• 19 Jung MA1, Kim HJ, Kang YJ, Kim YJ. Development of a modified naturalistic action test for Korean patients with impaired cognition. Ann Rehabil Med. 2013;37(1):57-65. doi:10.5535/arm.2013.37.1.57
• 20 Cockshell SJ, Mathias JL. Cognitive functioning in people with chronic fatigue syndrome: a comparison between subjective and objective measures. Neuropsychology. 2014;28(3):394-405. doi:10.1037/neu0000025
• 21 Burdick KE, Endick CJ, Goldberg JF. Assessing cognitive deficits in bipolar disorder: are self-reports valid? Psychiatry Res. 2005;136(1):43-50. doi:10.1016/j.psychres.2004.12.009
• 22 Gall TL, Evans DR, Howard J. The retirement process: changes in the well-being of male retirees across time. J Gerontol B Psychol Sci Soc Sci. 1997;52B:110-7. doi:10.1093/geronb/52B.3.P110
• 23 Quick HE, Moen P. Gender, employment, and retirement quality: A life course approach to the differential experiences of men and women. J Occup Health Psychol. 1998;3(1):44-64. doi:10.1037/1076-8998.3.1.44
• 24 Wallace JC, Vodanovich SJ. Workplace safety performance: conscientiousness, cognitive failure and their interaction. J Occup Health Psychol. 2003;8(4):316-27. doi:10.1037/1076-8998.8.4.316
• 25 Snyder HR, Miyake A, Hanking BL. Advancing understanding of executive function impairments and psychopathology: bridging the gap between clinical and cognitive approaches. Front Psychol. 2015;26(6):328. doi:10.3389/fpsyg.2015.00328

Publication History

Received: 22 June 2015

Accepted: 17 September 2015

Article published online:
06 September 2023