Subscribe to RSS
An Exploratory Analysis of the Internal Structure of Test Through a Multimethods Exploratory Approach of the ASQ:SE in BrazilFunding None.
Background A wide range of exploratory methods is available in psychometrics as means of gathering insight on existing data and on the process of establishing the number and nature of an internal structure factor of a test. Exploratory factor analysis (EFA) and principal component analysis (PCA) remain well-established techniques despite their different theoretical perspectives. Network analysis (NA) has recently gained popularity together with such algorithms as the Next Eigenvalue Sufficiency Test. These analyses link statistics and psychology, but their results tend to vary, leading to an open methodological debate on statistical assumptions of psychometric analyses and the extent to which results that are generated with these analyses align with the theoretical basis that underlies an instrument. The current study uses a previously published data set from the Ages & Stages Questionnaires: Social-Emotional to explore, show, and discuss several exploratory analyses of its internal structure. To a lesser degree, this study furthers the ongoing debate on the interface between theoretical and methodological perspectives in psychometrics.
Methods From a sample of 22,331 sixty-month-old children, 500 participants were randomly selected. Pearson and polychoric correlation matrices were compared and used as inputs in the psychometric analyses. The number of factors was determined via well-known rules of thumb, including the parallel analysis and the Hull method. Multidimensional solutions were rotated via oblique methods. R and Factor software were used, the codes for which are publicly available at https://luisfca.shinyapps.io/psychometrics_asq_se/.
Results Solutions from one to eight dimensions were suggested. Polychoric correlation overcame Pearson correlation, but nonconvergence issues were detected. The Hull method achieved a unidimensional structure. PCA and EFA achieved similar results. Conversely, six clusters were suggested via NA.
Conclusion The statistical outcomes for determining the factor structure of an assessment diverged, varying from one to eight domains, which allowed for different interpretations of the results. Methodological implications are further discussed.
Keywordspsychometrics - statistics - multivariate analysis - Ages & Stages Questionnaires - internal structure.
Data and codes are available at https://osf.io/z6gwv/.
Article published online:
11 February 2022
© 2022. Association for Helping Neurosurgical Sick People. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)
Thieme Medical and Scientific Publishers Pvt. Ltd.
A-12, 2nd Floor, Sector 2, Noida-201301 UP, India
- 1 Tukey JW. The future of data analysis. Ann Math Stat 1962; 33 (01) 1-67
- 2 Silberzahn R, Uhlmann EL, Martin DP. et al. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv Methods Pract Psychol Sci 2018; 1 (03) 337-356
- 3 Fried EI. Lack of theory building and testing impedes progress in the factor and network literature. Psychol Inq 2020; 31 (04) 271-288
- 4 Muthukrishna M, Henrich J. A problem in theory. Nat Hum Behav 2019; 3 (03) 221-229
- 5 Wijsen LD, Borsboom D. Perspectives on psychometrics interviews with 20 past psychometric society presidents. Psychometrika 2021; 86 (01) 327-343
- 6 Fabrigar LR, MacCallum RC, Wegener DT, Strahan EJ. Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods 1999; DOI: 10.1037/1082-989X.4.3.272.
- 7 Bandalos DL, Finney SJ. Factor analysis. In: The Reviewer's Guide to Quantitative Methods in the Social Sciences. New York, NY: Routledge; 2018: 98-122
- 8 Lilienfeld SO, Pinto MD. Risky tests of etiological models in psychopathology research: the need for meta-methodology. Psychol Inq 2015; 26 (03) 253-258
- 9 Edwards JR, Bagozzi RP. On the nature and direction of relationships between constructs and measures. Psychol Methods 2000; 5 (02) 155-174
- 10 Watkins MW. Exploratory factor analysis: a guide to best practice. J Black Psychol 2018; 44 (03) 219-246
- 11 Ehrenberg ASC. Some questions about factor analysis. Stat 1962; 12 (03) 191
- 12 Stevens SS. On the theory of scales of measurement. Science (80-) 1946 DOI: 10.1126/science.103.2684.677
- 13 Box GEP. Science and statistics. J Am Stat Assoc 1976; 71 (356) 791
- 14 Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011; 22 (11) 1359-1366
- 15 Anunciação L, Squires J, Clifford J, Landeira-Fernandez J. Confirmatory analysis and normative tables for the Brazilian Ages and Stages Questionnaires: Social-Emotional. Child Care Health Dev 2019; 45 (03) 387-393
- 16 Shi D, Lee T, Maydeu-Olivares A. Understanding the model size effect on SEM fit indices. Educ Psychol Meas 2019; 79 (02) 310-334
- 17 Ortell KK, Switonski PM, Delaney JR. FairSubset: a tool to choose representative subsets of data for use with replicates or groups of different sample sizes. J Biol Methods 2019; 6 (03) e118
- 18 Squires J, Bricker D, Heo K, Twombly E. Identification of social-emotional problems in young children using a parent-completed screening measure. Early Child Res Q 2001; 16 (04) 405-419
- 19 Singh A, Yeh CJ, Boone Blanchard S. Ages and Stages Questionnaire: a global screening scale. Bol Méd Hosp Infant México 2017; 74 (01) 5-12
- 20 Anunciação L, Chen C-Y, Pereira DA, Landeira-Fernandez J. Factor structure of a Social-Emotional screening instrument for preschool children. Psico-USF 2019; 24 (03) DOI: 10.1590/1413-82712019240304.
- 21 Chen C-Y, Xie H, Filgueiras A, Squires J, Anunciação L, Landeira-Fernandez J. Examining the psychometric properties of the Brazilian Ages & Stages Questionnaires-Social-Emotional: use in public child daycare centers in Brazil. J Child Fam Stud 2017; DOI: 10.1007/s10826-017-0770-0.
- 22 Gordon AM, Browne KW. Beginnings and beyond: foundations in early childhood education - Ann Miles Gordon, Kathryn Williams Browne - Google Livros. Belmont, CA: Wadsworth; 2014
- 23 Shulman C. Research and Practice in Infant and Early Childhood Mental Health. Switzerland: Springer International Publishing; 2016
- 24 Garrido LE, Abad FJ, Ponsoda V. A new look at Horn's parallel analysis with ordinal variables. Psychol Methods 2013; 18 (04) 454-474
- 25 Timmerman ME, Lorenzo-Seva U. Dimensionality assessment of ordered polytomous items with parallel analysis. Psychol Methods 2011; 16 (02) 209-220
- 26 Debelak R, Tran US. Comparing the effects of different smoothing algorithms on the assessment of dimensionality of ordered categorical items with parallel analysis. PLoS One 2016; 11 (02) e0148143
- 27 Achim A. Determining the number of factors using parallel analysis and its recent variants: Comment on Lim and Jahng (2019). Psychol Methods 2021; 26 (01) 69-73
- 28 Braeken J, van Assen MALM. An empirical Kaiser criterion. Psychol Methods 2017; 22 (03) 450-466
- 29 Lorenzo-Seva U, Timmerman ME, Kiers HAL. The Hull method for selecting the number of common factors. Multivariate Behav Res 2011; 46 (02) 340-364
- 30 Baghdarnia M, Soreh RF, Gorji R. The comparison of two methods of maximum likelihood (ML) and diagonally weighted least squares (DWLS) in testing construct validity of achievement goals. J Educ Manag Stude 2014; 4 (01) 22-38
- 31 Forero CG, Maydeu-Olivares A, Gallardo-Pujol D. Factor analysis with ordinal indicators: a Monte Carlo study comparing DWLS and ULS estimation. Struct Equ Model A Multidiscip J 2009; 16 (04) 625-641
- 32 Yang-Wallentin F, Joreskog K, Luo H. . Confirmatory factor analysis of ordinal variables with misspecified models. Struct Equ Model A Multidiscip J 2010; 17 (03) 392-423
- 33 Golino HF, Epskamp S. Exploratory graph analysis: a new approach for estimating the number of dimensions in psychological research. PLoS One 2017; 12 (06) e0174035
- 34 R Development Core Team, R Development Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria; 2020
- 35 Schmittmann VD, Cramer AOJ, Waldorp LJ, Epskamp S, Kievit RA, Borsboom D. Deconstructing the construct: a network perspective on psychological phenomena. New Ideas Psychol 2013; 31 (01) 43-53
- 36 Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006; 119 (02) 166.e7-166.e16
- 37 Coulacoglou, Carina, Donald HSaklofske. Psychometrics and Psychological Assessment: Principles and Applications. Elsevier/AP, Academic Press, an imprint of Elsevier, 2017
- 38 Anunciação L, Chieh-yu C, Squires J. et al. Screening for social and emotional delays in young children who live in poverty: a Brazilian example. J Child Dev Disord 2018; 4 (25) 3-6
- 39 Mukherjee SP, Sinha BK, Chattopadhyay AK. Factor analysis. In: Statistical Methods in Social Science Research. Singapore: Springer; 2018: 103-111
- 40 Santos RO, Gorgulho BM, Castro MA, Fisberg RM, Marchioni DM, Baltar VT. Principal component analysis and factor analysis: differences and similarities in nutritional epidemiology application. Rev Bras Epidemiol 2019; 22: e190041
- 41 van Kesteren E-J, Kievit RA. Exploratory factor analysis with structured residuals for brain network data. Netw Neurosci 2021; 5 (01) 1-27
- 42 Di Franco G, Marradi A. Factor Analysis and Principal Component Analysis. Milan, Italy: FrancoAngeli; 2013
- 43 Norris M, Lecavalier L. Evaluating the use of exploratory factor analysis in developmental disability psychological research. J Autism Dev Disord 2010; 40 (01) 8-20
- 44 Mvududu NH, Sink CA. Factor analysis in counseling research and practice. Couns Outcome Res Eval 2013; 4 (02) 75-98
- 45 Preacher KJ, Zhang G, Kim C, Mels G. Choosing the optimal number of factors in exploratory factor analysis: a model selection perspective. Multivariate Behav Res 2013; 48 (01) 28-56
- 46 Woods CM, Edwards MC. 12 factor analysis and related methods. In: Rao CR, Miller JP, Rao DC. eds. Epidemiology and Medical Statistics. 2007: 367-394 DOI: 10.1016/S0169-7161 (07)27012–9
- 47 Holgado-Tello FP, Chacón-Moscoso S, Barbero-García I, Vila-Abad E. Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Qual Quant 2010; 44 (01) 153-166
- 48 Goretzko D, Pham TTH, Bühner M. Exploratory factor analysis: current use, methodological developments and recommendations for good practice. Curr Psychol 2019; DOI: 10.1007/s12144-019-00300-2.
- 49 Lorenzo-Seva U, Ferrando PJ. Not positive definite correlation matrices in exploratory item factor analysis: causes, consequences and a proposed solution. Struct Equ Model 2021; DOI: 10.1080/10705511.2020.1735393.
- 50 Lloret S, Ferreres A, Tomás AH. The exploratory factor analysis of items: guided analysis based on empirical data and software. An Psicol/Ann Psychol 2017; DOI: 10.6018/analesps.33.2.270211.
- 51 Guyon H, Falissard B, Kop J-L. Modeling psychological attributes in psychology - an epistemological discussion: network analysis vs. latent variables. Front Psychol 2017; 8: 798
- 52 McKown C, Gumbiner LM, Russo NM, Lipton M. Social-emotional learning skill, self-regulation, and social competence in typically developing and clinic-referred children. J Clin Child Adolesc Psychol 2009; 38 (06) 858-871
- 53 Thompson RA, Virmani EA. Socioemotional development. In: Encyclopedia of Human Behavior. Burlington, Massachusetts; Elsevier: 2012: 504-511
- 54 Wright JD. International Encyclopedia of the Social & Behavioral Sciences. 2nd ed.. Elsevier; 2015
- 55 Glorfeld LW. An improvement on Horn's parallel analysis methodology for selecting the correct number of factors to retain. Educ Psychol Meas 1995; 55 (03) 377-393
- 56 Steger MF. An illustration of issues in factor extraction and identification of dimensionality in psychological assessment data. J Pers Assess 2006; 86 (03) 263-272
- 57 American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (Eds.) Standards for educational and psychological testing. American Educational Research Association; 2014