Summary
Objectives:
Questionnaires used in epidemiological studies should be validated. However, unclarity
exists about the appropriate statistical methods and interpretation of validation
studies. Thus, we investigated the theory and practice of statistical evaluation approaches.
Methods:
Using three platforms, a literature review, own simulations, and a validation study
performed by ourselves, we worked out relevant limitations, advantages, and new important
aspects of evaluation methods.
Results:
Our systematic literature review, based on physical activity questionnaires, revealed
that correlation coefficients are still the common approach in validation studies,
found in 41 of 46 reviewed publications (89.1%). This practice has been criticized
in the theoretically oriented literature for more than 20 years. Appropriate evaluation
methods as recommended by Bland and Altman were found in only ten publications (21.7
%).
We showed that serious bias in questionnaires can be revealed by Bland-Altman plots
but may remain undetected by correlation coefficients. With our simulations we refuted
the argument that correlation coefficients properly investigate whether a questionnaire
ranks the subjects sufficiently well. Further, with Bland-Altman analyses we could
evaluate differential errors with respect to case-control status in our validation
study. Yet, this was not possible with correlation coefficients, because they generally
do not identify systematic bias. In addition, we show a potential pitfall in the interpretation
of Bland-Altman plots that might occur in specific rare instances.
Conclusions:
The commonly used correlation approach can yield misleading conclusions in validation
studies. A more frequent and proper use of the Bland-Altman methods would be desirable
to improve epidemiological data quality.
Keywords
Validation - questionnaire - epidemiological methods - Bland-Altman plot - correlation
