The Evaluation of Bivariate Mixed Models in Meta-analyses of Diagnostic Accuracy Studies with SAS, Stata and RFinancial support for this study was provided in part by a grant from the Joint Program from the German Research Foundation (DFG) and the German Federal Ministry of Education and Research (BMBF) for meta-analyses for Felicitas Vogelgesang. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.
21 February 2017
accepted: 19 September 2017
02 May 2018 (online)
Background: Meta-analyses require a thoroughly planned procedure to obtain unbiased overall estimates. From a statistical point of view not only model selection but also model implementation in the software affects the results.
Objectives: The present simulation study investigates the accuracy of different implementations of general and generalized bivariate mixed models in SAS (using proc mixed, proc glimmix and proc nlmixed), Stata (using gllamm, xtmelogit and midas) and R (using reitsma from package mada and glmer from package lme4). Both models incorporate the relationship between sensitivity and specificity – the two outcomes of interest in meta-analyses of diagnostic accuracy studies – utilizing random effects.
Methods: Model performance is compared in nine meta-analytic scenarios reflecting the combination of three sizes for meta-analyses (89, 30 and 10 studies) with three pairs of sensitivity/specificity values (97%/87%; 85%/75%; 90%/93%).
Results: The evaluation of accuracy in terms of bias, standard error and mean squared error reveals that all implementations of the generalized bivariate model calculate sensitivity and specificity estimates with deviations less than two percentage points. proc mixed which together with reitsma implements the general bivariate mixed model proposed by Reitsma rather shows convergence problems. The random effect parameters are in general underestimated.
Conclusions: This study shows that flexibility and simplicity of model specification together with convergence robustness should influence implementation recommendations, as the accuracy in terms of bias was acceptable in all implementations using the generalized approach.
** These authors contributed equally to this work.
- 1 Leeflang MMG, Deeks JJ, Gatsonis C, Bossuyt PMM. Systematic Reviews of Diagnostic Test Accuracy. Annals of Internal Medicine 2008; 149 (12) 889-897.
- 2 Chu H, Cole SR. Bivariate meta-analysis with sparse data a generalized linear mixed model approach. Journal of Clinical Epidemiology 2006; 59 (12) 1331-1332.
- 3 Chu H, Guo H, Zhou Y. Bivariate Random Effects Meta-analysis of Diagnostic Studies Using Generalized Linear Mixed Models. Medical Decision Making 2010; 30 (04) 499-508.
- 4 Reitsma JB, Glas AS, Rutjes AWS, Scholten RJPM, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology 2005; 58 (10) 982-990.
- 5 Dahabreh IJ, Trikalinos TA, Lau J, Schmid C. An Empirical Assessment of Bivariate Methods for Meta-Analysis of Test Accuracy. Method Research Report (prepared by Tufts Evidence-based Practice Center. 2012 (12(13)-EHC136-EF2) [cited 2013 May 22]. Available from: www.effectivehealthcare.ahrq.gov/reports/final/cfm
- 6 Menke J. Bivariate Random-effects Meta-analysis of Sensitivity and Specificity with SAS PROC GLIMMIX. Methods of Information in Medicine 2010; 49 (01) 54-64.
- 7 Van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Statistics in Medicine 2002; 21 (04) 589-624.
- 8 Harbord R, Whiting P, Sterne JAC, Egger M, Deeks JJ, Shang A. et al. An empirical comparison of methods for meta-analysis of diagnostic accuracy showed hierarchical models are necessary. Journal of Clinical Epidemiology 2008; 61 (11) 1095-1103.
- 9 Riley RD, Abrams KR, Sutton AJ, Lambert PC, Thompson JR. Bivariate random-effects meta-analysis and the estimation of between-study correlation. BMC Medical Research Methodology 2007; 07 (01) 1-15.
- 10 Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23 (09) 1351-1375.
- 11 Schuetz GM, Zacharopoulou NM, Schlattmann P, Dewey M. Meta-analysis: Noninvasive Coronary Angiography Using Computed Tomography Versus Magnetic Resonance Imaging. Annals of Internal Medicine 2010; 152 (03) 167-177.
- 12 Burton A, Altman DG, Royston P, Holder RL. The design of simulation studies in medical statistics. Statistics in Medicine 2006; 25 (24) 4279-4292.
- 13 Arends LR, Hamza TH, Van Houwelingen JC, Heijenbrok-Kal MH, Hunink MGM, Stijnen T. Bivariate Random Effects Meta-Analysis of ROC Curves. Medical Decision Making 2008; 28 (05) 621-638.
- 14 StataCorp L. Stata Longitudinal-Data/Panel-Data Reference Manual: Release 12. StataCorp LP: 2011
- 15 Dwamena BA. midas: A program for Meta-analytical Integration of Diagnostic Accuracy Studies in Stata. 2007. Division of Nuclear Medicine, Department of Radiology, University of Michigan Medical School; Ann Arbor, Michigan.:
- 16 Rabe-Hersketh S, Skrondal A, Pickles A. GLLAMM Manual. UC Berkeley Division of Biostatistics Working Paper Series. 2004 (Working Paper 160) [cited 2017 Nov 22]. Available from: http://biostats.bepress.com/ucbbiostat/paper160
- 17 bSas Institute. SAS/Stat 9.2 User’s Guide. Cary, North Carolina: SAS Institute; 2008 Chap. 38 – The GLIMMIX Procedure; chap. 56 – The MIXED Procedure; chap. 61 – The NLMIXED Procedure..
- 18 Bates D, Maechler M, Bolker B. Package lme4. 2017 Version 1.1–13..
- 19 Doebler P. Package mada. 2015 Version 0.5.7..
- 20 Rabe-Hesketh S, Skrondal A, Pickles A. Reliable estimation of generalized linear mixed models using adaptive quadrature. The Stata Journal 2002; 02 (01) 1-21.
- 21 Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary roc curve: Data-analytic approaches and some additional considerations. Statistics in Medicine 1993; 12 (14) 1293-1316.
- 22 Zapf A, Hoyer A, Kramer K, Kuss O. Nonparametric meta-analysis for diagnostic accuracy studies. Statistics in Medicine 2015; 034 (29) 3831-3841.
- 23 Schuetz GM, Schlattmann P, Dewey M. Use of 3x2 tables with an intention to diagnose approach to assess clinical performance of diagnostic tests: meta-analytical evaluation of coronary CT angiography studies. BMJ 2012; 10 (345) e6717.
- 24 Hamza T, Arends L, van Houwelingen H, Stijnen T. Multivariate randome effects meta-analysis of diagnostic tests with multiple thresholds. BMC Medical Research Methodology 2009; 09 (01) 73.
- 25 Putter H, Fiocco M, Stijnen T. Meta-Analysis of Diagnostic Test Accuracy Studies with Multiple Thresholds using Survival Methods. Biometrical Journal 2010; 52 (01) 95-110.
- 26 Dwamena BA. Evidence-based Radiology: Step 3 – Diagnostic Systematic Review and Meta-analysis (Critical Appraisal). Seminars in Roentgenology 2009; 44 (03) 170-179.
- 27 Kiernan K, Tao J, Gibbs P. Tips and strategies for mixed modeling with SAS/STAT procedures. Proceedings SAS Global Forum 2012 (Paper 332–2012). 2012
- 28 Diaz M. Performance measures of the bivariate random effects model for meta-analyses of diagnostic accuracy. Computational Statistics and Data Analysis 2015; 83: 82-90.
- 29 Riley RD, Ahmed I, Debray T, Willis BH, Noordzij JP, Higgins J. et al. Summarising and validating test accuracy results across multiple studies for use in clinical practice. Statistics in Medicine 2015; 34 (13) 2081-2103.