Thumbnail Image
Item

The effect of publication bias on the Q test and assessment of heterogeneity

Augusteijn,Hilde E M
; 
van Aert,Robbie C M
; 
van Assen,M.A.L.M.
Abstract
One of the main goals of meta-analysis is to test for and estimate the heterogeneity of effect sizes. We examined the effect of publication bias on the Q test and assessments of heterogeneity as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The present study has two main contributions and is relevant to all researchers conducting meta-analysis. First, we show when and how publication bias affects the assessment of heterogeneity. The expected values of heterogeneity measures H² and I² were analytically derived, and the power and Type I error rate of the Q test were examined in a Monte Carlo simulation study. Our results show that the effect of publication bias on the Q test and assessment of heterogeneity is large, complex, and nonlinear. Publication bias can both dramatically decrease and increase heterogeneity in true effect size, particularly if the number of studies is large and population effect size is small. We therefore conclude that the Q test of homogeneity and heterogeneity measures H² and I² are generally not valid when publication bias is present. Our second contribution is that we introduce a web application, Q-sense, which can be used to determine the impact of publication bias on the assessment of heterogeneity within a certain meta-analysis and to assess the robustness of the meta-analytic estimate to publication bias. Furthermore, we apply Q-sense to 2 published meta-analyses, showing how publication bias can result in invalid estimates of effect size and heterogeneity. (PsycINFO Database Record (c) 2018 APA, all rights reserved).
Description
Date
2019
Journal Title
Journal ISSN
Volume Title
Publisher
Collections
Tilburg University Research Output Collection
Show all metadata
Files
Thumbnail Image
MTO_Augusteijn_the_effect_of_publication_bias_PM_2018_Preprint.pdf
Adobe PDF, 1.53 MB
Research Projects
Organizational Units
Journal Issue
Keywords
DECISIONS, DISTRIBUTIONS, EFFECT SIZE, METAANALYSIS, P VALUES, Q test, RECOMMENDATIONS, VARIANCE ESTIMATORS, extreme homogeneity, heterogeneity, meta-analysis, publication bias
Citation
Augusteijn, H E M, van Aert, R C M & van Assen, M A L M 2019, 'The effect of publication bias on the Q test and assessment of heterogeneity', Psychological Methods, vol. 24, no. 1, pp. 116-134. https://doi.org/10.1037/met0000197
URI
https://hdl.handle.net/20.500.14602/70278
License
info:eu-repo/semantics/closedAccess
Embedded videos