Loading...
Times are changing, bias isn't: A meta-meta-analysis on publication bias detection practices, prevalence rates, and predictors in industrial/organizational psychology
Siegel,Magdalena Eder,Junia Sophia Nur Wicherts,Jelte M. Pietschnig,Jakob
Siegel,Magdalena
Eder,Junia Sophia Nur
Wicherts,Jelte M.
Pietschnig,Jakob
Abstract
Effect misestimations plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplines. It has been suggested that particularly in I/O Psychology, appropriate publication bias detection methods are underused. In this meta-meta-analysis, we present prevalence estimates, predictors, and time trends of publication bias in 128 meta-analyses that were published in the Journal of Applied Psychology (7,263 effect sizes, 3,000,000+ participants). Moreover, we reanalyzed data of 87 meta-analyses and applied nine standard and more modern publication bias detection methods. We show that (a) the bias detection method applications are underused (only 41% of meta-analyses use at least one method) but have increased in recent years, (b) those meta-analyses that apply such methods now use more, but mostly inappropriate methods, and (c) the prevalence of potential publication bias is concerning but mostly remains undetected. Although our results indicate somewhat of a trend toward higher bias awareness, they substantiate concerns about potential publication bias in I/O Psychology, warranting increased researcher awareness about appropriate and state-of-the-art bias detection and triangulation. Embracing open science practices such as data sharing or study preregistration is needed to raise reproducibility and ultimately strengthen Psychological Science in general and I/O Psychology in particular.
Description
The research by MS, JSNE, and JP was funded by the Austrian Science Fund (FWF) project number P31002. The research by JMW was supported by a Consolidator Grant 726361 (IMPROVE) from the European Research Council (ERC)
Date
2022
Journal Title
Journal ISSN
Volume Title
Publisher
Files
Research Projects
Organizational Units
Journal Issue
Keywords
dissemination bias, publication bias, meta-meta-analysis, industrial and organizational psychology, decline effect, WORK-FAMILY CONFLICT, COGNITIVE-ABILITY TESTS, MEMBER EXCHANGE LMX, FILE DRAWER PROBLEM, JOB-PERFORMANCE, METAANALYTIC EXAMINATION, 5-FACTOR MODEL, COMPREHENSIVE METAANALYSIS, ORGANIZATIONAL COMMITMENT, CONDUCTING METAANALYSES, SDG 3 - Good Health and Well-being
Citation
Siegel, M, Eder, J S N, Wicherts, J M & Pietschnig, J 2022, 'Times are changing, bias isn't : A meta-meta-analysis on publication bias detection practices, prevalence rates, and predictors in industrial/organizational psychology', Journal of Applied Psychology, vol. 107, no. 11, pp. 2013-2039. https://doi.org/10.1037/apl0000991