Loading...
Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size
Bakker,Marjan Veldkamp,Coosje L.S. van den Akker,Olmo R. van Assen,Marcel A.L.M. Crompvoets,Elise Ong,How Hwee Wicherts,Jelte M.
Bakker,Marjan
Veldkamp,Coosje L.S.
van den Akker,Olmo R.
van Assen,Marcel A.L.M.
Crompvoets,Elise
Ong,How Hwee
Wicherts,Jelte M.
Abstract
In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so.
Description
The data and materials for this study are available at https://osf.io/94f2w/.
Date
2020
Journal Title
Journal ISSN
Volume Title
Publisher
Research Projects
Organizational Units
Journal Issue
Keywords
INCENTIVES, METAANALYSIS, PSYCHOLOGICAL-RESEARCH, PUBLICATION DECISIONS, REPLICABILITY, SCIENCE, STATISTICAL POWER, TESTS, TRUTH
Citation
Bakker, M, Veldkamp, C L S, van den Akker, O R, van Assen, M A L M, Crompvoets, E, Ong, H H & Wicherts, J M 2020, 'Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size', PLOS ONE, vol. 15, no. 7, e0236079. https://doi.org/10.1371/journal.pone.0236079
License
info:eu-repo/semantics/openAccess