Loading...
Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study
Hardwicke,Tom E. Bohn,Manuel MacDonald,Kyle Hembacher,Emily Nuijten,Michèle B. Peloquin,Benjamin N. deMayo,Benjamin E. Long,Bria Yoon,Erica J. Frank,Michael C.
Hardwicke,Tom E.
Bohn,Manuel
MacDonald,Kyle
Hembacher,Emily
Nuijten,Michèle B.
Peloquin,Benjamin N.
deMayo,Benjamin E.
Long,Bria
Yoon,Erica J.
Frank,Michael C.
Abstract
For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one ‘major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility.
Description
Funding T.E.H.'s contribution was enabled by a general support grant awarded to the Meta-Research Innovation Center at Stanford (METRICS) from the Laura and John Arnold Foundation and a grant from the Einstein Foundation and Stiftung Charité awarded to the Meta-Research Innovation Center Berlin (METRIC-B).
Date
2021
Journal Title
Journal ISSN
Volume Title
Publisher
Research Projects
Organizational Units
Journal Issue
Keywords
AVAILABILITY, journal policy, meta-research, open badges, open data, open science, reproducibility
Citation
Hardwicke, T E, Bohn, M, MacDonald, K, Hembacher, E, Nuijten, M B, Peloquin, B N, deMayo, B E, Long, B, Yoon, E J & Frank, M C 2021, 'Analytic reproducibility in articles receiving open data badges at the journal Psychological Science : An observational study', Royal Society Open Science, vol. 8, no. 1, 201494. https://doi.org/10.1098/rsos.201494