A tutorial on safe anytime-valid inference: Practical maximally flexible sampling designs for experiments based on e-values
Ly,Alexander Böhm,Udo Grünwald,Peter Ramdas,Aaditya van Ravenzwaaij,Don
Ly,Alexander
Böhm,Udo
Grünwald,Peter
Ramdas,Aaditya
van Ravenzwaaij,Don
Abstract
We demonstrate how e-values simplify both experimental design and the inference process. With e-values researchers can perform anytime-valid tests and construct confidence intervals that maintain type I error control regardless of the sample size. This enables real-time monitoring of evidence as data are collected, permitting early termination of experiments without intolerably inflating the risk of false discoveries. Early stopping not only conserves resources, but also mitigates risk for participants in clinical settings. Anytime-valid tests allow for optional continuation, that is, the extension of an experiment, for instance if more funds become available, or even if the evidence looks promising and the funding agency, a reviewer, or an editor urges the experimenter to collect more data. Analogously, a researcher can be assured that a 95% anytime-valid confidence interval will, with at least95% probability, cover the true effect size regardless of how, or even if, data collection is stopped. We use the free and open-source software package safestats implemented in R to illustrate the practical benefits of this novel inference framework.
Description
Date
2025
Journal Title
Journal ISSN
Volume Title
Publisher
PsyArXiv Preprints
Research Projects
Organizational Units
Journal Issue
Keywords
adaptive sampling design, evidence, reproducible science, research waste reduction, sequential analysis
Citation
Ly, A, Böhm, U, Grünwald, P, Ramdas, A & van Ravenzwaaij, D 2025 'A tutorial on safe anytime-valid inference : Practical maximally flexible sampling designs for experiments based on e-values' Behavior Research Methods, PsyArXiv Preprints. https://doi.org/10.31234/osf.io/h5vae_v3