Loading...
Thumbnail Image
Publication

Sequential Experiment Design via Investing

Citations
Altmetric:
Abstract
Online FDR methods have recently been developed to address the need for procedures that maintain FDR control for a sequence of tests when the test statistics are not all known at one time. State-of-the-art online FDR control, ``$alpha$-investing'', methods do not address the need for testing when the cost of data is not negligible. We propose a novel $alpha$-investing method for a setting that takes into account the cost of data sample collection, the sample size choice, and prior beliefs about the probability of rejection. Our specific contributions are a theoretical analysis of the long term asymptotic behavior of $alpha$-wealth in an $alpha$-investing procedure, a generalized $alpha$-investing procedure for sequential testing that simultaneously optimizes sample size and $alpha$-level using game-theoretic principles, and a non-myopic $alpha$-investing procedure that maximizes the expected reward over a finite horizon of tests. Empirical results show that a cost-aware ERO decision rule correctly rejects more false null hypotheses than other methods for a fixed sample size of $n=1$. On real data sets from biological experiments, empirical results show that cost-aware ERO balances the allocation of samples to an individual test against the allocation of samples across multiple tests. A recent perspective on sequential testing, named ``testing by betting'', poses the process as a repeated betting game between the investigator and nature. The investigator's wealth process across the repeated games can be used to provide continuous (time-uniform) control of the false positive rate, termed emph{safe, anytime-valid inference}. We draw a fundamental connection between concepts in mathematical finance and sequential testing by treating the test wealth process as an asset. Our work builds on this notion to construct derivative contracts on the wealth process, in particular, options contracts. These assets and options allow the investigator to hedge against the risk of ruin while maintaining anytime-valid error guarantees, providing the first forward-looking contracts that provably protect against ruin. Empirical results demonstrate that these derivative contracts can eliminate the risk of ruin without significant impact to the test's power. Modern A/B testing platforms offer tools to continuously monitor data and adaptively update the treatment assignment policy. When the purpose of the A/B test is to perform statistical inference of the Average Treatment Effect (ATE), an investigator would like to design an adaptive experiment to minimize uncertainty in the ATE estimate while maintaining time-uniform error control to accommodate data-dependent stopping times. We provide a central limit theorem, under weaker assumptions than previous literature, for a semiparametric efficient Adaptive Augmented Inverse-Probability Weighted estimator, enabling its use in more general settings, and derive both asymptotic and nonasymptotic confidence sequences that are considerably tighter than previous methods while maintaining time-uniform error control.
Type
Dissertation (Open Access)
Date
2024-05
Publisher
License
Attribution 4.0 International
License
http://creativecommons.org/licenses/by/4.0/
Research Projects
Organizational Units
Journal Issue
Embargo Lift Date
2025-05-17
Publisher Version
Embedded videos
Related Item(s)