A 5% test is only 5% if you look once, count each user once, and test one metric.

The cost of looking early

A fixed-horizon p-value is valid at the sample size the experiment was designed for, and nowhere else. These are A/A experiments - there is no effect to find - analysed with the same data and the same schedule of looks under two decision rules: the ordinary t-test read repeatedly, and a sequential test whose guarantee holds at every look.

The first row is the control case, where there is nothing to correct and the two rules should agree: 4.9% (±0.3%) against a nominal 5%. By 20 the naive rule reaches 25.3% - one A/A experiment in 4 declared a winner - while msprt never exceeds 1.2% anywhere in the range. Every cell carries its Monte Carlo error, and at these run counts a single cell two or three sigmas from its target is what sampling looks like rather than a finding.

The p-value is not the only casualty. Among the experiments that were stopped as significant, the reported effect is inflated too, because stopping happens precisely on the noisy excursions: a mean absolute effect of 0.175 when peeking against 0.074 with a single look, in a world where the true effect is exactly zero. Both are false positives; the peeked one claims to be more than twice as large.

The cost of counting a user more than once

A/A experiments with no effect anywhere, where each user contributes several rows that move together. Analysed row by row, every extra row looks like extra evidence and the variance of the difference is understated; analysed with a cluster-robust standard error, on the same draws, the rate holds. One row per user is the control case, where the correction must cost nothing at all.

The first row is the control case, where there is nothing to correct and the two rules should agree: 6.3% (±0.5%) against a nominal 5%. By 20 the naive rule reaches 44.0% - one A/A experiment in 2 declared a winner - while cluster-robust standard error never exceeds 6.3% anywhere in the range. Every cell carries its Monte Carlo error, and at these run counts a single cell two or three sigmas from its target is what sampling looks like rather than a finding.

The cost of asking more than one question

A/A experiments measured on several independent metrics at once, each read at alpha. The chance that at least one comes back significant is 1 - (1 - alpha) to the power of the family size - the same arithmetic as peeking, run across metrics instead of across time. Holm holds the family-wise rate at alpha whatever the family size.

The first row is the control case, where there is nothing to correct and the two rules should agree: 5.1% (±0.6%) against a nominal 5%. By 20 the naive rule reaches 65.7% - one A/A experiment in 2 declared a winner - while holm never exceeds 6.0% anywhere in the range. Every cell carries its Monte Carlo error, and at these run counts a single cell two or three sigmas from its target is what sampling looks like rather than a finding.

How do I know this code is right?

Every method is checked twice: against a reference implementation or hand arithmetic in the test suite, and against a simulated world where the truth is known. The second is the one that matters, because it tests the choice of formula and not just its transcription. 10,000 simulated experiments per row, seed 20260721; "MC error" is the standard error of the empirical rate - the noise floor of the run itself.

Note the claim in the last row. A fixed-horizon test promises its false positive rate equals alpha; an anytime-valid test promises only that it stays at most alpha, and the mSPRT is measurably conservative. The first version of the validation script judged it by the equality criterion and reported correct behaviour as a failure - the fix was to make the claim explicit per row, and the rule deciding it now lives on the result object so this page, the script and the test suite cannot disagree about what "pass" means.

What this package will not do for you

Independence is optional, but you have to ask for it. The default tests assume one observation per unit, and nothing detects when that is false. ab_lab.cluster corrects it when you say so - and its own estimator is anti-conservative below about forty units per arm, which the result reports rather than hides.

The mSPRT is conservative. Its measured false positive rate under ten looks is well under the nominal 5%. Validity is bought with power, and a correctly executed group-sequential design would stop sooner - the reasoning for choosing it anyway is in ADR 0003.

The bootstrap's p-value has a floor of 2/(n_resamples+1). A "p < 0.001" read off a 1 000-resample bootstrap is an artefact.

Normal approximations are used for proportions and are unreliable at very low rates with small samples - at least ten successes and ten failures expected per arm is the rule of thumb.

Nothing decides what "the family" is. ab_lab.multiplicity corrects across a set of metrics, but which metrics belong in one family is a judgement, not a computation - and correcting a subset while reading the rest raw controls nothing at all. The library makes you name the members; it cannot make that the right list.

Sample sizes may differ by a few percent from an online calculator, which usually applies the absolute-difference formula rather than Cohen's h. For the same reason a one-point drop and a one-point lift are not the same experiment, so a guardrail metric has to be sized in the direction it can actually move.