Adding more animals after seeing p = 0.07: why it inflates false positives, and how to plan interim looks

Quick answer. If you test, see p = 0.07, add animals and test again, the final p-value no longer means what it says: each extra look gives chance one more opportunity to produce p < 0.05. In our simulation of studies with no real effect, adding 4 mice per group whenever p was between 0.05 and 0.10 raised false positives from 5.0% to 6.1%, and adding 4 at a time until p < 0.05 (up to 20 per group) raised them to 11.7%; analysing the cohorts as blocks did not bring them back to 5%. Before the study, fix the sample size from a power calculation, or plan the interim look in advance as a group sequential design with a stricter threshold at each look (for two equally spaced looks and a two-sided 5% overall rate, Pocock’s threshold of p < 0.0294 at each look). If you already added animals, say so: report the planned n, the first result, why animals were added and how many, and the final estimate with its confidence interval; treat the result as provisional; and analyse the cohort as a block.

Terms used below. Sample size (n): the number of animals per group. Interim look: a test run before all planned animals are in. False positive: a “significant” result when there is no real effect. α: the false-positive rate a test is designed to have, usually 5%. Optional stopping: deciding whether to collect more data after looking at the result. Group sequential design: a design with interim looks planned in advance, each with a stricter threshold so that the overall false-positive rate stays at α. Pocock boundary: one such rule, using the same threshold at every look. Cohort: animals studied at the same time; a second cohort added later can differ from the first in ways unrelated to treatment. Block: a set of animals (here a cohort) within which groups are compared. Power: the chance of detecting an effect that is really there. d: the true difference between groups in standard-deviation units. 95% confidence interval (CI): the range of effects compatible with the data. p-value: how surprising the data would be if there were no real effect. Nominal rate: the false-positive rate a test claims (5% at p < 0.05), which holds only if the analysis was fixed in advance. Two-sided: counting differences in either direction. Futility stop: ending a study early because the interim result makes a useful effect unlikely. Linear model: a regression; here it compares groups while adjusting for cohort. Batch: another word for a cohort, such as animals delivered or tested together. Cohort term: the cohort entered as a block in the model. SD: standard deviation, the spread of values within a group. Student’s t-test: compares two means assuming equal spread; Welch’s t-test does not assume it. Equally spaced looks: interim analyses after equal numbers of new animals. Confirmatory study: one whose design and analysis were fixed in advance to test a stated hypothesis.

What to do, by situation

Situation	What to do	Report
Planning a study	Fix n from a power calculation (how many biological replicates)	n and how it was decided (ARRIVE 2.0 item 2b)
Planning a study, and you want the option to stop early or continue	Group sequential design written in the protocol: when you will look, the threshold at each look, the maximum n	The design, each look’s result and the threshold used
You already added animals after seeing the result	Analyse all animals with cohort as a block; report both looks and why animals were added; call the result provisional	Planned n, first p, number added, final estimate with CI
You want a confirmatory answer after an inconclusive study	A new, independent experiment with n fixed in advance; the first study informs the effect size	Both experiments, each with its own result
Animals lost or a batch failed (reason unrelated to the result)	Replacing them by a rule written before seeing outcomes is not optional stopping; excluding or replacing animals because their values weakened the result is optional stopping	The losses, the replacement rule and when it was set

ARRIVE 2.0 item 2b asks you to “explain how the sample size was decided”, and item 19 to describe the rationale for any deviations from the protocol “so that readers can take this into account when assessing the findings” (Percie du Sert et al. 2020).

Why one more look changes the error rate

Armitage and colleagues showed that “if significance tests at a fixed level are repeated at stages during the accumulation of data the probability of obtaining a significant result when the null hypothesis is true rises above the nominal significance level” (Armitage et al. 1969). Simmons and colleagues simulated the practice directly: a researcher who adds 10 observations per condition after a non-significant result “increases the false-positive rate by approximately 50%” (Simmons et al. 2011). The practice is common: in a survey of psychologists, about 56% admitted “deciding whether to collect more data after looking to see whether the results were significant” (John et al. 2012). For animal research, Neumann and colleagues describe how interim analyses are applied informally when “a statistically significant effect is desired but not found”, and conclude that “this practice inflates false-positive rates” (Neumann et al. 2017).

Worked example: p = 0.084 with 8 mice per group, p = 0.042 with 12

Constructed data: time on a rotarod in a control and a drug group. The plan was 8 mice per group; after p = 0.08, 4 more per group were tested as a second cohort. The data were generated with a real drug effect of 4 s and an SD of 8 s in both groups (d = 0.5), so the effect exists; the problem is what the final p-value can claim. By chance, the drug group in cohort 1 was unusually tight (SD 7.29 s in controls, 2.01 s with the drug), which is why Welch’s test is shown too; the tests below are Student’s t-tests, which match how the data were generated, and Welch’s t-test gives the same pattern (p = 0.099, then 0.043). The random seed was chosen to give this sequence of results: seed 58 was the first from 1 upward giving a first-look p between 0.06 and 0.09 and an enlarged-sample p between 0.025 and 0.045. Stored code and data: rotarod.csv, make_data.py, analysis.py, analysis.R and the analysis output; analysis.py and analysis.R print the same output.

Analysis	Result
Mean (SD), cohort 1	Control 45.88 (7.29) s, drug 50.86 (2.01) s
Planned analysis, 8 per group	Difference 4.98 s, p = 0.084
After adding 4 per group (unplanned), Student’s t-test	Difference 5.03 s, p = 0.042 (not adjusted for the interim look, so not a 5% test)
Both cohorts, cohort as a block	Difference 5.03 s (95% CI 0.14 to 9.91), p = 0.044 (adjusted for cohort, not for the interim look; provisional)
Second cohort alone, 4 per group	Difference 5.12 s, p = 0.34 (far too small to interpret; not evidence of no effect)

The estimate barely moved; what changed is that a second look was taken because the first fell short. The honest summary is the estimate with its CI, both looks, and the reason for adding animals; the test that produced p = 0.044 does not have a nominal 5% false-positive rate, because the decision to add animals depended on the first result. Treat it as evidence to confirm in an independent experiment with n fixed in advance. The cohort term removes any shift between cohorts (a different delivery batch or week); these constructed cohorts have none, so the blocked and simple analyses agree, but with real batches they often differ.

In 100,000 simulated experiments for each strategy (simulation.py, the simulation output):

Strategy	False positives (no real effect; nominal 5%)	Power (d = 1)	Mean mice per group (d = 1)
Fixed n = 8	5.0%	46.4%	8.0
Fixed n = 16	5.1%	78.1%	16.0
n = 8; if 0.05 ≤ p < 0.10, add 4 and test again	6.1%	56.6%	8.6
The same rule, with the second test using cohort as a block	6.1%	56.5%	8.6
n = 8; while p ≥ 0.05, add 4, up to 20	11.7%	89.9%	12.1
Planned two-stage design (8, then 16), p < 0.0294 at each look	5.1%	72.7%	13.1

The worked example’s effect is d = 0.5, smaller than the d = 1 used for the power columns, so the worked example’s power is lower than any figure shown. The planned two-stage design kept the false-positive rate at its nominal 5% (5.1% in simulation) and, when the effect was real, used on average 18% fewer mice than a fixed 16 per group, at 72.7% instead of 78.1% power; for equal power, the maximum n of a sequential design has to be somewhat larger than the fixed n. Neumann and colleagues report a similar saving for animal studies: with d = 1 and n = 18 per group, “sequential frequentist analysis consumes in the long run only around 80% of the planned number of experimental units” (Neumann et al. 2017). For equally spaced looks, a two-sided overall α of 0.05, and a design written into the protocol in advance, the Pocock threshold is p < 0.0294 at each of two looks and p < 0.0221 at each of three (Lakens 2014; Neumann et al. 2017); our pocock.py reproduces both thresholds (the Pocock output). For other spacings or rules, compute the thresholds with software such as the R packages rpact or gsDesign.

# Condensed: analysis.py also prints the two simple t-tests and the second cohort alone
import pandas as pd
import statsmodels.formula.api as smf

d = pd.read_csv("rotarod.csv")                       # columns: mouse, group, cohort, rotarod_s; one row per mouse
d["group"] = pd.Categorical(d["group"], categories=["control", "drug"])   # control is the reference level
fit = smf.ols("rotarod_s ~ C(group) + C(cohort)", d).fit()          # cohort as a block
print(fit.params["C(group)[T.drug]"], fit.conf_int().loc["C(group)[T.drug]"].values, fit.pvalues["C(group)[T.drug]"])

d <- read.csv("rotarod.csv")
d$group <- factor(d$group, levels = c("control", "drug")); d$cohort <- factor(d$cohort)
fit <- lm(rotarod_s ~ group + cohort, data = d)      # cohort as a block
coef(summary(fit))["groupdrug", ]; confint(fit)["groupdrug", ]
# Planning a group sequential design: the rpact or gsDesign packages compute boundaries for any number of looks

Before you start the study

1. Write the sample size and how you chose it in the protocol or ethics application.
2. If you want an interim look, write the whole rule in advance: when (after how many animals), the threshold at each look, whether you may stop for futility, and the maximum n. For two or three equally spaced looks with Pocock boundaries, use p < 0.0294 or p < 0.0221 at each look; for anything else, compute the thresholds with software (pocock.py above, or rpact or gsDesign in R).
3. Keep the person who decides whether to continue blind to the results by group, or let the planned rule decide.
4. Plan cohorts as blocks. If animals will come in batches anyway, include every group in every batch and analyse the batch (cohort) as a block (randomisation and blinding).
5. Replacing lost animals by a rule set before outcomes are seen (for example a failed surgery) is legitimate; say so. Excluding or replacing animals because their values weakened the result is optional stopping, not replacement.

Check what an AI assistant ran

1. An assistant suggests adding a few more animals because p = 0.07, with no rule planned in advance. Ask for the false-positive rate of that procedure, or a group sequential design for the next study.
2. A final p-value is reported as if n had been planned. Ask for both looks and the planned n in the Methods.
3. New animals are pooled without a cohort term. Ask for cohort as a block.
4. A pilot study is merged into the main study. Keep the pilot separate; use it only to choose the main study’s n.

Report it

If animals were added after an interim result:

The planned sample size was [8] mice per group, based on [a power calculation for a difference of X s, the smallest difference worth detecting, taken from a pilot, published data or judgement; say which]. After the planned analysis (p = [0.084]), [4] further mice per group were tested as a second cohort; this addition was not specified in advance. All mice were analysed together with cohort as a block (linear model; [software and version]). Because the decision to add animals depended on the first result, the p-value below does not come from a test with a nominal 5% false-positive rate and the result should be considered provisional.

The drug increased time on the rotarod by [5.0] s (95% CI [0.1] to [9.9]; p = [0.044], adjusted for cohort but not for the interim look, n = [12] per group in two cohorts); because animals were added after an interim p of [0.084], this is not a nominal 5% test and the result is provisional.

For a planned two-stage design:

A two-stage group sequential design was specified in advance: an interim analysis after [8] mice per group and a final analysis when [16] mice per group had been tested in total (the [8] from the interim plus [8] more), each at p < 0.0294 (Pocock boundary, overall two-sided α = 0.05).

Replace the bracketed values with your own.

References

• Percie du Sert N, Ahluwalia A, Alam S, et al. (2020). Reporting animal research: explanation and elaboration for the ARRIVE guidelines 2.0. PLOS Biology 18:e3000411. doi:10.1371/journal.pbio.3000411
• Armitage P, McPherson CK, Rowe BC (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society A 132:235–244. doi:10.2307/2343787
• Simmons JP, Nelson LD, Simonsohn U (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:1359–1366. doi:10.1177/0956797611417632
• John LK, Loewenstein G, Prelec D (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23:524–532. doi:10.1177/0956797611430953
• Neumann K, Grittner U, Piper SK, et al. (2017). Increasing efficiency of preclinical research by group sequential designs. PLOS Biology 15:e2001307. doi:10.1371/journal.pbio.2001307
• Lakens D (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology 44:701–710. doi:10.1002/ejsp.2023