Reviewer asks you to justify your sample size: how to respond

Quick answer. First decide which situation you are in. (1) You calculated the sample size before the study: give every number the calculation used — test, effect size and why it matters, the SD and where it came from, α, power and the resulting n — so anyone can repeat it. (2) You did not: say so plainly, give the real reason for your n (for example the animals or time available, or previous studies), and add a sensitivity analysis: the smallest effect your n detects with 80% power, next to the confidence interval for the effect you found. (3) The reviewer asks for the power of the completed study: do not compute “observed power” from your own result — it is a restatement of your p-value — and offer the sensitivity analysis instead. Templates for each are below.

Terms used below. Power is the chance that a study detects an effect of a given size if it is real. An a priori calculation is done before the data are collected. A sensitivity analysis fixes the sample size you had and asks which effects it detects with a chosen power. Observed (or post hoc) power plugs the effect you found back into a power formula. d is a difference divided by the standard deviation (SD); d = 1 means the groups differ by one SD. The confidence interval (CI) is the range of effects compatible with your data. α is the false-positive rate you accept, usually 0.05; two-sided means the test looks for a difference in either direction. The pooled SD is the SD of both groups combined (code below).

Which situation are you in?

If you cannot find the original numbers, use Situation 2. Reconstructing a calculation afterwards and presenting it as planned is exactly what the comment is meant to catch.

Question	If yes	If no
Did you calculate n before collecting data, and can you find the numbers you used (protocol, ethics application, grant, dated notebook)?	Situation 1	Go on to the next question
Is the reviewer asking for the power of the study you already ran, or calling it “underpowered” because a result was not significant?	Situation 3; if you also had no calculation, use the combined reply at the end of Situation 3	Situation 2

What the reviewer is checking

ARRIVE 2.0, the reporting guideline for animal research, item 2b: “Explain how the sample size was decided. Provide details of any a priori sample size calculation, if done” (ARRIVE item 2b). Its explanation is direct about the case without a calculation: “If no power calculation was used to determine the sample size, state this explicitly and provide the reasoning that was used to decide on the sample size per group” (Percie du Sert et al. 2020).

So the reviewer wants a reason they can check, not a particular number. Published calculations can fail that test: of 95 published sample size calculations done in G*Power, an audit reproduced 2 (2%) without making assumptions (Thibault et al. 2024, preprint, not peer reviewed).

What to do: answer with the reason and its numbers, and add one sentence to Methods. Every template below does both.

Situation 1: you calculated n in advance

Report each item the ARRIVE explanation lists for an a priori calculation (Percie du Sert et al. 2020), plus the result:

Item	Example
Analysis method	Two-sided two-sample t-test, α = 0.05
Effect size of interest, and why that size matters	15 U/L increase in ALT, the smallest change we would treat as liver injury
Variability and where it came from	SD 8 U/L, from the vehicle group of our previous study (ref.)
Power	80%
Result, and any allowance for losses	6 per group (d = 1.88; achieved power 83%); 7 per group allowing one loss

You can check the numbers with the sample size calculator, which prints the same items as a paragraph.

Reply. We thank the reviewer. The sample size was calculated before the study, and we have added the details to Methods: a two-sided two-sample t-test at α = 0.05 with 80% power to detect a 15 U/L difference in ALT, assuming an SD of 8 U/L from the vehicle group of [previous study], which required 6 mice per group (achieved power 83%) (Methods, line X).

Methods. Sample size was calculated a priori for a two-sided two-sample t-test (α = 0.05, power 80%) to detect a difference of 15 U/L with an SD of 8 U/L ([source]); this required 6 mice per group (achieved power 83%).

Copy both only if you ran this calculation before collecting data and can point to the record (protocol, ethics application, grant, dated notebook); then replace every bracketed item and example value with the ones from that record. If you cannot, use Situation 2.

Situation 2: you did not calculate n

Say so; it is allowed, and inventing a calculation is not. Then name the reason you actually had. Lakens (2022) lists six ways to justify a sample size (Lakens 2022):

Justification	What to write
Measuring (almost) the entire population	“All 14 animals of the line were studied.”
Resource constraints	The animals, time, budget or samples available, and why more were not possible
A priori power analysis	Situation 1
Planning for accuracy	The CI width you aimed for
Heuristics	A convention, such as a journal minimum or a previous study’s n; name it
No justification	Say so explicitly

Then add the sensitivity analysis: it tells the reviewer, and your reader, which effects the study was large enough to detect.

Worked example. Plasma ALT in 6 vehicle and 6 treated mice (constructed data): vehicle mean 47.2 U/L (SD 7.9), treated 58.2 U/L (SD 8.1). Welch’s t-test (the unequal-variance t-test) gives a difference of 11.0 U/L, 95% CI 0.7 to 21.3, t(9.99) = 2.37, p = 0.039.

With 6 per group, a two-sided test at α = 0.05 has 80% power only for differences of d = 1.80 or more. With the pooled SD of these data (8.04 U/L), that is 14.4 U/L; an SD from 12 animals is itself imprecise, so if a larger previous study or the literature gives an SD for this measure, use that one and name its source. The power formula is the standard equal-variance one used by power software and by our calculator; for Welch’s test, which we ran, power at that effect was 79% in our simulation (200,000 runs).

n per group	Smallest d with 80% power	with 90% power
3	3.07	3.59
4	2.38	2.77
5	2.02	2.35
6	1.80	2.08
8	1.51	1.74
10	1.32	1.53
12	1.20	1.39
15	1.06	1.23
20	0.91	1.05

Two-sample t-test, two-sided α = 0.05, equal group sizes. Python (SciPy) and R (pwr) agree to three decimals, and the values match the sample size calculator.

Read the example honestly: the effect was detected (a significant result), but the study had 80% power only for differences of about 14 U/L or more, and the CI runs from 0.7 to 21.3 U/L. The reply says that rather than hiding it. If your result was not significant, the same reply works: your CI will include zero, so say which effects remain compatible with the data (see Situation 3).

Get the difference, its 95% CI and the pooled SD from your data:

import numpy as np
from scipy import stats

vehicle = np.array([38.0, 45.0, 52.0, 41.0, 60.0, 47.0])   # replace with your values (U/L)
treated = np.array([55.0, 49.0, 71.0, 58.0, 64.0, 52.0])

res = stats.ttest_ind(treated, vehicle, equal_var=False)    # Welch's t-test
ci = res.confidence_interval(0.95)
print(round(treated.mean() - vehicle.mean(), 1), round(ci.low, 1), round(ci.high, 1))   # 11.0 0.7 21.3
sd_pooled = np.sqrt((vehicle.var(ddof=1) + treated.var(ddof=1)) / 2)                   # equal group sizes
print(round(sd_pooled, 2))                                                             # 8.04

vehicle <- c(38, 45, 52, 41, 60, 47)   # replace with your values (U/L)
treated <- c(55, 49, 71, 58, 64, 52)

res <- t.test(treated, vehicle)        # Welch's t-test (R's default)
round(c(mean(treated) - mean(vehicle), res$conf.int), 1)   # 11.0 0.7 21.3
round(sqrt((var(vehicle) + var(treated)) / 2), 2)           # pooled SD 8.04

Run it for your own n and SD:

from statsmodels.stats.power import TTestIndPower

d = TTestIndPower().solve_power(effect_size=None, nobs1=6, alpha=0.05, power=0.80,
                                ratio=1.0, alternative="two-sided")
print(round(d, 2))              # smallest standardized difference detectable with 80% power
print(round(d * 8.04, 1))       # in U/L, using the pooled SD of your data (8.04 here)

library(pwr)
d <- pwr.t.test(n = 6, power = 0.80, sig.level = 0.05, type = "two.sample")$d
round(d, 2)          # smallest standardized difference detectable with 80% power
round(d * 8.04, 1)   # in U/L, using the pooled SD of your data (8.04 here)

Both print 1.8 and 14.4. For paired designs, use TTestPower in Python or type = "paired" in R, with the SD of the differences.

Reply. We thank the reviewer. No formal sample size calculation was performed; the group size of 6 was set by [the number of animals available from the breeding cohort / the group size used in reference X]. We now state this in Methods. To show what the study could detect, we added a sensitivity analysis: with 6 mice per group, a two-sided t-test at α = 0.05 had 80% power to detect a difference of d = 1.8, about 14 U/L given the observed SD. The observed difference was 11.0 U/L (95% CI 0.7 to 21.3), and we have revised the Discussion to note that the size of the increase is estimated imprecisely (line X).

Methods. No a priori sample size calculation was performed; group size (n = 6) was determined by [reason]. In a sensitivity analysis (two-sided two-sample t-test, α = 0.05), this group size gave 80% power to detect standardized differences of d ≥ 1.8, about 14 U/L at an assumed SD of 8 U/L ([source of the SD]); the analysis itself used Welch’s t-test.

Copy both, then replace the bracketed reason and the numbers with your own.

Situation 3: the reviewer asks for the study’s power

“Observed power” uses the effect you found to compute power, and it adds nothing beyond your p-value. Hoenig and Heisey (2001) call post-experiment power calculations used to interpret results “fundamentally flawed” (Hoenig & Heisey 2001). The table shows why, for a two-sample t-test with 6 per group:

Two-sided p	Observed power (two-sample t-test, 6 per group)
0.01	81%
0.05	52%
0.10	37%
0.20	24%
0.50	10%

For a given test and sample size, each p-value maps to exactly one observed power, so a non-significant result always comes with low observed power. The CI and the sensitivity analysis show which real effects the study would have detected; observed power adds nothing to them. Bacchetti (2010) adds that reports of completed studies should focus on estimates and confidence intervals rather than discuss power (Bacchetti 2010).

Keep the courteous tone: the reviewer’s concern, that a non-significant result hides a real effect, is valid; only the tool is wrong. For how to word a non-significant result, see what a non-significant p-value means.

Reply. We thank the reviewer. Power computed from the observed effect is determined by the p-value and so cannot indicate whether a true effect was missed (Hoenig & Heisey 2001). We instead report (i) the 95% confidence interval for the difference, [lower to upper, in units], which shows the effects compatible with our data, and (ii) a sensitivity analysis: with 6 mice per group the study had 80% power for differences of d = 1.8 (about 14 U/L). The data remain compatible with smaller effects, and we now state this in the Discussion (line X).

Copy the reply and put your own interval and detectable effect in the brackets.

Combined reply (no calculation was done, and the reviewer asks for power). We thank the reviewer. No a priori sample size calculation was performed; group size (n = 6) was set by [reason], and we now state this in Methods. Power computed from the observed effect is determined by the p-value and cannot show whether a true effect was missed (Hoenig & Heisey 2001). We instead report the 95% confidence interval for the difference, [lower to upper, in units], and a sensitivity analysis: with 6 mice per group the study had 80% power for differences of d = 1.8 (about [d × SD] [units] at an SD of [SD] from [source]) (line X).

If the reviewer asks for more animals

Extra animals added because the first result missed raise the false-positive rate, whichever way you analyse them, unless the extension follows a stricter threshold set for two looks (last row). In our simulations (no true difference):

What you do	False positives
Test the original 6 per group once	4.9%
If p ≥ 0.05, add 3 per group to the same experiment and test again	7.4%
If p ≥ 0.05, run a second experiment of 6 per group and test it alone	9.6%
If p ≥ 0.05, run a second experiment and analyse both with experiment as a factor	8.4%
Same, but use p < 0.029 at each look (Pocock’s two-look boundary)	5.1%

The first two rows use Welch’s t-test (200,000 runs); the others use Student’s t-test and the combined model (100,000 runs). Pocock’s boundary for two equally spaced looks is a nominal two-sided 0.0294 at each look (Pocock 1977); we recomputed it and simulated the rate above.

What to do: choose one of two honest routes, and state in Methods that the extension was decided after the first result. (a) Keep the original analysis as the primary result, reported as it was, and present the new experiment as a replication, judged on its own; do not report a finding because the replication alone reached p < 0.05 after the first missed (that is the 9.6% row). (b) Treat the two experiments as a planned two-look design and use the stricter threshold at each look (p < 0.029 for two looks), reporting both looks.

Check what an AI assistant wrote

If an assistant drafted your reply or ran the power calculation, check four things:

1. Did it compute observed power? Look for power calculated from your measured effect size or p-value (for example solve_power(effect_size=<your observed d>, nobs1=..., power=None)). Replace it with the sensitivity analysis.
2. Is every parameter stated? Test, sidedness, α, power, effect size, SD and its source, n.
3. Does the n match your experimental unit? The calculation is per independent unit: animals, not cells or wells (what counts as n).
4. Does it claim a calculation you never did? Delete it and use Situation 2.

Then send the reply together with the revised Methods sentence.

References

• Percie du Sert N, et al. (2020). Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0. PLoS Biology 18:e3000411. doi:10.1371/journal.pbio.3000411
• ARRIVE guidelines, item 2 (sample size). arriveguidelines.org
• Lakens D (2022). Sample size justification. Collabra: Psychology 8:33267. doi:10.1525/collabra.33267
• Hoenig JM, Heisey DM (2001). The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician 55:19–24. doi:10.1198/000313001300339897
• Bacchetti P (2010). Current sample size conventions: flaws, harms, and alternatives. BMC Medicine 8:17. doi:10.1186/1741-7015-8-17
• Pocock SJ (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika 64:191–199. doi:10.1093/biomet/64.2.191
• Thibault RT, et al. (2024). An evaluation of reproducibility and errors in published sample size calculations performed using G*Power. medRxiv (preprint, not peer reviewed). doi:10.1101/2024.07.15.24310458