Reviewer asks why you did not use a non-parametric test: how to respond

Quick answer. When a reviewer writes “the data are not normally distributed — why not a non-parametric test?”, first decide which of three situations you are in. (1) The outcome is a measurement and you used Welch’s t-test, on log values if it is a positively skewed quantity such as a concentration: keep your analysis and explain, with references, that a rank test answers a different question and that Welch’s test already handles unequal spreads; report a rank test, if at all, as a clearly labelled sensitivity analysis beside the unchanged primary result. If you ran Student’s t-test instead (SciPy’s default), re-run it as Welch’s test and disclose the change as in situation 2. (2) The outcome is an ordinal score (for example a 0–4 histology grade) or the result is driven by a few values that remain extreme on the log scale: the reviewer is right — switch to a rank test (Mann–Whitney for two independent groups; Wilcoxon signed-rank for paired data) and say so, keeping the original result visible. (3) You have three independent animals per group (not three wells): the smallest exact two-sided Mann–Whitney p is 0.10, so explain this and report the t-test estimate with its confidence interval, stating where its normality assumption comes from, because three values cannot check it.

Terms used below. A non-parametric or rank test, such as the Mann–Whitney U test (also called the Wilcoxon rank-sum test), uses only the order of the values. The probability of superiority is the chance that a random value from one group is larger than one from the other, with ties counted as half; the Mann–Whitney test asks whether that probability differs from ½. Welch’s t-test compares means without assuming equal spreads. A ratio of geometric means is what a t-test on log values estimates: the typical fold difference. A sensitivity analysis is a second analysis that checks whether the conclusion depends on a choice; the planned (primary) analysis stays the main result. Exact p-values are computed from all possible rank arrangements rather than from an approximation. Residuals are each value minus its group mean. A QQ plot plots the residuals against the values a normal distribution would give; points close to the diagonal mean roughly normal residuals. The Hodges–Lehmann shift is the median of all treated-minus-control differences. Ties are equal values, which are common in scores. When there are ties or the groups are large, software replaces the exact calculation with a normal approximation (method="asymptotic" in SciPy, exact = FALSE in R); a continuity correction is a small adjustment to that approximation. The Brunner–Munzel test is a rank test that tolerates unequal spreads (scipy.stats.brunnermunzel).

Which situation are you in?

Answer from the study as it was planned, before the p-values were known. If no protocol recorded the test, say so in the reply, describe the test as chosen without a written plan, and report the alternative analysis beside it rather than claiming it was pre-specified.

Question	If yes	If no
Is the outcome an ordinal score (grades, ranks, a short Likert scale)?	Situation 2	Next question
Do one or two extreme values drive the result even after you log-transform a positive quantity? (Check: re-run the planned analysis without them as a labelled sensitivity check; if the conclusion changes, yes.)	Situation 2	Next question
Do you have three independent animals (or fewer) in a group?	Situation 3	Next question
Did you run Student’s t-test rather than Welch’s?	Re-run as Welch’s test, disclose the change as in situation 2, then use situation 1	Situation 1

If the outcome is a measurement (not an ordinal score) and the reviewer’s real concern is unequal spreads between groups, you are in situation 1 as long as you used Welch’s test: Welch’s test is built for that concern.

Then use the section for your situation below. This page covers two independent groups; for the same animals measured twice, the rank alternative is the Wilcoxon signed-rank test (paired vs unpaired t-test).

Situation 1: keep the planned analysis and explain

A log scale needs every value to be above zero; it suits concentrations and fold changes. If some values are zero or below the detection limit, do not add an arbitrary constant: report how many there are per group. Zeros in counts call for a count model (ANOVA assumptions not met). Values below the detection limit are censored, not zero: a pre-specified rank test can include them as tied at the lowest rank, and that choice must be stated in the Methods.

What each test answers. The t-test and the Mann–Whitney test answer different questions. The t-test compares means (or, on log values, geometric means); the Mann–Whitney test is “a test of both location and shape” (Hart 2001), and counter-examples show that it compares the probability of superiority rather than medians (Divine et al. 2018). If your claim is “the treatment roughly doubled IL-6”, only the analysis of means on the log scale estimates that.

Unequal spreads call for Welch’s test. For the Wilcoxon–Mann–Whitney test, “small differences in variances and moderate degrees of skewness can produce large deviations from the nominal type I error rate” (Fagerland & Sandvik 2009, Statistics in Medicine). In a simulation of five two-sample tests for skewed data with unequal variances, a Welch-type t-test computed on the ranks of the data (the “Welch U test”; not Welch’s t-test on raw or log values) was “recommended most frequently”, although no single test was best in every scenario (Fagerland & Sandvik 2009, Contemporary Clinical Trials). The case for Welch’s test as the default is set out in Delacre et al. 2017. If you need a rank method that tolerates unequal spreads, use the Brunner–Munzel test (scipy.stats.brunnermunzel; Mann–Whitney vs t-test).

Be fair about the evidence. Fagerland (2012) concludes that rank tests are most useful for small studies, and that t-tests with confidence intervals should be used in large studies even for heavily skewed data (Fagerland 2012). So base your reply at small n on the question you asked (a difference in means or a fold change) and on the log scale you chose for a positively skewed quantity.

Reply with a residual plot rather than a normality test. Choosing between the t-test and a rank test from a normality test changes the error rates of whichever test is chosen (Rochon et al. 2012); our normality guide shows the effect with unequal spreads. Show the reviewer a residual QQ plot instead, and keep the planned test. Draw it on the scale you analysed (log values here): points near the line support the analysis; a strongly curved pattern or a single far point is worth a sentence in the reply, not a change of test.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats

d = pd.read_csv("il6.csv")                      # columns: group, il6
d["log_il6"] = np.log(d["il6"])
resid = d["log_il6"] - d.groupby("group")["log_il6"].transform("mean")   # residuals
stats.probplot(resid, dist="norm", plot=plt)
plt.savefig("qq_residuals.png", dpi=150)
print("saved qq_residuals.png")

d <- read.csv("il6.csv")                         # columns: group, il6
fit <- lm(log(il6) ~ group, data = d)
png("qq_residuals.png"); qqnorm(resid(fit)); qqline(resid(fit)); dev.off()
cat("saved qq_residuals.png\n")

What to add. A rank test can go into the reply as a sensitivity analysis. Call it pre-specified when your protocol named it, and post hoc otherwise; either way the planned result stays primary, whichever analysis gives the smaller p-value.

Reply (situation 1, rank test pre-specified):

We thank the reviewer. The primary analysis was specified before the data were analysed in [protocol, date]: Welch’s t-test on log-transformed [IL-6], because concentrations vary proportionally and Welch’s test allows unequal variances. A rank test would answer a different question — whether values in one group tend to be larger — rather than estimate the fold change our conclusion concerns (Hart 2001; Divine et al. 2018), and its error rate is sensitive to differences in variance and shape between the groups (Fagerland & Sandvik 2009). As pre-specified, we also ran an exact Mann–Whitney test as a sensitivity analysis: [U = 60, exact p = 0.0019; probability that a treated mouse exceeded a control mouse 0.94]. The conclusion is unchanged. We have added the residual QQ plot as Supplementary Figure [X].

Reply (situation 1, no rank test in the protocol): use the same text, but replace “As pre-specified, we also ran” with “At the reviewer’s request, we ran a post hoc exact Mann–Whitney test”, and add: “The primary analysis and its estimate are unchanged.”

Reply (situation 1, no written protocol at all): replace the first sentence with “We chose Welch’s t-test on log-transformed [IL-6] before seeing the group comparison, because concentrations vary proportionally and Welch’s test allows unequal variances; this choice was not recorded in a dated protocol.” Then continue with the post hoc version above. Keep the Methods sentence free of “specified before analysis”.

Replace every bracketed item with your own details before sending; if your rank test disagrees with the primary result, report both and say so plainly rather than choosing between them.

Situation 2: ordinal scores or outlier-driven results — switch

For an ordinal score, a mean of grades has no clear meaning, and the rank test answers the question that fits the data: do treated animals tend to have higher grades? The reply and Methods sentence below are for two independent groups; for paired scores (the same animal before and after), name the Wilcoxon signed-rank test in both instead of Mann–Whitney. The same applies when one or two extreme values drive a t-test even on a log scale. Switching now changes the primary analysis after the results were seen, so the reply names the original test and its result and labels the change openly.

We thank the reviewer and agree. [The histology grade is an ordinal score / The result was driven by two values that remain extreme on the log scale], so after review we changed the primary comparison to a two-sided Mann–Whitney U test and now report medians and the probability of superiority. Treated mice had higher grades (median [2.5] vs [1.0]; probability that a treated mouse had a higher grade than a control [0.89]; U = [57], p = [0.0078], normal approximation with continuity correction because of ties). The submitted analysis ([t-test], [estimate, 95% CI, p]) is reported in Supplementary Table [X] and labelled as the original analysis. [The conclusion is unchanged. / The conclusion changes: (revised claim).] The Methods and Results have been revised accordingly.

With ties, R 4.3.3 switches to the normal approximation and prints a warning (newer R versions may compute exact p-values with ties; read the method line in your output); the Mann–Whitney guide covers the details. Say so in the reply if the switch changes your conclusion, and revise the claim; for three or more groups, see Kruskal–Wallis vs ANOVA.

Situation 3: three animals per group

With three values per group, ranks can be arranged in only 20 ways, so even complete separation gives an exact two-sided p of 2/20 = 0.10. GraphPad’s documentation gives the same two-sided p of 0.10 for this case (GraphPad Prism statistics guide). Choose four or more animals per group when a rank test is the planned analysis. If software reports p below 0.10 for 3 vs 3, it used a normal approximation, which is not valid at this size.

The floor explains why a rank test cannot help; it does not make the t-test safe. At n = 3 Welch’s t-test still assumes roughly normal values (on the log scale for a positive quantity), and three points cannot check that, so the reply must say where the assumption comes from (n = 3 guide).

Group sizes	Smallest possible exact two-sided p
3 vs 3	0.10
3 vs 4	0.057
4 vs 4	0.029
4 vs 5	0.016
5 vs 5	0.0079

We thank the reviewer. With three animals per group, the smallest p-value an exact two-sided Mann–Whitney test can produce is 0.10, reached when every treated value exceeds every control value. We therefore report the planned Welch’s t-test [on log values], which assumes approximately normal values on that scale; we base this assumption on [prior data from the same assay / the nature of the measurement], because three values per group cannot test it. We report the estimate with its 95% confidence interval and state that with n = 3 per group the study can reliably detect only very large effects.

See the n = 3 guide for which effects such a study can detect.

Same data, three analyses

A constructed example (the ordinal grades in situation 2 are also constructed: download scores.csv): IL-6 (pg/mL) in 8 control and 8 treated mice, drawn from lognormal distributions with a seed fixed in advance and rounded to two decimals so that no two values are tied (download il6.csv).

Analysis	Estimate (95% CI)	Test	p
Welch’s t-test on log values (planned)	Ratio of geometric means 2.17 (1.26 to 3.75)	t(14.0) = 3.05	0.0086
Welch’s t-test on raw values (for comparison)	Difference in means 24.8 pg/mL (5.3 to 44.4)	t(8.4) = 2.91	0.019
Exact Mann–Whitney U test	Probability of superiority 0.94; Hodges–Lehmann shift 18.6 pg/mL (3.4 to 52.5)	U = 60	0.0019

The Hodges–Lehmann shift summarises the data well when the two distributions have a similar shape; here the treated values are much more spread out on the raw scale, so read the shift as a rough summary and the ratio as the better summary. Here all three analyses agree on the direction and on p < 0.05, but they report different things: a fold change, a mean difference in pg/mL, and the chance that a treated mouse exceeds a control mouse. Report the planned analysis in the abstract, and the others where the reviewer asked for them.

import numpy as np
import pandas as pd
from scipy import stats

d = pd.read_csv("il6.csv")
treated = d.loc[d.group == "treated", "il6"].to_numpy()
control = d.loc[d.group == "control", "il6"].to_numpy()

# Planned analysis: Welch's t-test on log values -> ratio of geometric means
res = stats.ttest_ind(np.log(treated), np.log(control), equal_var=False)
ci = res.confidence_interval(0.95)
ratio = np.exp(np.log(treated).mean() - np.log(control).mean())
print(f"ratio {ratio:.2f}, 95% CI {np.exp(ci.low):.2f} to {np.exp(ci.high):.2f}, "
      f"t({res.df:.1f}) = {res.statistic:.2f}, p = {res.pvalue:.4f}")

# Rank-based sensitivity analysis (report it as planned only if your protocol named it)
mw = stats.mannwhitneyu(treated, control, method="exact")
print(f"U = {mw.statistic:.0f}, P(treated > control) = {mw.statistic / (len(treated) * len(control)):.2f}, "
      f"exact p = {mw.pvalue:.4f}")
hl = np.median(np.subtract.outer(treated, control))   # Hodges–Lehmann shift (its exact CI: use R below)
print(f"Hodges-Lehmann shift {hl:.2f} pg/mL")

# Situation 2 example: ordinal grades (scores.csv, columns: group, score; 8 vs 8 mice)
s = pd.read_csv("scores.csv")
st, sc = s.loc[s.group == "treated", "score"], s.loc[s.group == "control", "score"]
mw2 = stats.mannwhitneyu(st, sc, method="asymptotic", use_continuity=True)   # ties: approximation
print(f"grades: median {st.median()} vs {sc.median()}, U = {mw2.statistic:.0f}, "
      f"P(treated > control) = {mw2.statistic / (len(st) * len(sc)):.2f}, p = {mw2.pvalue:.4f}")

d <- read.csv("il6.csv")
treated <- d$il6[d$group == "treated"]
control <- d$il6[d$group == "control"]

# Planned analysis: Welch's t-test (R's default) on log values -> ratio of geometric means
lt <- t.test(log(treated), log(control))
print(round(exp(c(ratio = unname(lt$estimate[1] - lt$estimate[2]), lower = lt$conf.int[1],
                  upper = lt$conf.int[2])), 2))
print(lt$p.value)

# Rank-based sensitivity analysis (report it as planned only if your protocol named it)
wilcox.test(treated, control, exact = TRUE, conf.int = TRUE)

# Situation 2 example: ordinal grades (scores.csv, columns: group, score; 8 vs 8 mice)
s <- read.csv("scores.csv")
wilcox.test(score ~ group, data = transform(s, group = factor(group, c("treated", "control"))),
            exact = FALSE, correct = TRUE)   # ties: normal approximation with continuity correction

Both print a ratio of 2.17 (95% CI 1.26 to 3.75) and an exact Mann–Whitney p of 0.0019; both give the Hodges–Lehmann shift of 18.6 pg/mL, and R adds its exact 95% interval, 3.4 to 52.5. For the grades, both give U = 57 (R prints the statistic as W) and p = 0.0078. To use your own file, keep the column names group and il6 (or score), or change them in the code.

Methods sentences

• Situation 1: “IL-6 was compared between groups with Welch’s t-test on log-transformed values, specified before analysis; results are reported as the ratio of geometric means with its 95% confidence interval. [An exact Mann–Whitney U test was run as a pre-specified / post hoc sensitivity analysis.]”
• Situation 2: “Histology grades (ordinal, 0–4) were compared with a two-sided Mann–Whitney U test (normal approximation with continuity correction because of ties); we report medians and the probability of superiority. [This comparison replaced the submitted t-test after peer review; the original analysis is in Supplementary Table X.]”
• Situation 3: “With three animals per group, groups were compared with Welch’s t-test [on log values]; the smallest attainable exact rank-test p at this size is 0.10. Approximate normality on the log scale was assumed from [prior data / the nature of the measurement].”

Check what an AI assistant ran

If an assistant drafted your analysis or your reply:

1. Did it choose the test from a normality test? Code that runs shapiro and then picks ttest_ind or mannwhitneyu is the two-stage procedure this page advises against; choose the test from the design and the type of outcome.
2. Which t-test, on which scale? SciPy’s ttest_ind runs Student’s t-test by default (pass equal_var=False for Welch’s); R’s t.test runs Welch’s test by default. Check whether the p-value came from raw or log values.
3. What is n? Each value must be an independent animal; values from wells or cells of the same animal are not (what counts as n).
4. Exact or approximate? At small n without ties, check that the Mann–Whitney p-value is exact (method="exact" in SciPy; exact = TRUE in R); with ties, software uses a normal approximation, which the Methods section should state.
5. Does the reply swap the primary result? A drafted reply that reports only the rank test, or calls a post hoc analysis “pre-specified”, must be corrected before you send it.

References

• Delacre M, Lakens D, Leys C (2017). Why psychologists should by default use Welch’s t-test instead of Student’s t-test. International Review of Social Psychology 30(1):92–101. doi:10.5334/irsp.82 (a correction has been published: doi:10.5334/irsp.661)
• Divine GW, Norton HJ, Barón AE, Juarez-Colunga E (2018). The Wilcoxon–Mann–Whitney procedure fails as a test of medians. The American Statistician 72(3):278–286. doi:10.1080/00031305.2017.1305291
• Fagerland MW (2012). t-tests, non-parametric tests, and large studies—a paradox of statistical practice? BMC Medical Research Methodology 12:78. doi:10.1186/1471-2288-12-78
• Fagerland MW, Sandvik L (2009). The Wilcoxon–Mann–Whitney test under scrutiny. Statistics in Medicine 28(10):1487–1497. doi:10.1002/sim.3561
• Fagerland MW, Sandvik L (2009). Performance of five two-sample location tests for skewed distributions with unequal variances. Contemporary Clinical Trials 30(5):490–496. doi:10.1016/j.cct.2009.06.007
• GraphPad Software. The power of nonparametric tests. GraphPad Prism Statistics Guide (accessed 10 November 2026). graphpad.com
• Hart A (2001). Mann-Whitney test is not just a test of medians: differences in spread can be important. BMJ 323(7309):391–393. doi:10.1136/bmj.323.7309.391
• Rochon J, Gondan M, Kieser M (2012). To test or not to test: preliminary assessment of normality when comparing two independent samples. BMC Medical Research Methodology 12:81. doi:10.1186/1471-2288-12-81