# Reviewer asks why you did not use a non-parametric test: how to respond

> Decide which of three situations you are in — a measurement analysed with Welch's test, ordinal or outlier-driven data, or three animals per group — then use the matching reply, Methods sentence and code.

Canonical: https://www.licklider.ai/guides/reviewer-asks-why-not-non-parametric/
Section: Guides › Responding to reviewers
Author: Licklider · Published 2026-11-10 · Last reviewed 2026-11-10 (UTC)

> **Quick answer.** When a reviewer writes "the data are not normally distributed — why not a non-parametric test?", first decide which of three situations you are in. **(1) The outcome is a measurement and you used Welch's t-test, on log values if it is a positively skewed quantity such as a concentration:** keep your analysis and explain, with references, that a rank test answers a different question and that Welch's test already handles unequal spreads; report a rank test, if at all, as a clearly labelled sensitivity analysis beside the unchanged primary result. If you ran Student's t-test instead (SciPy's default), re-run it as Welch's test and disclose the change as in situation 2. **(2) The outcome is an ordinal score (for example a 0–4 histology grade) or the result is driven by a few values that remain extreme on the log scale:** the reviewer is right — switch to a rank test (Mann–Whitney for two independent groups; Wilcoxon signed-rank for paired data) and say so, keeping the original result visible. **(3) You have three independent animals per group (not three wells):** the smallest exact two-sided Mann–Whitney p is 0.10, so explain this and report the t-test estimate with its confidence interval, stating where its normality assumption comes from, because three values cannot check it.

**Terms used below.** A *non-parametric* or *rank* test, such as the Mann–Whitney U test (also called the Wilcoxon rank-sum test), uses only the order of the values. The *probability of superiority* is the chance that a random value from one group is larger than one from the other, with ties counted as half; the Mann–Whitney test asks whether that probability differs from ½. *Welch's t-test* compares means without assuming equal spreads. A *ratio of geometric means* is what a t-test on log values estimates: the typical fold difference. A *sensitivity analysis* is a second analysis that checks whether the conclusion depends on a choice; the planned (*primary*) analysis stays the main result. *Exact* p-values are computed from all possible rank arrangements rather than from an approximation. *Residuals* are each value minus its group mean. A *QQ plot* plots the residuals against the values a normal distribution would give; points close to the diagonal mean roughly normal residuals. The *Hodges–Lehmann shift* is the median of all treated-minus-control differences. *Ties* are equal values, which are common in scores. When there are ties or the groups are large, software replaces the exact calculation with a *normal approximation* (`method="asymptotic"` in SciPy, `exact = FALSE` in R); a *continuity correction* is a small adjustment to that approximation. The *Brunner–Munzel test* is a rank test that tolerates unequal spreads (`scipy.stats.brunnermunzel`).

## Which situation are you in?

Answer from the study as it was planned, before the p-values were known. If no protocol recorded the test, say so in the reply, describe the test as chosen without a written plan, and report the alternative analysis beside it rather than claiming it was pre-specified.

| Question | If yes | If no |
| --- | --- | --- |
| Is the outcome an ordinal score (grades, ranks, a short Likert scale)? | Situation 2 | Next question |
| Do one or two extreme values drive the result even after you log-transform a positive quantity? (Check: re-run the planned analysis without them as a labelled sensitivity check; if the conclusion changes, yes.) | Situation 2 | Next question |
| Do you have three independent animals (or fewer) in a group? | Situation 3 | Next question |
| Did you run Student's t-test rather than Welch's? | Re-run as Welch's test, disclose the change as in situation 2, then use situation 1 | Situation 1 |

If the outcome is a measurement (not an ordinal score) and the reviewer's real concern is unequal spreads between groups, you are in situation 1 as long as you used Welch's test: Welch's test is built for that concern.

Then use the section for your situation below. This page covers two independent groups; for the same animals measured twice, the rank alternative is the Wilcoxon signed-rank test ([paired vs unpaired t-test](/guides/paired-vs-unpaired-t-test/)).

## Situation 1: keep the planned analysis and explain

A log scale needs every value to be above zero; it suits concentrations and fold changes. If some values are zero or below the detection limit, do not add an arbitrary constant: report how many there are per group. Zeros in counts call for a count model ([ANOVA assumptions not met](/guides/anova-assumptions-not-met/)). Values below the detection limit are censored, not zero: a pre-specified rank test can include them as tied at the lowest rank, and that choice must be stated in the Methods.

**What each test answers.** The t-test and the Mann–Whitney test answer different questions. The t-test compares means (or, on log values, geometric means); the Mann–Whitney test is "a test of both location and shape" ([Hart 2001](https://doi.org/10.1136/bmj.323.7309.391)), and counter-examples show that it compares the probability of superiority rather than medians ([Divine et al. 2018](https://doi.org/10.1080/00031305.2017.1305291)). If your claim is "the treatment roughly doubled IL-6", only the analysis of means on the log scale estimates that.

**Unequal spreads call for Welch's test.** For the Wilcoxon–Mann–Whitney test, "small differences in variances and moderate degrees of skewness can produce large deviations from the nominal type I error rate" ([Fagerland & Sandvik 2009, *Statistics in Medicine*](https://doi.org/10.1002/sim.3561)). In a simulation of five two-sample tests for skewed data with unequal variances, a Welch-type t-test computed on the ranks of the data (the "Welch U test"; not Welch's t-test on raw or log values) was "recommended most frequently", although no single test was best in every scenario ([Fagerland & Sandvik 2009, *Contemporary Clinical Trials*](https://doi.org/10.1016/j.cct.2009.06.007)). The case for Welch's test as the default is set out in [Delacre et al. 2017](https://doi.org/10.5334/irsp.82). If you need a rank method that tolerates unequal spreads, use the Brunner–Munzel test (`scipy.stats.brunnermunzel`; [Mann–Whitney vs t-test](/guides/mann-whitney-vs-t-test/)).

**Be fair about the evidence.** Fagerland (2012) concludes that rank tests are most useful for small studies, and that t-tests with confidence intervals should be used in large studies even for heavily skewed data ([Fagerland 2012](https://doi.org/10.1186/1471-2288-12-78)). So base your reply at small n on the question you asked (a difference in means or a fold change) and on the log scale you chose for a positively skewed quantity.

**Reply with a residual plot rather than a normality test.** Choosing between the t-test and a rank test from a normality test changes the error rates of whichever test is chosen ([Rochon et al. 2012](https://doi.org/10.1186/1471-2288-12-81)); our [normality guide](/guides/should-you-test-for-normality/) shows the effect with unequal spreads. Show the reviewer a residual QQ plot instead, and keep the planned test. Draw it on the scale you analysed (log values here): points near the line support the analysis; a strongly curved pattern or a single far point is worth a sentence in the reply, not a change of test.

```python
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats

d = pd.read_csv("il6.csv")                      # columns: group, il6
d["log_il6"] = np.log(d["il6"])
resid = d["log_il6"] - d.groupby("group")["log_il6"].transform("mean")   # residuals
stats.probplot(resid, dist="norm", plot=plt)
plt.savefig("qq_residuals.png", dpi=150)
print("saved qq_residuals.png")
```

```r
d <- read.csv("il6.csv")                         # columns: group, il6
fit <- lm(log(il6) ~ group, data = d)
png("qq_residuals.png"); qqnorm(resid(fit)); qqline(resid(fit)); dev.off()
cat("saved qq_residuals.png\n")
```

**What to add.** A rank test can go into the reply as a sensitivity analysis. Call it pre-specified when your protocol named it, and post hoc otherwise; either way the planned result stays primary, whichever analysis gives the smaller p-value.

**Reply (situation 1, rank test pre-specified):**

> We thank the reviewer. The primary analysis was specified before the data were analysed in [protocol, date]: Welch's t-test on log-transformed [IL-6], because concentrations vary proportionally and Welch's test allows unequal variances. A rank test would answer a different question — whether values in one group tend to be larger — rather than estimate the fold change our conclusion concerns (Hart 2001; Divine et al. 2018), and its error rate is sensitive to differences in variance and shape between the groups (Fagerland & Sandvik 2009). As pre-specified, we also ran an exact Mann–Whitney test as a sensitivity analysis: [U = 60, exact p = 0.0019; probability that a treated mouse exceeded a control mouse 0.94]. The conclusion is unchanged. We have added the residual QQ plot as Supplementary Figure [X].

**Reply (situation 1, no rank test in the protocol):** use the same text, but replace "As pre-specified, we also ran" with "At the reviewer's request, we ran a post hoc exact Mann–Whitney test", and add: "The primary analysis and its estimate are unchanged."

**Reply (situation 1, no written protocol at all):** replace the first sentence with "We chose Welch's t-test on log-transformed [IL-6] before seeing the group comparison, because concentrations vary proportionally and Welch's test allows unequal variances; this choice was not recorded in a dated protocol." Then continue with the post hoc version above. Keep the Methods sentence free of "specified before analysis".

Replace every bracketed item with your own details before sending; if your rank test disagrees with the primary result, report both and say so plainly rather than choosing between them.

## Situation 2: ordinal scores or outlier-driven results — switch

For an ordinal score, a mean of grades has no clear meaning, and the rank test answers the question that fits the data: do treated animals tend to have higher grades? The reply and Methods sentence below are for two independent groups; for paired scores (the same animal before and after), name the Wilcoxon signed-rank test in both instead of Mann–Whitney. The same applies when one or two extreme values drive a t-test even on a log scale. Switching now changes the primary analysis after the results were seen, so the reply names the original test and its result and labels the change openly.

> We thank the reviewer and agree. [The histology grade is an ordinal score / The result was driven by two values that remain extreme on the log scale], so after review we changed the primary comparison to a two-sided Mann–Whitney U test and now report medians and the probability of superiority. Treated mice had higher grades (median [2.5] vs [1.0]; probability that a treated mouse had a higher grade than a control [0.89]; U = [57], p = [0.0078], normal approximation with continuity correction because of ties). The submitted analysis ([t-test], [estimate, 95% CI, p]) is reported in Supplementary Table [X] and labelled as the original analysis. [The conclusion is unchanged. / The conclusion changes: (revised claim).] The Methods and Results have been revised accordingly.

With ties, R 4.3.3 switches to the normal approximation and prints a warning (newer R versions may compute exact p-values with ties; read the method line in your output); the [Mann–Whitney guide](/guides/mann-whitney-vs-t-test/) covers the details. Say so in the reply if the switch changes your conclusion, and revise the claim; for three or more groups, see [Kruskal–Wallis vs ANOVA](/guides/kruskal-wallis-vs-anova/).

## Situation 3: three animals per group

With three values per group, ranks can be arranged in only 20 ways, so even complete separation gives an exact two-sided p of 2/20 = 0.10. GraphPad's documentation gives the same two-sided p of 0.10 for this case ([GraphPad Prism statistics guide](https://www.graphpad.com/guides/prism/latest/statistics/stat_the_power_of_nonparametric_tes.htm)). Choose four or more animals per group when a rank test is the planned analysis. If software reports p below 0.10 for 3 vs 3, it used a normal approximation, which is not valid at this size.

The floor explains why a rank test cannot help; it does not make the t-test safe. At n = 3 Welch's t-test still assumes roughly normal values (on the log scale for a positive quantity), and three points cannot check that, so the reply must say where the assumption comes from ([n = 3 guide](/guides/t-test-with-n-3/)).

| Group sizes | Smallest possible exact two-sided p |
| --- | --- |
| 3 vs 3 | 0.10 |
| 3 vs 4 | 0.057 |
| 4 vs 4 | 0.029 |
| 4 vs 5 | 0.016 |
| 5 vs 5 | 0.0079 |

> We thank the reviewer. With three animals per group, the smallest p-value an exact two-sided Mann–Whitney test can produce is 0.10, reached when every treated value exceeds every control value. We therefore report the planned Welch's t-test [on log values], which assumes approximately normal values on that scale; we base this assumption on [prior data from the same assay / the nature of the measurement], because three values per group cannot test it. We report the estimate with its 95% confidence interval and state that with n = 3 per group the study can reliably detect only very large effects.

See the [n = 3 guide](/guides/t-test-with-n-3/) for which effects such a study can detect.

## Same data, three analyses

A constructed example (the ordinal grades in situation 2 are also constructed: [download scores.csv](/guides/reviewer-asks-why-not-non-parametric/scores.csv)): IL-6 (pg/mL) in 8 control and 8 treated mice, drawn from lognormal distributions with a seed fixed in advance and rounded to two decimals so that no two values are tied ([download il6.csv](/guides/reviewer-asks-why-not-non-parametric/il6.csv)).

| Analysis | Estimate (95% CI) | Test | p |
| --- | --- | --- | --- |
| Welch's t-test on log values (planned) | Ratio of geometric means 2.17 (1.26 to 3.75) | t(14.0) = 3.05 | 0.0086 |
| Welch's t-test on raw values (for comparison) | Difference in means 24.8 pg/mL (5.3 to 44.4) | t(8.4) = 2.91 | 0.019 |
| Exact Mann–Whitney U test | Probability of superiority 0.94; Hodges–Lehmann shift 18.6 pg/mL (3.4 to 52.5) | U = 60 | 0.0019 |

The Hodges–Lehmann shift summarises the data well when the two distributions have a similar shape; here the treated values are much more spread out on the raw scale, so read the shift as a rough summary and the ratio as the better summary. Here all three analyses agree on the direction and on p < 0.05, but they report different things: a fold change, a mean difference in pg/mL, and the chance that a treated mouse exceeds a control mouse. Report the planned analysis in the abstract, and the others where the reviewer asked for them.

```python
import numpy as np
import pandas as pd
from scipy import stats

d = pd.read_csv("il6.csv")
treated = d.loc[d.group == "treated", "il6"].to_numpy()
control = d.loc[d.group == "control", "il6"].to_numpy()

# Planned analysis: Welch's t-test on log values -> ratio of geometric means
res = stats.ttest_ind(np.log(treated), np.log(control), equal_var=False)
ci = res.confidence_interval(0.95)
ratio = np.exp(np.log(treated).mean() - np.log(control).mean())
print(f"ratio {ratio:.2f}, 95% CI {np.exp(ci.low):.2f} to {np.exp(ci.high):.2f}, "
      f"t({res.df:.1f}) = {res.statistic:.2f}, p = {res.pvalue:.4f}")

# Rank-based sensitivity analysis (report it as planned only if your protocol named it)
mw = stats.mannwhitneyu(treated, control, method="exact")
print(f"U = {mw.statistic:.0f}, P(treated > control) = {mw.statistic / (len(treated) * len(control)):.2f}, "
      f"exact p = {mw.pvalue:.4f}")
hl = np.median(np.subtract.outer(treated, control))   # Hodges–Lehmann shift (its exact CI: use R below)
print(f"Hodges-Lehmann shift {hl:.2f} pg/mL")

# Situation 2 example: ordinal grades (scores.csv, columns: group, score; 8 vs 8 mice)
s = pd.read_csv("scores.csv")
st, sc = s.loc[s.group == "treated", "score"], s.loc[s.group == "control", "score"]
mw2 = stats.mannwhitneyu(st, sc, method="asymptotic", use_continuity=True)   # ties: approximation
print(f"grades: median {st.median()} vs {sc.median()}, U = {mw2.statistic:.0f}, "
      f"P(treated > control) = {mw2.statistic / (len(st) * len(sc)):.2f}, p = {mw2.pvalue:.4f}")
```

```r
d <- read.csv("il6.csv")
treated <- d$il6[d$group == "treated"]
control <- d$il6[d$group == "control"]

# Planned analysis: Welch's t-test (R's default) on log values -> ratio of geometric means
lt <- t.test(log(treated), log(control))
print(round(exp(c(ratio = unname(lt$estimate[1] - lt$estimate[2]), lower = lt$conf.int[1],
                  upper = lt$conf.int[2])), 2))
print(lt$p.value)

# Rank-based sensitivity analysis (report it as planned only if your protocol named it)
wilcox.test(treated, control, exact = TRUE, conf.int = TRUE)

# Situation 2 example: ordinal grades (scores.csv, columns: group, score; 8 vs 8 mice)
s <- read.csv("scores.csv")
wilcox.test(score ~ group, data = transform(s, group = factor(group, c("treated", "control"))),
            exact = FALSE, correct = TRUE)   # ties: normal approximation with continuity correction
```

Both print a ratio of 2.17 (95% CI 1.26 to 3.75) and an exact Mann–Whitney p of 0.0019; both give the Hodges–Lehmann shift of 18.6 pg/mL, and R adds its exact 95% interval, 3.4 to 52.5. For the grades, both give U = 57 (R prints the statistic as W) and p = 0.0078. To use your own file, keep the column names `group` and `il6` (or `score`), or change them in the code.

## Methods sentences

- **Situation 1:** "IL-6 was compared between groups with Welch's t-test on log-transformed values, specified before analysis; results are reported as the ratio of geometric means with its 95% confidence interval. [An exact Mann–Whitney U test was run as a pre-specified / post hoc sensitivity analysis.]"
- **Situation 2:** "Histology grades (ordinal, 0–4) were compared with a two-sided Mann–Whitney U test (normal approximation with continuity correction because of ties); we report medians and the probability of superiority. [This comparison replaced the submitted t-test after peer review; the original analysis is in Supplementary Table X.]"
- **Situation 3:** "With three animals per group, groups were compared with Welch's t-test [on log values]; the smallest attainable exact rank-test p at this size is 0.10. Approximate normality on the log scale was assumed from [prior data / the nature of the measurement]."

## Check what an AI assistant ran

If an assistant drafted your analysis or your reply:

1. **Did it choose the test from a normality test?** Code that runs `shapiro` and then picks `ttest_ind` or `mannwhitneyu` is the two-stage procedure this page advises against; choose the test from the design and the type of outcome.
2. **Which t-test, on which scale?** SciPy's `ttest_ind` runs Student's t-test by default (pass `equal_var=False` for Welch's); R's `t.test` runs Welch's test by default. Check whether the p-value came from raw or log values.
3. **What is n?** Each value must be an independent animal; values from wells or cells of the same animal are not ([what counts as n](/guides/what-counts-as-n/)).
4. **Exact or approximate?** At small n without ties, check that the Mann–Whitney p-value is exact (`method="exact"` in SciPy; `exact = TRUE` in R); with ties, software uses a normal approximation, which the Methods section should state.
5. **Does the reply swap the primary result?** A drafted reply that reports only the rank test, or calls a post hoc analysis "pre-specified", must be corrected before you send it.

## References

- Delacre M, Lakens D, Leys C (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. *International Review of Social Psychology* 30(1):92–101. [doi:10.5334/irsp.82](https://doi.org/10.5334/irsp.82) (a correction has been published: [doi:10.5334/irsp.661](https://doi.org/10.5334/irsp.661))
- Divine GW, Norton HJ, Barón AE, Juarez-Colunga E (2018). The Wilcoxon–Mann–Whitney procedure fails as a test of medians. *The American Statistician* 72(3):278–286. [doi:10.1080/00031305.2017.1305291](https://doi.org/10.1080/00031305.2017.1305291)
- Fagerland MW (2012). t-tests, non-parametric tests, and large studies—a paradox of statistical practice? *BMC Medical Research Methodology* 12:78. [doi:10.1186/1471-2288-12-78](https://doi.org/10.1186/1471-2288-12-78)
- Fagerland MW, Sandvik L (2009). The Wilcoxon–Mann–Whitney test under scrutiny. *Statistics in Medicine* 28(10):1487–1497. [doi:10.1002/sim.3561](https://doi.org/10.1002/sim.3561)
- Fagerland MW, Sandvik L (2009). Performance of five two-sample location tests for skewed distributions with unequal variances. *Contemporary Clinical Trials* 30(5):490–496. [doi:10.1016/j.cct.2009.06.007](https://doi.org/10.1016/j.cct.2009.06.007)
- GraphPad Software. The power of nonparametric tests. *GraphPad Prism Statistics Guide* (accessed 10 November 2026). [graphpad.com](https://www.graphpad.com/guides/prism/latest/statistics/stat_the_power_of_nonparametric_tes.htm)
- Hart A (2001). Mann-Whitney test is not just a test of medians: differences in spread can be important. *BMJ* 323(7309):391–393. [doi:10.1136/bmj.323.7309.391](https://doi.org/10.1136/bmj.323.7309.391)
- Rochon J, Gondan M, Kieser M (2012). To test or not to test: preliminary assessment of normality when comparing two independent samples. *BMC Medical Research Methodology* 12:81. [doi:10.1186/1471-2288-12-81](https://doi.org/10.1186/1471-2288-12-81)

## Related

- [Mann–Whitney vs t-test](https://www.licklider.ai/guides/mann-whitney-vs-t-test/)
- [Should you test for normality first?](https://www.licklider.ai/guides/should-you-test-for-normality/)
- [Welch's t-test vs Student's t-test](https://www.licklider.ai/guides/welch-vs-student-t-test/)
- [Statistical test chooser](https://www.licklider.ai/tools/statistical-test-chooser/)

## How this page was made

- Written and published by Licklider. Recommendations cite peer-reviewed methods papers and official software documentation listed under References; cited passages were checked against the source text or abstract.
- The worked example uses constructed data (seed fixed in advance). Its numbers were produced by stored code (Python with SciPy 1.18.1; R 4.3.3), and the two implementations agree.
- Separate review passes checked the statistics, the citations and readability before publication.