Kruskal–Wallis vs one-way ANOVA: which to use, and what each tells you

Quick answer. For three or more independent groups, pick the test that answers your question. Kruskal–Wallis asks whether values in some groups are systematically higher than in others; use it for ordinal scores (histology grades, clinical scores) or when ranks are the scale you care about, and follow it with Dunn’s test with Holm’s adjustment. It compares medians only when the groups have the same shape and spread. For positive, right-skewed measurements such as concentrations, analyse the logarithms with Welch’s ANOVA instead, which gives you ratios with confidence intervals. Neither rank test nor classical ANOVA fixes unequal spreads: with the most variable group also the smallest (groups of 12, 8 and 4; SD four times larger in the group of 4), Kruskal–Wallis gave 11% false positives in our simulation, against 6% for Welch’s ANOVA.

Terms used below. Independent groups: different animals in each group, one value per animal. Rank: an observation’s position when all values are sorted (1 = smallest). Ordinal score: a category with an order but no fixed distance between steps (grade 0–4). Spread: how variable a group is (its SD). False-positive rate: how frequently a test gives p < 0.05 when there is no real difference. Holm’s adjustment: a correction that keeps the chance of any false positive across several comparisons at 5%. Geometric mean: the back-transformed mean of the logarithms; for skewed positive data it is close to the median. Dunn’s test: pairwise comparisons that reuse the ranks from the Kruskal–Wallis test. Welch’s ANOVA: a one-way ANOVA that does not assume equal spreads. Games–Howell: its pairwise follow-up. H: the Kruskal–Wallis statistic, compared with a chi-square distribution with (groups − 1) degrees of freedom. IQR: interquartile range, the middle 50% of values. Permutation p-value: the share of all reshufflings of the group labels that give a result at least as extreme. Stochastic homogeneity: no group tends to rank higher than another. Stochastic homogeneity: if you pick one animal at random from each of two groups, either is equally likely to have the higher value; Kruskal–Wallis tests whether this holds for every pair of groups. Chi-square approximation: the shortcut most software uses to turn H into a p-value; it is unreliable with fewer than 5 animals per group. Exact or permutation p-value: a p-value computed by counting (or sampling) every way the values could have been split between the groups.

What each test answers

	Welch’s ANOVA (our recommended default; SciPy’s f_oneway runs the classical test unless you pass equal_var=False)	Kruskal–Wallis
Question	Do the group means differ?	Are values in some groups systematically higher than in others?
Works on	The values	Their ranks
Effect you can report	Mean difference (or ratio on the log scale) with a CI	Medians and IQRs per group; for a pair, the probability that a random animal from one group scores higher than one from the other
Follow-up	Games–Howell (or Tukey if you assume equal spreads)	Dunn’s test on the pooled ranks, Holm-adjusted
Needs	Independent animals; roughly symmetric values within groups	Independent animals; similar shape and spread if you want to call it a median comparison

Vargha and Delaney (1998) showed that the test can only detect departures from stochastic homogeneity, which is equivalent to equal expected mean ranks: it is a test that values in some groups rank higher than in others. Calling it a test of medians is correct only when the groups differ by a pure shift with identical shapes, as Hart (2001) explains for the two-group version, and Divine et al. (2018) give real and constructed counter-examples where it fails as a median test.

SciPy’s own documentation describes the null hypothesis as “the population median of all of the groups are equal”. Write “values were higher in…” (in rank terms) unless you have checked that the shapes and spreads match.

What to do: decide from the question and the scale. Ordinal scores → Kruskal–Wallis. Measurements where you want means or fold differences → Welch’s ANOVA, on the log scale for positive skewed data.

Ordinal scores: a worked example

Histology damage grade (0–4) in three groups of 8 mice, one grade per mouse (constructed data; grades.csv).

Group	Grades	Median (IQR)
Vehicle	0, 1, 1, 1, 2, 1, 0, 2	1 (0.75–1.25)
Low dose	1, 2, 2, 1, 3, 2, 2, 1	2 (1–2)
High dose	2, 3, 3, 4, 2, 3, 4, 3	3 (2.75–3.25)

Kruskal–Wallis gives H(2) = 13.71, p = 0.0011 (chi-square approximation, corrected for ties). A permutation p-value from 100,000 random relabellings is 0.0002, so with these heavily tied scores the approximation is conservative here.

import pandas as pd
from scipy import stats
import scikit_posthocs as sp

g = pd.read_csv("grades.csv")          # columns: dose, grade (one row per mouse)
groups = [g.loc[g["dose"] == k, "grade"] for k in ["vehicle", "low", "high"]]
print(stats.kruskal(*groups))           # H corrected for ties, chi-square p-value
print(g.groupby("dose")["grade"].median())

# Dunn's test on the pooled ranks. p_adjust defaults to None (no adjustment): set it.
print(sp.posthoc_dunn(g, val_col="grade", group_col="dose", p_adjust="holm"))

g <- read.csv("grades.csv")             # columns: dose, grade (one row per mouse)
g$dose <- factor(g$dose, levels = c("vehicle", "low", "high"))
kruskal.test(grade ~ dose, data = g)     # H corrected for ties, chi-square p-value
tapply(g$grade, g$dose, median)

# Dunn's test on the pooled ranks; rstatix adjusts with Holm by default.
rstatix::dunn_test(g, grade ~ dose, p.adjust.method = "holm")

For an exact-style p-value in R, coin::kruskal_test(grade ~ dose, data = g, distribution = coin::approximate(nresample = 100000)) gives a Monte Carlo permutation p (0.00017 with set.seed(1)).

What to do: report the medians with IQRs, H with its degrees of freedom and p, and then the follow-up comparisons below.

Follow up with Dunn’s test, not separate Mann–Whitney tests

Dunn’s test compares groups using the ranks from the one combined ranking of all animals, the ranking Kruskal–Wallis used; separate Mann–Whitney tests re-rank each pair on its own (Dunn 1964 proposed the combined ranking as the alternative to separate rankings). On the same grades:

Comparison	p (Dunn, Holm-adjusted)	p (separate Mann–Whitney, Holm-adjusted)	Probability a mouse in the first group scores higher
Low vs vehicle	0.16	0.080	0.75
High vs vehicle	0.0007	0.0045	0.97
High vs low	0.046	0.019	0.88

The last column is the probability that a random mouse from the first group has a higher grade than one from the second, counting ties as half (the Vargha–Delaney A, computed from each pair’s Mann–Whitney U). The two procedures disagree in both directions. Choose Dunn’s test before you see the results so that the follow-up matches the omnibus test, and adjust for the number of comparisons.

Python’s scikit_posthocs.posthoc_dunn returns unadjusted p-values unless you pass p_adjust; R’s rstatix::dunn_test uses Holm by default. Both gave identical Holm-adjusted values here.

What to do: run Dunn’s test with Holm’s adjustment and say so; if only comparisons with vehicle matter, decide that in advance and adjust over those two only (which correction to use).

Skewed measurements: same data, four answers

IL-6 in three groups of 6 mice (constructed log-normal data, seed fixed in advance; il6.csv). Vehicle includes one low value (4.4 pg/mL). The scripts behind both worked-example tables on this page: example.py (output) and example.R (output).

Analysis	Result
Classical ANOVA on raw values	F(2, 15) = 9.03, p = 0.0027
Welch’s ANOVA on raw values	F(2, 9.13) = 6.39, p = 0.018
Welch’s ANOVA on log values	F(2, 9.18) = 7.28, p = 0.013
Kruskal–Wallis	H(2) = 11.18, p = 0.0037

All four detect a difference, but only the log-scale analysis gives an effect in the units a biologist reports: geometric means 17.7, 30.8 and 63.1 pg/mL, and high dose / vehicle = 3.6-fold (95% CI 1.6 to 7.9, unadjusted). The raw-scale classical ANOVA has the smallest p because it ignores the larger spread of the high-dose group.

Python (SciPy 1.16 or later) and R, with il6.csv in the working folder; both print the Welch F above, the geometric means, the 3.6-fold ratio and the Games–Howell follow-up on the log scale (high / vehicle 3.56-fold, 95% CI 1.33 to 9.56, adjusted p = 0.015; low / vehicle 1.74-fold, 0.67 to 4.50, p = 0.26; high / low 2.05-fold, 1.06 to 3.94, p = 0.034):

import numpy as np
import pandas as pd
from scipy import stats   # f_oneway(equal_var=False) needs SciPy 1.16 or later

d = pd.read_csv("il6.csv")
order = ["vehicle", "low", "high"]
logs = {g: np.log(d.loc[d.dose == g, "il6"].to_numpy()) for g in order}
print(stats.f_oneway(*logs.values(), equal_var=False))            # Welch's ANOVA on log values
print({g: round(float(np.exp(v.mean())), 1) for g, v in logs.items()})   # geometric means
w = stats.ttest_ind(logs["high"], logs["vehicle"], equal_var=False)
ci = w.confidence_interval()
print(f"high / vehicle: {np.exp(logs['high'].mean() - logs['vehicle'].mean()):.1f}-fold "
      f"(95% CI {np.exp(ci.low):.1f} to {np.exp(ci.high):.1f}, unadjusted)")
gh = stats.tukey_hsd(*logs.values(), equal_var=False)   # Games–Howell on log values (SciPy 1.18)
gci = gh.confidence_interval()
for i, j in [(1, 0), (2, 0), (2, 1)]:
    print(f"{order[i]} / {order[j]}: {np.exp(logs[order[i]].mean() - logs[order[j]].mean()):.2f}-fold "
          f"(95% CI {np.exp(gci.low[i, j]):.2f} to {np.exp(gci.high[i, j]):.2f}, adjusted p = {gh.pvalue[i, j]:.4f})")

d <- read.csv("il6.csv")
d$dose <- factor(d$dose, levels = c("vehicle", "low", "high"))
oneway.test(log(il6) ~ dose, data = d)                     # Welch's ANOVA on log values
round(exp(tapply(log(d$il6), d$dose, mean)), 1)            # geometric means
w <- t.test(log(il6) ~ dose, data = droplevels(subset(d, dose %in% c("high", "vehicle"))))
round(c(ratio = unname(exp(diff(w$estimate))), exp(-rev(w$conf.int))), 1)   # high / vehicle, unadjusted
gh <- rstatix::games_howell_test(transform(d, log_il6 = log(il6)), log_il6 ~ dose)   # Games–Howell on log values
transform(gh[, c("group1", "group2")], ratio = round(exp(gh$estimate), 2),
          lower = round(exp(gh$conf.low), 2), upper = round(exp(gh$conf.high), 2), p.adj = signif(gh$p.adj, 3))

What to do: for positive, right-skewed measurements, decide in advance to use Welch’s ANOVA on the logarithms and report ratios of geometric means (Welch’s ANOVA); keep Kruskal–Wallis for scores and for the question of which group ranks higher.

Unequal spreads: the rank test is not a fix

Ranks remove the influence of extreme values, not of different spreads. Zimmerman (2000) found that unequal variances bias the significance level of Kruskal–Wallis even with equal group sizes, raising false-positive rates by as much as 40–50% and sometimes 300%.

Our simulation (normal data, all groups with the same mean and median, 20,000 runs per row, seed 20261110):

Groups (n)	SDs	Kruskal–Wallis	Classical ANOVA	Welch’s ANOVA
8, 8, 8	1, 1, 1	4.5%	5.0%	4.7%
8, 8, 8	1, 1, 4	6.5%	8.6%	5.0%
12, 8, 4	1, 1, 4 (smallest group most variable)	11.3%	26.1%	5.7%
4, 8, 12	1, 1, 4 (largest group most variable)	3.1%	2.0%	5.0%

What to do: use Welch’s ANOVA when the spreads differ and you want to compare means (on the log scale for positive skewed data). If you need a rank-based answer with unequal spreads, state that your question is which group ranks higher and use pairwise Brunner–Munzel tests, which do not assume equal spreads (Brunner and Munzel 2000; scipy.stats.brunnermunzel in SciPy), with Holm’s adjustment, at 8 or more animals per group: in our simulation (10,000 runs) they kept the familywise false-positive rate at 5.1% with groups of 8 and SDs 1, 1 and 4 (Dunn’s test: 5.9%), but reached 11.6% with groups of 12, 8 and 4 and the group of 4 most variable. SciPy’s documentation adds that below about 10 values per group a permutation version of the test is preferable to the default t approximation.

from scipy import stats
from statsmodels.stats.multitest import multipletests

a, b, c = [12.1, 9.8, 11.4, 10.6, 13.0, 10.2, 11.7, 9.9], [10.4, 11.9, 12.8, 9.5, 11.1, 10.9, 12.2, 10.0], [12.0, 19.5, 14.1, 18.8, 10.2, 16.6, 21.9, 15.3]
p = [stats.brunnermunzel(x, y).pvalue for x, y in [(a, b), (a, c), (b, c)]]
print([round(float(x), 3) for x in p])                                   # unadjusted p: a-b, a-c, b-c
print([round(float(x), 3) for x in multipletests(p, method="holm")[1]])   # Holm-adjusted p

With smaller or very unequal groups, compare means with Welch’s ANOVA instead.

Very small groups: the floor and the approximation

With few animals, a rank test has only so many arrangements. Counting every way to split the ranks 1–9 into three groups of 3 (1,680 arrangements), the most extreme result, perfectly separated groups, has an exact p of 0.0036; the chi-square approximation that R and SciPy report for the same data is 0.027.

Animals per group (3 groups, no ties)	Smallest exact p	Chi-square p for the same data
3	0.0036	0.027
4	0.00017	0.0073
5	0.0000079	0.0019

SciPy’s documentation states the usual rule that “each sample must have at least 5 measurements” for the chi-square approximation. Below that, the reported p-value is too large for extreme data, so a perfectly separated 3 × 3 experiment cannot reach p < 0.01 with the default output; report the exact or permutation p-value instead.

Power is close to ANOVA’s when data really are normal: in our simulation with three groups of 8 mice whose means were 0, 0.5 and 1 SD apart, Kruskal–Wallis detected the difference 33% of the time and classical ANOVA 37%. Hodges and Lehmann (1956) proved that, for large samples and pure location shifts, the rank test’s asymptotic relative efficiency (the ratio of sample sizes needed for the same power) never falls below 0.864 compared with the F-test; it says nothing about small groups or unequal spreads. Simulation code and output: small_n_and_simulation.py, output; Brunner–Munzel check: brunner_munzel.py, output (seed 20261110).

What to do: report a permutation p-value when you have fewer than 5 animals per group (coin in R, or a permutation loop in Python) and the group values themselves.

A permutation loop in Python, with grades.csv in the working folder (it prints H = 13.71 and a permutation p of 0.00012 with this seed; a different seed or number of relabellings moves the last digits):

import numpy as np
import pandas as pd
from scipy import stats

g = pd.read_csv("grades.csv")                      # columns: dose, grade
groups = [g.loc[g.dose == k, "grade"].to_numpy() for k in g.dose.unique()]
h_obs = stats.kruskal(*groups).statistic
pooled, sizes = np.concatenate(groups), [len(x) for x in groups]
rng = np.random.default_rng(1)                     # fixed seed: the p-value is a Monte Carlo estimate
n_perm, count = 100_000, 0
for _ in range(n_perm):
    parts = np.split(rng.permutation(pooled), np.cumsum(sizes)[:-1])
    count += stats.kruskal(*parts).statistic >= h_obs - 1e-12
print(f"H = {h_obs:.2f}, permutation p = {(count + 1) / (n_perm + 1):.5f}")

Choose before you see a normality test

A Shapiro–Wilk result should not decide between the two tests: at small n it rarely detects non-normality, and choosing a test after seeing the data changes the error rates (should you test for normality?). Decide from the outcome type and the question when you plan the analysis. For paired or repeated measurements on the same animals, neither test applies; see nested and repeated measurements.

What to do: write the planned test and follow-up in your analysis plan or lab notebook before unblinding.

Check what your software or AI assistant ran

Check	Why it matters
Was the choice made from a normality test?	Ask for the reason; choose from the outcome and question instead
Does the text say “medians differ”?	Correct only with equal shapes; otherwise “values ranked higher in…”
Which follow-up ran: Dunn, or separate Mann–Whitney tests?	They rank differently and give different p-values
Was p_adjust set in posthoc_dunn?	Python’s default is no adjustment
Are there fewer than 5 animals per group?	Ask for a permutation p-value
Are the values cells or wells rather than animals?	The test assumes independent animals (what counts as n)

What to do: check each row against the code and output before you report the result.

Report it so a reviewer can check it

Histology grades were compared between groups with the Kruskal–Wallis test (n = 8 mice per group; one grade per mouse), followed by Dunn’s test on the pooled ranks with Holm’s adjustment. Grades tended to be higher with dose (rank-based comparison; median [IQR]: vehicle 1 [0.75–1.25], low 2 [1–2], high 3 [2.75–3.25]; H(2) = 13.71, p = 0.001). High dose differed from vehicle (Holm-adjusted p < 0.001) and from low dose (p = 0.046); low dose did not differ clearly from vehicle (p = 0.16). Analyses used R 4.3.3 (rstatix 0.7.2).

Report, for a skewed measurement analysed on the log scale, the ratio of geometric means with its confidence interval and the Welch F with its fractional denominator degrees of freedom.

References

• Brunner E, Munzel U (2000). The nonparametric Behrens–Fisher problem: asymptotic theory and a small-sample approximation. Biometrical Journal 42(1):17–25. doi:10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U
• Divine GW, Norton HJ, Barón AE, Juarez-Colunga E (2018). The Wilcoxon–Mann–Whitney procedure fails as a test of medians. The American Statistician 72(3):278–286. doi:10.1080/00031305.2017.1305291
• Dunn OJ (1964). Multiple comparisons using rank sums. Technometrics 6(3):241–252. doi:10.1080/00401706.1964.10490181
• Hart A (2001). Mann-Whitney test is not just a test of medians: differences in spread can be important. BMJ 323:391–393. doi:10.1136/bmj.323.7309.391
• Hodges JL, Lehmann EL (1956). The efficiency of some nonparametric competitors of the t-test. Annals of Mathematical Statistics 27(2):324–335. doi:10.1214/aoms/1177728261
• Kruskal WH, Wallis WA (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association 47(260):583–621. doi:10.1080/01621459.1952.10483441
• Vargha A, Delaney HD (1998). The Kruskal-Wallis test and stochastic homogeneity. Journal of Educational and Behavioral Statistics 23(2):170–192. doi:10.3102/10769986023002170
• Zimmerman DW (2000). Statistical significance levels of nonparametric tests biased by heterogeneous variances of treatment groups. Journal of General Psychology 127(4):354–364. doi:10.1080/00221300009598589
• SciPy 1.18.1 documentation: scipy.stats.kruskal, scipy.stats.brunnermunzel. R 4.3.3 documentation: kruskal.test. scikit-posthocs posthoc_dunn; rstatix dunn_test.