Welch's ANOVA vs one-way ANOVA: which to use for three or more groups

Quick answer. To compare the means of three or more independent groups — one value per animal, with technical replicates averaged first — use Welch’s ANOVA by default, and follow it with the Games–Howell test, which compares every pair of groups using each pair’s own spreads. The classical one-way ANOVA assumes all groups are equally variable; Welch’s version does not, and when they are equally variable it gives almost the same answer. Welch’s ANOVA still needs roughly symmetric values: with small, clearly skewed groups, analyse logarithms or use a rank test. R’s oneway.test and statsmodels’ anova_oneway run Welch’s ANOVA by default. R’s aov and SciPy’s stats.f_oneway run the classical test; in SciPy 1.16 and later, stats.f_oneway(..., equal_var=False) runs Welch’s.

Terms used below. F is the ratio of the spread between group means to the spread within groups; larger F means stronger evidence of a difference. Variance is the SD squared. Degrees of freedom (df) measure how much information the spread estimates carry; Welch’s second df is a decimal and smaller than the classical one. A false positive is a significant result when there is no true difference; the Type I error rate is how often that happens, and a test is conservative when that rate falls below the 5% you set (fewer false positives, more missed effects). Power is the chance of detecting a real difference. Games–Howell compares every pair of groups without assuming equal spreads.

Decide in four questions

1. Three or more independent groups, one factor? Different animals in each group, one grouping variable (for example dose): continue. Two groups: use Welch’s t-test — with two groups, Welch’s ANOVA gives the same p-value (its F is the square of Welch’s t). Two factors such as genotype × treatment: use a two-way ANOVA. The same animals measured repeatedly: use a repeated-measures or mixed model.
2. What is one observation? Count the units you randomised — usually animals, not wells or cells. Average technical replicates within each animal first (code below).
3. Is the outcome a continuous measurement, roughly symmetric within each group? Then continue. With clearly skewed values and small groups, analyse logs or use a rank test (details).
4. Which comparisons will you report? Decide now: all pairs (Games–Howell) or each group against a control. That choice, made before you see the data, fixes the follow-up test.

If you reached the end: use Welch’s ANOVA, then Games–Howell, decided before you look at the data.

Why Welch’s ANOVA is the safer default

The classical F-test pools the spread (variance) of all groups into one number. When the groups really are equally variable, that is efficient. When they are not and the group sizes differ, the test misjudges the noise: it becomes too eager when the smaller groups are the more variable ones and too cautious when the larger groups are, as earlier simulation studies summarised by Delacre et al. (2019) found.

We simulated 100,000 experiments per row with three normally distributed groups and no true difference, so every “significant” result is a false positive. The intended rate is 5%. The code and its output are published: simulation.py (seed 20261110) and simulation_output.json.

Groups (n)	Standard deviations	Classical F	Welch’s F
8, 8, 8	equal	5.0%	4.8%
6, 10, 14	smallest group twice as variable	12.6%	5.5%
6, 10, 14	largest group twice as variable	2.7%	4.9%
10, 10, 10	one group three times as variable	7.7%	5.0%
10, 9, 5 (as in the example below)	7, 3.5, 11.6	12.3%	5.5%

Welch’s test stays close to 5% in every row (groups of 5 to 14; for 3–5 per group see below). Equal group sizes help the classical test only partly: with 10 per group and one group three times as variable, it still flags 7.7% of experiments.

The protection costs little at moderate sizes. With equal spreads and 8 animals per group, a shift of 1.4 standard deviations in one group was detected in 77% of experiments by the classical test and 73% by Welch’s. After 3,840 simulated scenarios, Delacre and colleagues recommend Welch’s test “instead of the F-test to compare groups means”, because “the loss of power when using the W-test is very small (and often even negligible), and … the gain in Type I error control is considerable” (Delacre et al. 2019, Recommendations). The test itself goes back to Welch (1951).

Use Welch’s ANOVA unless you have a specific, pre-stated reason to assume equal spreads — and check its remaining conditions first: independent observations and roughly symmetric values within groups. With small, skewed groups, analyse logarithms or use a rank test (details).

Same data, two answers

A constructed example of the case where the two tests disagree: plasma ALT (U/L) in mice given vehicle, a low dose or a high dose, one value per mouse. The high-dose group is the smallest and by far the most variable.

Group	n	Mean	SD
Vehicle	10	99.96	6.94
Low dose	9	103.98	3.49
High dose	5	115.16	11.61

import scipy
from scipy import stats
print(scipy.__version__)   # equal_var in f_oneway needs SciPy 1.16 or later; otherwise upgrade,
                           # or use statsmodels.stats.oneway.anova_oneway (Welch by default)

vehicle = [101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7]
low = [109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8]
high = [95.9, 127.1, 115.4, 118.0, 119.4]

classical = stats.f_oneway(vehicle, low, high)                # default: equal variances
welch = stats.f_oneway(vehicle, low, high, equal_var=False)   # Welch (SciPy 1.16+)
print(f"Classical: F = {classical.statistic:.2f}, p = {classical.pvalue:.4f}")
print(f"Welch:     F = {welch.statistic:.2f}, p = {welch.pvalue:.4f}")

Classical: F = 7.61, p = 0.0033
Welch:     F = 3.62, p = 0.0719

The same comparison in R, where oneway.test runs Welch’s version by default:

alt <- c(101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7,
         109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8,
         95.9, 127.1, 115.4, 118.0, 119.4)
group <- factor(rep(c("vehicle", "low", "high"), c(10, 9, 5)),
                levels = c("vehicle", "low", "high"))

oneway.test(alt ~ group)                    # Welch: F = 3.6242, num df = 2, denom df = 8.6481, p = 0.07191
oneway.test(alt ~ group, var.equal = TRUE)  # classical: F = 7.6138, num df = 2, denom df = 21, p = 0.003261

With your own data. Save one row per mouse with columns mouse, dose and alt (the example as alt.csv); the order you list the groups in sets the sign of every pairwise difference below.

import pandas as pd
from scipy import stats

df = pd.read_csv("alt.csv")                      # one row per mouse: columns mouse, dose, alt
g = df.groupby("dose")["alt"]
groups = [g.get_group(k) for k in ["vehicle", "low", "high"]]   # fixed order sets the sign of differences
print(stats.f_oneway(*groups, equal_var=False))  # Welch's ANOVA

If you measured several wells per mouse, average them first so that each mouse gives one value:

import pandas as pd

wells = pd.DataFrame({"mouse": ["V1", "V1", "V1", "V2", "V2", "V2"],
                      "dose": ["vehicle"] * 6,
                      "alt": [100.2, 101.5, 101.6, 96.0, 97.4, 97.3]})   # one row per well
per_mouse = wells.groupby(["mouse", "dose"], as_index=False)["alt"].mean()
print(per_mouse)                                  # one row per mouse: use this for the test

wells <- data.frame(mouse = rep(c("V1", "V2"), each = 3), dose = "vehicle",
                    alt = c(100.2, 101.5, 101.6, 96.0, 97.4, 97.3))   # one row per well
per_mouse <- aggregate(alt ~ mouse + dose, data = wells, FUN = mean)
per_mouse                                        # one row per mouse: use this for the test

R 4.3.3 and SciPy 1.18.1 agree to every printed digit; statsmodels’ anova_oneway gives the same Welch result with its default settings.

What changed. The classical test pools the three spreads into one, so the five highly variable high-dose mice look more precisely measured than they are, and p = 0.003. Welch’s test lets each group keep its own spread and reports p = 0.072 — not significant, but not evidence of no effect: the high-dose mean is 15.2 U/L above vehicle (Games–Howell 95% CI −2.6 to 33.0). Its denominator degrees of freedom (df, which reflect how much information the spread estimates carry) fall from 21 to 8.6, because the small, variable group carries most of the uncertainty. With spreads like these and no true effect, the classical test would flag about 12% of experiments (table above).

How to read it. Welch’s result does not show that the doses have no effect: the high-dose mean is 15 U/L above vehicle, and the data are too noisy to pin that down. Report the estimate with its interval (next section) and plan more animals for the high-dose group.

Follow up with Games–Howell, not Tukey

Tukey’s test, the usual follow-up to a classical ANOVA, pools the variances in the same way. The Games–Howell test compares each pair with its own Welch-type standard error and degrees of freedom; it is named after the study of pairwise procedures for unequal group sizes and variances by Games & Howell (1976). It is designed to control the error rate across all pairs on its own, so run it whatever the ANOVA p-value; a significant ANOVA with no significant pair is a normal outcome.

Comparison	Difference (U/L)	Games–Howell 95% CI	Games–Howell p	Tukey p
High − vehicle	15.2	−2.6 to 33.0	0.086	0.002
High − low	11.2	−7.0 to 29.4	0.196	0.027
Low − vehicle	4.0	−2.5 to 10.5	0.272	0.452

Tukey’s test would report two “significant” pairs that rest on treating the high-dose group as precise as the others.

res = stats.tukey_hsd(vehicle, low, high, equal_var=False)  # Games–Howell (equal_var=False)
print(res)   # "(2 - 0)" = high minus vehicle: groups are numbered in the order passed (0 vehicle, 1 low, 2 high)
ci = res.confidence_interval()
print(f"High - vehicle: {res.statistic[2, 0]:.1f}, 95% CI {ci.low[2, 0]:.1f} to {ci.high[2, 0]:.1f}, p = {res.pvalue[2, 0]:.3f}")

If tukey_hsd rejects equal_var, your SciPy is older: use pingouin.pairwise_gameshowell or the R function below, not plain tukey_hsd(...), which runs Tukey’s test. In R, base stats has no Games–Howell function. This one, written from the published definition, reproduces SciPy’s and Pingouin’s results to every printed digit:

games_howell <- function(y, g) {
  lv <- levels(g); k <- length(lv); out <- NULL
  for (i in 1:(k - 1)) for (j in (i + 1):k) {
    a <- y[g == lv[i]]; b <- y[g == lv[j]]
    va <- var(a) / length(a); vb <- var(b) / length(b)
    d <- mean(b) - mean(a); se <- sqrt(va + vb)
    df <- (va + vb)^2 / (va^2 / (length(a) - 1) + vb^2 / (length(b) - 1))
    crit <- qtukey(0.95, k, df) / sqrt(2)
    out <- rbind(out, data.frame(pair = paste(lv[j], "-", lv[i]), diff = d, df = df,
      lwr = d - crit * se, upr = d + crit * se,
      p = ptukey(abs(d) / se * sqrt(2), k, df, lower.tail = FALSE)))
  }
  out
}
games_howell(alt, group)

If your plan is to compare each dose with vehicle only, say so before the analysis and use a many-to-one comparison instead; see which correction to use. Dunnett’s test pools the variances, so with unequal spreads compare each dose with vehicle by Welch t-tests and adjust with Holm’s method (in R, multcomp::glht() with vcov = sandwich::vcovHC is an alternative):

alt <- c(101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7,
         109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8,
         95.9, 127.1, 115.4, 118.0, 119.4)
group <- factor(rep(c("vehicle", "low", "high"), c(10, 9, 5)), levels = c("vehicle", "low", "high"))
p <- c(low  = t.test(alt[group == "low"],  alt[group == "vehicle"])$p.value,   # Welch by default
       high = t.test(alt[group == "high"], alt[group == "vehicle"])$p.value)
round(p.adjust(p, method = "holm"), 4)
# low 0.1288, high 0.0784

from scipy import stats
from statsmodels.stats.multitest import multipletests
vehicle = [101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7]
low = [109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8]
high = [95.9, 127.1, 115.4, 118.0, 119.4]
p = [stats.ttest_ind(low, vehicle, equal_var=False).pvalue,
     stats.ttest_ind(high, vehicle, equal_var=False).pvalue]
print(multipletests(p, method="holm")[1].round(4))   # Holm-adjusted: low, high

Decide before you look at the variances

A common habit is to run Levene’s test first and use the classical ANOVA when it is “not significant”. Use Welch’s ANOVA from the start instead. Variance tests are weak at lab group sizes: they miss real differences in spread most of the time, so the two-step procedure inherits the classical test’s false positives. For two groups the two-stage procedure “fails to protect the significance level and usually makes the situation worse” (Zimmerman 2004, abstract), and Delacre et al. (2019) give the same reason for skipping it with three or more groups: “tests of the equal variances assumption will often fail to detect differences when these are present”.

Plot the data instead — every value, by group — and use the picture to describe the spread, not to choose the test.

When the default needs more thought

Welch’s ANOVA removes one assumption, equal spreads. Its other conditions still apply: independent observations and roughly symmetric values within each group.

• Very small groups (3–5 per group). In our simulation, Welch’s test was slightly conservative with equal spreads — 4.2% false positives at 4 per group and 3.8% at 3 per group — and it costs more power than at moderate sizes: with 5 per group and a 1.8-SD shift, 66% against 74% for the classical test. The classical test, in turn, gave 8.8% false positives at 4 per group when one group was three times as variable (Welch: 5.1%). Choose in advance, and lead with the estimates and their intervals.
• Skewed values with small groups. Delacre et al. (2019) found that with fewer than about 50 observations per group (comparing up to four groups), Welch’s test “will not control Type I error rate when skewness is present”. For right-skewed measurements such as concentrations, analyse the logarithms, which turns differences into ratios; for ordinal scores or strong outliers, use the Kruskal–Wallis test. Kruskal–Wallis asks whether some groups tend to have higher values; it does not fix unequal spreads, which can distort it when group sizes also differ.
• Technical replicates and clusters. Three wells from each of five mice is n = 5, not 15. Average within each mouse; if animals share cages or litters, use a mixed model.
• Repeated measurements or two factors. Welch’s ANOVA is for one factor with independent groups. Use a repeated-measures or mixed model for the same animals over time, and a two-way ANOVA for crossed factors.

Write whichever of these applies into your analysis plan before you see the results.

Check which test your software or AI ran

Software (checked 10 Nov 2026)	Default for one-way ANOVA	To get Welch’s ANOVA	Games–Howell
R 4.3.3 oneway.test	Welch (var.equal = FALSE)	Nothing to change	Not in base R (function above)
R aov, anova(lm(...))	Classical	Use oneway.test	—
SciPy 1.18 stats.f_oneway	Classical (equal_var=True)	equal_var=False (SciPy 1.16+)	stats.tukey_hsd(..., equal_var=False)
statsmodels 0.15 anova_oneway	Welch (use_var="unequal")	Nothing to change	—
Pingouin 0.7	anova is classical	welch_anova	pairwise_gameshowell

Sources: R oneway.test, SciPy f_oneway, SciPy tukey_hsd, statsmodels anova_oneway, Pingouin pairwise_gameshowell documentation.

Three checks on a result you were given:

1. Look at the second degrees of freedom. The classical test reports exactly N − k, a whole number (24 animals − 3 groups = 21). Welch’s denominator df is usually a decimal and smaller (8.65). F(2, 21) means the classical test ran.
2. Search the code. f_oneway(...) without equal_var=False, aov(...), or oneway.test(..., var.equal = TRUE) ran the classical test. tukey_hsd(...) without equal_var=False ran Tukey, not Games–Howell.
3. Read the output header. R prints One-way analysis of means (not assuming equal variances) for Welch’s ANOVA.

A prompt that names the test and its parameters leaves less to the assistant’s defaults:

Compare the mean of [outcome] across [k] independent groups (one value per animal) using
Welch's one-way ANOVA, then Games–Howell for all pairwise comparisons. In Python use
scipy.stats.f_oneway(..., equal_var=False) and scipy.stats.tukey_hsd(..., equal_var=False).
Report group means and SDs, Welch's F with both degrees of freedom and the exact p-value,
and each pairwise difference with its 95% confidence interval.

Report it so a reviewer can check it

Methods sentence (keep the bracketed parts only if they are true):

Group means were compared with Welch’s one-way ANOVA (unequal variances), followed by the Games–Howell procedure for all pairwise comparisons[, specified before the data were analysed]; the unit of analysis was the [animal]. Analyses used [software and version].

Results sentence (the example above):

Plasma ALT was 100.0 U/L (SD 6.9, n = 10) after vehicle, 104.0 U/L (SD 3.5, n = 9) after the low dose and 115.2 U/L (SD 11.6, n = 5) after the high dose; Welch’s F(2, 8.65) = 3.62, p = 0.072. The high-dose minus vehicle difference was 15.2 U/L (Games–Howell 95% CI −2.6 to 33.0, p = 0.086).

Give both degrees of freedom — the decimal one tells the reader Welch’s test was used — the exact p-value, and the pairwise differences with their intervals, which carry the size of the effect.

References

• Delacre M, Leys C, Mora YL, Lakens D (2019). Taking parametric assumptions seriously: arguments for the use of Welch’s F-test instead of the classical F-test in one-way ANOVA. International Review of Social Psychology 32(1):13. doi:10.5334/irsp.198
• Games PA, Howell JF (1976). Pairwise multiple comparison procedures with unequal N’s and/or variances: a Monte Carlo study. Journal of Educational Statistics 1(2):113–125. doi:10.3102/10769986001002113
• Welch BL (1951). On the comparison of several mean values: an alternative approach. Biometrika 38(3–4):330–336. doi:10.1093/biomet/38.3-4.330
• Zimmerman DW (2004). A note on preliminary tests of equality of variances. British Journal of Mathematical and Statistical Psychology 57(1):173–181. doi:10.1348/000711004849222
• Software documentation accessed 10 November 2026: R stats::oneway.test; SciPy 1.18 scipy.stats.f_oneway and scipy.stats.tukey_hsd; statsmodels 0.15 anova_oneway; Pingouin 0.7 pairwise_gameshowell.