# Welch's ANOVA vs one-way ANOVA: which to use for three or more groups

> Use Welch's ANOVA by default to compare the means of three or more independent groups, then Games–Howell for pairs. Why, with a worked example in R and Python, a simulation and the software defaults that differ.

Canonical: https://www.licklider.ai/guides/welch-anova/
Section: Guides › Comparing three or more groups
Author: Licklider · Published 2026-11-10 · Last reviewed 2026-11-10 (UTC)

> **Quick answer.** To compare the means of three or more independent groups — one value per animal, with technical replicates averaged first — use **Welch's ANOVA** by default, and follow it with the **Games–Howell** test, which compares every pair of groups using each pair's own spreads. The classical one-way ANOVA assumes all groups are equally variable; Welch's version does not, and when they are equally variable it gives almost the same answer. Welch's ANOVA still needs roughly symmetric values: with small, clearly skewed groups, analyse logarithms or use a rank test. R's `oneway.test` and statsmodels' `anova_oneway` run Welch's ANOVA by default. **R's `aov` and SciPy's `stats.f_oneway` run the classical test**; in SciPy 1.16 and later, `stats.f_oneway(..., equal_var=False)` runs Welch's.

**Terms used below.** *F* is the ratio of the spread between group means to the spread within groups; larger F means stronger evidence of a difference. *Variance* is the SD squared. *Degrees of freedom (df)* measure how much information the spread estimates carry; Welch's second df is a decimal and smaller than the classical one. A *false positive* is a significant result when there is no true difference; the *Type I error rate* is how often that happens, and a test is *conservative* when that rate falls below the 5% you set (fewer false positives, more missed effects). *Power* is the chance of detecting a real difference. *Games–Howell* compares every pair of groups without assuming equal spreads.

## Decide in four questions

1. **Three or more independent groups, one factor?** Different animals in each group, one grouping variable (for example dose): continue. Two groups: use [Welch's t-test](/guides/welch-vs-student-t-test/) — with two groups, Welch's ANOVA gives the same p-value (its F is the square of Welch's t). Two factors such as genotype × treatment: use a [two-way ANOVA](/guides/two-way-anova-vs-multiple-t-tests/). The same animals measured repeatedly: use a repeated-measures or mixed model.
2. **What is one observation?** Count the units you randomised — usually animals, not wells or cells. Average technical replicates within each animal first (code below).
3. **Is the outcome a continuous measurement, roughly symmetric within each group?** Then continue. With clearly skewed values and small groups, analyse logs or use a rank test ([details](#when-the-default-needs-more-thought)).
4. **Which comparisons will you report?** Decide now: all pairs (Games–Howell) or each group against a control. That choice, made before you see the data, fixes the follow-up test.

If you reached the end: **use Welch's ANOVA, then Games–Howell**, decided before you look at the data.

## Why Welch's ANOVA is the safer default

The classical F-test pools the spread (variance) of all groups into one number. When the groups really are equally variable, that is efficient. When they are not and the group sizes differ, the test misjudges the noise: it becomes too eager when the smaller groups are the more variable ones and too cautious when the larger groups are, as earlier simulation studies summarised by [Delacre et al. (2019)](https://doi.org/10.5334/irsp.198) found.

We simulated 100,000 experiments per row with three normally distributed groups and no true difference, so every "significant" result is a false positive. The intended rate is 5%. The code and its output are published: [simulation.py](/guides/welch-anova/simulation.py) (seed 20261110) and [simulation_output.json](/guides/welch-anova/simulation_output.json).

| Groups (n) | Standard deviations | Classical F | Welch's F |
| --- | --- | --- | --- |
| 8, 8, 8 | equal | 5.0% | 4.8% |
| 6, 10, 14 | smallest group twice as variable | **12.6%** | 5.5% |
| 6, 10, 14 | largest group twice as variable | 2.7% | 4.9% |
| 10, 10, 10 | one group three times as variable | 7.7% | 5.0% |
| 10, 9, 5 (as in the example below) | 7, 3.5, 11.6 | **12.3%** | 5.5% |

Welch's test stays close to 5% in every row (groups of 5 to 14; for 3–5 per group see [below](#when-the-default-needs-more-thought)). Equal group sizes help the classical test only partly: with 10 per group and one group three times as variable, it still flags 7.7% of experiments.

The protection costs little at moderate sizes. With equal spreads and 8 animals per group, a shift of 1.4 standard deviations in one group was detected in 77% of experiments by the classical test and 73% by Welch's. After 3,840 simulated scenarios, Delacre and colleagues recommend Welch's test "instead of the F-test to compare groups means", because "the loss of power when using the W-test is very small (and often even negligible), and … the gain in Type I error control is considerable" ([Delacre et al. 2019](https://doi.org/10.5334/irsp.198), Recommendations). The test itself goes back to [Welch (1951)](https://doi.org/10.1093/biomet/38.3-4.330).

Use Welch's ANOVA unless you have a specific, pre-stated reason to assume equal spreads — and check its remaining conditions first: independent observations and roughly symmetric values within groups. With small, skewed groups, analyse logarithms or use a rank test ([details](#when-the-default-needs-more-thought)).

## Same data, two answers

A constructed example of the case where the two tests disagree: plasma ALT (U/L) in mice given vehicle, a low dose or a high dose, one value per mouse. The high-dose group is the smallest and by far the most variable.

| Group | n | Mean | SD |
| --- | --- | --- | --- |
| Vehicle | 10 | 99.96 | 6.94 |
| Low dose | 9 | 103.98 | 3.49 |
| High dose | 5 | 115.16 | 11.61 |

```python
import scipy
from scipy import stats
print(scipy.__version__)   # equal_var in f_oneway needs SciPy 1.16 or later; otherwise upgrade,
                           # or use statsmodels.stats.oneway.anova_oneway (Welch by default)

vehicle = [101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7]
low = [109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8]
high = [95.9, 127.1, 115.4, 118.0, 119.4]

classical = stats.f_oneway(vehicle, low, high)                # default: equal variances
welch = stats.f_oneway(vehicle, low, high, equal_var=False)   # Welch (SciPy 1.16+)
print(f"Classical: F = {classical.statistic:.2f}, p = {classical.pvalue:.4f}")
print(f"Welch:     F = {welch.statistic:.2f}, p = {welch.pvalue:.4f}")
```

```text
Classical: F = 7.61, p = 0.0033
Welch:     F = 3.62, p = 0.0719
```

The same comparison in R, where `oneway.test` runs Welch's version by default:

```r
alt <- c(101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7,
         109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8,
         95.9, 127.1, 115.4, 118.0, 119.4)
group <- factor(rep(c("vehicle", "low", "high"), c(10, 9, 5)),
                levels = c("vehicle", "low", "high"))

oneway.test(alt ~ group)                    # Welch: F = 3.6242, num df = 2, denom df = 8.6481, p = 0.07191
oneway.test(alt ~ group, var.equal = TRUE)  # classical: F = 7.6138, num df = 2, denom df = 21, p = 0.003261
```

**With your own data.** Save one row per mouse with columns `mouse`, `dose` and `alt` (the example as [alt.csv](/guides/welch-anova/alt.csv)); the order you list the groups in sets the sign of every pairwise difference below.

```python
import pandas as pd
from scipy import stats

df = pd.read_csv("alt.csv")                      # one row per mouse: columns mouse, dose, alt
g = df.groupby("dose")["alt"]
groups = [g.get_group(k) for k in ["vehicle", "low", "high"]]   # fixed order sets the sign of differences
print(stats.f_oneway(*groups, equal_var=False))  # Welch's ANOVA
```

If you measured several wells per mouse, average them first so that each mouse gives one value:

```python
import pandas as pd

wells = pd.DataFrame({"mouse": ["V1", "V1", "V1", "V2", "V2", "V2"],
                      "dose": ["vehicle"] * 6,
                      "alt": [100.2, 101.5, 101.6, 96.0, 97.4, 97.3]})   # one row per well
per_mouse = wells.groupby(["mouse", "dose"], as_index=False)["alt"].mean()
print(per_mouse)                                  # one row per mouse: use this for the test
```

```r
wells <- data.frame(mouse = rep(c("V1", "V2"), each = 3), dose = "vehicle",
                    alt = c(100.2, 101.5, 101.6, 96.0, 97.4, 97.3))   # one row per well
per_mouse <- aggregate(alt ~ mouse + dose, data = wells, FUN = mean)
per_mouse                                        # one row per mouse: use this for the test
```

R 4.3.3 and SciPy 1.18.1 agree to every printed digit; statsmodels' `anova_oneway` gives the same Welch result with its default settings.

**What changed.** The classical test pools the three spreads into one, so the five highly variable high-dose mice look more precisely measured than they are, and p = 0.003. Welch's test lets each group keep its own spread and reports p = 0.072 — not significant, but not evidence of no effect: the high-dose mean is 15.2 U/L above vehicle (Games–Howell 95% CI −2.6 to 33.0). Its denominator degrees of freedom (df, which reflect how much information the spread estimates carry) fall from 21 to 8.6, because the small, variable group carries most of the uncertainty. With spreads like these and no true effect, the classical test would flag about 12% of experiments (table above).

**How to read it.** Welch's result does not show that the doses have no effect: the high-dose mean is 15 U/L above vehicle, and the data are too noisy to pin that down. Report the estimate with its interval (next section) and plan more animals for the high-dose group.

## Follow up with Games–Howell, not Tukey

Tukey's test, the usual follow-up to a classical ANOVA, pools the variances in the same way. The Games–Howell test compares each pair with its own Welch-type standard error and degrees of freedom; it is named after the study of pairwise procedures for unequal group sizes and variances by [Games & Howell (1976)](https://doi.org/10.3102/10769986001002113). It is designed to control the error rate across all pairs on its own, so run it whatever the ANOVA p-value; [a significant ANOVA with no significant pair](/guides/anova-significant-but-post-hoc-not/) is a normal outcome.

| Comparison | Difference (U/L) | Games–Howell 95% CI | Games–Howell p | Tukey p |
| --- | --- | --- | --- | --- |
| High − vehicle | 15.2 | −2.6 to 33.0 | 0.086 | 0.002 |
| High − low | 11.2 | −7.0 to 29.4 | 0.196 | 0.027 |
| Low − vehicle | 4.0 | −2.5 to 10.5 | 0.272 | 0.452 |

Tukey's test would report two "significant" pairs that rest on treating the high-dose group as precise as the others.

```python
res = stats.tukey_hsd(vehicle, low, high, equal_var=False)  # Games–Howell (equal_var=False)
print(res)   # "(2 - 0)" = high minus vehicle: groups are numbered in the order passed (0 vehicle, 1 low, 2 high)
ci = res.confidence_interval()
print(f"High - vehicle: {res.statistic[2, 0]:.1f}, 95% CI {ci.low[2, 0]:.1f} to {ci.high[2, 0]:.1f}, p = {res.pvalue[2, 0]:.3f}")
```

If `tukey_hsd` rejects `equal_var`, your SciPy is older: use `pingouin.pairwise_gameshowell` or the R function below, not plain `tukey_hsd(...)`, which runs Tukey's test. In R, base `stats` has no Games–Howell function. This one, written from the published definition, reproduces SciPy's and Pingouin's results to every printed digit:

```r
games_howell <- function(y, g) {
  lv <- levels(g); k <- length(lv); out <- NULL
  for (i in 1:(k - 1)) for (j in (i + 1):k) {
    a <- y[g == lv[i]]; b <- y[g == lv[j]]
    va <- var(a) / length(a); vb <- var(b) / length(b)
    d <- mean(b) - mean(a); se <- sqrt(va + vb)
    df <- (va + vb)^2 / (va^2 / (length(a) - 1) + vb^2 / (length(b) - 1))
    crit <- qtukey(0.95, k, df) / sqrt(2)
    out <- rbind(out, data.frame(pair = paste(lv[j], "-", lv[i]), diff = d, df = df,
      lwr = d - crit * se, upr = d + crit * se,
      p = ptukey(abs(d) / se * sqrt(2), k, df, lower.tail = FALSE)))
  }
  out
}
games_howell(alt, group)
```

If your plan is to compare each dose with vehicle only, say so before the analysis and use a many-to-one comparison instead; see [which correction to use](/guides/which-multiple-comparison-correction/). Dunnett's test pools the variances, so with unequal spreads compare each dose with vehicle by Welch t-tests and adjust with Holm's method (in R, `multcomp::glht()` with `vcov = sandwich::vcovHC` is an alternative):

```r
alt <- c(101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7,
         109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8,
         95.9, 127.1, 115.4, 118.0, 119.4)
group <- factor(rep(c("vehicle", "low", "high"), c(10, 9, 5)), levels = c("vehicle", "low", "high"))
p <- c(low  = t.test(alt[group == "low"],  alt[group == "vehicle"])$p.value,   # Welch by default
       high = t.test(alt[group == "high"], alt[group == "vehicle"])$p.value)
round(p.adjust(p, method = "holm"), 4)
# low 0.1288, high 0.0784
```

```python
from scipy import stats
from statsmodels.stats.multitest import multipletests
vehicle = [101.1, 96.9, 97.5, 85.4, 110.8, 106.9, 98.0, 104.6, 101.7, 96.7]
low = [109.9, 102.1, 102.0, 99.2, 106.7, 103.4, 107.3, 100.4, 104.8]
high = [95.9, 127.1, 115.4, 118.0, 119.4]
p = [stats.ttest_ind(low, vehicle, equal_var=False).pvalue,
     stats.ttest_ind(high, vehicle, equal_var=False).pvalue]
print(multipletests(p, method="holm")[1].round(4))   # Holm-adjusted: low, high
```

## Decide before you look at the variances

A common habit is to run Levene's test first and use the classical ANOVA when it is "not significant". Use Welch's ANOVA from the start instead. Variance tests are weak at lab group sizes: they miss real differences in spread most of the time, so the two-step procedure inherits the classical test's false positives. For two groups the two-stage procedure "fails to protect the significance level and usually makes the situation worse" ([Zimmerman 2004](https://doi.org/10.1348/000711004849222), abstract), and Delacre et al. (2019) give the same reason for skipping it with three or more groups: "tests of the equal variances assumption will often fail to detect differences when these are present".

Plot the data instead — every value, by group — and use the picture to describe the spread, not to choose the test.

## When the default needs more thought

Welch's ANOVA removes one assumption, equal spreads. Its other conditions still apply: independent observations and roughly symmetric values within each group.

- **Very small groups (3–5 per group).** In our simulation, Welch's test was slightly conservative with equal spreads — 4.2% false positives at 4 per group and 3.8% at 3 per group — and it costs more power than at moderate sizes: with 5 per group and a 1.8-SD shift, 66% against 74% for the classical test. The classical test, in turn, gave 8.8% false positives at 4 per group when one group was three times as variable (Welch: 5.1%). Choose in advance, and lead with the estimates and their intervals.
- **Skewed values with small groups.** Delacre et al. (2019) found that with fewer than about 50 observations per group (comparing up to four groups), Welch's test "will not control Type I error rate when skewness is present". For right-skewed measurements such as concentrations, analyse the logarithms, which turns differences into ratios; for ordinal scores or strong outliers, use the Kruskal–Wallis test. Kruskal–Wallis asks whether some groups tend to have higher values; it does not fix unequal spreads, which can distort it when group sizes also differ.
- **Technical replicates and clusters.** Three wells from each of five mice is n = 5, not 15. Average within each mouse; if animals share cages or litters, use a mixed model.
- **Repeated measurements or two factors.** Welch's ANOVA is for one factor with independent groups. Use a repeated-measures or mixed model for the same animals over time, and a [two-way ANOVA](/guides/two-way-anova-vs-multiple-t-tests/) for crossed factors.

Write whichever of these applies into your analysis plan before you see the results.

## Check which test your software or AI ran

| Software (checked 10 Nov 2026) | Default for one-way ANOVA | To get Welch's ANOVA | Games–Howell |
| --- | --- | --- | --- |
| R 4.3.3 `oneway.test` | Welch (`var.equal = FALSE`) | Nothing to change | Not in base R (function above) |
| R `aov`, `anova(lm(...))` | **Classical** | Use `oneway.test` | — |
| SciPy 1.18 `stats.f_oneway` | **Classical** (`equal_var=True`) | `equal_var=False` (SciPy 1.16+) | `stats.tukey_hsd(..., equal_var=False)` |
| statsmodels 0.15 `anova_oneway` | Welch (`use_var="unequal"`) | Nothing to change | — |
| Pingouin 0.7 | `anova` is classical | `welch_anova` | `pairwise_gameshowell` |

Sources: [R `oneway.test`](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/oneway.test.html), [SciPy `f_oneway`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.f_oneway.html), [SciPy `tukey_hsd`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.tukey_hsd.html), [statsmodels `anova_oneway`](https://www.statsmodels.org/stable/generated/statsmodels.stats.oneway.anova_oneway.html), [Pingouin `pairwise_gameshowell`](https://pingouin-stats.org/generated/pingouin.pairwise_gameshowell.html) documentation.

**Three checks on a result you were given:**

1. **Look at the second degrees of freedom.** The classical test reports exactly N − k, a whole number (24 animals − 3 groups = 21). Welch's denominator df is usually a decimal and smaller (8.65). F(2, 21) means the classical test ran.
2. **Search the code.** `f_oneway(...)` without `equal_var=False`, `aov(...)`, or `oneway.test(..., var.equal = TRUE)` ran the classical test. `tukey_hsd(...)` without `equal_var=False` ran Tukey, not Games–Howell.
3. **Read the output header.** R prints `One-way analysis of means (not assuming equal variances)` for Welch's ANOVA.

A prompt that names the test and its parameters leaves less to the assistant's defaults:

```text
Compare the mean of [outcome] across [k] independent groups (one value per animal) using
Welch's one-way ANOVA, then Games–Howell for all pairwise comparisons. In Python use
scipy.stats.f_oneway(..., equal_var=False) and scipy.stats.tukey_hsd(..., equal_var=False).
Report group means and SDs, Welch's F with both degrees of freedom and the exact p-value,
and each pairwise difference with its 95% confidence interval.
```

## Report it so a reviewer can check it

**Methods sentence** (keep the bracketed parts only if they are true):

> Group means were compared with Welch's one-way ANOVA (unequal variances), followed by the Games–Howell procedure for all pairwise comparisons[, specified before the data were analysed]; the unit of analysis was the [animal]. Analyses used [software and version].

**Results sentence (the example above):**

> Plasma ALT was 100.0 U/L (SD 6.9, n = 10) after vehicle, 104.0 U/L (SD 3.5, n = 9) after the low dose and 115.2 U/L (SD 11.6, n = 5) after the high dose; Welch's F(2, 8.65) = 3.62, p = 0.072. The high-dose minus vehicle difference was 15.2 U/L (Games–Howell 95% CI −2.6 to 33.0, p = 0.086).

Give both degrees of freedom — the decimal one tells the reader Welch's test was used — the exact p-value, and the pairwise differences with their intervals, which carry the size of the effect.

## References

- Delacre M, Leys C, Mora YL, Lakens D (2019). Taking parametric assumptions seriously: arguments for the use of Welch's F-test instead of the classical F-test in one-way ANOVA. *International Review of Social Psychology* 32(1):13. [doi:10.5334/irsp.198](https://doi.org/10.5334/irsp.198)
- Games PA, Howell JF (1976). Pairwise multiple comparison procedures with unequal N's and/or variances: a Monte Carlo study. *Journal of Educational Statistics* 1(2):113–125. [doi:10.3102/10769986001002113](https://doi.org/10.3102/10769986001002113)
- Welch BL (1951). On the comparison of several mean values: an alternative approach. *Biometrika* 38(3–4):330–336. [doi:10.1093/biomet/38.3-4.330](https://doi.org/10.1093/biomet/38.3-4.330)
- Zimmerman DW (2004). A note on preliminary tests of equality of variances. *British Journal of Mathematical and Statistical Psychology* 57(1):173–181. [doi:10.1348/000711004849222](https://doi.org/10.1348/000711004849222)
- Software documentation accessed 10 November 2026: R `stats::oneway.test`; SciPy 1.18 `scipy.stats.f_oneway` and `scipy.stats.tukey_hsd`; statsmodels 0.15 `anova_oneway`; Pingouin 0.7 `pairwise_gameshowell`.

## Related

- [Welch's t-test vs Student's t-test](https://www.licklider.ai/guides/welch-vs-student-t-test/)
- [ANOVA significant but post hoc not](https://www.licklider.ai/guides/anova-significant-but-post-hoc-not/)
- [Which multiple-comparison correction?](https://www.licklider.ai/guides/which-multiple-comparison-correction/)
- [Statistical test chooser](https://www.licklider.ai/tools/statistical-test-chooser/)
- [ANOVA assumptions not met: what to do](https://www.licklider.ai/guides/anova-assumptions-not-met/)

## How this page was made

- Written and published by Licklider. The recommendation and every figure from the literature are cited to peer-reviewed studies or official software documentation listed under References; cited passages were checked against the source text.
- The worked example uses constructed data. Its numbers and our simulation were produced by stored code (Python with SciPy 1.18.1 and statsmodels 0.15.0; R 4.3.3); Python and R agree, and the Games–Howell results were cross-checked against three implementations.
- Separate review passes checked the statistics, the citations and readability before publication.