Two-way ANOVA vs separate t-tests: does the effect differ between groups?

Quick answer. “The drug worked in wild type (p < 0.05) but not in knockout (p > 0.05)” is a claim that the drug effect differs between genotypes. Test that difference directly: fit a two-way ANOVA (or linear model) with genotype, treatment and their interaction, and read the interaction. Use the interaction instead of two separate t-tests: one significant and one non-significant result can come from drug effects that are almost the same. If you only want to say the drug works in wild type, a test inside wild type answers that; the moment you contrast the genotypes, you need the interaction. This applies to independent mice with one measurement each; if the same mice are measured repeatedly (genotype × time), use a repeated-measures or mixed model instead.

Terms used below. The interaction asks whether the drug effect differs between genotypes. A main effect is the drug effect averaged over both genotypes. A simple effect is the drug effect inside one genotype. Type I, II and III sums of squares are three ways of splitting the variation in the ANOVA table; they give different main-effect rows only when group sizes differ. A sum of squares is the share of the variation a row of the table explains. The residual (pooled) error is the leftover variation, used as the yardstick for every test in the model. Sum-to-zero contrasts code each factor so that groups are compared with the overall level, which Type III tables need. HC3 is a correction that stops assuming equal SDs. In emmeans, revpairwise means the second level minus the first (drug − vehicle).

Why two t-tests mislead

Gelman and Stern put the problem in one line: the difference between “significant” and “not significant” is not itself statistically significant. Large changes in p can correspond to small, non-significant changes in the effect itself [Gelman & Stern 2006]. The mistake is common in top journals. Of 513 neuroscience articles in Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience, 157 compared two effects; 79 of them used the incorrect two-separate-tests procedure and 78 the correct one. The authors’ additional analysis suggests the error is even more common in cellular and molecular neuroscience [Nieuwenhuis et al. 2011].

Here it is with numbers. Constructed data: grip strength in wild-type (WT) and knockout (KO) mice given vehicle or drug, 8 mice per group, one measurement per mouse.

Group	Mean	SD	n
WT, vehicle	10.00	2.00	8
WT, drug	13.00	2.00	8
KO, vehicle	10.00	2.00	8
KO, drug	12.00	2.00	8

Analysis	Result	What it can claim
Welch t-test inside WT	drug − vehicle = 3.0, t(14.0) = 3.00, p = 0.0095	The drug raises grip strength in WT
Welch t-test inside KO	drug − vehicle = 2.0, t(14.0) = 2.00, p = 0.065	Not enough evidence in KO alone — not evidence that the drug fails in KO, and not a comparison with WT
Interaction, two-way ANOVA	F(1, 28) = 0.50, p = 0.48; difference in drug effect (WT − KO) = 1.0, 95% CI −1.9 to 3.9	Could not tell whether the drug effect differs between genotypes (the CI includes no difference and large differences)
Main effect of treatment	F(1, 28) = 12.50, p = 0.0014	Averaged over genotypes, the drug raises grip strength

The drug effects are 3.0 and 2.0 units, and the test of their difference gives p = 0.48. The “significant versus not” pattern came from noise around two similar effects. The interval matters as much as the p-value: −1.9 to 3.9 includes no difference and includes a KO effect much smaller than the WT effect. Write “we could not tell whether the effect differs”, not “the effect is the same”.

What to do: run the interaction test whenever your sentence compares an effect across groups (“stronger in”, “only in”, “abolished in”), and report its estimate with a confidence interval. This applies to independent mice with one measurement each; for repeated measurements on the same mice use a mixed model.

Which question are you asking?

A two-way design answers three different questions. Match your sentence to the right row.

Your sentence	Quantity	How to test it
“The drug raises grip strength” (overall)	Main effect of treatment: drug − vehicle, averaged over genotypes	Treatment row of the two-way ANOVA
“The drug effect is smaller in KO” / “the knockout abolishes the effect”	Interaction: (WT drug − WT vehicle) − (KO drug − KO vehicle)	Interaction row, plus its estimate and CI
“The drug works in KO mice”	Simple effect: drug − vehicle within KO	Contrast from the fitted model (below)

Read the simple effects rather than the main effects when the interaction is clearly present: an average over genotypes describes neither genotype. Correct for the simple effects if you test several — see which multiple-comparison correction to use.

Run it in R and Python

Long-format data, one row per mouse, columns genotype, treatment and y. Download the example data as grip.csv to run the code as printed.

d <- read.csv("grip.csv")
d$genotype  <- factor(d$genotype,  levels = c("WT", "KO"))
d$treatment <- factor(d$treatment, levels = c("vehicle", "drug"))
fit <- lm(y ~ genotype * treatment, data = d)       # * = both main effects and the interaction
car::Anova(fit, type = 2)                           # genotype:treatment row is the interaction

tapply(d$y, list(d$genotype, d$treatment), sd)      # cell SDs: check they are similar
plot(fit, which = 1)                                # residuals vs fitted: look for a fan shape or outliers

library(emmeans)
emm <- emmeans(fit, ~ treatment | genotype)
confint(contrast(emm, "revpairwise"), adjust = "none")       # simple effects (drug − vehicle) with 95% CIs
summary(contrast(emm, "revpairwise"), adjust = "none")       # their p-values
confint(contrast(emm, interaction = c("revpairwise", "pairwise"), by = NULL))  # difference in drug effect between genotypes

import pandas as pd
import statsmodels.formula.api as smf
from statsmodels.stats.anova import anova_lm

d = pd.read_csv("grip.csv")
fit = smf.ols("y ~ C(genotype, Sum) * C(treatment, Sum)", data=d).fit()
print(anova_lm(fit, typ=2))   # state the type; the default is Type I
print(d.groupby(["genotype", "treatment"])["y"].std())   # cell SDs: check they are similar

# Simple effects and the interaction contrast from a cell-means model (works for unequal group sizes too)
cells = smf.ols("y ~ C(genotype):C(treatment) - 1", data=d).fit()
order = [(g, t) for g, t in (s.replace("C(genotype)[", "").replace("C(treatment)[", "").replace("]", "").split(":")
                             for s in cells.params.index)]
def vec(weights):
    return [weights.get(cell, 0) for cell in order]
contrasts = {
    "Drug - vehicle in WT": vec({("WT", "drug"): 1, ("WT", "vehicle"): -1}),
    "Drug - vehicle in KO": vec({("KO", "drug"): 1, ("KO", "vehicle"): -1}),
    "WT - KO difference in drug effect": vec({("WT", "drug"): 1, ("WT", "vehicle"): -1, ("KO", "drug"): -1, ("KO", "vehicle"): 1}),
}
for name, c in contrasts.items():
    t = cells.t_test(c)
    low, high = t.conf_int()[0]
    print(f"{name}: {t.effect[0]:.2f}, 95% CI {low:.2f} to {high:.2f}, t({int(cells.df_resid)}) = {t.tvalue[0][0]:.2f}, p = {t.pvalue:.4f}")

Both give F(1, 28) = 0.501, p = 0.485 for the interaction and F(1, 28) = 12.50, p = 0.0014 for treatment on the example data. With a balanced design like this one, Types I, II and III give identical tables.

Simple effects from the model. The model’s pooled error (residual mean square 4.00 on 28 degrees of freedom) gives each within-genotype contrast a standard error of √(4.00 × 2/8) = 1.00:

Contrast	Estimate	95% CI	t(28)	p
Drug − vehicle in WT	3.00	0.95 to 5.05	3.00	0.0056
Drug − vehicle in KO	2.00	−0.05 to 4.05	2.00	0.055
Difference (interaction)	1.00	−1.90 to 3.90	0.71	0.48

These use 28 error degrees of freedom instead of 14, so p-values differ slightly from the separate t-tests. The emmeans lines in R and the cell-means block in Python print this table (R prints the interaction as estimate 1, SE 1.41, 95% CI −1.9 to 3.9); the Python contrasts work for unequal group sizes too; with several simple effects, adjust them as a family (unadjusted here because each answers its own planned question).

The two-way ANOVA assumes independent animals, roughly symmetric residuals and similar SDs in the four cells; it pools one residual SD for every test, including the simple effects. Check this with the two lines in the code above (cell SDs and the residual plot). If one cell’s SD is clearly larger than the others (for example more than twice the smallest), or the residual plot fans out, use heteroscedasticity-robust tests (car::Anova(fit, type = 2, white.adjust = "hc3")) or a model with a separate variance per cell (nlme::gls(y ~ genotype * treatment, weights = varIdent(form = ~ 1 | genotype * treatment), data = d)).

What to do: for independent mice with one measurement each, run one model with the interaction, check the cell SDs and the residual plot, report the interaction row, then report only the simple effects your hypothesis needs (repeated measurements on the same mice need a mixed model instead).

Plan for the interaction: it needs about four times the animals

In a balanced 2 × 2 design, the interaction is a difference of two differences, so its standard error is twice that of a main effect of the same size. For 80% power at two-sided α = 0.05 (exact noncentral t; our stored calculation):

Effect to detect (in SD units)	Main effect: mice per group (80% power, two-sided α 0.05, balanced 2 × 2)	Interaction: mice per group (same settings)
1.0 SD	9 (36 in total)	32 (128 in total)
0.5 SD	32 (128 in total)	127 (508 in total)

A study sized for the drug’s main effect will usually miss an interaction of the same size. That is why “no significant interaction” in a small study is weak evidence of no difference.

What to do: plan the sample size for the interaction if “the effect differs between genotypes” is your hypothesis, and state that in Methods.

Unequal group sizes: say which type of ANOVA table you used

When groups are unequal, the main-effect rows depend on how the sums of squares are calculated. Drop four mice from the example (n = 28; grip_unbalanced.csv) and the genotype row changes:

Table	Genotype F (p)	Treatment F (p)	Interaction F (p)
Type I, genotype entered first	1.26 (0.27)	9.27 (0.0056)	0.62 (0.44)
Type I, treatment entered first	0.43 (0.52)	10.10 (0.0041)	0.62 (0.44)
Type II	0.43 (0.52)	9.27 (0.0056)	0.62 (0.44)
Type III (sum-to-zero contrasts)	0.54 (0.47)	8.73 (0.0069)	0.62 (0.44)

The interaction row is the same in every table; the main effects are not. This code reproduces the table:

u <- read.csv("grip_unbalanced.csv")                 # four mice fewer (n = 28)
u$genotype  <- factor(u$genotype,  levels = c("WT", "KO"))
u$treatment <- factor(u$treatment, levels = c("vehicle", "drug"))
anova(lm(y ~ genotype * treatment, data = u))        # Type I, genotype first
anova(lm(y ~ treatment * genotype, data = u))        # Type I, treatment first
fit_u <- lm(y ~ genotype * treatment, data = u)
car::Anova(fit_u, type = 2)                          # Type II
contrasts(u$genotype)  <- contr.sum(2)               # sum-to-zero contrasts, needed for Type III
contrasts(u$treatment) <- contr.sum(2)
car::Anova(lm(y ~ genotype * treatment, data = u), type = 3)

Software defaults differ:

Software	Default
R anova(lm(...))	Sequential (Type I): each term is added in the order of the formula [R anova.lm help]
R car::Anova()	Type II; Type III is available but only sensible with sum-to-zero (or other orthogonal) contrasts, not R’s default treatment contrasts [car manual]
Python statsmodels anova_lm()	Type I unless you pass typ=2 or typ=3 [statsmodels source and docstring]

Langsrud (2003) argues for Type II over Type III for unbalanced data. Whichever you choose, the decision rule is the same: interpret the interaction first, and name the table type in Methods so others can reproduce your numbers.

When separate t-tests are fine

• Your claims stay inside each group (“the drug works in WT”) and you never compare the effects. A t-test within WT answers that; correct if you run several (see which correction).
• You have one pre-specified comparison between two groups and no second factor. That is a plain two-group comparison.

Use a repeated-measures or mixed model with group, time and their interaction instead of an ordinary two-way ANOVA if the same mice are measured repeatedly (for example genotype × time); see repeated measures ANOVA vs mixed model. The statistical test chooser does not yet cover designs that combine a between-animal and a within-animal factor.

Check what an AI assistant ran

AI-generated code can run one t-test per genotype. Check four things:

1. Is there an interaction term? In R or statsmodels formulas, genotype * treatment or a genotype:treatment term. genotype + treatment has none.
2. Is the claim about the interaction? “Only in WT” or “abolished in KO” needs the interaction row, not two p-values.
3. Which table type? anova() in R and anova_lm() in statsmodels default to Type I; with unequal groups the main effects depend on formula order.
4. Type III with the right contrasts? If the code asks for Type III, check for contr.sum in R or C(x, Sum) in statsmodels.

What to do: copy these four checks into your conversation with the assistant and ask it to show the interaction row and its confidence interval.

Report it

We analysed grip strength with a two-way ANOVA with genotype (WT, KO), treatment (vehicle, drug) and their interaction (n = 8 mice per group; one measurement per mouse). The drug increased grip strength (main effect F(1, 28) = 12.50, p = 0.0014). The data could not show whether the drug effect differed between genotypes (interaction F(1, 28) = 0.50, p = 0.48; difference in drug effect, WT − KO, 1.0 units, 95% CI −1.9 to 3.9). Analyses used R 4.3.3; with equal group sizes, Type I, II and III sums of squares are identical.

Replace the third sentence with the interaction estimate and the simple effects you planned, with the correction you used, if the interaction is present.

Corrections

11 November 2026: this page said that the statistical test chooser routes designs such as genotype × time, in which the same mice are measured repeatedly, to the right analysis. The chooser has no route that combines a between-animal and a within-animal factor: it offers either a repeated-measures model without the group factor or an ordinary two-way ANOVA. The sentence now points to the repeated-measures guide and says what the chooser does not cover.

References

• Gelman A, Stern H (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician 60(4):328–331. doi:10.1198/000313006X152649
• Nieuwenhuis S, Forstmann BU, Wagenmakers EJ (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nature Neuroscience 14(9):1105–1107. doi:10.1038/nn.2886
• Langsrud Ø (2003). ANOVA for unbalanced data: use Type II instead of Type III sums of squares. Statistics and Computing 13:163–167. doi:10.1023/A:1023260610025
• R Core Team. anova.lm: ANOVA for linear model fits (R help page; behaviour checked in R 4.3.3). stat.ethz.ch/R-manual/R-patched/library/stats/html/anova.lm.html
• Fox J, Weisberg S, et al. car: Companion to Applied Regression, Anova reference (manual for version 3.1-5, 2026; example run with 3.1-2). cran.r-project.org/package=car
• statsmodels developers. statsmodels.stats.anova.anova_lm (version 0.15.0). statsmodels.org/stable/generated/statsmodels.stats.anova.anova_lm.html