# Nested data (cells within mice): average per animal or fit a mixed model?

> Several cells, wells or slices per animal? Compare groups with one mean per animal, or with a mixed model that has a random intercept for each animal. Worked example in R and Python.

Canonical: https://www.licklider.ai/guides/nested-data-mixed-model/
Section: Guides › Nested measurements
Author: Licklider · Published 2026-11-10 · Last reviewed 2026-11-10 (UTC)

> **Quick answer.** When you measure several cells from each mouse and each mouse belongs to one treatment group, the mouse is the unit you compare. Two routes give a valid test. **Route A:** average the cells within each mouse and run Welch's t-test (or ANOVA) on the mouse means. **Route B:** fit a linear mixed model with a random intercept for each mouse, taking its p-value from a small-sample t-based method (R's lmerTest) rather than a z-test when you have fewer than about 50 mice. Use Route A when every mouse contributes a similar number of cells, and always when there are 3 or fewer mice per group; use Route B when cell counts differ a lot between mice or you need cell-level covariates. If mice share cages or litters within a group, or if both conditions were applied inside each mouse, the model needs an extra term (see the table).

> **Terms used below.** *Unit*: the animal, culture or experiment that was treated independently. *Welch's t-test*: a two-group test that does not assume equal spreads. *ANOVA*: the extension of the t-test to three or more groups. *Linear mixed model*: a regression that knows which cells came from which mouse. *Random intercept*: lets each mouse have its own baseline level, so cells from one mouse are not treated as independent. *Covariate*: an extra measured variable, such as cell size. *z-test vs t-test*: a z-test assumes evidence from many units; a t-test corrects for having few. *Degrees of freedom (df)*: a count that reflects how many independent units the test thinks it has. *Intraclass correlation (ICC)*: the share of the total variation that lies between mice.

## Which route fits your data

First settle what the independent unit is — the animal, culture or experiment that was treated separately ([what counts as n](/guides/what-counts-as-n/)). This page starts after that: you have several measurements inside each unit.

| Your data | Route | In R | In Python |
| --- | --- | --- | --- |
| Similar number of cells per mouse, one measurement of interest | A: mouse means, then Welch's t-test | `aggregate()` then `t.test()` | `groupby().mean()` then `ttest_ind(equal_var=False)` |
| Cell counts differ a lot between mice (for example 8 vs 25; a rough guide is more than about twice as many cells in one mouse as another) | B: random intercept per mouse | `lmerTest::lmer(y ~ group + (1 \| mouse))` | `smf.mixedlm(..., groups=mouse)` plus the t-based p-value in the code below |
| A cell-level covariate (cell size, cell type, time in culture) | B, with the covariate in the model | `lmer(y ~ group + size + (1 \| mouse))` | `smf.mixedlm("y ~ group + size", ...)` |
| Three or more groups | A with Welch's ANOVA, or B with `anova(fit)` | `oneway.test()` / `anova(fit)` | Route A: `anova_oneway()` |
| Both conditions applied inside each animal or experiment | Paired means, or add `(1 \| mouse:condition)` | `lmer(y ~ condition + (1 \| mouse) + (1 \| mouse:condition))` | Route A on paired means: `ttest_rel()` (worked code in [SuperPlots](/guides/superplots/)) |
| 3 or fewer mice per group (this row takes precedence over unequal cell counts) | A | as above | as above |
| Several mice share a cage or litter | B with cage or litter added, `(1 \| cage/mouse)`, or cage means as the unit when the treatment was given per cage | `lmer(y ~ group + (1 \| cage/mouse))` | Route A on cage means |

Both routes answer the same question: do mice given the treatment differ from control mice? Neither lets you count cells as independent replicates. The random intercept `(1 | mouse)` alone fits designs where each animal belongs to one group; when both conditions are applied inside each animal or experiment, use paired means or add `(1 | mouse:condition)` ([SuperPlots](/guides/superplots/) shows that design).

**What to do:** name the unit in your plan, then pick the row that matches how many measurements each unit gives you.

## What goes wrong when every cell counts

The worked example has 5 control and 5 treated mice and 154 neurite-length measurements, with 8 to 25 cells per mouse (constructed data: [cells.csv](/guides/nested-data-mixed-model/cells.csv); the seed was chosen to show the contrast, and the simulation below gives the general rate). Here is the same data analysed three ways:

| Analysis | Treated − control (µm) | Test | p |
| --- | --- | --- | --- |
| Every cell as a replicate (wrong) | 5.83, 95% CI 3.57 to 8.09 | Welch t(152.0) = 5.10 | 0.000001 |
| Route A: Welch's t-test on 10 mouse means | 4.42, 95% CI −2.25 to 11.10 | Welch t(8.0) = 1.53 | 0.165 |
| Route B: mixed model, random intercept (lmerTest) | 4.59, 95% CI −2.09 to 11.28 | t(7.97) = 1.59 | 0.15 |

The cell-level test claims overwhelming evidence; the two valid analyses find an effect of plausible size whose 95% CI, with 10 mice, still runs from below zero to about 11 µm. The cell-level test has 152 degrees of freedom because it treats 154 cells as 154 independent observations. The valid tests have about 8, because there are 10 mice.

How often does the cell-level test raise a false alarm? In this design we simulated 10,000 times with **no** treatment effect (between-mouse SD 4 µm, within-mouse SD 6 µm, the same cell counts):

| Analysis | Share of runs with p < 0.05 when there is no effect |
| --- | --- |
| Every cell as a replicate (Welch) | 43.8% |
| Route A: Welch's t-test on mouse means | 4.5% |
| Route B: lmerTest, Satterthwaite (2,000 runs) | 5.4% |
| Route B: statsmodels MixedLM z-test (2,000 runs) | 8.5% |

Published reviews find the same pattern. Aarts et al. (2014) found nested data in 53% of 314 neuroscience papers and showed that, in the worst cases they examined, ignoring the dependency can push the false-positive rate as high as 80%. Eisner (2021) simulated 5 cells from each of 3 animals per group (15 cells per group) and found p < 0.05 in 29% of runs with no real difference.

**What to do:** run Route A or B before you interpret the p-value whenever your current analysis has cells, wells or slices as n.

## Run it in R and Python

The data file has one row per cell, with columns `group`, `mouse` and `length` ([cells.csv](/guides/nested-data-mixed-model/cells.csv); the complete scripts are [page_snippet.R](/guides/nested-data-mixed-model/page_snippet.R) and [page_snippet.py](/guides/nested-data-mixed-model/page_snippet.py)). Install once with `install.packages("lmerTest")` in R or `pip install pandas scipy statsmodels` in Python.

```r
library(lmerTest)                        # loads lme4 and adds Satterthwaite df
d <- read.csv("cells.csv")
d$group <- factor(d$group, levels = c("control", "treated"))

m <- aggregate(length ~ group + mouse, data = d, FUN = mean)   # Route A
t.test(length ~ group, data = m)                                # Welch by default

fit <- lmer(length ~ group + (1 | mouse), data = d)             # Route B
summary(fit)                                                    # Satterthwaite df and p

ct <- summary(fit)$coefficients["grouptreated", ]                # estimate, SE, df, t, p
ct["Estimate"] + c(-1, 1) * qt(0.975, ct["df"]) * ct["Std. Error"]   # 95% CI
vc <- as.data.frame(VarCorr(fit))$vcov                          # mouse variance, cell variance
vc[1] / sum(vc)                                                 # ICC
```

```python
import pandas as pd
from scipy import stats
import statsmodels.formula.api as smf

d = pd.read_csv("cells.csv")            # one row per cell: group, mouse, length

# Route A: one mean per mouse, then Welch's t-test on the means
m = d.groupby(["group", "mouse"], as_index=False)["length"].mean()
print(stats.ttest_ind(m.loc[m.group == "treated", "length"],
                      m.loc[m.group == "control", "length"], equal_var=False))

# Route B: random intercept per mouse (statsmodels reports a z-test)
fit = smf.mixedlm("length ~ group", d, groups=d["mouse"]).fit(reml=True)
print(fit.summary())                   # the p-value column is a z-test: do not report it with few mice

# Small-sample p-value and CI for Route B: estimate/SE against t on (mice - groups) df
est, se = fit.params["group[T.treated]"], fit.bse["group[T.treated]"]
df = d["mouse"].nunique() - 2                       # 10 mice, 2 groups -> 8
t = est / se
half = stats.t.ppf(0.975, df) * se
print(f"treated - control = {est:.2f}, 95% CI {est - half:.2f} to {est + half:.2f}, "
      f"t({df}) = {t:.2f}, p = {2 * stats.t.sf(abs(t), df):.3f}")
icc = fit.cov_re.iloc[0, 0] / (fit.cov_re.iloc[0, 0] + fit.scale)
print(f"ICC = {icc:.2f}")
```

What the output shows:

| Output (Python reports treated − control; R's `t.test` prints control − treated) | R | Python |
| --- | --- | --- |
| Route A, Welch's t-test on mouse means | t = −1.53, df = 8.0, p = 0.165 (control − treated) | t = 1.53, df = 8.0, p = 0.165 |
| Route B, treatment estimate and SE | 4.59, SE 2.90 | 4.59, SE 2.90 |
| Route B, test | t(7.97) = 1.59, p = 0.152 | summary: z = 1.58, p = 0.113; t-based lines: t(8) = 1.58, p = 0.152, 95% CI −2.09 to 11.28 |
| ICC | 0.33 | 0.33 |
| Mouse variance, cell variance | 18.3, 36.7 | 18.3, 36.7 |

The estimate and its standard error agree, but the reference distribution changes the p-value: 0.152 with a t distribution, 0.113 with statsmodels' default z-test. The next section explains why the t-based value is the one to report.

**What to do:** copy the block for your language, replace `length`, `group` and `mouse` with your column names, and check the number of animals: statsmodels prints it as `No. Groups`, R's `summary(fit)` as `groups: mouse, 10`.

## Few animals: take the p-value from a t-based method

A mixed model's evidence about the treatment comes from the animals, so its degrees of freedom are close to the number of mice minus the number of groups (here 7.97, from 10 mice). R's lmerTest uses the Satterthwaite approximation for these degrees of freedom by default. statsmodels' `MixedLM` compares the estimate with a normal distribution instead (the summary column is headed `z`), which treats the evidence as if it came from many animals. With 10 mice that gave 8.5% false positives in our simulation instead of 5%.

The lme4 documentation flags the same point: its list of p-value options marks the methods with "finite-size corrections (important when the number of groups is <50)".

- **In R:** load `lmerTest` before fitting (Satterthwaite is its default; `ddf = "Kenward-Roger"` is the alternative).
- **In Python with fewer than about 50 animals:** report Route A, or use the t-based lines in the Python block above (estimate and standard error against a t distribution on mice − groups degrees of freedom) and say so in Methods.

**What to do:** check which distribution your software used for the p-value; if it says `z` and you have a few dozen animals or fewer, switch to one of the two options above.

## When the two routes agree, and when they differ

**Equal cells per mouse.** To show the equivalence we kept the first 8 cells recorded for every mouse (a convenience selection, `d.groupby("mouse").head(8)`, used only for this demonstration; [cells_balanced.csv](/guides/nested-data-mixed-model/cells_balanced.csv)). In this balanced subset, Student's t-test on the mouse means and the random-intercept model give the same test; with unequal cell counts they differ, as the next paragraph shows:

| Balanced subset (8 cells per mouse) | Estimate | SE | Test | p |
| --- | --- | --- | --- | --- |
| Student's t-test on mouse means | 4.00 | 2.76 | t(8) = 1.449 | 0.185 |
| Random-intercept model (lmerTest) | 4.00 | 2.76 | F(1, 8) = 2.10 (= 1.449²) | 0.185 |
| Random-intercept model (statsmodels, z-test) | 4.00 | 2.76 | z = 1.449 | 0.147 |

The last row is the small-sample issue from the previous section in its purest form: identical estimate and standard error, smaller p. Welch's version, the site's default, differs from Student's only when the two groups' spreads differ.

**Unequal cells per mouse.** Route A gives every mouse the same weight; Route B gives more weight to mice with more cells, in proportion to how much those extra cells add. That is why the estimates differ slightly (4.42 vs 4.59 µm). Eisner (2021) points out that averaging "takes no account of the fact that some animals provide more cells than others" and recommends a hierarchical or mixed model for that reason. Lazic (2010, Table 1) lists both "include random effects" and "average over observations" as solutions for nested data.

**What to do:** with similar cell counts use Route A (or Route B, which gives nearly the same answer); with very unequal counts use Route B — unless there are 3 or fewer mice per group, where Route A is the safer choice.

## How much do extra cells buy you?

The intraclass correlation (ICC) is the share of the total variation that lies between animals. In the example it is 18.3 ÷ (18.3 + 36.7) = 0.33. The design effect, 1 + (cells per animal − 1) × ICC, says how many cells carry the information of one independent cell:

| Cells per animal | ICC 0.05 | ICC 0.1 | ICC 0.3 | ICC 0.5 |
| --- | --- | --- | --- | --- |
| 5 | 1.2 | 1.4 | 2.2 | 3.0 |
| 10 | 1.45 | 1.9 | 3.7 | 5.5 |
| 20 | 1.95 | 2.9 | 6.7 | 10.5 |
| 50 | 3.45 | 5.9 | 15.7 | 25.5 |

In the example (about 15 cells per mouse, ICC 0.33) the design effect is about 5.8, so the 154 cells carry about as much information on the overall mean as 27 independent cells would. That is not the n for the treatment comparison, which rests on the 10 mice. Aarts et al. (2014) reach the same design conclusion: improving a nested design "nearly always concerns collection of more truly independent observations, rather than more observations from one research object".

**What to do:** plan for more animals rather than more cells per animal; use the [sample size calculator](/tools/sample-size-calculator/) with the number of animals as n.

## Assumptions, and what to do when one fails

Both routes assume that animals are independent of each other. Mice housed or born together within one group are not independent: model the cage or litter, or use it as the unit ([what counts as n](/guides/what-counts-as-n/)). Both routes also assume that the values, or the mouse means, are roughly symmetric. Route B also assumes that the spread between mice and the spread between cells are similar in both groups.

| If | Then |
| --- | --- |
| Lengths, intensities or counts are right-skewed | Analyse `log(length)`; the difference becomes a ratio of geometric means. For counts that include zeros, the log is undefined: with fewer than about 20 mice per group, sum the counts per mouse and fit a quasi-Poisson model with t-based inference (an offset for cells or area counted; [tests for count data](/guides/statistical-tests-for-count-data/)), or analyse per-mouse proportions or rates with Welch's t-test ([comparing percentages between groups](/guides/comparing-percentages-between-groups/)); count mixed models (Poisson, negative binomial or binomial with a random intercept) give z-tests that run above 5% false positives at these sizes |
| The groups' spreads clearly differ | Route A with Welch's t-test, which does not assume equal spreads |
| Several mice share a cage or litter | Add the cage or litter as a further random effect, `(1 \| cage/mouse)`, or use cage means as the unit when the treatment was given per cage |
| The model reports a singular fit (mouse variance 0) | Report Route A as well; a zero between-mouse variance with few mice is an estimation limit, not proof that mice do not differ |
| Very few animals (3 or fewer per group) | Route A; with so few animals the between-mouse variance rests on a handful of values |

**What to do:** decide the route and the scale (raw or log) before you look at the result, and write both into the analysis plan.

## Check what an AI assistant ran

AI-written code can pass every cell to `ttest_ind` or `t.test` and report hundreds of degrees of freedom. Before you use the result, check three things:

1. **Degrees of freedom.** For a two-group comparison they should be close to the number of animals minus 2, not the number of cells.
2. **A grouping term.** Look for `(1 | mouse)` in R or `groups=` in statsmodels. A plain `lm()` or `ols()` on cells has no random effect. If mice share cages or litters within a group, the cage or litter needs a term too.
3. **The distribution behind the p-value.** `z` in a statsmodels summary means a large-sample test; with few animals use one of the options in the section above.

**What to do:** ask for the Route A or Route B code from this page, with your column names, if any check fails.

## Report it

> (Constructed example; replace every number with yours.) Neurite length was measured in 8–25 cells per mouse (154 cells, 5 mice per group). To account for cells being nested within mice, we compared groups with a linear mixed model with a random intercept for mouse (lme4 1.1-35.1, lmerTest 3.1-3, R 4.3.3; Satterthwaite degrees of freedom). Treated mice had longer neurites than controls by 4.6 µm (95% CI −2.1 to 11.3; t(8.0) = 1.59, p = 0.15). The intraclass correlation was 0.33. A Welch's t-test on per-mouse means gave the same conclusion (difference 4.4 µm, 95% CI −2.3 to 11.1; p = 0.165).

State n as the number of animals in the figure legend, and show the cells and the animal means together ([SuperPlots](/guides/superplots/)).

## Corrections

- **10 November 2026:** the row for skewed counts previously recommended a Poisson or negative binomial mixed model without a size condition. With few mice per group those models' z-tests give more than 5% false positives, so the row now recommends per-mouse totals with quasi-Poisson and t-based inference, or per-mouse rates with Welch's t-test, below about 20 mice per group.

## References

- Aarts E, Verhage M, Veenvliet JV, Dolan CV, van der Sluis S (2014). A solution to dependency: using multilevel analysis to accommodate nested data. *Nature Neuroscience* 17:491–496. [doi:10.1038/nn.3648](https://doi.org/10.1038/nn.3648)
- Bates D, Mächler M, Bolker B, Walker S (2015). Fitting linear mixed-effects models using lme4. *Journal of Statistical Software* 67(1):1–48. [doi:10.18637/jss.v067.i01](https://doi.org/10.18637/jss.v067.i01)
- Eisner DA (2021). Pseudoreplication in physiology: more means less. *Journal of General Physiology* 153(2):e202012826. [doi:10.1085/jgp.202012826](https://doi.org/10.1085/jgp.202012826)
- Kuznetsova A, Brockhoff PB, Christensen RHB (2017). lmerTest package: tests in linear mixed effects models. *Journal of Statistical Software* 82(13):1–26. [doi:10.18637/jss.v082.i13](https://doi.org/10.18637/jss.v082.i13)
- Lazic SE (2010). The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis? *BMC Neuroscience* 11:5. [doi:10.1186/1471-2202-11-5](https://doi.org/10.1186/1471-2202-11-5)
- lme4 documentation, help page `pvalues` ("Getting p-values for fitted models"), lme4 1.1-35.1.
- statsmodels 0.15.0 documentation, `MixedLM` and `MixedLMResults`.

## Related

- [How to report a mixed model](https://www.licklider.ai/guides/how-to-report-a-mixed-model/)
- [What counts as n: biological vs technical replicates](https://www.licklider.ai/guides/what-counts-as-n/)
- [SuperPlots: show cells and animals in one figure](https://www.licklider.ai/guides/superplots/)
- [Welch's t-test vs Student's t-test](https://www.licklider.ai/guides/welch-vs-student-t-test/)
- [Repeated measures ANOVA vs mixed model](https://www.licklider.ai/guides/repeated-measures-anova-vs-mixed-model/)
- [Microbiome relative abundance](https://www.licklider.ai/guides/microbiome-relative-abundance/)
- [Calcium imaging analysis](https://www.licklider.ai/guides/calcium-imaging-analysis/)
- [Sholl analysis](https://www.licklider.ai/guides/sholl-analysis/)

## How this page was made

- Written and published by Licklider. Recommendations cite peer-reviewed methods papers and software documentation listed under References; cited figures were checked against the source text.
- The worked example uses constructed data. Its numbers and the false-positive simulations were produced by stored code (Python 3.13 with SciPy 1.18.1 and statsmodels 0.15.0; R 4.3.3 with lme4 1.1-35.1 and lmerTest 3.1-3), and the two languages agree on every estimate and standard error; where their default p-values differ, the page says so and shows how to match them.
- Separate review passes checked the statistics, the citations and readability before publication.