Clinical scores over time (EAE, arthritis): how to compare groups without a test on every day

Quick answer. A test on every day of a clinical-score time course finds “significant” days by chance: with 22 daily tests and 10 mice per group, our simulation gave at least one “significant” day in 21% of experiments with no real effect. Instead, decide before the experiment which per-animal summaries answer your question, for example the cumulative score (sum of daily scores, the area under each animal’s curve), the maximum score, incidence (how many animals developed disease) and day of onset, and compare each with a method that suits it: a rank-based test such as Mann–Whitney for cumulative and maximum scores, Fisher’s exact test for incidence, and a log-rank test for onset, keeping animals that never became ill (cumulative score 0) and using the same days for every animal. Clinical scores such as the 0–5 scale for experimental autoimmune encephalomyelitis (EAE) or the 0–16 arthritis score are ordinal: a step from 1 to 2 is not necessarily the same as a step from 3 to 4. Decide in advance how you will score animals that die or reach the humane endpoint, and report the rule you used.

Terms used below. Ordinal score: a score with ordered categories but no fixed distance between them. Cumulative score: the sum of an animal’s daily scores, equal to the area under its score curve when scores are recorded daily. Maximum score: the highest score an animal reached. Incidence: the number of animals that developed disease (here, a score of 1 or more on any day). Day of onset: the first day with a score of 1 or more (a half-step of 0.5 does not count here; choose and state your threshold in advance, because it changes the results). Censored: an animal whose onset was not observed, either because it was still free of disease at the end of the study or because it was removed earlier for another reason; its onset day lies after its last day of observation. Mann–Whitney test: a rank-based comparison of two groups. Hodges–Lehmann shift: the median of all pairwise differences between an animal in one group and an animal in the other, an effect size that goes with the Mann–Whitney test. Fisher’s exact test: compares two proportions in small samples. Log-rank test: compares the time to an event between groups. Ordinal mixed model: a regression for ordinal scores with a term for each animal (a cumulative link mixed model). Humane endpoint: the condition at which an animal must be euthanised. False positive: a “significant” result when there is no real effect. Interquartile range: the middle half of the values, from the 25th to the 75th percentile. 95% confidence interval (CI): the range of effects compatible with the data. Power: the chance of detecting an effect that is really there. Kaplan–Meier curve: the share of animals not yet ill over time. Kruskal–Wallis test: the rank-based test for three or more groups. Post hoc test: a comparison made after an overall test, here on each day. (1 | mouse): model notation for a separate baseline for each mouse.

Which summary for your question

Your question	Per-animal summary	Test	Report
How much disease overall?	Cumulative score (area under the curve) over a fixed period, the same days for every animal	Mann–Whitney (Kruskal–Wallis for three or more groups)	Medians with interquartile ranges, Hodges–Lehmann shift with 95% CI
How severe at its worst?	Maximum score	Mann–Whitney	Medians
How many animals became ill?	Incidence	Fisher’s exact test	Counts and percentages
Did disease start later?	Day of onset; animals still free of disease at the end are censored at the last day, and animals removed earlier (for reasons unrelated to disease) at their last day of observation	Log-rank test	Kaplan–Meier median onset if reached; Kaplan–Meier curves
How does the score change day by day?	All daily scores	Ordinal mixed model (ordinal::clmm in R)	The model’s treatment effect

Fleming and colleagues describe three types of question in EAE studies: the level or extent of disease, the time until a specified event such as onset, and rates or percentages such as incidence (Fleming et al. 2005); Emerson and colleagues give “the area under the curve of daily clinical scores” as an example of an extent-of-disease measure (Emerson et al. 2009). Baker and colleagues reported that “as many as 55%” of the EAE studies they assessed used statistical tests they considered inappropriate, and recommend that scores from “arbitrary scale measurements” be compared with non-parametric tests (Baker et al. 2014). The Mann–Whitney test compares medians only when the groups’ distributions have a similar shape and spread; otherwise describe the result as one group tending to have higher scores (Kruskal–Wallis vs ANOVA).

Worked example: 10 vehicle-treated and 10 drug-treated mice, days 7 to 28

Constructed data: daily EAE-style scores (0–5 in steps of 0.5). By construction, the drug lowers incidence (from 90% to 70%), lowers the maximum score (mean 3.0 to 2.2) and delays onset by a day; in this sample all 20 mice became ill, so the incidence comparison and the censoring for onset are not illustrated here. The random seed was the first one tried, not hand-picked. Stored code and data: scores.csv, model.py (the disease-course model), make_data.py, analysis.py, analysis.R and the analysis output; analysis.py and analysis.R print the same output.

Summary	Vehicle	Drug	Test
Incidence	10 of 10	10 of 10	Fisher p = 1.00
Cumulative score, median (interquartile range)	45.0 (33.0 to 49.9)	30.0 (22.1 to 38.1)	Mann–Whitney p = 0.089; Hodges–Lehmann shift −13.0 (95% CI −25.5 to 2.5; hl_ci.R)
Maximum score, median	3.5	2.75	Mann–Whitney p = 0.055
Day of onset, median (all mice became ill)	12.5	13.5	Log-rank p = 0.55
Separate test on each of 22 days			p < 0.05 on four days (16, 21, 22, 24)

The planned summary is inconclusive: the cumulative score was about 13 points lower with the drug (the Hodges–Lehmann shift, the median of all mouse-to-mouse differences, not the 15-point gap between the medians; about 0.6 points per day over 22 days), but with 10 mice per group the interval runs from a 25-point reduction to a 2.5-point increase. Separate daily tests would instead have reported four “significant” days, a result the simulation shows to be unreliable evidence. In 10,000 simulated experiments of this design (simulation.py, the simulation output):

Scenario (rate of p < 0.05)	Cumulative score	Maximum score	At least one of 22 daily tests (unadjusted)
No drug effect (false-positive rate)	4.4%	4.4%	21.1%
Drug effect as constructed (power)	59.6%	51.9%	80.5%

The daily arm of the simulation used a Mann–Whitney test on each day; a post hoc test on each day after a repeated-measures ANOVA is the same kind of procedure. The daily tests look more powerful only because they also find “effects” in a fifth of experiments where there are none. The 21.1% is lower than 22 independent tests would give (68%) because a mouse’s scores on neighbouring days are strongly related and early days with all scores at 0 cannot be tested. With 10 mice per group, even the planned cumulative-score comparison had only about 60% power here; plan the group size for the primary summary in advance (how many biological replicates).

# Condensed: analysis.py also prints incidence, the maximum score, onset and the daily tests; the CI comes from hl_ci.R
# Your file: one row per mouse per day, columns mouse, group ("vehicle"/"drug"), day, score; every mouse scored on the same days
# Wide sheet (one column per day)? d = wide.melt(id_vars=["mouse", "group"], var_name="day", value_name="score")
import numpy as np, pandas as pd
from scipy import stats

d = pd.read_csv("scores.csv")                        # columns: mouse, group, day, score; one row per mouse per day
cum = d.groupby(["group", "mouse"]).score.sum()      # cumulative score = area under each mouse's daily curve
v, g = cum["vehicle"].to_numpy(), cum["drug"].to_numpy()
print(np.median(v), np.median(g), stats.mannwhitneyu(g, v, method="asymptotic").pvalue)
print(np.median(np.subtract.outer(g, v)))           # Hodges-Lehmann shift (drug - vehicle)

d <- read.csv("scores.csv")
cum <- aggregate(score ~ mouse + group, data = d, sum)                   # cumulative score per mouse
wilcox.test(score ~ factor(group, levels = c("drug", "vehicle")), data = cum,
            conf.int = TRUE, exact = FALSE)                               # p, shift (drug - vehicle) and 95% CI
# Day-by-day modelling of the ordinal scores (not run in this example):
# ordinal::clmm(factor(score, ordered = TRUE) ~ group * day + (1 | mouse), data = d)

Before you analyse

1. Choose the summaries and the period in advance, for example cumulative score from day 7 to day 28 as the primary outcome, maximum score, incidence and onset as secondary. Name one as primary. If the data are already collected, state which summary you chose and when, and report the others as exploratory.
2. Score blind to group, with the same scorer where possible, and record the scale and its half-steps exactly.
3. Decide how to score animals that die or are euthanised. Common rules are to carry the last score forward or to assign the scale maximum (5 for EAE, not the animal’s own highest score) for the remaining days; each changes the cumulative score, so state the rule before unblinding and report how many animals it affected.
4. Keep animals that never became ill. They have a cumulative score of 0, count towards incidence and enter the onset analysis as censored. Summaries such as mean peak score “of those mice within an experimental group that displayed clinical signs” (Miller & Karpus 2007) describe only the ill animals; label them as such, and use the log-rank test with censoring for onset.
5. Show every animal’s curve or the daily scores, not only group means with error bars.
6. Same days for every animal. The cumulative score is comparable only over the same days; if some animals were scored on fewer days, restrict the period or state how missing days were filled, and report how many animals this affected.
7. Several summaries are several tests. Name the primary one, or adjust (which correction to use).

Check what an AI assistant ran

1. The assistant ran a two-way repeated-measures ANOVA with a post hoc test on every day. Ask for the planned per-animal summaries instead.
2. The assistant ran a t-test on mean scores. Scores are ordinal; ask for a rank-based test with a Hodges–Lehmann shift, or an ordinal model.
3. The assistant silently dropped animals without disease, or animals that died. Ask how many were excluded and why.
4. Mean peak score among ill animals reported as the treatment effect. Ask for incidence and the cumulative score over all animals as well.

Report it

Methods:

Mice were scored daily from day [7] to day [28] on a 0–5 scale by an observer blind to treatment. The primary outcome, [specified in advance / chosen after data collection but before analysis], was the cumulative clinical score (sum of daily scores); secondary outcomes were the maximum score, incidence (score ≥ 1) and day of onset. Cumulative and maximum scores were compared with Mann–Whitney tests, incidence with Fisher’s exact test and onset with a log-rank test, with mice that did not develop disease censored at day [28] (R [4.3.3]). Mice reaching the humane endpoint ([n] mice) were assigned [the rule] for the remaining days; mice removed for other reasons were censored at their last day of observation in the onset analysis.

Results (the bracketed values are from the constructed example):

The median cumulative score was [30.0] (interquartile range [22.1] to [38.1]) with the drug and [45.0] ([33.0] to [49.9]) with vehicle (Hodges–Lehmann shift [−13.0], 95% CI [−25.5] to [2.5]; Mann–Whitney p = [0.089]). The median maximum score was [2.75] with the drug and [3.5] with vehicle (p = [0.055]; if the groups’ distributions differ in shape, write instead that scores tended to be higher with vehicle); incidence was [10/10] in both groups, and median onset was day [13.5] with the drug and day [12.5] with vehicle (log-rank p = [0.55]).

Replace the bracketed values with your own.

References

• Fleming KK, Bovaird JA, Mosier MC, Emerson MR, LeVine SM, Marquis JG (2005). Statistical analysis of data from studies on experimental autoimmune encephalomyelitis. Journal of Neuroimmunology 170:71–84. doi:10.1016/j.jneuroim.2005.08.020
• Emerson MR, Gallagher RJ, Marquis JG, LeVine SM (2009). Enhancing the ability of experimental autoimmune encephalomyelitis to serve as a more rigorous model of multiple sclerosis through refinement of the experimental design. Comparative Medicine 59:112–128. PMC2703151
• Baker D, Lidster K, Sottomayor A, Amor S (2014). Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLOS Biology 12:e1001756. doi:10.1371/journal.pbio.1001756
• Miller SD, Karpus WJ (2007). Experimental autoimmune encephalomyelitis in the mouse. Current Protocols in Immunology 77, Unit 15.1. doi:10.1002/0471142735.im1501s77