Published

Renaming treatment groups changes agricolae’s REGW result

With observations and group membership unchanged, renaming groups changes a p-value from 0.0363 to 0.0791 and reverses a 5% decision.

The names change; the observations do not

Treatment names identify groups in an experiment. Renaming them should leave the comparison of the same observations unchanged. In agricolae, an R package for analysing experiments,REGW.test() compares treatment groups. In our example, swapping two group names changes a reported p-value from 0.0363 to 0.0791. The first result is below the 5% significance threshold; the second is above it.

The displayed interval and the letters used to summarize group differences also change. Every observation remains in the same numeric group. The effect comes from renaming the groups, without adding, removing or changing data.

Tasuku Kobayashi sent the report and reproduction files to the agricolae maintainer by email on September 10, 2026. Reported by email — upstream confirmation pending.The submission date is recorded by the sender. There is no public upstream discussion or accepted fix to link to for this report; the technical materials below are published by Licklider for readers to inspect and run.

A small example crosses the 5% threshold

The example has four groups of six observations, with means of 0, 1.625, 4 and 8. Within each group, observations alternate one unit below and above its mean. We first name the groups A, B, C and D, then exchange the names B and D. The first two numeric groups are therefore compared as A–B before renaming and A–D afterward.

The same numeric pair before and after renaming
Group labelsCompared pairp-valueDisplayed lower boundDisplayed upper bound
A, B, C, DA–B0.0363−3.15369694116−0.0963030588423
A, D, C, BA–D0.0791−3.395202650940.145202650939

The difference between the compared means is −1.625 in both cases. Both fitted models have 20 residual degrees of freedom and residual mean square 1.2. The first displayed interval excludes zero; the second includes it. With group=TRUE, the same two groups change from distinct letters to a shared letter.

Run the reported comparison

The recorded environment is CRAN agricolae 1.3-7 on R4.6.1, Linux x86_64. R prints the package version as1.3.7. The example uses the documented model interface, so the function extracts the model’s error parameters itself.

library(agricolae)
y <- rep(c(0, 13/8, 4, 8), each=6) + rep(c(-1, 1), 12)
run <- function(labels) {
  d <- data.frame(y=y, trt=factor(rep(labels, each=6)))
  REGW.test(aov(y ~ trt, data=d), "trt", group=FALSE)$comparison
}
run(c("A", "B", "C", "D"))["A - B", ]
run(c("A", "D", "C", "B"))["A - D", ]

The full script checks that the compared observations are identical after renaming, prints both pairwise comparisons and group displays, and records the R/package versions and model error parameters. It makes four REGW calls using an unmodified installed package, with no random inputs or external data.

With agricolae installed, run Rscript --vanilla reproduce.R in a clean session.

The apparent cause is an ordering by labels

In the inspected source,Omeans <- order(means[, 1], decreasing = TRUE) orders the first column, which contains treatment labels. The response means are in the second column. The resulting ranks affect the range size used for p-values and displayed intervals.

For the reported pair, the span in label order changes from two to four while its span in mean order stays two. The report asks whether this ordering should use the response means. That identifies a source location to inspect; it does not establish a complete correction to the REGW procedure.

What this finding establishes

The check is a simple invariance requirement: a one-to-one renaming should preserve inference for the same observations and group membership. Neither observed p-value is treated as a certified reference answer. This example does not estimate how often the issue occurs, prove the procedure’s overall false-positive control, or validate a production patch.

For Licklider, the work demonstrates how a small reproducible example and source inspection can expose a consequential change in statistical output. It does not add REGW verification to nomue. See thecurrent verification scope.

The report and reproducer were prepared with AI assistance. The published observations come from native R execution. Upstream confirmation and any eventual correction will be recorded separately from this reproduction.

Package documentation and reporting route