Checking multiple-testing procedures against their original papers

Current status

Six original papers now have a completed primary-source review in nomue Protocol's preparation for multi-group comparisons. The review checked the recorded claims against the original pages and independently recomputed numerical evidence. It returned GO within that six-paper scope.

The resulting proposal in PR #196 treats one source requirement as ready for closure, with a separate review supporting that treatment. The subsequent cumulative source record documents limited steward acceptance of SR-C; this is acceptance of the bounded source treatment, not adoption of the methods. This is public research evidence for Release 3 preparation. September 9 update: the current Welch scope has distinct public local Record verification and limited hosted access. Later source work is reported separately; the six-paper findings below retain their original scope.

What the originals resolved

Our earlier cataloguecovered 49 multi-group procedures. Many entries still depended on unread sources. This follow-up examined papers by Simes, Hochberg, Hommel, Shaffer, Rom, and Holland–Copenhaver to determine exactly which claims their procedures support.

Multiple testing asks how to control errors when a family contains several hypotheses. Procedures with similar names or thresholds can make different promises. A result about whether any hypothesis in a family is false does not automatically justify naming individual false hypotheses. A guarantee proved under independence also cannot be silently extended to dependent tests.

The source review made those distinctions explicit. Simes's global test and its exploratory individual rejections are separated. Hochberg and Hommel retain the validity conditions required by the Simes tests used in their arguments. Shaffer's procedures distinguish how many hypotheses can remain true from which specific hypotheses have already been rejected.

Those details affect executable rules. Hommel's shortcut uses a strict comparison during its search and a non-strict comparison for final rejection. Its general procedure and its logically restricted variant also give different results in the paper's example: three rejections and five, respectively. The review checked both against the printed example. Adjusted-output formulas derived by the investigators remain labelled as derivations, separate from statements printed in the papers.

When a table disagrees with its equation

Rom's 1990 paper defines critical values through a recurrence and presents a table obtained from it. In Table 1 on page 664, the modified-Hochberg entry for ten tests at α = 0.01 prints 1.01 × 10⁻³. Recomputing from equation (2) gives0.001004472598983613…, which rounds to 1.00 × 10⁻³ at the table's three significant digits.

The primary-source reviewer checked the page image, confirmed the discrepancy, and independently integrated the relevant order-statistic probability using exact polynomial arithmetic. The recurrence constants satisfied the defining probability identity exactly for one through ten tests. The other nineteen modified-Hochberg cells agreed at their printed precision. The conflict is therefore recorded as a disagreement within the source, rather than attributed to PDF extraction.

The approved bounded treatment preserves both values in the evidence record. It uses the defining equation and procedure to characterize the method, while excluding the printed table as authority for future numerical constants. Any implementation would require separately reviewed constants and numerical evidence. No formal erratum was verified, and this work makes no claim to be the first to have noticed the discrepancy.

What this advances in nomue

The practical gain is a more precise basis for deciding what a future verifier could promise. Source statements, investigator derivations, and conflicting numerical evidence now have distinct recorded roles. A later implementation can be reviewed against those roles instead of inheriting an ambiguous method name or copying a table without checking its definition.

PR #196 applies that treatment to the source requirement labelled SR-C. Its original candidate ledger recorded four requirements marked CLOSED and ten still incomplete; that historical count is not a count of four formally accepted hold closures. The subsequent bounded SR-C acceptance leaves the overall source set incomplete and the narrowed program scope remains in place. Selecting a procedure, proving its implementation, and opening a Release 3 public RFC are separate steps.

Evidence and review

The primary-source completion review in PR #195 checked six supplied originals against recorded file hashes and printed-page references. It reported no blocker or should-fix finding within that scope. PR #197 separately reviewed PR #196's application of the approved treatment and returned GO for that change; it also identified a public-language repair in the cited approval record outside the change.

Authoring and review used separate LLM-assisted contexts. The records disclose author-side OpenAI assistance, reviewer-side model provenance, and the steward's scoped independence decision. The steward retains responsibility for each bounded acceptance. These are project review records, not journal peer review. The papers themselves are not redistributed in the repository.

• Cumulative source record — Parts H and M record the limited SR-C acceptance and distinguish candidate counts from formally accepted holds.
• Source-acquisition result at the reviewed revision — Parts E–F contain the source interpretations and executable diagnostics; Part G records the bounded treatment and candidate ledger.
• Primary-source completion review, PR #195 — original-page checks, numerical recomputation, and scope.
• Proposal review, PR #197 — treatment, independence boundary, ledger, and remaining acceptance steps.
• PR #196 — live proposal and acceptance history.

Release 3 public discussion opened on September 9, 2026. The fixed proposal and evidence map limit the opening to supplied sources and preserve remaining scientific and numerical conditions; these historical findings keep their recorded scope.