Multiple Comparison Enforcement

The Test Configuration Guard's multiple comparison check operates at the test selection step —before any p-values are calculated. By requiring correction method selection upfront, it prevents the common sequence of "run all comparisons, see which ones are significant, then add correction only to those that are still significant after correction."

STEP 1 —The Pitfall

Selective correction —correcting only some comparisons, or correcting after seeing which survive —is statistically invalid. The correction method must be declared and applied to all comparisons within a family simultaneously. Comparing only significant pairs post-hoc, or applying different corrections to different subsets of the same figure, inflates the effective false positive rate in ways that are invisible to reviewers.

The Multiple Comparison Compliance dimension (4.1.5) addresses the output level. This guard addresses the input level: correction must be declared before analysis proceeds, not retrofitted after.

[Image placeholder: Timeline diagram showing two workflows. "Wrong" workflow: run all 6 comparisons xsee which are significant xapply correction only to borderline ones xreport. "Right" workflow: declare correction method xrun all 6 comparisons with correction applied uniformly xreport. Highlights where the FWER control is broken in the wrong workflow.]

Selective correction workflow (left) breaks FWER control. Uniform pre-declared correction (right) maintains it.

STEP 2 —Journal Requirement

The correction method must be declared in Methods and match the corrections applied to all reported comparisons. Post-hoc tests must be identified by name (Tukey HSD, Dunnett's, Holm, etc.). Journals with statistical review will request clarification if the number of comparisons and the correction method are inconsistent.

STEP 3 —Licklider Solution

Input

Number of groups in the comparison (auto-detected from group variable)
Comparison structure: all-pairwise, vs. control only (Dunnett's), planned contrasts
Analysis family membership from Analysis Family Ledger (for family-level FWER control)

Output

Correction method dialog shown at the point of test selection (not after results)
Available methods: Holm, Bonferroni, Benjamini-Hochberg (FDR), Tukey HSD, Dunnett's, None (exploratory)
FWER or FDR rate displayed for the chosen method and comparison count
Correction applied uniformly to all comparisons in the family
Comparison count and method logged in analysis record

Guard

Analysis cannot proceed past test selection for >=3 groups without a correction method selection. The "None —exploratory" option is available but triggers a disclosure log entry. In publication-ready mode, "None" requires explicit acknowledgment that the figure is labeled as exploratory and that FWER is uncontrolled.

[Image placeholder: Test configuration dialog appearing before any comparison results are shown. Shows: group count (4), comparison count (6 all-pairwise), correction method dropdown, and FWER display: "Without correction: FWER = 26.5%. With Holm: FWER 竕､ 5%." The "Run analysis" button is grayed out until a selection is made.]

Correction method required before analysis runs: group count, comparison count, FWER impact displayed upfront.

STEP 4 —Draft Output (Draft / Needs review)

Four groups were compared using one-way Kruskal-Wallis test, followed
by pairwise Dunn post-hoc tests with Holm correction for all six
pairwise comparisons (familywise error rate 竕､ 5%). Correction was
applied uniformly to all comparisons in the analysis family prior
to significance evaluation.