Multiple Comparison Compliance

When you run three pairwise comparisons at alpha=0.05, the probability of at least one false positive is nearly 14%. When you run ten comparisons, it is 40%. Multiple comparison correction is not an optional statistical nicety —it is the difference between a controlled and uncontrolled error rate.

STEP 1 —The Pitfall

In practice, multiple comparison correction is routinely omitted from figures with three, four, or five groups —often because the individual comparisons "look significant" and correction would reduce them. This is precisely backwards: correction is most important when results are borderline significant, because those are the results most likely to be false positives in the uncorrected test.

A figure showing five pairwise comparisons, each with an asterisk, but no mention of correction, is either reporting uncorrected results (inflated false positive rate) or correction without disclosure. Both are reporting errors.

[Image placeholder: Bar chart with 4 groups and 6 pairwise comparisons, all marked with stars. Annotation shows: uncorrected FWER = 26.5%. Then same chart with Holm correction applied —3 comparisons remain significant. Shows the practical impact of correction.]

Six pairwise comparisons without correction: FWER = 26.5%. After Holm correction, the number of significant pairs drops from 6 to 3.

STEP 2 —Journal Requirement

ANOVA without post-hoc correction is rejected by most statistics-literate reviewers. Nature, NEJM, and journal guidelines universally require disclosure of the correction method used for multiple comparisons. "Post-hoc Tukey HSD" or "Holm-corrected pairwise comparisons" must appear in Methods or figure legends.

STEP 3 —Licklider Solution

Input

Number of groups in the comparison (from analysis configuration)
Analysis family membership (from Analysis Family Ledger)
Correction method selection: Holm, Bonferroni, Benjamini-Hochberg (FDR), Tukey HSD, Dunnett's

Output

For >=3 group comparisons: correction method selection dialog shown before results display
Corrected p-values calculated and applied to significance annotations
Uncorrected p-values available for display (explicitly labeled as uncorrected)
Validity Score dimension: Fail if publication-ready export contains uncorrected pairwise comparisons across >=3 groups without explicit "no correction" justification
Correction method auto-included in figure legend and Methods text

Guard

The guard fires when: (1) three or more groups are present, (2) pairwise comparisons are requested, and (3) no correction method has been selected. The guard presents a selection dialog and does not proceed until a correction method is chosen or the user explicitly selects "No correction —exploratory analysis" with an acknowledgment that FWER is uncontrolled. In publication-ready mode, the "no correction" option is unavailable for inferential analyses.

[Image placeholder: Licklider correction method dialog —shows comparison count (6 pairwise), FWER without correction (26.5%), and correction method dropdown: Holm / Bonferroni / Tukey HSD / Benjamini-Hochberg. The "proceed without correction" button is grayed out in publication-ready mode.]

Correction method dialog: cannot proceed to publication-ready export without a selection.

STEP 4 —Draft Output (Draft / Needs review)

Pairwise comparisons were conducted using Holm's sequential Bonferroni
correction to control the familywise error rate (FWER). Corrected p-values
are reported. All six pairwise comparisons were tested; three comparisons
remained significant after correction (Groups A vs. C: p_adj = 0.018;
Groups A vs. D: p_adj = 0.031; Groups B vs. D: p_adj = 0.044).