Permutation Test

The permutation test is the most assumption-free significance test available. It requires only that the data be exchangeable under the null hypothesis —an assumption met by most randomized experiments. When parametric assumptions fail and simple rank-based summaries are insufficient, the permutation test is the principled alternative in methodological terms.

Product boundary: Full one-click permutation workflows and side-by-side rank panels are not all verified in the current public engine — align expectations with Non-Parametric Alternatives and Known Limitations.

STEP 1 —The Pitfall

Researchers facing assumption violations typically choose between: ignore the violation and use the parametric test, or switch to a Wilcoxon / Mann-Whitney–style rank test in external software. Both choices can be suboptimal. The parametric test may produce unreliable p-values; a rank-based test loses information about the actual magnitude of the effect and may not test the same hypothesis (it tests the distribution, not the mean difference).

The permutation test directly tests the hypothesis of interest —typically whether the observed difference between groups could have arisen by chance under random assignment —without requiring any distributional assumptions. Its main cost is computational, which is negligible for typical experimental sample sizes.

[Image placeholder: Permutation test visualization showing the null distribution of the test statistic (difference in means) across 10,000 permutations as a histogram, with the observed statistic marked as a vertical line. The proportion of permutations exceeding the observed value is labeled as the permutation p-value.]

Permutation null distribution: the observed test statistic (vertical line) vs. the distribution under 10,000 random permutations. p = the proportion of permutations exceeding the observed value.

STEP 2 —Journal Requirement

Permutation tests are recommended by ARRIVE 2.0 for small-sample animal studies with non-normal distributions. Statistical Analysis and Data Mining and similar journals recommend permutation tests when parametric assumptions cannot be verified. Many journals accept permutation p-values as the primary significance test for n < 15 per group.

STEP 3 —Licklider Solution

Input

Group structure and outcome variable from Variable & ID Mapping
Number of permutations: default 10,000 (user-adjustable to 100,000 for high precision)
Test statistic: difference in means (default), difference in medians, or ratio of means

Output

Permutation null distribution visualization
Permutation p-value (two-tailed by default; one-tailed if declared via One-Sided Disclosure)
Observed test statistic with its position in the null distribution
Permutation CI (optional): derived from the percentile of permuted statistics
Where supported: comparison with parametric results; rank-based panels are not guaranteed end-to-end

Guard

When the Normality & Homoscedasticity check (4.1.1) flags a violation, the intended product behavior is to surface stronger robust options; automatic one-click permutation as the sole primary analysis is not verified everywhere. Proceeding with the parametric test after a violation flag may require explicit justification where guards are active.

[Image placeholder: comparison panel showing p-values side by side: t-test, optional rank-based (if exposed), permutation — illustrative; not all columns are verified in-product today.]

Illustrative parallel comparison — verify which panels your build exposes.

STEP 4 —Draft Output (Draft / Needs review)

Group differences were assessed using a two-sample permutation test
(10,000 permutations, test statistic: difference in means) to avoid
parametric assumptions that were not met by the data
(Shapiro-Wilk: W = 0.84, p = 0.031). The observed difference
(Group A ~Group B = 4.2 units) exceeded 96.8% of permuted differences
(permutation p = 0.032, two-tailed).