Multiverse Analysis

Multiverse analysis is the systematic exploration of all defensible analytical paths for a given dataset and research question. Instead of asking "is my result significant?" it asks "across all analyses I could have reasonably done, what distribution of results would I have seen?" The answer reveals whether the published result is representative or cherry-picked.

STEP 1 —The Pitfall

Every analysis involves dozens of small decisions: how to handle outliers, which covariates to include, how to define the primary outcome, which test to use. Each decision is often presented as natural or obvious, but many are in fact arbitrary. When these decisions are made after seeing the data —or when the presentation of results implies they were made before —the effective Type I error rate of the published result is substantially higher than the declared alpha.

Multiverse analysis makes the full space of analytical decisions visible, showing the distribution of p-values and effect sizes across all defensible combinations. A p-value of 0.03 in a multiverse where 80% of paths produce significant results is a different claim from a p-value of 0.03 in a multiverse where 20% of paths produce significant results.

[Image placeholder: Multiverse plot (specification curve) showing all analytical paths on the x-axis sorted by effect size, with effect size on y-axis. Paths above the significance threshold shown in blue, below in grey. Dotted line at zero. The proportion of paths exceeding the threshold labeled as "% paths significant = 73%."]

Specification curve (multiverse plot): each point is one analytical path. The distribution of results across paths reveals whether the primary result is representative.

STEP 2 —Journal Requirement

Multiverse analysis is recommended by Steegen et al. (2016, Psychological Science) and increasingly expected in high-tier journals for studies making strong causal or clinical claims. PLOS ONE, Nature Human Behaviour, and Psychological Science have published editorial guidelines recommending multiverse analysis as a transparency tool for studies with substantial researcher degrees of freedom.

STEP 3 —Licklider Solution

Input

Dataset with group structure declared
Analytical decision space: user-defined or auto-generated from the analysis configuration
Decisions covered: outlier handling (none / ROUT Q=1% / IQR / Grubbs), test family (parametric; rank-based branches only where the engine supports them — see Non-Parametric Alternatives), covariate inclusion (if applicable), correction method

Output

Full enumeration of analytical paths (combinatorial product of decision options)
Effect size and p-value for each path
Specification curve visualization: paths sorted by effect size, significance threshold shown
Summary: % paths with significant result in same direction as primary, % with opposite direction, % non-significant
Primary analysis highlighted in the specification curve

Guard

Multiverse analysis is an advisory feature —it does not block export. However, if the primary result falls in the top 10% of effect sizes in the multiverse (i.e., the primary path was notably optimistic relative to the full space), the guard issues an advisory suggesting the author consider reporting a more conservative analytical path as the primary, or adding the multiverse visualization as a required supplementary figure.

[Image placeholder: Licklider multiverse analysis panel showing the full specification curve with 48 paths (3 outlier options x2 test families x2 covariate options x4 correction methods). The primary analysis is marked with a star. 65% of paths show significant results in the same direction.]

Multiverse panel: 48 analytical paths, primary analysis marked with a star, robustness summary badge shows "65% of paths consistent with primary conclusion."

STEP 4 —Draft Output (Draft / Needs review)

We conducted a multiverse analysis to assess the robustness of the
primary finding across 48 defensible analytical paths defined by
combinations of: outlier handling (none, ROUT Q=1%, IQRx.5),
test family (parametric, non-parametric), and multiple comparison
correction (Holm, Bonferroni, FDR, uncorrected). 65% of paths (31/48)
produced a significant result in the same direction as the primary
analysis. 4% (2/48) produced a significant result in the opposite
direction. The full specification curve is provided in Supplementary Figure 3.

Example journal-style text. A full parametric vs. rank multiverse is not guaranteed in-product — verify scope against Non-Parametric Alternatives.