Analysis Family Ledger

Running 20 tests on the same dataset without multiple comparison correction gives you a 64% chance of at least one false positive, even if every individual test is performed correctly. Licklider currently tracks claim-bearing figures generated from the same dataset lineage, requires an analysis intent before inferential or claim-bearing figure work, and warns at export when family-wide multiplicity remains unresolved for confirmatory or publication-ready intent. For exploratory intent, Licklider discloses the family size automatically without blocking export.

STEP 1 - The Pitfall: The Multiple Testing Blind Spot

In modern research, a single dataset rarely supports a single analysis. A researcher might run a primary comparison, a sensitivity analysis, and a post-hoc exploration of an unexpected pattern from the same uploaded CSV. Each result can appear statistically valid in isolation. Together, they form a family of claim-bearing outputs, and the probability of at least one spurious finding grows as more claims are produced from the same dataset.

The problem is not only the number of tests. It is the lack of visibility into how many claim-bearing outputs have been generated from the same dataset lineage and which policy, if any, governs family-wide multiplicity.

• 5 tests on the same dataset: P(at least one false positive) about 22.6% (at alpha = 0.05)
• 10 tests: about 40.1%
• 20 tests: about 64.2%

Exploratory follow-up work is particularly prone to this problem because it is often launched after looking at the data. A significant follow-up result discovered after inspection carries more uncertainty than its nominal p-value alone suggests.

STEP 2 - Journal Requirement

Nature and Science both require authors to distinguish pre-specified analyses from post-hoc analyses. JAMA's Statistical and Methods Reporting requirements state that if multiple comparisons or outcomes were assessed, the methods used to account for this should be described.

For clinical trials, ICH E9(R1) formalizes the estimand framework and expects primary and secondary objectives to be specified in advance. The ICMJE similarly expects authors to make clear when data mining or data dredging occurred and how exploratory findings should be interpreted.

In practice, that requires a reliable record of how many claim-bearing outputs were produced from the same dataset family and what analysis intent governed them. Licklider's current analysis family record is designed to make that family-level summary visible during export.

STEP 3 - Licklider's Solution

Input

• The uploaded dataset identifier and dataset lineage key. Claim-bearing figures derived from the same dataset lineage and sharing the same analysis intent are grouped into the same analysis family. Exploratory and confirmatory figures from the same dataset are tracked as separate families.
• The Data Contract analysis intent for inferential or claim-bearing figure work. Licklider currently distinguishes exploratory, confirmatory, and publication-ready intent.
• Subset and filter operations are tracked separately as part of exploratory analysis pattern detection rather than as independent rows in the family summary.

The lineage key is currently derived from the source artifact ID. A stable, persistent lineage identifier is planned for a future release.

Output

• Analysis family summary. A family-level summary of claim-bearing figures in the active dataset lineage, including family size, declared intent, and whether the current figure is included in that count.
• Family-wide policy summary. Licklider currently records whether the family proceeds under holm_family_correction, preregistered_family_plan_confirmed, descriptive_only, or single_primary_or_not_applicable.
• Export disclosure draft. At export, Licklider generates a short draft paragraph summarizing the active family size, analysis intent, and currently selected family-wide multiplicity policy.
• Inspector. The Analysis Family card in the Inspector is shown only when the analysis intent is set to confirmatory or publication-ready.

Guard

STEP 4 - Current Export Disclosure (Illustrative - actual draft fills in available session data and may require additional detail before submission)

Example: Confirmatory family with Holm policy

Analysis family summary: the active dataset lineage currently contains 2
claim-bearing figures for confirmatory use, including the current figure.
Holm family-wide correction acknowledged across 2 claim-bearing figures.

Example: Single primary figure

Analysis family summary: only one primary claim-bearing figure was identified
in the active confirmatory family. No additional family-wide multiplicity
policy was required for this figure.

Example: Unresolved policy

[Draft / Needs review] Analysis family summary: the active dataset lineage
currently contains 2 claim-bearing figures for confirmatory use, including
the current figure. No family-wide multiplicity policy has been recorded.
Select a correction policy or keep the output descriptive-only before export.

The draft above reflects the current implementation. Per-analysis rows and richer manuscript-ready family narratives are planned for a future release.

Related Features