Multiplicity and Analysis Families

How Licklider handles multiple comparisons within a figure and across claim-bearing figures in a project, and what you are asked to confirm before results can be used as claims.

When an analysis involves multiple comparisons, the risk of false positives increases. Licklider handles this at two distinct levels: within a single figure, and across all claim-bearing figures in a project.

These two levels are different problems with different solutions. This page explains both.

Within-figure multiple comparisons

When a single figure compares more than two groups — for example, a one-way ANOVA with three or more conditions — multiple pairwise comparisons are involved. Each comparison carries its own false positive risk, and running them without correction inflates the overall error rate.

For ANOVA-based analyses, post hoc correction methods (such as Tukey HSD, Holm, or Bonferroni) are selected in the statistics settings for the figure. For more detail on selecting a post hoc method → see One-Way ANOVA and Post Hoc.

In addition to the statistical correction applied to the results, Licklider asks you to confirm the disclosure policy for the comparison. When pairwise comparisons are present, you are asked to select one of the following:

Apply a family-wide correction A correction method has been applied to control the family-wise error rate across the pairwise comparisons. The correction is disclosed in the figure output.

Report the omnibus test only Only the overall test (e.g., the ANOVA F-test) is reported. Individual pairwise comparisons are not presented as claims.

Exploratory — no correction applied The comparisons are exploratory and will be disclosed as such. This option is available for exploratory analyses and is not eligible for claim-bearing export without further acknowledgment.

Analysis family tracking

Beyond a single figure, a project may contain multiple figures that each make a distinct inferential claim. If those figures test related hypotheses — drawn from the same dataset, contributing to the same research question — they form an analysis family.

Running multiple claim-bearing figures within the same family without accounting for the multiplicity across them inflates the project-level false positive rate, even if each figure individually applied within-figure correction.

Licklider can track families using project context, dataset lineage, and analysis intent, but it cannot fully determine the scientific boundary of a family from metadata alone. Whether two figures belong to the same claim family still depends on your research question, pre-specified plan, and how you intend the results to be interpreted together.

When this applies

Analysis family tracking activates when:

  1. The analysis intent is set to confirmatory or publication-ready, and
  2. Two or more claim-bearing figures have been generated within the same family

For exploratory analyses, a lightweight summary of the family structure is disclosed automatically without requiring confirmation.

What you are asked to confirm

When the conditions above are met, Licklider presents a confirmation step with three options:

Proceed with a Holm family-wide correction Apply Holm correction across the family of claims. The most conservative choice. Appropriate when multiple pre-specified hypotheses are being tested and you want to control the family-wise error rate across figures.

A preregistered family plan already covers this family A pre-registered analysis plan exists that specifies how the family of analyses will be handled. Appropriate when the study was pre-registered and the current analyses fall within that plan.

Show a descriptive figure only The family-wide multiplicity cannot be resolved at this time. The figure is kept in a descriptive state and is not eligible for inferential claims until the family policy is confirmed.

Holm is offered as the default family-wide option because it controls the family-wise error rate while avoiding some of the unnecessary power loss of a plain Bonferroni-style adjustment. It is a pragmatic default for projects that want strong error control without requiring users to choose among many advanced procedures before they can proceed.

Effect on export

When analysis intent is confirmatory or publication-ready and the family policy has not been confirmed, figures generated within that family are marked as provisional. Provisional figures can be exported with a disclosure noting the unresolved state, but they are not eligible for claim-bearing export until the family policy is resolved.

The family policy confirmation is visible in the Inspector alongside the figure's other disclosure requirements.

This stricter confirmation is limited to confirmatory and publication-ready work on purpose. Exploratory work often involves trying several analyses to understand the data-generating pattern, so Licklider surfaces the family structure as a disclosure first. When the user declares a stronger claim intent, unresolved project-level multiplicity becomes a gating issue rather than just a note.

How to set the analysis intent

Analysis family tracking depends on the analysis intent being set. If intent has not been set, family-level checks do not apply.

Set the analysis intent from the Inspector Overview tab or via Chat before running confirmatory analyses. For more detail — see Outcome Type and Analysis Intent.

If the intent is unset or set incorrectly, the family-level check may be too weak for a strong claim or unnecessarily strict for exploratory work. Licklider cannot infer your intended evidential standard unless you declare it.

What gets recorded

The within-figure correction method is recorded in the figure's disclosure and appears in the figure's export output.

The analysis family policy confirmation is recorded in the project's figsafe ledger. It applies to figures generated within the same family — defined by the project, the analysis intent, and the dataset lineage.

This information is separate from the Preprocessing Audit Log, which records data preparation steps rather than analysis policy decisions.

Design rationale and references

This page separates within-figure multiplicity from across-figure family tracking because they are not the same statistical problem. Correcting pairwise comparisons within one ANOVA does not by itself control the error rate across several figures that all support the same project-level claim.

Licklider asks for stronger confirmation only when the analysis intent is confirmatory or publication-ready because multiplicity is most consequential when results are being elevated to claim-bearing status. In exploratory work, the family structure is still disclosed, but the system does not force a project-level resolution before allowing iteration.

The software also does not attempt to fully automate the scientific definition of an analysis family. That boundary depends on the hypothesis structure and on how the reader should interpret the figures together. If the family is drawn too narrowly, the project can remain under-corrected; if drawn too broadly, the correction can become unnecessarily conservative.

References

  1. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.
  2. Bretz, F., Hothorn, T., & Westfall, P. (2010). Multiple Comparisons Using R. CRC Press.

What this page does not cover