Outlier Sensitivity Report

When outliers are excluded from a dataset, the question is not just whether the exclusion was justified — it is whether the conclusion changes depending on what was excluded. Licklider evaluates this automatically for every figure where outlier exclusions have been applied.

How the evaluation works

Licklider compares the results from three variants of the analysis:

Raw The analysis run on the original data, before any outlier exclusions.

Processed The analysis run on the data after exclusions have been applied. This is the version that appears in the figure.

Robust An analysis run using a method that is less sensitive to extreme values — for example, using rank-based statistics rather than means.

This three-way comparison is meant to answer two different questions at once: whether the reported conclusion depends on the exclusion decision, and whether a less outlier-sensitive analysis points in the same general direction. The robust variant is a sensitivity companion, not a universal replacement for the main method.

Licklider compares each pair of variants and detects the following changes:

• Whether statistical significance changed (a result that was significant became non-significant, or vice versa)
• Whether the direction of the effect changed (a positive effect became negative, or vice versa)
• Whether the confidence interval moved across the null value
• Whether the effect size changed substantially (a relative change of 25% or more is treated as material)

Licklider cannot automatically determine whether the exclusion rule itself was scientifically appropriate, pre-specified, or chosen only after seeing the data. It also cannot guarantee that the robust variant is the most appropriate alternative for every assay or endpoint. A stable sensitivity result reduces one class of concern, but it does not by itself justify the exclusion.

Severity levels

The results of the comparison are summarized as a severity level:

The 25% threshold is a product heuristic for flagging potentially meaningful instability in effect size, not a universal scientific cutoff. It is intended to surface results whose practical magnitude changes enough to deserve explicit review, even when significance does not flip.

The severity level is visible in the Inspector alongside the outlier exclusion disclosure.

What this means for your analysis

None or Low severity The conclusion does not depend on whether the outliers were excluded. The figure can be used in a claim-bearing context, with the exclusion disclosed.

Medium severity The effect size estimate changes substantially depending on what is excluded. This is worth noting in the methods section. The figure can still be used, but the sensitivity result should be acknowledged.

High severity The conclusion changes materially depending on whether the outliers are included. A significance flip or direction change means the result is not robust to the exclusion decision. Licklider marks the figure as provisional until the sensitivity result is acknowledged and a position is taken on which analysis to report.

That provisional state is intentional. When the claim changes depending on whether excluded rows are present, the software should not let the figure silently appear as if the result were stable.

Responding to a high severity result

When severity is high, you have several options:

• Report the processed result with full disclosure — acknowledge that the exclusion changes the conclusion and explain why the exclusion was appropriate. The processed result is then eligible for export with the sensitivity result disclosed.

• Report the raw result — report the unexcluded analysis and note the outlier presence. This is appropriate when the biological explanation for the outlier is uncertain.

• Keep the figure as descriptive only — if the sensitivity cannot be resolved, the figure can be used for illustration purposes only, not as a claim-bearing result.

Where to find the report

The sensitivity evaluation is shown in the Inspector when a figure has outlier exclusions applied. The report describes what was compared and what changed. If no outlier exclusions were applied, the sensitivity report does not appear.

The full disclosure — including the exclusion criterion, the number of excluded observations, and the sensitivity result — is included in the figure's export output when acknowledged.

Design rationale and references

Licklider compares Raw, Processed, and Robust variants because outlier handling creates more than one analytic degree of freedom. A result can look convincing after exclusion while being much less stable in the raw data, and a robust variant helps show whether the overall conclusion depends mainly on extreme observations.

Licklider treats flipped significance, reversed effect direction, and confidence intervals moving across the null as high-severity events because these changes alter the substantive conclusion a reader would take from the figure. That is why high-severity figures are treated as provisional until the user explicitly acknowledges the issue.

The report is not designed to certify that an outlier exclusion was justified. Its role is narrower: to make the dependence of the reported result on the exclusion decision visible enough that the reader can judge robustness and disclosure quality.

References

1. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632
2. Wilcox, R. R., & Rousselet, G. A. (2018). A guide to robust statistical methods in neuroscience. Current Protocols in Neuroscience, 82(1), e39. https://doi.org/10.1002/cpns.39

What this page does not cover

• How to define an outlier exclusion criterion → see Outlier and Exclusion Policy
• What the Outlier Exclusion Log records → see Outlier Exclusion Log
• How researcher degrees of freedom relate to outlier decisions → see Outliers and Researcher Degrees of Freedom