# When can you exclude an outlier? A rule, a check and what to write

> Exclude a value only for a documented reason that does not depend on the value itself, decided before you look at the group comparison. Otherwise keep it, use a robust analysis and report both versions.

Canonical: https://www.licklider.ai/guides/when-can-you-exclude-an-outlier/
Section: Guides › Choosing a test
Author: Licklider · Published 2026-11-10 · Last reviewed 2026-11-10 (UTC)

> **Quick answer.** Exclude a value only when you have a documented reason that does not depend on the value itself — a failed injection, a haemolysed sample, a protocol deviation noted in the lab book — and, where you can, write the criterion down before you look at the group comparison (before unblinding, if the study is blinded; ARRIVE 2.0 item 3a). Being far from the others is not a reason. If the value is extreme but you have no such record, keep it, report the analysis you planned as the main result, and show the result without it as a sensitivity analysis; when you expect extreme values, plan a robust analysis (log scale, rank-based or trimmed means) in the protocol.

> **Terms used below.** *Outlier*: a value far from the rest of its group. *False positive*: a "significant" result when there is no real difference; with α = 0.05 it should happen in about 5% of experiments. *z score*: how many standard deviations (SD) a value lies from its group mean. *Scaled MAD*: the median absolute deviation from the median multiplied by 1.4826, so that it matches the SD for normal data; it changes little when one value is extreme. *Robust analysis*: a test that is little affected by a few extreme values. *Sensitivity analysis*: running the analysis both with and without a value to show whether the conclusion depends on it. *Welch's t-test*: compares two means without assuming equal spreads. *Mann–Whitney test*: a rank-based comparison of two groups. *Trimmed mean / Yuen's test*: the mean after dropping the most extreme 20% at each end, and the t-test built on it. *Grubbs' test*: a test for one outlier, built for normally distributed data. *CI*: confidence interval, the range of effects compatible with the data. *Unblinding*: the point at which you learn which animal received which treatment. *Pre-specified*: written down, with a date, before anyone looked at the group comparison. *Haemolysed*: red blood cells burst in the sample, which distorts many plasma assays. *20% trimmed mean at n = 6*: drops the single lowest and single highest value (20% of 6 rounds down to 1), leaving 4 per group.

## Three questions before you touch the value

1. **Is there a record of something going wrong with this animal or sample?** A note in the lab book, a flagged run, a failed quality control, a deviation from the protocol. If yes, and the note was made without knowing how the value compared with the other group, you may exclude it: state the reason and report it (ARRIVE 2.0 item 3b).
2. **Is the value impossible?** A body weight of 250 g in a mouse, a negative concentration, a typing error. Correct it from the raw record if you can; if not, treat it as missing and report it.
3. **Neither?** Then it is a real measurement of a real animal. Keep it and run the analysis you planned (see below). An unusual animal can be the most informative one in the study.

The ARRIVE 2.0 guidelines ask you to "describe any criteria used for including or excluding animals (or experimental units) during the experiment, and data points during the analysis. Specify if these criteria were established a priori. If no criteria were set, state this explicitly", and, for each group, to "report any animals, experimental units, or data points not included in the analysis and explain why" ([ARRIVE item 3](https://arriveguidelines.org/arrive-guidelines/inclusion-and-exclusion-criteria); [Percie du Sert et al. 2020](https://doi.org/10.1371/journal.pbio.3000410)).

If the "outlier" is one technical replicate well within a sample, the question is different: average the technical replicates or drop the failed well by your assay's pre-set rule, then analyse the samples ([what counts as n](/guides/what-counts-as-n/)).

**What to do:** answer the three questions in writing before you look at the group comparison again.

## Why "it changes the p-value" is never a reason

Deleting a value because the result then looks better turns α = 0.05 into a much larger false-positive rate. We simulated 100,000 experiments with no true difference between two groups, analysed with Welch's t-test, and applied common habits:

| Strategy (no true effect) | 6 per group | 10 per group | 30 per group |
| --- | --- | --- | --- |
| Analyse every value, as planned | 4.6% | 4.8% | 4.9% |
| If p > 0.05, drop the most extreme value and test again | 12.4% | 11.0% | 8.4% |
| If p > 0.05, drop values with \|z\| > 2 and test again | 4.7% | 8.9% | 10.9% |
| Try dropping each value in turn and keep any significant version | 16.5% | 15.0% | 11.0% |
| Always drop values more than 3 scaled MADs (MAD × 1.4826) from the group median, decided in advance | 10.1% | 8.7% | 6.6% |
| Delete nothing, but if p > 0.05 switch to Mann–Whitney, then Yuen's test, and keep the first significant result | 7.3% | 7.2% | 6.9% |

*Normally distributed data; two-sided α = 0.05; stored code: [simulation.py](/guides/when-can-you-exclude-an-outlier/simulation.py) and [its output](/guides/when-can-you-exclude-an-outlier/simulation_output.json) (seed 20261110, fixed in advance; equal SDs in both groups). With right-skewed (lognormal) data at 6 or 10 per group, the planned analysis stayed at 2.6–3.2%, the deletion habits reached up to 14%, and switching between four tests (adding the log scale) gave about 7%.*

Three results surprise people:

- **A rule written in advance does not make deletion safe.** The MAD rule above was decided before any data were seen and applied to every dataset, yet it raised the false-positive rate at every n we simulated (10.1% at 6 per group, 6.6% at 30); at 6 per group it removed at least one value in 38% of null experiments. The cause: after values are removed, the t-test computes the standard error as if the remaining values were an ordinary random sample, which understates the uncertainty. Yuen's trimmed-mean test makes that correction, which is why trimming inside Yuen's test is valid and deleting before a t-test is not. Use a statistical screen to find values worth checking against your records, not to delete them.
- **Switching tests is the same problem in disguise.** Keeping every value but trying another test whenever p > 0.05 also inflates false positives (7% instead of 5% in the last row). A robust test helps only when it is the analysis you planned.
- **"|z| > 2" barely works at n = 6, for a structural reason.** In a sample of n values, no value can lie more than (n − 1)/√n SD from the mean ([Shiffler 1988](https://doi.org/10.1080/00031305.1988.10475530)). At n = 6 that ceiling is 2.04, so "|z| > 3" can never flag anything and "|z| > 2" flags only the most lopsided samples; look at a dot plot and your records instead.

| Values in the group | 3 | 4 | 5 | 6 | 8 | 10 | 12 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Largest possible \|z\| | 1.15 | 1.50 | 1.79 | 2.04 | 2.48 | 2.85 | 3.18 |

Published simulations point the same way. In a study of independent-samples t-tests on psychology test scores, removing values beyond |z| = 2 gave "Type I error rates of above 20%" (false-positive rates) in a short and difficult test, and choosing the threshold after seeing the results made it worse; the authors recommend Mann–Whitney or robust Yuen–Welch tests without removing outliers ([Bakker & Wicherts 2014](https://doi.org/10.1037/met0000014), abstract). Simmons, Nelson and Simonsohn (2011) showed with simulations that flexibility in data collection, analysis and reporting "dramatically increases actual false-positive rates" ([abstract](https://doi.org/10.1177/0956797611417632)).

**What to do:** decide the analysis before you look at the group comparison (before unblinding, if the study is blinded), and treat any exclusion you did not plan as a sensitivity analysis, never as the main result.

## A worked example: keep the value, use the analysis you planned

IL-6 in plasma (pg/mL), 6 control and 6 treated mice. The protocol, written before the experiment, specified Welch's t-test on log-transformed values, because concentrations such as IL-6 can be right-skewed, and named a Mann–Whitney test and a repeat of the main analysis with the most extreme animal removed as sensitivity analyses. Treated mouse T6 reads 118 pg/mL; the other treated mice read 49–61. The lab book has no note about T6. (Constructed data, chosen by hand to show the dilemma.)

- Control: 40, 44, 37, 46, 42, 39
- Treated: 52, 58, 49, 61, 55, 118

| Analysis | Result | p |
| --- | --- | --- |
| **Planned:** Welch's t-test on log values, all values | treated/control ratio 1.51, 95% CI 1.08 to 2.12; t(5.62) = 3.05 | 0.025 |
| Sensitivity (named in the protocol): Mann–Whitney, all values (exact) | U = 36: every treated value is above every control value | 0.0022 |
| Sensitivity (named in the protocol): planned analysis without T6 | ratio 1.33, 95% CI 1.18 to 1.49; t(8.36) = 5.62 | 0.00042 |
| For comparison only: Welch's t-test on raw values, all values | difference 24.2 pg/mL, 95% CI −3.2 to 51.5; t(5.16) = 2.25 | 0.072 |
| For comparison only: Yuen's test on 20% trimmed means, all values | trimmed difference 15.3 pg/mL; t(4.74) = 4.97 | 0.0049 |

The planned analysis keeps T6 and shows an increase, and both pre-named sensitivity analyses point the same way, so deleting T6 adds nothing. T6's z score is 2.01 — just under the n = 6 ceiling of 2.04 — and Grubbs' statistic (2.01) exceeds its two-sided 5% critical value (1.89); neither fact explains what happened to T6. A statistical flag is a prompt to check the records.

The order of decisions is what makes this honest. If the protocol had specified Welch's test on raw values, the main result would be p = 0.072, reported as such, with the Mann–Whitney, Yuen and log-scale results labelled as sensitivity analyses. Choosing whichever test crosses 0.05 after seeing 0.072 is the switching habit in the simulation table above.

```python
import numpy as np
from scipy import stats

control = np.array([40, 44, 37, 46, 42, 39.0])
treated = np.array([52, 58, 49, 61, 55, 118.0])   # T6 = 118

lt = stats.ttest_ind(np.log(treated), np.log(control), equal_var=False)         # planned: log scale, all values
ci = lt.confidence_interval()
print(lt, f"ratio {np.exp(np.log(treated).mean() - np.log(control).mean()):.2f} (95% CI {np.exp(ci.low):.2f} to {np.exp(ci.high):.2f})")
print(stats.mannwhitneyu(treated, control, method="exact"))                      # sensitivity: rank-based
print(stats.ttest_ind(np.log(treated[:-1]), np.log(control), equal_var=False))  # sensitivity: without T6
print(stats.ttest_ind(treated, control, equal_var=False))                        # comparison: raw scale
print(stats.ttest_ind(treated, control, equal_var=False, trim=0.2))              # comparison: Yuen
```

```r
control <- c(40, 44, 37, 46, 42, 39)
treated <- c(52, 58, 49, 61, 55, 118)   # T6 = 118

lt <- t.test(log(treated), log(control))   # planned: log scale, all values
lt; round(c(ratio = exp(unname(lt$estimate[1] - lt$estimate[2])), exp(lt$conf.int)), 2)
wilcox.test(treated, control)            # sensitivity: rank-based (exact for n < 50 without ties)
t.test(log(treated[-6]), log(control))   # sensitivity: without T6
t.test(treated, control)                 # comparison: raw scale
```

Base R has no built-in Yuen test; the full example code, including a short Yuen function that matches SciPy, is in [example.R](/guides/when-can-you-exclude-an-outlier/example.R) and [example.py](/guides/when-can-you-exclude-an-outlier/example.py) (outputs: [Python](/guides/when-can-you-exclude-an-outlier/example_output.json), [R](/guides/when-can-you-exclude-an-outlier/example_R_output.txt)). Each robust method answers a slightly different question — Mann–Whitney whether treated values tend to be higher, Yuen about trimmed means, the log scale about a ratio of typical values — so pick the one that fits your claim, and write it into the protocol, before you see the data ([Mann–Whitney vs t-test](/guides/mann-whitney-vs-t-test/)).

**What to do:** report the planned analysis with all values as the main result, whatever its p-value, and show the version without the value next to it.

## Detection tests: what they can and cannot tell you

| Method | What it assumes or measures | Use it for |
| --- | --- | --- |
| Grubbs' test | "A single outlier in a univariate data set that follows an approximately normal distribution" ([NIST 1.3.5.17.1](https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h1.htm)) | Flagging one suspect value in otherwise near-normal data; for two or more suspects NIST recommends other tests |
| \|z\| > 2 or 3 | Distance in SDs from the mean; capped at (n − 1)/√n | Little at n ≤ 10 (see the ceiling table above) |
| MAD screen | Distance in scaled MADs (MAD × 1.4826) from the median, which the extreme value barely moves ([Leys et al. 2013](https://doi.org/10.1016/j.jesp.2013.03.013), by their title) | Choosing which raw records to re-check; in the example T6 sits 9.2 scaled MADs from the treated median |
| Dot plot of every animal | Shape of the whole group | Seeing skew, two clusters or two extreme values at once |

Two extreme values can hide each other: a test for one outlier can miss both ("masking"), so NIST advises complementing formal tests with plots ([NIST 1.3.5.17](https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm)). If the outcome has a long right tail, high values are expected; analyse on the log scale rather than trimming the tail ([ANOVA assumptions not met](/guides/anova-assumptions-not-met/)).

**What to do:** use a detection method only to decide which records to re-check, then follow the three questions above.

## If you do exclude

Exclude whole animals for reasons recorded during the experiment (death, failed surgery, wrong dose); exclude single data points for reasons recorded during measurement (instrument fault, sample flagged by quality control). In both cases:

- report the number excluded per group and the reason (ARRIVE 2.0 item 3b);
- give the n actually analysed in every table and figure legend;
- keep the excluded values in the deposited data, marked, so others can repeat both analyses;
- report the result with the value included as well.

**What to do:** keep an exclusions table in your notebook from the first day of the experiment: animal, date, reason, who decided, and whether the group comparison had been seen.

## Check what an AI assistant ran

Code from an AI assistant can drop rows without saying so — through an outlier filter, a `dropna()`, or a z-score rule. Before you use its result:

1. Compare the number of rows read with the number analysed in each group.
2. Search the code for `zscore`, `quantile`, `IQR`, `std` thresholds, `dropna` or `drop`.
3. Ask it: *"List every row you removed, imputed or changed, with the rule you used. Then run the analysis on all rows and report both."*
4. If it proposes removing values "beyond 3 SD" or "outside 1.5 × IQR" without a recorded reason, decline and ask for a rank-based or log-scale analysis on all values instead.

**What to do:** accept an exclusion only when you can justify it from your lab records; otherwise ask for the analysis on all rows.

## Report it

**Methods (no exclusions):**

> [If you wrote criteria down beforehand:] Exclusion criteria were defined before the experiment: animals were to be excluded for [failed injection / weight loss > 20% / …]. [If not:] No exclusion criteria were set in advance. No animals or data points were excluded; all n = 6 mice per group were analysed.

**Methods (one documented exclusion):**

> One treated mouse's plasma sample was haemolysed (recorded on [date]) and gave no valid measurement, so the mouse was excluded before unblinding, as specified in the protocol. Results with this mouse included are given in Supplementary Table S1 and lead to the same conclusion.

**Methods (extreme value, no recorded cause — kept):**

> No data were excluded. One treated mouse had an IL-6 concentration of 118 pg/mL (others 49–61 pg/mL) with no recorded technical problem; it was retained. As specified in the protocol before the experiment, groups were compared with Welch's t-test on log-transformed values (ratio 1.51, 95% CI 1.08 to 2.12, p = 0.025). Pre-specified sensitivity analyses gave the same conclusion (Mann–Whitney p = 0.002; without this mouse, ratio 1.33, 95% CI 1.18 to 1.49).

Use "as specified in the protocol" only if a dated protocol named that analysis before anyone looked at the group comparison. Otherwise, report the analysis you used first as the main result and call every other analysis a sensitivity analysis; if you have already looked at the comparison, say so.

**Methods (extreme value, no recorded cause, no dated protocol):**

> No data were excluded. One treated mouse had an IL-6 concentration of [118] pg/mL (others [49–61] pg/mL) with no recorded technical problem; it was retained. No analysis plan was registered before the data were collected; the main analysis is the one first carried out, [Welch's t-test on raw values: result]. The following were run afterwards and are reported as sensitivity analyses: [log-scale Welch: result]; [Mann–Whitney: result]; [without this mouse: result]. The conclusion [is unchanged / differs between analyses: describe how].

**Analysing in GraphPad Prism or Excel:** transform the column to logarithms, run an unpaired t-test with Welch's correction on the transformed values, then back-transform the mean difference and the two ends of its CI to get the ratio and its interval. Keep the base consistent: natural log (Excel `LN()`) back-transforms with `exp`, log10 (Excel `LOG()`, or Prism's log10 option) with `10^x`; mixing them gives a wrong ratio. Excel's ToolPak "t-Test: Two-Sample Assuming Unequal Variances" prints no CI: compute SE = √(var₁/n₁ + var₂/n₂) of the log values and use difference ± `T.INV.2T(0.05, df)` × SE with the df the ToolPak reports.

**Figure legend:** "Each dot is one mouse; n = 6 per group; no values were excluded." Show every animal, including the extreme one.

**Response to a reviewer who asks why you did not remove the outlier:**

> We did not remove the value because there was no recorded technical or procedural reason, and removing values based on their magnitude increases the false-positive rate (Bakker & Wicherts 2014; our simulation above). A smaller p-value without the value is the expected effect of deleting it, and a deletion decided after seeing which way it moves the result is exactly what raises false positives. [If pre-specified: Our protocol specified a log-scale analysis for this right-skewed outcome, which gave [result].] [If not: Our main analysis, with all values, gave [result]; as sensitivity analyses, a rank-based test gave [result] and the analysis without this value gave [result].] [The conclusion is unchanged. / The conclusion depends on this value: we report the main analysis and state this.]

Replace every item in square brackets with your own details.

## References

- Bakker M, Wicherts JM (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: the power of alternatives and recommendations. *Psychological Methods* 19:409–427. [doi:10.1037/met0000014](https://doi.org/10.1037/met0000014) (abstract)
- Leys C, Ley C, Klein O, Bernard P, Licata L (2013). Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. *Journal of Experimental Social Psychology* 49:764–766. [doi:10.1016/j.jesp.2013.03.013](https://doi.org/10.1016/j.jesp.2013.03.013) (title)
- NIST/SEMATECH e-Handbook of Statistical Methods, sections 1.3.5.17 (Detection of outliers) and 1.3.5.17.1 (Grubbs' test). [itl.nist.gov](https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm), accessed 10 November 2026.
- Percie du Sert N, Hurst V, Ahluwalia A, et al. (2020). The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research. *PLOS Biology* 18:e3000410. [doi:10.1371/journal.pbio.3000410](https://doi.org/10.1371/journal.pbio.3000410); item 3 at [arriveguidelines.org](https://arriveguidelines.org/arrive-guidelines/inclusion-and-exclusion-criteria), accessed 10 November 2026.
- Shiffler RE (1988). Maximum Z scores and outliers. *The American Statistician* 42:79–80. [doi:10.1080/00031305.1988.10475530](https://doi.org/10.1080/00031305.1988.10475530) (abstract)
- Simmons JP, Nelson LD, Simonsohn U (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. *Psychological Science* 22:1359–1366. [doi:10.1177/0956797611417632](https://doi.org/10.1177/0956797611417632) (abstract)

## Related

- [What counts as n?](https://www.licklider.ai/guides/what-counts-as-n/)
- [Welch's t-test vs Student's t-test](https://www.licklider.ai/guides/welch-vs-student-t-test/)
- [Mann–Whitney vs t-test](https://www.licklider.ai/guides/mann-whitney-vs-t-test/)
- [Preference index vs chance](https://www.licklider.ai/guides/preference-index-vs-chance/)

## How this page was made

- Written and published by Licklider. The rules cite the ARRIVE 2.0 guidelines, the NIST/SEMATECH e-Handbook and peer-reviewed methods papers listed under References; cited passages were checked against the source text, and three papers are cited at the level of their abstract or title.
- The simulation and the worked example use constructed data and were produced by stored code (Python 3.13 with SciPy 1.18.1; R 4.3.3); the two languages agree on every number shown.
- Separate review passes checked the statistics, the citations and readability before publication.