Published
After the data choose the report: what multiplicity, selective, sequential and post hoc guarantees still support
This review states what a report may still claim from a multiple-testing, selective or sequential guarantee once the data choose what to report, with an exact counterexample for each loss.
Preprint v1 — not peer reviewed. A narrative review by Tasuku Kobayashi, published on Zenodo with a supplement and a separate archive of exact computations.
Analysts often decide after seeing the results what goes into the report: part of a list, the results that were selected, the moment to stop, or the level at which a result counts as significant. Multiple-testing, selective and sequential guarantees are stated for a procedure applied as planned. This review shows researchers, reviewers and readers what a report may still claim once such a choice has been made.
Its central point is that a guarantee does not carry over to a data-dependent choice by its name. A guarantee supports a specific sentence about the procedure in repeated use, and that sentence must be derived again after each choice. For example, control of the false discovery rate (FDR) at level q supports a statement about the average proportion of false discoveries over repeated analyses. It does not say that at most a fraction q of the reported discoveries are false.
What a guarantee still supports
For each guarantee, the review writes out four fields: the error event it controls, the times over which it is controlled, what it is conditioned on, and the sentence a report may make. Writing the fields out makes the change visible when an analyst reports only some results, stops at a data-chosen time or picks a level afterwards.
For each of the four choices below, the review then states the sentence each guarantee still supports, the assumptions that sentence now needs, and what is lost, with an exact small counterexample. It also describes what the alternatives, including e-value and post hoc guarantees, control instead. Each relation is credited to its source or marked as derived in the review.
Four choices made after the data
- Reporting part of a list — presenting some of the tested results rather than all of them.
- Reporting what was selected — making statements about the results that a selection rule picked out.
- Stopping or continuing — ending or extending data collection or a sequence of tests at a time that depends on the results so far.
- Choosing the level after the data — deciding at which level a result counts as significant once the result is known.
Three exact examples
The values below are exact fractions from the paper, all at level 1/10.
| Setting | What the guarantee supports | What happens after the choice |
|---|---|---|
| Holm procedure, three true null hypotheses | Familywise error rate of 2611/27000, about 0.097 | Given that anything is reported, every reported result is false: conditional probability 1 |
| Benjamini–Hochberg, one real effect and two true null hypotheses | FDR of 1/15: q times the two-thirds share of true null hypotheses | The list contains at least one false discovery with probability 13/100 |
| Online FDR control at every fixed time | FDR of 1/10 at each time fixed in advance | At a stopping time chosen from the data, every reported discovery is false with probability 19/100 |
The Holm guarantee is a statement about repeated analyses: in fewer than one in ten, the procedure rejects any true hypothesis. It does not describe the results that one report presents once something has been found. With Benjamini–Hochberg, the average proportion of false discoveries is 1/15, yet with probability 13/100 the reported list contains at least one false discovery. Online FDR control at fixed times does not carry over to a time the analyst chose by looking at the data.
Scope and status
This is a review of existing guarantees and their relations, with small exact witnesses for each loss. It does not report new empirical data or estimate how often these choices occur in published research.
The review belongs to Licklider’s research on the conditions under which statistical results support claims. It does not define SciGround’s checks or add supported analyses. The author is Licklider’s founder and CEO.
Read the paper and check the values
- Zenodo v1: article and supplement — DOI 10.5281/zenodo.24286123.
- Download the article PDF — 39 pages.
- Download the supplement PDF — 39 pages: fixed-wording statements, the derived relations with their credit and checks, documented misreadings, the literature search and the record of which source versions were read, and the construction of every exact witness.
- Exact computations, code v1.0.1 — DOI 10.5281/zenodo.24127934; one archive,
after-the-data-choose-the-report-code-v1.0.1.zip.
The code archive checks those exact values. It uses only the Python standard library and was tested with Python 3.13. Every script uses exact rational arithmetic, except for one value, 13/24, which is checked by numerical integration in floating point. python3 -I run_all.py runs all 61 scripts in about 13 minutes; each script exits with status 0 when its assertions hold.
The paper and supplement are licensed CC BY 4.0; the code is licensed MIT. The Zenodo record was created on November 16, 2026 UTC and lists its publication date as November 17, 2026.
Related research note: why a statistical method name is not enough · All research