Reviewer asks you to correct for multiple comparisons: how to respond

Quick answer. First decide which situation you are in. (1) Several confirmatory comparisons, any of which you would report as a finding: the reviewer is right — apply Holm’s correction (or Tukey or Dunnett for group comparisons after an ANOVA), report the adjusted p-values and say which conclusions change. (2) One comparison named as primary before the data were collected: keep it unadjusted, show where it was named, and label the other tests exploratory. (3) An exploratory screen of many outcomes: control the false discovery rate (Benjamini–Hochberg) and present the hits as leads to confirm. In every case, add a sentence to Methods stating whether and how you adjusted.

Terms used below. Confirmatory tests are the ones your conclusions rest on; exploratory tests generate ideas to check later. The familywise error rate is the chance of even one false “significant” result among all the tests you report — what Holm, Tukey and Dunnett control. The false discovery rate is the expected share of false results among those you call hits — what Benjamini–Hochberg controls. An adjusted p-value has already been corrected; compare it with 0.05 (your significance level, α) directly. Holm–Šidák is a close variant of Holm that is slightly less strict when the tests are independent. A screen is an experiment run to find candidates for later confirmation. Pre-specified means named in a dated document before the data were collected.

Which situation are you in?

Answer three questions about the analysis as it was planned, before the results came in.

Question	If yes	If no
Did you test more than one hypothesis and would you report any significant one as a result?	Go on to the next question	No correction needed (only one hypothesis was tested); say so in Methods
Was one comparison written down as primary before the data came in (protocol, registration, grant, dated notebook)?	Situation 2 for that comparison; the rest are exploratory	Go on to the next question
Are the tests a screen to pick candidates for follow-up experiments?	Situation 3	Situation 1

If you have no dated document naming the primary comparison, use Situation 1, or Situation 3 if the study was genuinely a screen. A primary comparison chosen after seeing the data is exactly what the reviewer is worried about.

What the reviewer is checking

Testing several hypotheses raises the chance that at least one comes out “significant” by luck. If all five of your outcomes had no true effect and the tests were independent, the chance of at least one p < 0.05 would be 1 − 0.95⁵ = 23%, not 5%.

Reporting guidelines ask you to say what you did about this, which is why the comment is so common:

• SAMPL asks authors to “indicate whether and how any allowance or adjustments were made for multiple comparisons (performing multiple hypothesis tests on the same data)” (Lang & Altman 2013).
• The ARRIVE 2.0 explanation for item 7 (statistical methods) lists “adjustment for multiple comparisons” among the details to report, and says the analysis methods “should be decided in advance before data are collected” (ARRIVE item 7; Percie du Sert et al. 2020).
• The Nature Portfolio reporting summary asks for “a description of any assumptions or corrections, such as tests of normality and adjustment for multiple comparisons” (Nature Portfolio).
• For confirmatory clinical trials, ICH E9 says “adjustment should always be considered and the details of any adjustment procedure or an explanation of why adjustment is not thought to be necessary should be set out in the analysis plan” (ICH E9, section 5.6).

Notice the word whether: every one of these accepts a reasoned “no adjustment, because …”. What none accepts is silence. So your reply always contains either the corrected analysis or the reason, plus a Methods sentence.

Situation 1: apply the correction

Worked example. Treated and vehicle mice (n = 8 each) were compared on five outcomes, each with a two-sided Welch’s t-test (the unequal-variance t-test). The authors reported four of the five as significant. Here is what Holm’s correction does to them:

Outcome	Difference (treated − vehicle), unadjusted 95% CI	p	Holm-adjusted p	Benjamini–Hochberg (FDR)
IL-6 (pg/mL)	−64.2 (−99.4 to −28.9)	0.0016	0.008	0.008
TNF-α (pg/mL)	−16.1 (−26.3 to −5.9)	0.0044	0.018	0.011
Paw thickness (mm)	−0.31 (−0.54 to −0.07)	0.017	0.051	0.028
CRP (µg/mL)	−2.31 (−4.31 to −0.32)	0.026	0.053	0.033
Body weight (g)	−0.46 (−2.16 to 1.24)	0.57	0.57	0.57

Two outcomes stay below 0.05 after Holm; paw thickness and CRP move just above it. Holm’s method works by ordering the p-values from smallest to largest and multiplying them by 5, 4, 3, 2 and 1 in turn (keeping the sequence non-decreasing). It is valid however the outcomes are correlated, which matters here because all five come from the same mice; R’s documentation notes that plain Bonferroni “is dominated by Holm’s method, which is also valid under arbitrary assumptions” (R p.adjust). If your comparisons are between groups after an ANOVA, use Tukey’s test for all pairs or Dunnett’s for each group against a control instead; see which correction to use.

Reply to the reviewer

We thank the reviewer for this point and agree. We tested five outcomes and would have reported any of them as an effect of treatment, so we now control the familywise error rate with Holm’s procedure across the five tests (Methods, page [x], lines [y–z]). After adjustment, IL-6 (adjusted p = 0.008) and TNF-α (adjusted p = 0.018) remain significant; paw thickness (adjusted p = 0.051) and CRP (adjusted p = 0.053) no longer reach the 0.05 threshold. We have revised the Results and Abstract accordingly and now describe the paw and CRP changes as consistent with the cytokine findings but not statistically established on their own; their estimates and confidence intervals are unchanged in Table [n].

Methods sentence

Each outcome was compared between groups with a two-sided Welch’s t-test. Because five outcomes were tested, p-values were adjusted with Holm’s step-down procedure to control the familywise error rate at 0.05; adjusted p-values are reported.

Code

# R: p.adjust uses Holm unless you choose another method
p <- c(il6 = 0.00162, tnf = 0.00443, paw = 0.01705, crp = 0.02631, weight = 0.56740)
round(p.adjust(p, method = "holm"), 4)
#    il6    tnf    paw    crp weight
# 0.0081 0.0177 0.0512 0.0526 0.5674

# Python: name the method; statsmodels' default is Holm–Šidák, not Holm
from statsmodels.stats.multitest import multipletests
p = [0.00162, 0.00443, 0.01705, 0.02631, 0.56740]
reject, p_holm, _, _ = multipletests(p, alpha=0.05, method="holm")
print(p_holm.round(4))   # [0.0081 0.0177 0.0512 0.0526 0.5674]

Run the correction on the full-precision p-values from your software output; rounded inputs can shift the adjusted values. The table shows p-values rounded for reading; the code uses 0.00162, 0.00443, 0.01705, 0.02631 and 0.56740.

Situation 2: one comparison was primary from the start

Only if a dated document — protocol, registration, grant or dated notebook — named that comparison as primary before the data were collected may it be tested at 0.05 without adjustment, with everything else labelled secondary or exploratory; with no such document, use Situation 1. This matches the guidance: Bender and Lange conclude that “adjustments for multiple testing are required in confirmatory studies whenever results from multiple tests have to be combined in one final conclusion and decision” (Bender & Lange 2001) — with a single primary test there is no such combination. Rothman argues further that routine adjustment trades fewer false positives for more missed real effects (Rothman 1990); cite it for that trade-off, not as a licence to skip adjustment for several confirmatory claims.

Reply to the reviewer

Use this reply only if you can cite the dated document; without it, use the Situation 1 reply. Replace every [bracketed] item with your own details and delete the options you did not use; never send a reply with brackets still in it.

We thank the reviewer. The primary outcome, IL-6 at day 7, was specified before the experiment in [the protocol registered at … / the grant proposal, page … / the dated lab notebook entry of …], and the study was powered for it. We therefore report it without adjustment, as planned. The remaining four outcomes were secondary; we now state this in Methods (page [x]) and label them as exploratory in the Results and Table [n], so that no conclusion rests on them alone. For transparency, Holm-adjusted p-values for the four secondary outcomes are given in Supplementary Table [m].

Methods sentence

Use this sentence only if the dated document exists; otherwise use the Situation 1 Methods sentence.

IL-6 at day 7 was the pre-specified primary outcome (named in [document, date]) and was tested at a two-sided α of 0.05 without adjustment. The four secondary outcomes are reported as exploratory, with Holm-adjusted p-values in Supplementary Table [m].

The Holm-adjusted p-values for the four secondary outcomes:

p_secondary <- c(tnf = 0.00443, paw = 0.01705, crp = 0.02631, weight = 0.56740)
round(p.adjust(p_secondary, method = "holm"), 4)
# 0.0177 0.0512 0.0526 0.5674

Offering the adjusted values for the secondary outcomes costs you nothing and answers the reviewer’s concern directly.

Situation 3: an exploratory screen

When you test many outcomes to decide what to follow up — twenty cytokines, a panel of genes — the goal is a short list worth confirming, not a set of proven effects. Only in that case — the output is a list of candidates for a separate confirmatory experiment — control the false discovery rate (the expected share of your “hits” that are false) with the Benjamini–Hochberg procedure instead of the familywise rate, and say the results need confirmation. If any outcome would be reported as a finding in its own right, use Situation 1; choosing FDR because it keeps more results is the after-the-fact switch the reviewer is asking about.

Reply to the reviewer

We agree that the [20] cytokines require adjustment. Because this panel was a screen to select candidates, we now control the false discovery rate at 5% with the Benjamini–Hochberg procedure (Methods, page [x]). [Number] cytokines meet this criterion (Table [n]); we describe them as candidates and have removed causal language from the Discussion. [If you have one: Two of them are tested in the confirmatory experiment in Figure [k].]

Methods sentence

The 20 cytokines were analysed as an exploratory screen; p-values were adjusted with the Benjamini–Hochberg procedure to control the false discovery rate at 0.05, and cytokines with adjusted p < 0.05 are reported as candidates for confirmation.

p <- c(0.00162, 0.00443, 0.01705, 0.02631, 0.56740)   # replace with your full-precision p-values
round(p.adjust(p, method = "BH"), 4)
# 0.0081 0.0111 0.0284 0.0329 0.5674

from scipy import stats
from statsmodels.stats.multitest import multipletests

p = [0.00162, 0.00443, 0.01705, 0.02631, 0.56740]   # replace with your full-precision p-values
print(stats.false_discovery_control(p).round(4))       # Benjamini–Hochberg
print(multipletests(p, method="fdr_bh")[1].round(4))  # same values from statsmodels

When a result loses significance

Report it. Adjusting changes the claim, while the estimate and its confidence interval stay the same, so keep both in the table. In the example, paw thickness still fell by 0.31 mm (95% CI 0.07 to 0.54 before adjustment); the honest sentence is that the change is consistent with the other findings but not established on its own.

Choose the method from the situation, never from the result. The last column of the table shows why: Benjamini–Hochberg would keep all four “significant”, but switching to it after Holm removed two findings is the selective analysis the reviewer is guarding against. If your study really was a screen, say so and justify FDR on those grounds; otherwise stay with the familywise method.

Check which correction your software ran

The same request — “correct these p-values” — gives different procedures depending on the tool:

Tool	Call without a method	What you get
R	p.adjust(p)	Holm (the first method listed) (R docs)
Python, statsmodels 0.15	multipletests(p)	Holm–Šidák (method='hs'): adjusted p = 0.0081, 0.0176, 0.0503, 0.0519, 0.5674 here (statsmodels docs)
Python, SciPy 1.18	false_discovery_control(p)	Benjamini–Hochberg: false discovery rate, not familywise error (SciPy docs)

Holm–Šidák adjusts each step with 1 − (1 − p)^k instead of k × p; it is valid and slightly less strict than Holm when the tests are independent, but it is not what your Methods sentence says if you wrote “Holm”. Always pass the method explicitly and name it in Methods exactly as run.

If an AI assistant wrote the reply or the code

Before you send anything, check three things against your own output:

1. The method in the code matches the method in the text. Look for method= in Python; a bare multipletests(p) is Holm–Šidák, and false_discovery_control is FDR.
2. The family is right. The correction should cover every test you would report as a finding — here all five outcomes, not only the four that were significant.
3. No invented history. A reply claiming a pre-specified primary outcome must point to a real dated document. If the assistant wrote “as pre-specified” and you have no such record, use the Situation 1 reply.

Then re-run the numbers yourself from your own software output.

Letter mechanics

Quote the reviewer’s comment, answer it directly in the first sentence with the decision you reached above — “We agree and have applied Holm’s correction” (Situation 1), “The primary outcome was pre-specified in [document]; we report it unadjusted and label the other outcomes exploratory” (Situation 2), or “The outcomes were an exploratory screen; we now control the false discovery rate and present the results as candidates” (Situation 3) — then give the detail, and quote the changed Methods and Results text with page and line numbers. Starting each response with a direct answer is one of the ten rules in Noble’s guide to response letters, alongside responding to every point and being clear about what changed (Noble 2017). Before sending, check that the adjusted p-values in the letter, the revised Results, the tables and the Abstract are the same numbers.

References

• Bender R, Lange S (2001). Adjusting for multiple testing—when and how? Journal of Clinical Epidemiology 54(4):343–349. doi:10.1016/S0895-4356(00)00314-0
• International Council for Harmonisation (1998). ICH Topic E 9: Statistical principles for clinical trials (CPMP/ICH/363/96), section 5.6. PDF
• Lang T, Altman D (2013). Statistical Analyses and Methods in the Published Literature: the SAMPL Guidelines. EQUATOR Network. equator-network.org; published as Lang TA, Altman DG (2015), International Journal of Nursing Studies 52(1):5–9. doi:10.1016/j.ijnurstu.2014.09.006
• Nature Portfolio. Reporting Summary (statistics section). PDF
• Noble WS (2017). Ten simple rules for writing a response to reviewers. PLOS Computational Biology 13(10):e1005730. doi:10.1371/journal.pcbi.1005730
• Percie du Sert N, Ahluwalia A, Alam S, et al. (2020). Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0. PLOS Biology 18(7):e3000411. doi:10.1371/journal.pbio.3000411
• R Core Team. p.adjust: Adjust P-values for Multiple Comparisons (R documentation). stat.ethz.ch
• Rothman KJ (1990). No adjustments are needed for multiple comparisons. Epidemiology 1(1):43–46. doi:10.1097/00001648-199001000-00010
• SciPy. scipy.stats.false_discovery_control (SciPy 1.18 documentation). docs.scipy.org
• statsmodels. statsmodels.stats.multitest.multipletests (statsmodels 0.15 documentation). statsmodels.org