In the ISIS-2 trial — one of the most important cardiovascular RCTs ever conducted — aspirin reduced 35-day vascular mortality by 23% (p<0.00001). A landmark result. But buried in the supplementary analyses, the investigators noted something peculiar: patients born under the astrological signs of Gemini or Libra appeared to derive no benefit from aspirin. The finding was, of course, nonsense. It was the product of running so many subgroup analyses that at least one spurious result was statistically inevitable. The investigators published it deliberately, as a warning: when you test enough hypotheses, any dataset will eventually confess to patterns that don’t exist. The multiple testing problem is not a theoretical concern — it is the mechanism behind a significant portion of the medical literature’s non-replicable results. Understanding it is not optional for any researcher who wants their findings to be believed.
What Is the Multiple Testing Problem?
Every statistical test is performed at a pre-set significance level — conventionally α = 0.05. This means that if the null hypothesis is true (there is no real effect), there is a 5% probability that the test will nevertheless produce p<0.05, generating a false positive. One test at α = 0.05 carries a 5% false-positive risk. This is the Type I error rate: acceptable for a single, pre-specified hypothesis.
The problem compounds immediately when multiple tests are run on the same dataset. If you run two independent tests at α = 0.05 and both null hypotheses are true, the probability that at least one returns p<0.05 by chance is not 5% — it is 1 — (1 − 0.05)² = 9.75%. Run 10 independent tests and the probability climbs to 40.1%. Run 20 tests and it reaches 64.2%.
This cumulative false-positive probability across a family of tests is called the family-wise error rate (FWER). The formula for k independent tests at per-comparison error rate α is:
Family-Wise Error Rate Formula
FWER = 1 − (1 − α)⁽ᴸ where k = number of tests, α = per-comparison error rate
FWER by Number of Tests
| Tests Run (k) | FWER at α=0.05 | Visualisation | Interpretation |
|---|
| 1 | 5.0% | | Acceptable — one pre-specified test |
| 2 | 9.8% | | Nearly double — already requires consideration |
| 5 | 22.6% | | 1-in-4 chance of at least one false positive |
| 10 | 40.1% | | Nearly coin-flip odds of a spurious result |
| 20 | 64.2% | | More likely than not to have ≥1 false positive |
| 50 | 92.3% | | Near-certain false positive somewhere |
| 100 | 99.4% | | Virtually guaranteed spurious significant result |
When Does the Multiple Testing Problem Arise?
The multiple testing problem is not limited to the scenario of running many tests simultaneously and choosing the one that is significant. It arises in many common research designs — some of which are not intuitively obvious:
- Multiple outcome variables: Testing 12 clinical outcomes (mortality, LOS, readmission, wound infection, pain, function…) and reporting only those that achieve p<0.05
- Multiple biomarkers or laboratory values: Correlating a panel of 22 blood tests with an outcome, each at α=0.05
- Multiple time points: Comparing groups at 1 week, 4 weeks, 8 weeks, 12 weeks, and 24 weeks independently, each as a separate test
- Multiple subgroup analyses: Testing whether the treatment effect differs by age, sex, smoking status, diabetes, baseline severity, prior treatment, BMI, and region — eight subgroup comparisons
- Post-hoc pairwise comparisons: After ANOVA or Kruskal-Wallis for k groups, there are k(k−1)/2 possible pairwise tests (6 for k=4, 10 for k=5)
- Exploratory regression: Stepwise or forward selection testing 20 candidate predictors in a logistic regression model
- Interim analyses: Checking for significance repeatedly during a clinical trial as data accumulate
- Genome-wide association studies (GWAS): Testing up to 10 million SNP associations, where the required Bonferroni threshold is p < 5 × 10⁻⁸
The hidden multiple testing problem: Multiple testing violations are not always visible in a published paper. A researcher who tests 30 outcomes and reports only the 2 that are significant has committed a multiple testing violation — but readers cannot detect this from the paper. Pre-registration (ClinicalTrials.gov, ISRCTN, OSF) creates an immutable public record of which outcomes were planned before data collection, making selective reporting detectable. Reviewers at top journals now routinely check whether reported outcomes match the registered protocol.
Correction Methods for Multiple Comparisons
Bonferroni Correction
The Bonferroni correction is the simplest and most widely understood approach. It adjusts the significance threshold by dividing α by the number of tests: α_adjusted = α / k. For 10 tests at α=0.05, each individual test must achieve p<0.005 to be declared significant. Equivalently, each observed p-value can be multiplied by k (adjusted p = p × k, capped at 1.0) and compared to the original α=0.05.
The Bonferroni correction strictly controls the FWER — the probability of any false positive across all k tests is guaranteed to be ≤ α. It is the correct choice for small numbers of pre-specified, independent comparisons. Its main limitation is conservatism when tests are positively correlated: if 10 inflammatory biomarkers all tend to rise together, testing all 10 is not equivalent to testing 10 truly independent hypotheses, and Bonferroni over-corrects.
Holm-Bonferroni Sequential Procedure
The Holm-Bonferroni procedure (Holm, 1979) achieves the same strict FWER control as Bonferroni but is uniformly more powerful — it detects at least as many true effects as Bonferroni, and often more. The procedure works sequentially:
- Order the k p-values from smallest to largest: p(1) ≤ p(2) ≤ … ≤ p(k)
- Compare p(1) to α/k. If p(1) > α/k, stop — no tests are significant.
- If p(1) ≤ α/k, declare test 1 significant and compare p(2) to α/(k−1).
- Continue: at step i, compare p(i) to α/(k−i+1). Stop when p(i) exceeds its threshold; all remaining tests are non-significant.
Because the threshold relaxes with each successive step (denominator decreases), Holm-Bonferroni is always at least as liberal as Bonferroni — and should be preferred over standard Bonferroni whenever k > 3.
Šidák Correction
The Šidák correction computes the adjusted threshold as: α_Šidák = 1 − (1−α)^(1/k). For k=10 and α=0.05, α_Šidák = 0.00512 vs Bonferroni’s 0.00500 — slightly more liberal. The Šidák correction is exact when tests are independent; Bonferroni is a conservative approximation. In practice, the difference is negligible for k ≤ 20.
Bonferroni
Strict FWER
α_adj = α / k
Simplest correction. Controls FWER strictly. Conservative when tests are correlated. Best for small k with independent comparisons. Universally understood by reviewers.
Holm-Bonferroni
Strict FWER
p(i) vs α/(k−i+1) sequentially
Same FWER control as Bonferroni but uniformly more powerful. Should be preferred over standard Bonferroni when k > 3. Step-down sequential procedure.
Šidák
Strict FWER (exact)
α_adj = 1−(1−α)^(1/k)
Exact correction for independent tests. Slightly less conservative than Bonferroni. Negligible practical difference for k ≤ 20; choose Holm for preference.
Benjamini-Hochberg (BH)
FDR Control
p(i) ≤ (i/k) × Q
Controls the expected proportion of false positives among all significant findings (FDR). More powerful than FWER methods. Preferred for exploratory analyses, omics studies, large biomarker panels.
False Discovery Rate: The Benjamini-Hochberg Procedure
FWER methods ask: “What is the probability of making even one false positive?” For confirmatory research, this strict standard is appropriate. But for exploratory research — biomarker panels, proteomics, genomics, hypothesis-generating studies — FWER control is often too conservative. It misses too many true effects in pursuit of zero false positives.
The false discovery rate (FDR) controls a different quantity: the expected proportion of false positives among all rejected hypotheses. If you declare 20 biomarkers significant at FDR = 0.05, you expect about 1 (5%) to be a false positive — acceptable in a discovery context, where the false positives will be filtered out in subsequent replication studies.
The Benjamini-Hochberg (BH) procedure controls FDR at a pre-specified level Q (typically 0.05 or 0.20 for exploratory work):
BH Step-by-Step: Worked Example
A metabolomics study tests k=8 metabolites for association with disease status. Results, ranked by p-value, with Q=0.05:
| Metabolite | Rank (i) | p-value p(i) | BH Threshold (i/k)×Q | p(i) ≤ Threshold? | Decision |
|---|
| Lactate | 1 | 0.003 | (1/8)×0.05 = 0.00625 | YES | Significant |
| Pyruvate | 2 | 0.009 | (2/8)×0.05 = 0.01250 | YES | Significant |
| Glutamine | 3 | 0.021 | (3/8)×0.05 = 0.01875 | NO | Stop here — not significant |
| Alanine | 4 | 0.034 | (4/8)×0.05 = 0.02500 | NO | Not significant |
| Citrate | 5 | 0.041 | (5/8)×0.05 = 0.03125 | NO | Not significant |
| Succinate | 6 | 0.058 | (6/8)×0.05 = 0.03750 | NO | Not significant |
| Fumarate | 7 | 0.12 | (7/8)×0.05 = 0.04375 | NO | Not significant |
| Malate | 8 | 0.34 | (8/8)×0.05 = 0.05000 | NO | Not significant |
Result: Lactate and Pyruvate survive BH-FDR correction at Q=0.05. Glutamine (p=0.021) does not — its p-value exceeds its rank-adjusted BH threshold. Note the key rule: once the p-value first exceeds the BH threshold working down from rank 1, all subsequent tests are non-significant, even if a later p-value happens to fall below its own threshold.
Interpreting q-values: When BH-FDR is applied, the result is often expressed as a q-value — the minimum FDR at which a given test would be declared significant. A q-value of 0.08 for Glutamine means: “Among all findings that would be called significant at FDR=0.08, we expect 8% to be false positives.” The q-value is analogous to the adjusted p-value but refers to a proportion of false discoveries rather than a probability of any false discovery.
Clinical Examples
1
22-Biomarker Sepsis Study: What Survives Correction?
A single-centre observational study of 120 patients admitted with sepsis measures 22 inflammatory and metabolic biomarkers at admission and correlates each with 30-day mortality using logistic regression. Four biomarkers reach p<0.05 without correction. The investigators conclude “four biomarkers were independently associated with mortality.”
67.7%
FWER (at least 1 false positive)
Uncorrected Results (p < 0.05)
CRP: OR=1.08 (95%CI: 1.01–1.16), p = 0.031
IL-6: OR=1.12 (95%CI: 1.02–1.23), p = 0.022
PCT: OR=1.09 (95%CI: 1.00–1.19), p = 0.047
Ferritin: OR=1.06 (95%CI: 1.00–1.12), p = 0.038
Expected False Positives (Without Correction)
Expected false positives = 22 × 0.05 = 1.1
FWER = 1 − (1 − 0.05)²² = 67.7%
Interpretation: >2 in 3 chance that at least one of the 22 tests is a false positive.
With 4 significant findings and ~1.1 expected by chance, 1–2 of the 4 are plausibly false.
Bonferroni Correction (α/22 = 0.00227)
IL-6 p=0.022 > 0.00227 → NOT significant after correction
CRP p=0.031 > 0.00227 → NOT significant after correction
Ferritin p=0.038 > 0.00227 → NOT significant after correction
PCT p=0.047 > 0.00227 → NOT significant after correction
Result: 0 of 4 survive Bonferroni correction.
BH-FDR Correction (Q = 0.20 — discovery threshold)
Rank 1: IL-6 p=0.022; BH threshold (1/22)×0.20 = 0.0091 → 0.022 > 0.0091 FAIL
Rank 2: CRP p=0.031; BH threshold (2/22)×0.20 = 0.0182 → 0.031 > 0.0182 FAIL
Rank 3: Ferritin p=0.038; BH threshold (3/22)×0.20 = 0.0273 → 0.038 > 0.0273 FAIL
Rank 4: PCT p=0.047; BH threshold (4/22)×0.20 = 0.0364 → 0.047 > 0.0364 FAIL
Result: 0 of 4 survive even the liberal FDR correction at Q=0.20.
Correct Interpretation
The four uncorrected p-values are consistent with the number expected by chance
when running 22 tests. Neither Bonferroni nor BH-FDR supports claiming these
as validated biomarkers. These results are hypothesis-generating only —
they require independent replication in a new cohort before clinical conclusions
can be drawn.
Conclusion after correction: zero biomarkers survive either Bonferroni (α/22 = 0.00227) or BH-FDR (Q=0.20) correction. The four “significant” uncorrected findings are consistent with chance inflation. Correct framing: “Four biomarkers (CRP, IL-6, PCT, ferritin) showed nominally significant associations with 30-day mortality (all p<0.05 uncorrected). After correction for 22 simultaneous comparisons (Bonferroni α=0.00227), no association remained statistically significant, and these findings should be regarded as hypothesis-generating pending independent replication.”
The key methodological lesson: Reporting four nominally significant associations from a 22-variable screen without correction — and labelling them “independent predictors” — is one of the most common errors in observational clinical research. The FWER of 67.7% means the investigators were more likely than not to generate at least one false-positive before even starting. No exploratory biomarker study should claim associations without correction, and no correction should be omitted without explicit acknowledgement of the number of tests performed.
2
Post-Hoc Comparisons in a 4-Group Antihypertensive Trial
An RCT compares four antihypertensive drug combinations (A = standard, B = ARB+diuretic, C = CCB+ARB, D = triple therapy) on systolic blood pressure (SBP) reduction at 12 weeks. One-way ANOVA: F(3,76)=4.82, p=0.004. The researcher now wants to know which specific groups differ, and runs all pairwise comparisons.
26.5%
FWER (6 uncorrected tests)
p=0.004
Omnibus ANOVA F-test
All Pairwise p-Values (6 comparisons)
D vs A: p = 0.003 ← clearly significant by any method
D vs B: p = 0.018
C vs A: p = 0.041
C vs B: p = 0.087
D vs C: p = 0.14
B vs A: p = 0.38
FWER for 6 independent comparisons = 1 − (0.95)^6 = 26.5%
Expected false positives at α=0.05: 6 × 0.05 = 0.30 (low, but possible)
LSD (Least Significant Difference — NO correction, inappropriate for k=4)
All 3 comparisons with p < 0.05 declared significant: D vs A ✓, D vs B ✓, C vs A ✓
BUT: FWER is uncontrolled at 26.5% — "C vs A: p=0.041" could be a false positive
Bonferroni Post-Hoc (α/6 = 0.00833)
D vs A: p=0.003 ≤ 0.00833 → Significant ✓
D vs B: p=0.018 > 0.00833 → NOT significant ✗
C vs A: p=0.041 > 0.00833 → NOT significant ✗
→ Only 1 of 3 uncorrected findings survives. Bonferroni may over-correct here.
Tukey HSD (designed for balanced pairwise comparisons after ANOVA)
D vs A: q = 4.81, p(Tukey) = 0.002 → Significant ✓
D vs B: q = 3.51, p(Tukey) = 0.041 → Significant ✓
C vs A: q = 3.01, p(Tukey) = 0.084 → NOT significant ✗
→ Tukey detects 2 differences vs Bonferroni's 1 — less conservative, same FWER
Holm-Bonferroni (Sequential)
Step 1: p(1)=0.003 vs α/6=0.00833 → 0.003 ≤ 0.00833 → Significant ✓ (D vs A)
Step 2: p(2)=0.018 vs α/5=0.0100 → 0.018 > 0.0100 → STOP
→ Same as Bonferroni here: 1 significant. But for a 5th p-value of 0.012,
Holm would find: 0.012 ≤ 0.0100 → no... still stops. Holm advantage is clearer
when middle-ranked p-values cluster near the threshold.
The correct analysis uses Tukey HSD (designed for balanced post-hoc after ANOVA): D vs A (p=0.002) and D vs B (p=0.041) are significant; C vs A (p=0.084) is not. Using uncorrected LSD would falsely claim three significant differences; Bonferroni over-corrects to one. The choice of post-hoc method directly changes the clinical conclusion about which treatment combinations differ significantly.
The post-hoc selection issue: With four groups, researchers face a genuine choice among Tukey HSD, Bonferroni, Holm-Bonferroni, and Scheffé’s test — each giving different results. The decision must be made in advance and stated in the Methods, not selected after seeing the data. Reporting LSD p-values when a corrected method gives non-significant results, or switching from Tukey to Bonferroni when Tukey misses a comparison you wanted to find, are forms of selective reporting. Tukey HSD is the standard recommendation for balanced designs; Holm-Bonferroni is preferred for pre-specified, non-exhaustive subsets of comparisons.
3
Subgroup Analysis Over-Claim in an RCT: The Elderly Patient Trap
A phase III RCT of n=800 patients tests a novel anti-inflammatory agent for reducing 30-day mortality in pneumonia. The primary ITT analysis shows a non-significant result. The investigators pre-specified 8 subgroup analyses. One subgroup — patients aged ≥65 — appears strikingly different from the overall null result.
HR=0.88
Primary ITT result (p=0.14)
33.7%
FWER for 8 subgroup tests
Primary Endpoint (ITT, all 800 patients)
HR = 0.88 (95%CI: 0.74–1.04), p = 0.14
Conclusion: non-significant — trial fails to meet its primary endpoint
Subgroup Analysis: Age ≥ 65 vs Age < 65
Age ≥65 (n=380): HR = 0.71 (95%CI: 0.54–0.93), p = 0.013 ← "significant"
Age <65 (n=420): HR = 1.08 (95%CI: 0.84–1.38), p = 0.56 ← not significant
Interaction test (age × treatment): p(interaction) = 0.048
Why This Is Probably a False Positive
8 subgroup tests → FWER = 1 − (0.95)^8 = 33.7%
8 interaction tests → need 8 interaction p-values; at α=0.05 each, >1 expected false
Bonferroni-corrected interaction threshold: α/8 = 0.00625
p(interaction) = 0.048 > 0.00625 → NOT significant after correction
The observed interaction (elderly benefit, young harm) is biologically implausible
as a mechanism and is not supported by prior pharmacological evidence.
The Astrological Parallel (ISIS-2 Warning)
ISIS-2 investigators tested aspirin subgroups by star sign.
Gemini/Libra patients: no benefit (OR ≈ 1.00, CI crossed null)
The finding was published to illustrate: with enough subgroups,
any pattern is achievable by chance.
Correct Interpretation
Primary analysis: negative trial (HR=0.88, p=0.14).
The age ≥65 subgroup finding (p=0.013) does NOT rescue the trial.
The interaction test does not survive Bonferroni correction for 8 subgroup tests.
Correct framing: "Pre-specified subgroup analysis suggested a possible treatment
benefit in patients aged ≥65 (HR=0.71, p=0.013); however, this finding must be
interpreted with caution given 8 subgroup comparisons were conducted
(Bonferroni-corrected threshold: p<0.00625) and the interaction test
was borderline (p=0.048). This subgroup finding is hypothesis-generating
and requires confirmation in a dedicated elderly-population trial."
The primary trial is negative. The apparently exciting elderly subgroup result does not survive correction for 8 subgroup comparisons (Bonferroni threshold p<0.00625; observed interaction p=0.048). Treating this as a positive finding for elderly patients — and changing clinical practice accordingly — would be methodologically indefensible. A dedicated confirmatory trial in patients aged ≥65 is needed.
The clinical stakes of subgroup over-claiming: Subgroup analyses from negative trials that “rescue” a drug for a particular demographic have a poor record of replication. The ILLUMINATE trial (torcetrapib) showed subgroup benefits that evaporated in confirmatory studies. CAST (encainide/flecainide) showed subgroup antiarrhythmic success that preceded a definitive trial demonstrating increased mortality. A subgroup finding from a negative trial that is not pre-specified and not corrected for multiplicity has the same evidentiary value as a fishing expedition — it is a hypothesis to be tested, not a conclusion to be acted upon.
Pre-Specification: The Primary Defence
The most powerful protection against multiple testing inflation is not a statistical correction — it is pre-specification of the analysis plan before any data are examined. When the primary outcome, the number of secondary outcomes, and the analysis strategy are registered publicly before data collection begins, selective reporting becomes detectable and the multiple testing context is known to readers.
Pre-specification works because it shifts the question from “among all the things I tested, which were significant?” to “did this pre-specified hypothesis test achieve significance?” The latter is a single, pre-planned test; no correction is needed. The former is always a multiple testing problem.
- Primary outcome: One pre-specified primary outcome requires no correction — it is a single planned test at α=0.05
- Secondary outcomes: Pre-specify the number and hierarchy; apply Bonferroni or Holm-Bonferroni if testing all simultaneously; or use a hierarchical gate where secondaries are tested only if the primary is significant
- Exploratory analyses: Any analysis not pre-specified must be labelled as exploratory/hypothesis-generating; correction does not fully repair the credibility of post-hoc exploration, but transparent disclosure is mandatory
- Interim analyses: Each look at accumulating data is an additional test; O’Brien-Fleming or Lan-DeMets spending functions allocate alpha across planned interim looks while maintaining overall α = 0.05
P-hacking and its consequences: P-hacking — running multiple analyses and reporting only significant ones — is the primary mechanism behind the replication crisis in medical and biomedical research. A 2015 analysis of psychology studies found that only 36% of results replicated when independently tested. A 2013 survey of pre-clinical cancer research found that only 11% of reported “landmark” findings were replicable. Multiple testing without correction or disclosure is the statistical engine of non-replicable findings. Journals, funders, and institutional review boards are increasingly requiring pre-registration as a condition of publication and funding.
Thesis Writing Recommendations
Model Methods Paragraph — Multiple Comparisons
“The primary outcome (30-day mortality) was tested using a single pre-specified logistic regression model at α = 0.05; no correction for multiple comparisons was applied to the primary analysis. Five secondary outcomes were pre-specified (LOS, readmission, wound infection, functional score, quality-of-life VAS); to control the family-wise error rate across these five comparisons, the Holm-Bonferroni sequential procedure was applied. Post-hoc pairwise comparisons following one-way ANOVA were conducted using Tukey’s HSD, which controls the FWER for all k(k−1)/2 pairwise comparisons. Exploratory biomarker analyses (k=14 biomarkers) were corrected using the Benjamini-Hochberg false discovery rate procedure at Q=0.20; findings from these analyses are presented as hypothesis-generating and require independent replication.”
Model Results Disclosure — Corrected Findings
“Of five pre-specified secondary outcomes, two survived Holm-Bonferroni correction: 30-day readmission rate (OR=0.51, 95%CI=0.33–0.78, p=0.002, Holm-adjusted p=0.010) and wound infection (OR=0.44, 95%CI=0.26–0.75, p=0.002, Holm-adjusted p=0.010). The remaining three secondary outcomes (LOS, functional score, quality-of-life VAS) did not achieve significance after correction (all Holm-adjusted p>0.05). In the exploratory biomarker analysis, IL-6 and CRP survived BH-FDR correction at Q=0.20 (unadjusted p=0.003 and p=0.009 respectively); these findings are hypothesis-generating and not considered confirmatory.”
Common Mistakes Researchers Make
Mistake 1: Running Multiple Tests Without Any Correction and Reporting All Significant Results as Findings
A researcher tests 15 candidate biomarkers for association with outcome, finds 3 at p<0.05, and reports them as confirmed predictors. With FWER = 1−(0.95)¹⁵ = 53.7%, up to 2 of the 3 “significant” results are plausibly false. The paper cites no correction method, reports no number of tests performed, and the conclusion section presents all three as clinical recommendations.
Fix: Always report the total number of tests performed (not just the significant ones), state explicitly whether any correction was applied, and label uncorrected exploratory findings as hypothesis-generating. If p-values are reported without correction, state the FWER in the Limitations section: “As 15 biomarkers were tested without correction, the family-wise error rate was 53.7%; findings should be considered exploratory.”
Mistake 2: Using Least Significant Difference (LSD) Post-Hoc for 4+ Groups
Fisher’s LSD does not control the FWER when four or more groups are compared — it was designed for the specific case of k=3 groups where only 3 pairwise comparisons are made (and the omnibus F-test provides partial protection). For k=4 groups and 6 pairwise comparisons, LSD produces a FWER of 26.5% — yet it is commonly used in clinical research because it is the default in some software outputs and produces the smallest p-values.
Fix: Use Tukey HSD for balanced groups (equal or near-equal group sizes) after ANOVA; use Games-Howell for unequal variances; use Holm-Bonferroni for pre-specified subsets of comparisons. Never use LSD as the only post-hoc method for 4+ groups. If LSD is reported, explicitly note that it does not control FWER and repeat the key conclusions with a corrected method.
Mistake 3: Treating a Significant Subgroup Finding from a Negative Trial as Confirmation
A trial with a non-significant primary endpoint reports a subgroup analysis (elderly patients, women, patients with baseline CRP>5 mg/L) that achieves p<0.05 and presents this as evidence that the treatment works in that subgroup. This is not confirmatory evidence. Subgroup analyses from negative trials are exploratory by definition; without correction for the number of subgroups tested and without a significant a priori interaction hypothesis, a significant subgroup p-value is likely to reflect chance.
Fix: Subgroup analyses should be pre-specified, numbered, and corrected for multiplicity. Report the interaction test (treatment × subgroup variable) rather than within-subgroup p-values alone. Interpret significant interaction tests cautiously when multiple interactions were tested. If the primary was negative, state clearly: “The trial failed to demonstrate efficacy in the overall population; the [subgroup] analysis is exploratory and does not constitute evidence of efficacy in that population.”
Mistake 4: Applying Bonferroni When Tests Are Highly Correlated
Standard Bonferroni divides α by k assuming the k tests are independent. When tests are highly correlated — for example, testing 10 measures of the same physiological system that all move together — Bonferroni is overly conservative: it assumes 10 independent pieces of evidence when the 10 tests carry much less than 10 independent units of information. A researcher correcting 10 highly correlated inflammation markers with Bonferroni (α=0.005) may reject associations that are genuinely meaningful.
Fix: When tests are correlated, consider Holm-Bonferroni (slightly more powerful) or permutation-based methods that account for the actual correlation structure of the tests. For large biomarker panels, BH-FDR is more appropriate than FWER methods because it accommodates the typical correlation structure of omics data. The correlation structure should be described and its impact on correction acknowledged in the Methods.
Mistake 5: Checking Significance at Multiple Interim Time Points Without Alpha Spending
In clinical trials and longitudinal observational studies, researchers sometimes check whether a result is significant at 6 months, 12 months, and 18 months, and report significance at any point where p<0.05 is achieved. Each look is an additional test of the same hypothesis; three interim looks at α=0.05 carry FWER ≈ 14% — nearly triple the nominal rate. This is especially problematic because stopping early when a result first crosses p=0.05 tends to overestimate the true effect size.
Fix: For longitudinal data, use a repeated-measures or mixed-model approach that tests all time points simultaneously within a single model (the multiple time points are then part of the model structure, not separate tests). For clinical trials with planned interim analyses, pre-specify the number of interim looks and apply O’Brien-Fleming or Lan-DeMets alpha-spending functions, which allocate the Type I error budget across interim and final analyses while maintaining overall α=0.05.
Mistake 6: Defining the “Family” of Tests Too Narrowly
Researchers sometimes apply correction within a single analysis (e.g., correcting for 5 post-hoc pairwise comparisons) but fail to recognise that the same dataset is used for many other tests reported throughout the results chapter — different outcomes tested on the same cohort, different subgroup analyses, different time points. The true “family” of tests is the full set of tests performed on the same dataset, not just the set within one table.
Fix: Define the family of tests in the Methods section at the outset: “The family of tests consists of the primary outcome, k pre-specified secondary outcomes, and m pre-specified subgroup analyses.” Apply the correction to the entire family. If analyses span multiple sections or tables, the cumulative number of tests must be acknowledged. A conservative but transparent approach: report FWER for the total number of tests performed in the Results chapter, even when individual families are corrected separately.
Scientific Reporting Standards
- CONSORT 2010 (Item 6a): Requires that all outcomes (primary and secondary) be pre-specified, including the primary outcome clearly identified as such. Multiple secondary outcomes tested simultaneously without correction must be acknowledged. Item 17b requires that all pre-specified outcomes be reported, including non-significant ones — selective reporting of only significant secondary outcomes is a CONSORT violation.
- CONSORT Subgroup Analysis Guidance: Subgroup analyses should be labelled pre-specified or post-hoc; the number of subgroup analyses performed should be stated; interaction tests (not within-subgroup tests alone) should be the basis of subgroup inference; subgroup findings should be interpreted with explicit reference to the multiple testing context.
- STROBE (observational studies): Item 16 requires that the number of participants at each stage of analysis be reported, and that the analyses conducted be fully disclosed. A biomarker study that tested 40 variables but reports 5 significant ones without disclosing the total is in violation of STROBE’s full disclosure requirement.
- ICH E9 (clinical trials): Requires pre-specification of the primary endpoint and the analysis strategy for secondary endpoints, including whether multiplicity adjustment is planned. Exploratory analyses must be distinguished from confirmatory analyses in the Statistical Analysis Plan.
- Reporting uncorrected results: When correction is not applied (e.g., for exploratory analyses transparently labelled), the total number of tests performed must be stated alongside the results, and the implied FWER should be noted in the Limitations: “As 14 biomarkers were tested simultaneously without correction, the family-wise error rate was 51.2%; findings are hypothesis-generating.”
Practical Guidance for Researchers
Identify Your Primary Outcome Before Data Collection
The single most protective decision is identifying one primary outcome — and registering it publicly — before any data are examined. A single pre-specified primary test at α=0.05 has exactly a 5% Type I error rate. No correction is needed, no FWER inflation exists. Everything tested after this becomes secondary or exploratory.
Count Your Tests and Compute FWER Before Writing Up
Before writing your Results, count the total number of hypothesis tests you ran on the primary dataset. Compute FWER = 1 − (0.95)^k. If FWER > 20%, you must either correct, label findings as exploratory, or both. This calculation takes 30 seconds and prevents a thesis examiner from asking: "Have you accounted for the multiple testing problem?"
Use Holm-Bonferroni Instead of Standard Bonferroni
Holm-Bonferroni achieves the same strict FWER control as Bonferroni but is uniformly more powerful — it detects at least as many true effects and sometimes more. Implementing it requires only sorting your p-values and comparing each to a decreasing threshold. Most statistical software (R: p.adjust(method="holm"), SPSS: syntax adjustment) computes it automatically.
Choose FDR for Discovery, FWER for Confirmation
The choice between FWER and FDR methods should match the research context. Confirmatory research (regulatory submission, primary RCT endpoint, definitive clinical question): use FWER control — even one false positive is costly. Discovery research (biomarker screening, omics, hypothesis generation, exploratory secondary analyses): use BH-FDR — maximising sensitivity to true effects is more important than eliminating every false positive.
Report Both Corrected and Uncorrected P-Values
For transparency, report both the uncorrected and corrected p-values (or adjusted significance thresholds) for every test subject to multiple comparison correction. This allows readers to see the original evidence alongside the corrected interpretation. Format: "IL-6: OR=1.12, p=0.022 uncorrected; Bonferroni-adjusted threshold p<0.00227, not significant."
Treat Subgroup Analyses as Exploratory Unless Pre-Specified
Any subgroup analysis not pre-specified and registered before data collection is exploratory by default, regardless of how biologically plausible it appears post-hoc. Pre-specified subgroup analyses require an interaction test (not within-subgroup tests) and correction for the number of planned subgroup comparisons. Post-hoc subgroup findings require independent replication before influencing clinical practice.
Frequently Asked Questions
What is the multiple testing problem? +
The multiple testing problem is the inflation of the Type I error rate (probability of a false positive) that occurs when many statistical tests are performed on the same dataset. Each test at α=0.05 has a 5% chance of a false positive. When k tests are performed, the family-wise error rate — the probability of at least one false positive — is FWER = 1 − (1 − 0.05)ᴸ. For k=10 tests, FWER = 40.1%; for k=20, FWER = 64.2%; for k=50, FWER = 92.3%. This means exploratory studies testing many outcomes, biomarkers, or subgroups without correction are near-certain to produce at least one spurious significant finding. The problem underlies a major portion of non-replicable medical research.
What is the Bonferroni correction? +
The Bonferroni correction is the simplest method for controlling the FWER. The adjusted significance threshold is α_adjusted = α / k, where k is the number of tests. For 10 tests at α=0.05, each must achieve p<0.005 to be declared significant. Equivalently, multiply each p-value by k and compare the result to 0.05. Bonferroni strictly controls FWER but is conservative — especially when tests are correlated. It is appropriate for small numbers of pre-specified, independent comparisons. For k>3, the Holm-Bonferroni sequential procedure should be preferred: it achieves the same FWER control but is uniformly more powerful.
What is the difference between FWER and FDR? +
Family-wise error rate (FWER) is the probability of making even one false positive across all tests. FWER methods (Bonferroni, Holm) guarantee that the chance of any single false positive among all declared results is ≤5%. This is the right criterion for confirmatory research where any false discovery could cause harm — regulatory drug approval, primary RCT endpoints. False discovery rate (FDR) controls the expected proportion of false positives among all discoveries. At FDR=0.05, if you declare 20 results significant, you expect about 1 to be false. FDR methods (Benjamini-Hochberg) are more powerful and better suited to exploratory research, biomarker discovery, and genomics where missing true effects is costlier than occasional false positives.
How do I apply the Benjamini-Hochberg FDR procedure? +
Steps: (1) Run k tests and collect all p-values. (2) Rank them smallest to largest: p(1) ≤ p(2) ≤ … ≤ p(k). (3) For each rank i, compute the BH threshold: (i/k) × Q, where Q is your FDR level (commonly 0.05 or 0.20). (4) Starting from the largest rank, find the largest i* where p(i*) ≤ (i*/k) × Q. (5) Declare all tests with rank ≤ i* as significant. Once p(i) exceeds its threshold working top-to-bottom, all remaining tests are non-significant — even if a later p-value falls below its individual threshold. In R: p.adjust(p_values, method="BH"). In SPSS: available through syntax or the BH add-on.
Is Bonferroni too conservative? +
Bonferroni is conservative in two scenarios: (1) when tests are positively correlated (measuring related biological pathways means tests are not truly independent; Bonferroni treats them as if they are, over-correcting); and (2) when k is very large (GWAS with 10 million SNPs, where the Bonferroni threshold of p<5 × 10⁻⁸ is necessarily stringent). For small numbers of pre-specified, relatively independent comparisons (k ≤ 10), Bonferroni’s conservatism is modest and acceptable. When conservatism is a concern, use Holm-Bonferroni (same FWER control, more power) or BH-FDR (trades some FWER protection for greater power). Never respond to Bonferroni “missing” a result by switching to LSD or no correction — that is selecting a method by its output.
What is the Holm-Bonferroni procedure? +
Holm-Bonferroni (Holm, 1979) is a sequential step-down procedure that provides the same strict FWER control as standard Bonferroni but is uniformly more powerful — it detects at least as many true effects and usually more. Procedure: rank p-values smallest to largest. Compare p(1) to α/k. If significant, continue; compare p(2) to α/(k−1). At each step i, compare p(i) to α/(k−i+1). Stop when p(i) first exceeds its threshold — all remaining tests are non-significant. The threshold relaxes with each step (denominator decreases), giving Holm its power advantage. R: p.adjust(method="holm"). This should be the standard choice over Bonferroni for any analysis with k>3 pre-specified comparisons.
Do post-hoc tests after ANOVA correct for multiple comparisons? +
Yes — the major post-hoc tests are specifically designed for this: Tukey HSD controls FWER for all pairwise comparisons (balanced groups); Tukey-Kramer extends this to unequal group sizes; Scheffé’s controls FWER for all possible contrasts (most conservative); Holm-Bonferroni is appropriate for pre-specified, non-exhaustive subsets. LSD (Least Significant Difference) does NOT control FWER for k ≥ 4 groups — it should never be used as the only post-hoc method when four or more groups are compared. If you used LSD, re-run with Tukey HSD and report both; if Tukey does not confirm LSD’s findings, the LSD results are unreliable.
What is p-hacking and how does it relate to multiple testing? +
P-hacking is running multiple analyses and reporting only those that achieve p<0.05, without disclosing the number of tests attempted. It is a form of multiple testing inflation made invisible by selective reporting. Common manifestations: testing 12 outcomes and reporting 3 significant ones; testing 8 subgroups until one is significant; re-defining covariates until p<0.05 is achieved; stopping data collection when p first crosses the threshold. P-hacking is the primary driver of the replication crisis: studies that are fishing for significance will almost certainly catch something — but it won’t be real. Defence: pre-registration of all outcomes and analysis plans before data collection; full reporting of all tests in the paper (including non-significant ones); independent replication.
How should multiple testing be reported in a thesis? +
In Methods: state the number of planned comparisons, the correction method applied, and whether any analyses are exploratory. In Results: report both uncorrected and corrected p-values (or state the adjusted threshold); report all tests performed, not just significant ones; label exploratory findings explicitly. In Discussion: acknowledge the multiple testing context when interpreting results; do not present exploratory findings as confirmed effects; state the FWER for any uncorrected exploratory analyses. Thesis examiners will ask: “How many tests did you run in total?” and “Did your significant results survive correction?” — have the numbers ready and the correction method justified in the Methods.
When is it acceptable to not correct for multiple comparisons? +
Legitimate exceptions: (1) Single pre-specified primary outcome — one planned test needs no correction. (2) Hierarchical testing — gate secondary tests behind a significant primary; each test is protected by the prior gate. (3) Safety data reporting — FDA guidance permits individual safety event reporting without alpha correction because missing a safety signal costs more than a false positive. (4) Exploratory analyses transparently disclosed — no correction is applied, but the total number of tests and implied FWER must be stated, and findings must be labelled hypothesis-generating. (5) Descriptive analyses — baseline characteristic comparisons in an RCT are not hypothesis tests (they describe the population) and do not need correction. In all cases, the decision must be pre-specified and justified, not decided post-hoc after seeing that correction would remove “significant” results.
Check Your Multiple Testing Exposure
P-value adjustment tools, Bonferroni, Holm, and BH-FDR correction calculators for medical researchers.
Open StatClinic →