Why Statistical Comments Dominate Peer Review
Most medical journals now route submissions through a statistical reviewer or a statistically literate editor specifically because a flawed analysis can produce a confident, publishable-looking conclusion that is simply wrong — regardless of how important the clinical question is or how well the study was conducted at the bedside. A wrong test, an unadjusted comparison, or an underpowered sample size are each independently sufficient grounds for rejection at most journals, which is why these comments appear so consistently across specialties and journal tiers. The good news is that the same handful of issues recur constantly, which is exactly what makes them predictable — and preventable — before you ever submit.
1. "The Wrong Statistical Test Was Used"
Why Reviewers Ask This
Every parametric test carries assumptions — normality being the most commonly violated in medical data (lab values, length of stay, symptom scores) — and applying a parametric test to a variable that clearly violates those assumptions can distort the p-value and confidence interval in either direction. Reviewers with statistical training recognize certain variables (CRP, bilirubin, hospital stay, viral load) as reliably skewed from experience, even without seeing your histogram, and will flag the mismatch on sight.
How to Fix It
Run a formal normality test (Shapiro-Wilk for n < 50, either Shapiro-Wilk or Kolmogorov-Smirnov for larger samples) on the disputed variable, and let the result decide the test, not convenience or precedent. If the variable is confirmed non-normal, switch to the corresponding non-parametric test — Mann-Whitney U in place of the independent t-test, Wilcoxon signed-rank in place of the paired t-test, Kruskal-Wallis in place of one-way ANOVA — and report medians and IQR instead of means and SD throughout that comparison.
"We thank the reviewer for this important observation. We re-assessed the distribution of CRP using the Shapiro-Wilk test, which confirmed significant non-normality (p < 0.001) in both groups. We have therefore replaced the independent t-test with the Mann-Whitney U test, and revised Table 2 and the Results text (page 6) to report median (IQR) rather than mean ± SD for this variable. The revised result (U = 412.5, p = 0.008) is consistent with our original conclusion."
Run and report a normality test for every continuous variable before choosing an analysis, not after a reviewer asks — this single step, done up front, prevents the most common statistical comment in medical peer review. Use the normality test calculator and see How to Choose the Right Statistical Test for the full decision framework, or the Test Finder wizard to confirm your choice interactively.
2. "No Normality Assessment Was Reported"
Why Reviewers Ask This
Even when the correct test was ultimately used, an absent normality statement leaves the reviewer unable to verify that the choice was principled rather than accidental. This comment is really asking for transparency and reproducibility, both explicitly required by CONSORT and STROBE reporting checklists: a reader should be able to see exactly how each analytic decision was reached.
How to Fix It
Add one sentence to your Statistical Analysis subsection stating which normality test was used and the general decision rule applied — for example, "Continuous variables were assessed for normality using the Shapiro-Wilk test; parametric tests were used for normally distributed variables (p > 0.05) and non-parametric alternatives for non-normally distributed variables." This single sentence resolves the comment for every test in the paper at once, not just one instance.
"We agree this was insufficiently described. We have added the following sentence to the Statistical Analysis subsection (page 5): 'Normality of continuous variables was assessed using the Shapiro-Wilk test; variables with p > 0.05 were analyzed using parametric methods, and variables with p ≤ 0.05 were analyzed using the corresponding non-parametric test.' We have also added a supplementary table reporting the Shapiro-Wilk result for every continuous variable in the study."
Write the normality-testing sentence into your Methods draft before you run a single inferential test, as a standing part of your analysis template, rather than retrofitting it after results are in. See our normality testing guide for the full Shapiro-Wilk vs Kolmogorov-Smirnov comparison.
3. "No Sample Size Justification Was Provided"
Why Reviewers Ask This
Without a documented a priori sample size calculation, a reviewer cannot distinguish a true null result from a false negative caused simply by insufficient power — and a study that recruited "however many patients we could get in a year" invites exactly that suspicion. This is a mandatory item on both the CONSORT checklist for trials and STROBE for observational studies, and its absence is one of the most common reasons for a "major revision" verdict.
How to Fix It
If the sample size was pre-calculated but simply omitted from the manuscript, add the calculation with its assumed effect size, alpha, power, and the specific formula or software used. If no formal calculation was performed before data collection, perform and clearly report a post-hoc power analysis, and add an explicit limitation acknowledging that the study may be underpowered to detect smaller effects, especially for any non-significant secondary outcome.
"We thank the reviewer for raising this. The target sample size of 84 was calculated to detect a 15% difference in the primary outcome between groups, with 80% power and alpha = 0.05, using a two-sided test, which required 76 patients; we recruited 84 to allow for 10% attrition. This has been added to the Methods (page 4). We have also added a post-hoc power calculation for the non-significant secondary outcome and note this as a limitation in the Discussion."
Calculate your required sample size before starting recruitment, not after, and keep the calculation's assumptions (expected effect size, alpha, power) documented so they can be dropped directly into your Methods section later. Use the sample size calculator and see Sample Size for Cross-Sectional Studies or Sample Size for Case-Control Studies depending on your design.
4. "Confidence Intervals Are Missing"
Why Reviewers Ask This
A p-value only signals whether an effect is statistically distinguishable from zero; it conveys nothing about the plausible range of the true effect size. A 95% CI does that job, and both ICMJE recommendations and APA 7th edition treat it as a required companion to any p-value, not an optional extra — a reviewer who spots the omission is enforcing a near-universal journal requirement, not a personal preference.
How to Fix It
Go through every inferential result in the Results section and add the corresponding 95% CI: for a mean difference, an odds ratio, a hazard ratio, a correlation coefficient, or a regression coefficient, virtually all standard statistical software reports the CI alongside the point estimate by default — it usually just needs to be transcribed into the manuscript rather than recalculated.
"We agree and have added 95% confidence intervals throughout the Results and in Tables 2–4. For example, the primary outcome result now reads: 'mean difference 12.4 mmHg, 95% CI [9.85, 14.95], t(59) = 9.80, p < 0.001,' and the same has been applied to every other test statistic in the manuscript."
Adopt a fixed reporting template that includes the CI as a mandatory field for every result, so it is never an afterthought. See Confidence Intervals Explained and our full test-by-test reporting guide, or compute a CI directly with the confidence interval calculator.
5. "No Effect Size Was Reported"
Why Reviewers Ask This
Statistical significance and clinical importance are independent properties of a result, and large samples routinely produce tiny, clinically meaningless p-values — a reviewer raising this comment is testing whether the authors can distinguish the two, which is exactly the distinction a clinical reader needs to interpret the finding correctly.
How to Fix It
Calculate and report the appropriate effect size for each test — Cohen's d for a mean difference, eta-squared for ANOVA, an odds or hazard ratio for categorical or survival outcomes, r or r² for a correlation — and add a sentence in the Discussion explicitly interpreting whether the magnitude, not just the p-value, is likely to matter clinically.
"We thank the reviewer for this important point. We have calculated Cohen's d for the primary comparison (d = 0.21, a small effect by conventional benchmarks) and added this to the Results. We have also added a paragraph to the Discussion (page 11) explicitly addressing that, despite statistical significance, the absolute difference of 2.1 mmHg is unlikely to be clinically meaningful at the individual patient level, and we have revised our concluding statement accordingly."
Report an effect size alongside every p-value as a fixed habit, especially in large-sample studies where significance is easy to achieve. Use the effect size calculator and see Effect Size and Clinical Significance or Cohen's d Explained.
6. "Regression Assumptions Were Not Checked"
Why Reviewers Ask This
A regression model can produce a coefficient and p-value even when its underlying assumptions are badly violated — the software will not stop you — which means an unchecked model may report standard errors, confidence intervals, and significance levels that are simply wrong. Reviewers with methodological training know this failure mode is invisible in the output tables alone and can only be ruled out with diagnostic plots.
How to Fix It
Generate and review a residuals-versus-fitted plot (checking linearity and homoscedasticity) and a Q-Q plot of residuals (checking normality), and briefly describe the result in the Methods or as a supplementary figure. If an assumption is meaningfully violated, address it directly — a log transformation of a skewed outcome, a robust standard error correction, or switching to a more appropriate model — rather than silently proceeding.
"We thank the reviewer for raising this. We have added residual diagnostic plots as Supplementary Figure 2, confirming approximate linearity and homoscedasticity. Residuals showed mild right-skew, so we log-transformed the outcome variable and re-fitted the model; results were materially unchanged (log-transformed model: β = 0.31, 95% CI [0.18, 0.44] vs original β = 0.29, 95% CI [0.15, 0.43]), and we now report the log-transformed model as primary with a note in the Methods."
Treat residual diagnostics as a required step of fitting any regression model, not an optional check performed only if asked. Use the linear regression calculator and see Multivariate Analysis in Medical Research for building and validating multi-predictor models.
7. "Multicollinearity Was Not Assessed"
Why Reviewers Ask This
When two or more predictors in the same model are strongly correlated with each other, the model cannot reliably separate their individual effects — coefficients can become unstable, standard errors inflate, and a genuinely important predictor can appear falsely non-significant simply because it shares variance with a correlated variable in the same equation. This is invisible in the model's overall fit statistics and only shows up in a dedicated collinearity diagnostic.
How to Fix It
Check the Variance Inflation Factor (VIF) for every predictor — a VIF above 5 to 10 is generally treated as concerning. If problematic collinearity is found, address it directly: remove one of the correlated predictors on clinical or statistical grounds, combine them into a single composite variable, or use a penalized regression method (ridge or LASSO) that is more robust to correlated predictors, and report whichever solution was chosen and why.
"We thank the reviewer for this important point. VIF for BMI and waist circumference were 8.7 and 8.3 respectively, confirming problematic collinearity. As waist circumference is the more established predictor of visceral adiposity in our outcome of interest, we retained it and removed BMI from the final model. VIF values for all remaining predictors are now below 3 and are reported in Supplementary Table 3."
Check VIF for every multi-predictor model before interpreting individual coefficients, as a standard step alongside residual diagnostics. See Multivariate Analysis in Medical Research for building models with correlated candidate predictors.
8. "No Correction for Multiple Comparisons"
Why Reviewers Ask This
Every additional hypothesis test performed at the conventional 0.05 threshold adds to the overall chance of at least one false-positive result purely by chance — with 14 independent uncorrected tests, the probability of at least one spurious "significant" finding rises well above 50%. Reviewers flag this because uncorrected multiple testing is one of the most common ways an underpowered or exploratory analysis produces a headline result that fails to replicate.
How to Fix It
Apply an appropriate correction — Bonferroni for a small number of pre-specified comparisons, or a false discovery rate method (Benjamini-Hochberg) for a larger exploratory set — and clearly distinguish in the manuscript between your single pre-specified primary outcome (which does not require correction) and any secondary or exploratory outcomes that do. Explicitly label results that no longer reach significance after correction as hypothesis-generating rather than confirmatory.
"We thank the reviewer for this critical observation. We have applied a Bonferroni correction to the 14 secondary outcome comparisons (adjusted alpha = 0.0036). Of the 5 outcomes originally reported as significant, 2 remain significant after correction; we have revised Table 3 to show both unadjusted and Bonferroni-adjusted p-values, and revised our Discussion to describe the 3 no-longer-significant findings as exploratory and hypothesis-generating rather than confirmatory."
Pre-specify your primary outcome before data collection, and plan a correction method for any secondary or exploratory comparisons at the design stage rather than after seeing which ones turned out significant. See The Multiple Testing Problem Explained.
9. "Handling of Missing Data Was Not Described"
Why Reviewers Ask This
How missing data is handled can materially change a study's results — listwise deletion can introduce bias if data are not missing completely at random, and a reader has no way to judge whether this occurred if missingness is never quantified or explained. STROBE explicitly requires authors to report the number of participants with missing data for each variable of interest.
How to Fix It
Report the number and percentage of missing values for every key variable, state the assumed missingness mechanism (missing completely at random, missing at random, or missing not at random) and your justification for that assumption, and describe the handling method used — complete-case analysis, multiple imputation, or another approach — including how many cases were affected.
"We apologize for this omission. Missing data ranged from 0% to 8.3% across variables (detailed in new Supplementary Table 1). Given the plausibly random pattern of missingness (unrelated to outcome, confirmed via Little's MCAR test, p = 0.34), we used complete-case analysis for the primary outcome, affecting 6 of 110 participants (5.5%); this is now stated explicitly in the Methods (page 5) along with a sensitivity analysis using multiple imputation, which produced consistent results (reported in Supplementary Table 2)."
Run a frequency check on every variable immediately after data entry to quantify missingness before any analysis, and decide your handling strategy at the design stage rather than reactively. See Handling Missing Data in Medical Research, or use the Excel data cleaning tool to detect and quantify missingness automatically.
10. "Descriptive Statistics Are Inappropriate for the Data"
Why Reviewers Ask This
An SD that is close to or larger than the mean is a well-known visual signature of right-skew (a few extreme values, such as very long hospital stays, pulling the mean upward), and reporting mean ± SD for such a variable is both statistically misleading and immediately recognizable to an experienced reviewer — often flagged before any test result is even read.
How to Fix It
Formally test the disputed variable for normality, and if confirmed skewed, replace mean ± SD with median and interquartile range throughout the manuscript — not only in the summary table but consistently in every place that variable is discussed, including the abstract if it appears there.
"We thank the reviewer for this observation. Shapiro-Wilk confirmed significant non-normality for length of stay (p < 0.001). We have replaced mean ± SD with median (IQR) throughout the manuscript: length of stay is now reported as 4 days (IQR 3–7), consistent with Table 1, the Results text, and the Abstract."
Before finalizing Table 1, scan every continuous variable for an SD approaching or exceeding its mean as a quick red flag, then confirm formally with a normality test rather than defaulting to mean ± SD out of habit. See Median vs Mean: Which One Should You Report? and Outlier Detection in Medical Research.
11. "The Statistical Methods Section Lacks Sufficient Detail"
Why Reviewers Ask This
Reproducibility is a foundational requirement of the scientific method, and a Methods section that reads as a vague summary ("data were analyzed using SPSS") rather than a precise recipe prevents another researcher from independently verifying or replicating the analysis — this is exactly what a statistical reviewer is trained to check line by line.
How to Fix It
Expand the Statistical Analysis subsection to explicitly state: the software name, version, and manufacturer; the specific test used for each type of comparison (not a generic list); the alpha threshold and whether tests were one- or two-tailed; how continuous and categorical variables were each summarized; how missing data and multiple comparisons were handled; and, for any multivariable model, exactly how candidate covariates were chosen (a priori clinical relevance, univariate screening at a stated p-value threshold, or a specific variable-selection algorithm).
"We have substantially expanded the Statistical Analysis subsection (pages 4–5) to address each point: software (IBM SPSS Statistics v28.0), specific tests per comparison type, alpha = 0.05 two-tailed throughout, normality testing procedure, missing data handling, and the covariate selection process for the multivariable model (variables associated with the outcome at p < 0.10 on univariate analysis were entered into the multivariable model, per a priori protocol)."
Draft your Statistical Analysis subsection as a step-by-step protocol before data collection begins, treating it as a recipe a stranger could follow exactly, rather than writing it retrospectively as a brief summary after the results are known. See How to Report Statistical Results Correctly.
12. "The Conclusions Are Not Supported by the Data"
Why Reviewers Ask This
Study design determines what kind of claim the data can support, and this comment is almost always about a mismatch between an observational design (which can establish association, adjusted for measured confounders) and causal language (which properly requires randomization or very strong quasi-experimental evidence). Reviewers treat this as a scientific integrity issue, not a stylistic one, because overstated conclusions can directly influence clinical practice.
How to Fix It
Systematically replace causal verbs ("caused," "resulted in," "led to") with association language ("was associated with," "was correlated with") everywhere the study design does not support causation, add or expand a paragraph explicitly discussing plausible unmeasured confounders and other alternative explanations, and, where relevant, note that adjustment for measured confounders (via multivariable regression or propensity matching) does not eliminate the possibility of residual or unmeasured confounding.
"We thank the reviewer for this essential correction. We have revised all causal language throughout the Abstract, Results, and Discussion to association language (e.g., 'Drug X use was associated with improved 30-day outcomes' rather than 'caused'). We have also expanded the Discussion (page 12) to explicitly address residual confounding, including unmeasured variables such as disease severity at baseline and treatment adherence, as limitations of the retrospective design."
Match your language to your design at every draft stage — reserve causal verbs strictly for randomized designs, and build a habit of re-reading your own Abstract and Discussion specifically for overstated claims before submission. See Confounding and Effect Modification Explained and Common Statistical Mistakes in Medical Research.
The Reviewer's Checklist: What Statistical Reviewers Actually Scan For
Understanding the reviewer's own mental checklist is the fastest way to pre-empt it. This is roughly the sequence an experienced statistical reviewer runs through on a first pass.
| Reviewer Checks | What Triggers a Comment |
|---|---|
| Test selection | Parametric test used with no stated normality check, or a test that doesn't match the data type/design |
| Sample size | No a priori power calculation, especially alongside a non-significant primary result |
| Completeness of reporting | P-value present without test statistic, df, effect size, or 95% CI |
| Table 1 consistency | SD larger than the mean, or varying denominators suggesting unreported missing data |
| Regression rigor | No residual diagnostics, no VIF for multi-predictor models, unclear covariate selection |
| Multiple testing | Many comparisons with no correction and no primary/secondary outcome distinction |
| Claims vs design | Causal language in an observational study; over-generalized conclusions beyond the sample studied |
| Methods reproducibility | Software/version not stated; vague description that would not allow replication |
Pre-Submission Manuscript Checklist
Run your manuscript through this list before submission — it mirrors the 12 comments above directly.
Normality tested and reported
Every continuous variable checked, with the test and result stated in Methods.
Test choice matches the data
Parametric vs non-parametric decision documented, not assumed.
Sample size / power calculation included
Assumptions (effect size, alpha, power) stated explicitly in Methods.
95% CI reported for every primary result
Not just for the headline finding — for every test statistic quoted.
Effect size reported alongside every p-value
Cohen's d, OR, HR, r, or eta-squared as appropriate to the test.
Regression diagnostics performed
Linearity, homoscedasticity, normality of residuals checked and reported.
VIF checked for multi-predictor models
Any predictor above the collinearity threshold addressed explicitly.
Multiple comparisons corrected where needed
Primary outcome distinguished from secondary/exploratory comparisons.
Missing data quantified and explained
Percentage missing per variable, mechanism assumed, handling method stated.
Descriptive statistics match the distribution
Mean ± SD for normal data, median (IQR) for skewed data — no exceptions.
Methods section is fully reproducible
Software, version, tests, thresholds, and covariate selection all specified.
Conclusions match the study design
Causal language reserved for randomized designs; confounding discussed for observational studies.
Frequently Asked Questions
Catch These Issues Before a Reviewer Does
Run your data through StatClinic's calculators for normality testing, effect sizes, confidence intervals, and regression diagnostics — and get a journal-ready, reviewer-proof results sentence generated automatically. Free, no registration required.
Try StatClinic Free →