The Seven Mistakes at a Glance
Each mistake described in this guide represents a category of error identified repeatedly in published methodological audits and statistical peer reviews. They are ordered roughly from most to least frequently encountered in clinical research submissions.
Mistake 1: Selecting the Wrong Statistical Test
Applying a statistical test that does not match the data type, the number of groups, or the distributional assumptions of the outcome variable. The most common versions: using a t-test on ordinal or non-normal data, applying chi-square when expected cell frequencies are below 5, using Pearson correlation on a non-linear relationship, or running one-way ANOVA without checking the sphericity assumption for repeated measures.
A rheumatology study compares post-operative pain intensity — measured on a 0–10 visual analogue scale (VAS) — between two NSAID regimens using an independent samples t-test. The VAS is ordinal: the difference between a score of 3 and 4 is not necessarily the same as between 7 and 8, and in an acute pain population, scores cluster at the low end (floor effect). The correct test is the Mann-Whitney U test. Running a t-test on this data inflates the Type I error rate and produces an invalid p-value. The published conclusion — "Regimen A provides significantly greater analgesia" — is not supported by the appropriate analysis.
A wrong test produces a wrong p-value. The conclusion — significant or non-significant — may be incorrect in either direction. Statistical reviewers at major journals now use systematic checklists, and test-selection errors are a documented leading cause of peer-review rejection and post-publication corrections. A 2020 audit of 160 surgical RCTs found that 22% had used a parametric test on clearly non-normal outcomes without justification.
- Before choosing any test, classify your outcome variable explicitly: continuous (interval/ratio), ordinal, or categorical (nominal).
- Ask: Are groups independent or paired/matched? How many groups? Are there confounders requiring adjustment?
- Run a normality check (Shapiro-Wilk) before deciding between parametric and non-parametric.
- Use a validated statistical decision tree — the StatClinic test selection guide covers every branch of this decision.
- State your test selection rationale explicitly in the Methods section, including any assumptions checked.
Independent t-test on VAS pain scores
(ordinal, non-normal, floor-effected)
Mann-Whitney U test
(non-parametric, 2 independent groups)
Mistake 2: Ignoring Normality Assumptions
Applying the t-test, ANOVA, Pearson correlation, or linear regression without ever checking whether the outcome variable (or regression residuals) is approximately normally distributed. Many researchers assume that clinical data is normal by default, or perform the analysis first and add a normality check only if a reviewer requests it. This is methodologically backwards: the normality check must precede test selection, not follow it.
A hepatology team compares serum alanine aminotransferase (ALT) between alcoholic hepatitis patients and healthy controls using an independent t-test. ALT is structurally right-skewed — the majority of values cluster near the normal range (20–50 U/L) while a minority spike into the thousands during acute hepatic injury. No normality assessment is reported in the Methods. Shapiro-Wilk on the hepatitis group returns W=0.68, p<0.001. The t-test is invalid. The correct approach is Mann-Whitney U, with results reported as median (IQR) rather than mean ± SD.
Parametric tests applied to substantially skewed data produce incorrect p-values — typically underestimating variance in the tails, generating spuriously narrow confidence intervals and falsely low p-values. A conclusion of statistical significance may be an artefact of violating the normality assumption rather than a genuine treatment effect. A 2019 systematic review of 112 cardiology RCTs found that 48% applied parametric tests without reporting normality assessment.
- Run the Shapiro-Wilk test (n < 50) or Kolmogorov-Smirnov with Lilliefors correction (n > 50) on every outcome variable before selecting a test.
- Inspect a histogram and a Q-Q plot in SPSS — visual assessment is essential, especially in large samples where Shapiro-Wilk is oversensitive.
- Check the skewness statistic: values >1 or <−1 indicate meaningful skew warranting non-parametric tests.
- For variables known to be structurally non-normal (ALT, CRP, troponin, creatinine, LOS, costs), use non-parametric tests by default.
- Document in your Methods: "Normality was assessed using the Shapiro-Wilk test. Non-normally distributed variables were analysed using non-parametric tests and reported as median (IQR)."
t(48) = 2.31, p = 0.025
Mean ALT: 142 vs 28 U/L
U = 418, z = 3.84, p < 0.001
Median ALT: 124 (IQR 67–289) vs 26 (IQR 19–34) U/L
Mistake 3: Inadequate Sample Size
Beginning data collection without a prospective sample size calculation — or performing the calculation after data collection to justify the numbers available. Underpowered studies cannot reliably detect clinically important differences even when they exist (Type II error). The resulting false-negative conclusion can waste years of research, mislead systematic reviews, and delay effective treatments from reaching patients. The complementary error — vastly over-recruiting — wastes resources and unnecessarily exposes additional patients to study procedures.
A surgical team designs an RCT comparing surgical site infection (SSI) rates between standard wound care (expected 30%) and a new antimicrobial dressing (target reduction to 15%). They enrol n=20 per group. A properly powered study (two-sided chi-square, α=0.05, power=0.80) requires n=82 per group for this effect size. After analysis, the study finds SSI rates of 28% vs 19% (p=0.07). The team concludes: "No statistically significant benefit was demonstrated." This conclusion is unwarranted. The study was fatally underpowered — it had less than 30% power to detect a 15-percentage-point difference at this sample size. The dressing may genuinely reduce infections, but the study was too small to know.
Underpowered studies generate unduly wide confidence intervals that straddle the null, leading to false acceptance of no effect. These null results accumulate in systematic reviews, providing false scientific weight to ineffectiveness. A 2021 analysis of 119 orthopaedic RCTs found that 27% were underpowered to detect their primary outcome at 80% power. Post-hoc power analysis — calculating power after the fact to explain a non-significant result — is methodologically meaningless and is rejected by all major statistical bodies.
- Calculate sample size before data collection begins — always, without exception.
- Specify the expected effect size (from pilot data or published literature), α (typically 0.05), desired power (typically 0.80 or 0.90), and the appropriate test.
- Add a 10–20% inflation factor for expected dropout, non-compliance, and missing data.
- Report the complete formula and all inputs in the Methods section: "Sample size was calculated using [formula/tool], assuming [effect size], α=0.05, power=0.80, with 15% attrition adjustment, giving n=X per group."
- If a study is already complete and underpowered, report the 95% CI, acknowledge the limitation explicitly, and do not accept the null hypothesis based on p > 0.05 alone.
Mistake 4: Misusing the P-value
1. Treating p > 0.05 as "no effect." Absence of statistical significance is not evidence of absence of an effect. A non-significant result in a small sample may simply reflect insufficient power. 2. Treating p < 0.05 as clinical proof. Statistical significance and clinical importance are entirely different concepts. A large RCT can produce p < 0.001 for a difference that is too small to matter to any patient. 3. P-hacking: running multiple subgroup analyses, outcomes, or models and reporting selectively only those achieving p < 0.05, without correcting for multiple testing.
A cardiologist compares LDL reduction between atorvastatin 20 mg and 40 mg in n=30 per group. Mean difference: 8.2 mg/dL (95% CI: −0.4 to 16.8 mg/dL; p=0.062). The paper concludes: "There was no significant difference in LDL reduction between the two doses." The CI shows that the true effect could plausibly be as large as 16.8 mg/dL — a clinically important reduction. Declaring no effect based on p=0.062 alone is unjustified and misleading.
A researcher tests a drug's effect across 8 pre-specified subgroups (age >60, male sex, hypertension, diabetes, BMI >30, smoking, CKD, prior MI) without correction for multiple comparisons. One subgroup (males >60 with diabetes) achieves p=0.031. The discussion highlights this subgroup as a "clinically important finding." With 8 simultaneous tests at α=0.05, the probability of at least one false positive by chance alone is 1 − (0.95)⁴ ≈ 34%. Without Bonferroni correction (p < 0.006 per test), this result is unreliable.
P-value misuse is the statistical engine of the reproducibility crisis. Ioannidis's landmark 2005 analysis argued that the majority of published medical research findings may be false, with selective reporting and p-value misuse among the primary drivers. Journals including JAMA, The BMJ, and the New England Journal of Medicine have since updated their statistical reporting requirements to mandate effect sizes, confidence intervals, and pre-registration. Papers without these elements face increasingly difficult peer review.
- Always report the effect size alongside the p-value: Cohen's d, OR, RR, η², Cramér's V, or Pearson r as appropriate.
- Report 95% confidence intervals for every key estimate — the CI conveys what the p-value cannot: the direction and plausible magnitude of the effect.
- Pre-register your primary outcome and analysis plan in a trial registry (ClinicalTrials.gov, ISRCTN) or on OSF before data collection.
- Apply Bonferroni correction (p < α/k) or Benjamini-Hochberg FDR correction when conducting multiple simultaneous comparisons.
- Never write "no difference was found" when p > 0.05 in an underpowered study. Write instead: "The study was not sufficiently powered to detect a clinically important difference; no firm conclusion regarding equivalence can be drawn."
Mistake 5: Overfitting Regression Models
Including too many predictor variables for the sample size available in a multivariable regression model. An overfitted model learns the random noise in your specific dataset rather than the true underlying signal — producing inflated apparent performance metrics (R², AUC, sensitivity) that collapse dramatically when the model is tested in a new patient population. Overfitting is also caused by data-driven stepwise variable selection: iteratively adding or removing variables based on p-values, which capitalises on chance associations in the dataset.
A critical care team develops a logistic regression model to predict in-hospital mortality in sepsis patients. They have n=80 patients with 22 deaths (the outcome events). They include 18 predictors: age, sex, BMI, diabetes, hypertension, CKD, lactate, creatinine, bilirubin, temperature, heart rate, MAP, GCS, SOFA score, antibiotic delay, fluid volume, vasopressor use, and infection source. The model achieves AUC=0.91 in the training set. Two independent validation attempts achieve AUC=0.61 and 0.58 — barely better than random. The Events Per Variable (EPV) ratio was 22 ÷ 18 = 1.2 — catastrophically below the recommended minimum of 10. The model overfit massively to noise.
An overfitted clinical prediction model, when deployed in real-world practice, will perform unpredictably across different patient populations and clinical settings. Decisions about ICU escalation, early discharge, or treatment initiation based on such a model can directly harm patients. Overfitted models published without external validation have contributed to flawed clinical decision-support tools that failed at the bedside. Peer reviewers increasingly demand bootstrap validation or prospective external validation before publication.
- Apply the EPV rule: Events ÷ Number of Predictors ≥ 10 (conservative: aim for ≥ 20).
- Pre-specify your predictor variables based on biological plausibility and prior literature — never use stepwise, backward, or forward selection driven by p-values.
- Perform internal validation: bootstrap resampling (500+ iterations) or k-fold cross-validation to obtain corrected performance estimates.
- Perform external validation in an independent dataset before publishing a prediction model.
- Report calibration (Hosmer-Lemeshow test or calibration plot) alongside discrimination (AUC/C-statistic). A high AUC with poor calibration indicates an overfitted model.
18 predictors, n=80, 22 events
EPV = 1.2 → AUC = 0.91 (training)
2 predictors (SOFA + lactate)
EPV = 11 → Bootstrap-corrected AUC = 0.78
Mistake 6: Incorrect Questionnaire and Likert Scale Analysis
1. No reliability check: Applying a questionnaire without reporting Cronbach's alpha. 2. Parametric tests on individual Likert items: Treating a 5-point Likert item as if it were measured on a continuous interval scale and applying a t-test. 3. Summing unreliable items: Computing a composite score from items that do not measure the same construct (low internal consistency). 4. Skipping factorial validity: Assuming a multi-item scale measures a single construct without checking dimensionality via exploratory or confirmatory factor analysis.
A physiotherapy study evaluates patient satisfaction using a 5-item questionnaire (each item: 1=Strongly Disagree to 5=Strongly Agree). The researcher sums the five items into a composite score (range 5–25) and runs an independent t-test to compare two treatment regimens. No Cronbach's alpha is reported. SPSS analysis reveals α=0.43 — well below the accepted minimum of 0.70. The five items are measuring different and inconsistent constructs. Summing them is therefore statistically invalid. The t-test comparison of the composite is meaningless — the composite is noise. The paper was rejected during peer review on this specific methodological ground.
An unreliable questionnaire contains excessive random measurement error. When t-tests or ANOVA are applied to such data, they may return statistically significant results driven by noise rather than genuine group differences — or conversely, fail to detect genuine differences because the outcome measure is too imprecise. Research conclusions built on unvalidated or misanalysed PROMs cannot be trusted or replicated. Journal editors in nursing, rehabilitation, psychiatry, and primary care have significantly tightened requirements for PROM reporting following systematic identification of this problem.
- Use an existing validated questionnaire (SF-36, EQ-5D, HADS, PHQ-9, ODI, AUDIT) where possible — validity and reliability are pre-established.
- If using a new questionnaire, report Cronbach's α ≥ 0.70 as a minimum threshold for acceptable internal consistency; include this in the Methods section.
- For individual Likert items, use non-parametric tests: Mann-Whitney U (2 groups), Kruskal-Wallis (3+ groups), Spearman correlation.
- For validated composite scores that are approximately normally distributed, parametric tests may be defensible — check and report the distribution.
- Report dimensionality: cite the original validation study or run exploratory factor analysis if using the scale in a new population.
Composite score (5 items summed)
t(58)=2.14, p=0.036
No Cronbach's α reported
PHQ-9 composite (α=0.86)
Approx. normal; t(58)=2.14, p=0.036
or Mann-Whitney U for single items
Mistake 7: Incorrect Interpretation of Results
Even when analysis is technically correct, the interpretation in the Discussion section can undermine the entire paper. The four most dangerous interpretation errors: (1) inferring causation from association in observational data; (2) treating statistical significance as clinical significance; (3) reporting an odds ratio as if it were a relative risk when the outcome is common; (4) interpreting a weak correlation coefficient (e.g., r=0.28) as evidence of a "strong" or "meaningful" association.
A cross-sectional ICU study finds that serum albumin correlates with 28-day mortality: r=−0.39, p=0.001. The Discussion concludes: "Low albumin causes increased ICU mortality. Albumin supplementation should therefore be considered as a therapeutic strategy." Serum albumin is a marker of critical illness severity, not a modifiable cause of death in most patients. Confounding by illness severity is entirely ignored. Dozens of RCTs of albumin infusion based on exactly this logic have found no mortality benefit or harm in certain groups.
A case-control study finds OR=2.8 for a dietary risk factor. Outcome prevalence in controls is 38%. The paper states: "Exposure to this dietary factor more than doubled disease risk." With high baseline prevalence (38%), the OR substantially overestimates the relative risk. Using the Zhang & Yu conversion formula: RR ≈ OR ÷ [(1−P₀) + (P₀ × OR)] = 2.8 ÷ [(0.62) + (0.38 × 2.8)] ≈ 1.68. The risk increased by 68%, not 180%. This distinction has direct implications for policy, NNT calculations, and clinical counselling.
Interpretation errors in high-profile papers directly shape clinical practice. Physicians reading a well-designed study that incorrectly concludes causation may implement interventions that are ineffective or harmful. Misreported ORs in case-control studies of common diseases have been cited in guidelines and systematic reviews, inflating perceived risk magnitudes. A 2023 umbrella review of cardiovascular risk factor studies found that >30% of case-control papers reporting ORs for common outcomes (prevalence >15%) did not acknowledge the OR-RR discrepancy in their interpretation.
- Never use the word "causes" from observational data unless you have performed a formal causal analysis (propensity score matching, instrumental variable, Mendelian randomisation, or difference-in-differences).
- In case-control studies where outcome prevalence exceeds 10%, convert OR to RR using the Zhang & Yu formula and report both clearly.
- Compare your effect size to the published Minimum Clinically Important Difference (MCID) for your outcome. Acknowledge explicitly if the statistically significant effect does not reach the MCID.
- Apply the standard correlation benchmarks: r=0.1–0.3 = small, r=0.3–0.5 = moderate, r > 0.5 = large (Cohen, 1988). Do not describe a weak correlation as "strong" or "clinically meaningful" without empirical justification.
- State design limitations clearly in the Discussion: "As a cross-sectional study, this design cannot establish temporality or causation. Residual confounding by unmeasured variables cannot be excluded."
Journal Publication Consequences
Each of these mistakes has a documented track record of consequences at the peer review and post-publication stages. Understanding what reviewers look for helps you pre-empt rejection.
| # | Mistake | Reviewer Response | Severity |
|---|---|---|---|
| 01 | Wrong test selection | Major revision: reanalysis with correct test required; results may change significantly | Critical |
| 02 | Normality not checked | Major revision: normality assessment and potentially reanalysis required | Critical |
| 03 | Underpowered study | Rejection or major revision; conclusion of "no effect" challenged; more recruitment required | Critical |
| 04 | P-value misuse | Major revision: effect sizes + CIs mandated; selective reporting may trigger editorial investigation | Critical |
| 05 | Overfitting regression | Rejection: external validation or cross-validation required before prediction model is publishable | High |
| 06 | Questionnaire errors | Major revision: reliability stats required; potential reanalysis with non-parametric tests | High |
| 07 | Misinterpretation | Major revision: causal language must be removed; OR/RR distinction must be addressed | High |
Practical Prevention Checklist
Use this checklist as a pre-submission quality control gate. Every item below corresponds to one of the seven mistake categories. Clear all items before submitting to any journal.
Frequently Asked Questions
Analyse Your Data Without the Mistakes
StatClinic's AI-powered tool guides you through test selection, normality checking, and correct interpretation — all without SPSS.
Try StatClinic Free →