A physiotherapy research team develops a new handheld dynamometer to measure quadriceps strength in patients recovering from knee replacement surgery. Before using the device in a clinical trial, they need to know: do two different physiotherapists using the same device get the same reading on the same patient? And does the same physiotherapist get the same reading if they test the same patient one week later? These are reliability questions — and answering them incorrectly can invalidate an entire measurement protocol. Many researchers answer them by computing Pearson's correlation coefficient (r). This is a well-documented methodological mistake: Pearson's r measures whether two sets of numbers move together, but it is blind to systematic bias. Two raters who consistently differ by 20 N can show a perfect r = 1.00 while being clinically useless. The correct tool for quantifying measurement reliability on a continuous scale is the Intraclass Correlation Coefficient (ICC) — a statistic that captures both the degree of correlation and the magnitude of any systematic disagreement between measurements.
What Is the Intraclass Correlation Coefficient?
The Intraclass Correlation Coefficient (ICC) is a reliability statistic for continuous or ratio-level data that quantifies the proportion of total measurement variance attributable to true between-subject differences, relative to the total variance including measurement error. In plain terms, it answers: "Out of all the variation I observe in my measurements, how much of it reflects genuine differences between patients — and how much is just noise from the measuring process?"
The ICC was introduced by Fisher (1925) as an extension of the standard Pearson correlation to situations where observations within the same subject (or matched group) are not distinguishable by role — they are intraclass. Unlike Pearson's r, which requires two distinguishable variables (X and Y), ICC can handle any number of raters rating the same subjects, and it is sensitive to both the rank-order correlation between raters and any systematic offset between them.
The ICC formula is grounded in Analysis of Variance (ANOVA): it decomposes the total variance in a dataset into variance due to subjects (MS_between), variance due to raters (MS_rater), and residual error (MS_error), then expresses the subject variance as a proportion of the total in a way that reflects the study's design.
Inter-Rater vs Intra-Rater Reliability
The two most common reliability questions in medical research require different study designs but the same statistical tool — ICC — applied to different data structures:
Inter-Rater Reliability (Inter-Observer Reliability)
Inter-rater reliability measures how consistently two or more different observers produce the same measurement when independently assessing the same subjects. This is critical whenever a clinical measurement depends on who performs it — examples include:
- Two radiologists independently measuring tumour diameter on the same CT images
- Two physical therapists assessing range of motion in the same patients
- Two nurses recording systolic blood pressure using the same sphygmomanometer protocol
- Two pathologists grading mitotic activity per high-power field in the same tissue sections
For inter-rater ICC, each subject is measured by each of the k raters, and the design question is: if a different set of raters from the same training background were to measure these patients, would they agree similarly?
Intra-Rater Reliability (Test-Retest Reliability)
Intra-rater reliability measures how consistently the same observer produces the same measurement when assessing the same subjects on two separate occasions, separated by a washout interval long enough to prevent memory effects but short enough that the true underlying value has not changed. Examples include:
- The same physiotherapist re-measuring quadriceps force seven days later
- The same radiologist re-reading the same scans after a four-week washout period
- The same nurse re-recording blood pressure one hour later under standardised conditions
Study design tip: For intra-rater studies, the washout interval must be long enough to eliminate memory effects (usually ≥ 2 weeks for subjective assessments), but short enough that the measured variable genuinely has not changed (disease-free interval for stable conditions). Document the exact washout period in your methods section.
ICC Models: The Shrout and Fleiss Framework
The most important and most commonly misunderstood aspect of ICC analysis is that there is not one ICC — there are six, defined by Shrout and Fleiss (1979) based on three design decisions. Choosing the wrong model produces a different (often misleading) reliability estimate and is a common peer-review criticism.
The Three Decisions
ICC Model Selection: Three Sequential Questions
1
Who are the raters? If the raters in your study are a random sample from a broader pool of equivalent raters and you want to generalise to that pool — use a random effects model. If the raters are the specific people whose reliability you want to describe (and you do not intend to generalise) — use a mixed or fixed effects model.
2
Are systematic rater differences clinically important? If you want to detect systematic biases between raters (e.g., Rater A consistently reads higher than Rater B), choose absolute agreement. If you only care about consistency in rank ordering of subjects (relative reliability), choose consistency.
3
How will the measurement be used clinically? If each patient will be measured by a single rater in practice, report the single measures ICC. If the clinical protocol averages across k raters, report the average measures ICC (which is always higher and reflects that averaging reduces random error).
| Model |
Design |
Agreement Type |
When to Use |
SPSS Label |
| ICC(1,1) |
One-way random |
Absolute |
Each subject rated by a different random subset of raters; rater identity unknown |
One-Way Random, Single |
| ICC(2,1) |
Two-way random |
Absolute |
Most common choice — all subjects rated by same raters; raters are random sample; want to detect systematic bias |
Two-Way Random, Absolute, Single |
| ICC(3,1) |
Two-way mixed |
Consistency |
Raters are the specific raters of interest (fixed); ignore systematic rater bias; test-retest often uses this |
Two-Way Mixed, Consistency, Single |
| ICC(1,k) |
One-way random |
Absolute |
As (1,1) but clinical practice averages across k raters |
One-Way Random, Average |
| ICC(2,k) |
Two-way random |
Absolute |
As (2,1) but clinical practice averages across k raters |
Two-Way Random, Absolute, Average |
| ICC(3,k) |
Two-way mixed |
Consistency |
As (3,1) but clinical practice averages across k raters |
Two-Way Mixed, Consistency, Average |
The most common ICC model error: Using ICC(3,1) consistency when ICC(2,1) absolute agreement is appropriate. ICC(3,1) ignores systematic rater differences by removing rater variance from the denominator, producing a higher (more optimistic) ICC than ICC(2,1). For most inter-rater studies in clinical research, ICC(2,1) absolute agreement is the correct and more conservative choice. Koo and Li (2016) recommend ICC(2,1) for inter-rater studies and ICC(3,1) for test-retest when the raters are fixed.
Interpretation: The Koo and Li (2016) Benchmarks
The most widely used modern interpretation scale for ICC in medical research is from Koo and Li (2016), published in the Journal of Chiropractic Medicine. It is recommended over older scales because it was derived with specific consideration for clinical measurement contexts and is more conservative than some earlier benchmarks:
| ICC Value | Reliability Level | Clinical Interpretation |
|---|
| < 0.50 | Poor | Unacceptable for clinical use; major measurement protocol revision required |
| 0.50 – 0.75 | Moderate | Borderline; may be acceptable for group-level research but not individual patient decisions |
| 0.75 – 0.90 | Good | Acceptable for most clinical measurements; minimum standard for primary outcome instruments |
| > 0.90 | Excellent | High-precision measurements suitable for individual clinical decisions and regulatory submissions |
Always report the 95% CI alongside the ICC point estimate. An ICC of 0.78 with a 95% CI of 0.52–0.91 spans both "moderate" and "excellent" — a much less certain result than it appears. The lower bound of the CI is the most conservative, clinically relevant figure. State which interpretation scale you are using and cite it explicitly.
Other Interpretation Scales
Earlier scales are still cited in older literature but have been largely superseded by Koo and Li (2016) in contemporary clinical research:
- Cicchetti (1994): <0.40 = poor, 0.40–0.59 = fair, 0.60–0.74 = good, ≥0.75 = excellent
- Fleiss (1986): <0.40 = poor, 0.40–0.75 = fair to good, ≥0.75 = excellent
- Portney & Watkins (2008): <0.75 = poor to moderate, ≥0.75 = good to excellent
These differences matter: an ICC of 0.62 would be "good" by Cicchetti (1994) but only "moderate" by Koo and Li (2016). Always cite which scale you used; do not select the scale post-hoc to improve the appearance of your results.
Clinical Examples
1
Inter-Rater Reliability: Blood Pressure Measurement by Two Nurses
A hospital quality improvement team wants to assess whether systolic blood pressure measurements taken by two nurses using the same standardised auscultatory protocol are sufficiently reliable to be used interchangeably in a hypertension monitoring programme. They measure 60 patients sequentially on the same visit, with a 5-minute rest between measurements.
Summary statistics from the ANOVA decomposition of the 60-patient, 2-rater dataset:
142.3
Mean SBP, Nurse A (mmHg)
143.6
Mean SBP, Nurse B (mmHg)
ANOVA variance components (from SPSS output)
MS_between subjects = 312.4 mmHg²
MS_within subjects = 18.6 mmHg²
MS_rater = 26.4 mmHg² (systematic Nurse B − A bias of 1.3 mmHg)
MS_error = 14.2 mmHg²
ICC(2,1) — Two-Way Random, Absolute Agreement, Single Measures
ICC = (MS_B − MS_E) / (MS_B + (k−1)×MS_E + k×(MS_R − MS_E)/n)
= (312.4 − 14.2) / (312.4 + 1×14.2 + 2×(26.4−14.2)/60)
= 298.2 / (312.4 + 14.2 + 0.41) = 298.2 / 327.0 = 0.912
95% Confidence Interval
95% CI: 0.87 – 0.94 (from F-distribution)
Result: ICC(2,1) = 0.912 (95% CI: 0.87–0.94) — Excellent inter-rater reliability (Koo & Li 2016)
Interpretation: The two nurses showed excellent agreement (ICC(2,1) = 0.912) in systolic blood pressure measurement. The systematic difference of 1.3 mmHg between nurses (Nurse B consistently reading higher) is captured by the absolute agreement model and slightly depresses the ICC below what a consistency model would report. At this level of reliability, the two nurses' measurements can be considered interchangeable for clinical monitoring purposes — the 1.3 mmHg systematic bias is within the expected measurement error of auscultatory sphygmomanometry and is clinically negligible. The team can proceed with the hypertension monitoring programme using either nurse as the measurer.
2
Intra-Rater Reliability: Knee Flexion Range of Motion
A physiotherapy researcher develops a standardised goniometer protocol for measuring active knee flexion range of motion (ROM) in patients with osteoarthritis. To establish intra-rater reliability, the same physiotherapist measures 45 patients twice, with a two-week washout interval between sessions. The true underlying ROM is assumed stable over two weeks in this stable OA cohort.
108.4°
Mean ROM, Session 1
107.9°
Mean ROM, Session 2
Model selection
Intra-rater: same rater at 2 time points (fixed rater) → ICC(3,1) consistency
ANOVA-based ICC output (from SPSS)
MS_between subjects = 428.6 deg²
MS_error = 28.3 deg²
MS_session = 2.0 deg² (0.5° mean difference — trivial)
ICC(3,1) — Two-Way Mixed, Consistency, Single Measures
ICC = (MS_B − MS_E) / (MS_B + (k−1)×MS_E)
= (428.6 − 28.3) / (428.6 + 1×28.3)
= 400.3 / 456.9 = 0.876
95% CI and SEM
95% CI: 0.79 – 0.93
SEM = SD_pooled × √(1 − ICC) = 21.0 × √(0.124) = 7.4°
MDC₉₅ = SEM × 1.96 × √2 = 7.4 × 2.77 = 20.5°
Result: ICC(3,1) = 0.876 (95% CI: 0.79–0.93) — Good intra-rater reliability (Koo & Li 2016)
Interpretation: The physiotherapist demonstrated good intra-rater reliability (ICC = 0.876) for knee flexion ROM measurement. The Standard Error of Measurement (SEM = 7.4°) and Minimal Detectable Change at the 95% confidence level (MDC₉₅ = 20.5°) are clinically important companions to the ICC. They tell you: any ROM change of less than 20.5° between two sessions cannot be distinguished from measurement noise. This information must accompany the ICC in the thesis and be incorporated into the clinical protocol — if the expected treatment effect on ROM is 15°, this tool cannot detect it with confidence despite having "good" reliability.
3
Inter-Rater Reliability: Tumour Diameter on CT Imaging
A multicentre oncology trial requires radiologists at different sites to measure the longest diameter of lung tumours on CT scans (RECIST criteria). Two radiologists independently measure the same 80 target lesions. The trial protocol specifies that a single radiologist's measurement is used at each site, so single-measures ICC(2,1) is the appropriate model. Absolute agreement is required because systematic biases between sites would affect response classification.
28.4 mm
Mean diameter, Rad 1
29.1 mm
Mean diameter, Rad 2
Model
ICC(2,1) — Two-Way Random, Absolute Agreement, Single Measures
(raters represent the broader radiologist population at trial sites; absolute agreement required)
SPSS Output
ICC(2,1) point estimate = 0.934
95% CI: 0.906 – 0.954
F(79, 79) = 29.4, p < 0.001
SEM = 2.8 mm
MDC₉₅ = 7.8 mm
Result: ICC(2,1) = 0.934 (95% CI: 0.91–0.95) — Excellent inter-rater reliability (Koo & Li 2016)
Interpretation: Excellent radiologist agreement was demonstrated for RECIST tumour diameter measurement (ICC(2,1) = 0.934). However, the MDC₉₅ of 7.8 mm has important implications: a measured change of less than 7.8 mm between imaging time points cannot be reliably attributed to true tumour growth or shrinkage rather than measurement variability between radiologists. RECIST 1.1 defines partial response as ≥30% diameter decrease — for a 30 mm tumour this is 9 mm, just above the MDC. The trial team should factor this into their response adjudication protocol and use the same radiologist at each site where possible to eliminate inter-rater error entirely.
SEM and MDC: ICC's Clinical Companions
The ICC alone does not tell clinicians how much measurement error matters in practice. Two derived statistics that should accompany every ICC report are the Standard Error of Measurement (SEM) and the Minimal Detectable Change (MDC):
- SEM = SDpooled × √(1 − ICC) — the expected measurement error in the original units (mmHg, degrees, mm). A smaller SEM indicates a more precise instrument.
- MDC₅₅ = SEM × 1.96 × √2 — the minimum change between two measurements that exceeds measurement error with 95% confidence. Changes smaller than MDC₅₅ cannot be distinguished from measurement noise.
These values contextualise the ICC and allow clinicians to judge whether a measurement instrument is precise enough for its intended use. A high ICC can coexist with a large SEM if the subjects in the reliability study were very heterogeneous — always report SEM and MDC alongside ICC.
ICC vs Cohen's Kappa vs Pearson's r vs Bland-Altman
ICC
- Continuous or ratio data
- Sensitive to both correlation and systematic bias
- Can handle 2+ raters simultaneously
- Multiple model choices (1,1), (2,1), (3,1)
- Reports SEM and MDC
Cohen's Kappa
- Categorical (nominal/ordinal) data
- Corrects for chance agreement
- Binary or multi-category classification
- Weighted kappa ≈ ICC(2,1) for ordinal
- Cannot capture magnitude of numeric error
Pearson's r / Bland-Altman
- Pearson r: correlation only (not agreement)
- Blind to systematic bias between raters
- Bland-Altman: plots differences vs means
- Shows limits of agreement (LOA) in original units
- Use B-A to complement ICC — different questions
ICC and Bland-Altman are complementary, not alternatives. ICC tells you how much of the total variance is signal vs noise (a single reliability number). Bland-Altman analysis tells you the actual distribution of differences between raters in clinical units — whether the errors are random or show a systematic pattern (proportional bias), and whether they are clinically tolerable. For a complete reliability study, report both: ICC for the reliability coefficient and Bland-Altman for the limits of agreement in clinically interpretable units.
Thesis and Research Reporting Recommendations
A reliability study using ICC has specific structural requirements in the Methods, Results, and Discussion sections that differ from standard inferential analyses.
Methods Section: Specify Everything
State all of the following explicitly:
- Number of raters, their qualifications, experience level, and blinding to each other's measurements
- Whether the study measures inter-rater or intra-rater reliability (or both)
- For intra-rater: the exact washout interval and justification for why the target variable is stable over that period
- The ICC model chosen (e.g., ICC(2,1) two-way random absolute agreement single measures) and the reason for selecting it over alternatives
- The interpretation scale to be used (e.g., Koo & Li 2016) and the minimum acceptable ICC threshold pre-specified in the protocol
- How SEM and MDC will be calculated
Results Section: Report the Full Picture
Model Reporting Paragraph — Inter-Rater
"Inter-rater reliability for ultrasound-measured common femoral artery diameter was assessed between two vascular sonographers using ICC(2,1) (two-way random effects, absolute agreement, single measures) in accordance with the recommendations of Koo and Li (2016). Measurements were performed independently on 65 participants without communication between sonographers. ICC(2,1) was excellent at 0.923 (95% CI: 0.884–0.949, F(64,64) = 24.8, p < 0.001). The Standard Error of Measurement was 0.32 mm and the Minimal Detectable Change at the 95% confidence level was 0.89 mm."
Model Reporting Paragraph — Intra-Rater
"Intra-rater reliability was assessed for the same sonographer repeating measurements on the same 65 participants after a three-week washout period. ICC(3,1) (two-way mixed effects, consistency, single measures) was used as the rater was a fixed element of the study. Intra-rater reliability was excellent (ICC(3,1) = 0.947, 95% CI: 0.919–0.964, p < 0.001; SEM = 0.22 mm; MDC₅₅ = 0.61 mm), confirming that the sonographer's technique was highly reproducible across sessions."
Discussion Section
Interpret the ICC in clinical context. A high ICC does not automatically mean the measurement is fit for purpose: compare the MDC to the expected clinical treatment effect. If the MDC exceeds the expected effect of your intervention, the instrument cannot detect meaningful change — a fundamental threat to the study's validity that must be acknowledged.
Common Mistakes Researchers Make
Mistake 1: Using Pearson's r to Assess Measurement Reliability
Pearson's r detects whether two variables move together but is completely insensitive to systematic bias between raters. Two raters with a constant 15-unit difference produce r = 1.00 but ICC much less than 1. Using Pearson's r to "establish reliability" will almost always overestimate how well raters agree in absolute terms.
Fix: Always use ICC for continuous reliability studies. Report Pearson's r only if you specifically want to describe the correlation, and add ICC to describe agreement. They measure different things and both may be informative.
Mistake 2: Selecting the ICC Model After Seeing Results
ICC(2,1) (absolute agreement) and ICC(3,1) (consistency) can differ substantially when raters have systematic biases. Researchers sometimes report ICC(3,1) because it yields a higher value, without justifying the model choice. This is a form of outcome reporting bias and will be caught by experienced reviewers.
Fix: Pre-specify the ICC model in your study protocol before data collection. Justify the choice based on study design (are raters fixed or random? Does systematic bias matter clinically?). Report the model used with a clear rationale in the Methods section.
Mistake 3: Reporting ICC Without the 95% Confidence Interval
An ICC point estimate without a CI can be profoundly misleading in small samples. ICC = 0.80 from 20 subjects typically has a 95% CI of roughly 0.55–0.92 — the lower bound only reaches "good" reliability. Without the CI, readers cannot assess the precision of the estimate.
Fix: Always compute and report the 95% CI for ICC. In SPSS, the CI is reported automatically in the Reliability Analysis output. In R: icc() from the irr package returns the CI. Discuss the clinical implications of the lower bound of the CI.
Mistake 4: Failing to Report SEM and MDC
ICC is a dimensionless ratio that tells you about relative reliability in your sample. It does not tell clinicians how large the measurement error is in the actual units they care about (mmHg, degrees, mm). Without SEM and MDC, the reliability study cannot be used to plan a treatment study or interpret change scores.
Fix: Always compute and report SEM (= SD_pooled × √(1 − ICC)) and MDC₅₅ (= SEM × 1.96 × √2). Interpret the MDC clinically: "A change of X units or more is needed to be confident that true change has occurred beyond measurement noise."
Mistake 5: Using Homogeneous Subjects in a Reliability Study
ICC is the ratio of between-subject variance to total variance. If all subjects in the reliability study have very similar values (e.g., only healthy volunteers with similar blood pressure), between-subject variance is artificially low, and even excellent measurement precision produces a misleadingly low ICC. Conversely, a highly heterogeneous sample inflates ICC.
Fix: Select subjects for reliability studies who represent the full spectrum of the clinical population you intend to measure — including patients with low, medium, and high values on the target variable. Document the range of measurements in the sample and discuss whether the ICC generalises to other populations.
Mistake 6: Applying ICC to Categorical Data
ICC assumes continuous, interval-level data with meaningful numeric differences between values. Applying ICC to nominal categories (positive/negative, blood type, histological grade) violates the measurement assumption and produces a meaningless coefficient.
Fix: Use Cohen's Kappa for binary or nominal categorical outcomes. Use weighted kappa for ordered categorical outcomes with ≤5 levels. Use ICC when you have a genuinely continuous measurement scale (mmHg, degrees, cm, g/dL).
Mistake 7: Interpreting ICC Without Citing the Scale Used
The same ICC value of 0.62 is "good" by Cicchetti (1994) and "moderate" by Koo and Li (2016). Without stating which scale was used, readers cannot interpret the verbal label or compare results across studies. Many papers simply write "ICC = 0.72 indicating good reliability" without any citation.
Fix: Always state "using the [Author, Year] benchmarks" when labelling the ICC. Cite the interpretation scale in both the Methods and Results sections. Pre-specify the minimum acceptable ICC threshold in the protocol.
Scientific Paper Reporting Standards
The COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) initiative has produced the most comprehensive reporting guidelines for reliability studies in clinical research. Key elements required by COSMIN and leading clinical journals:
- Study design: Number of raters, number of subjects, measurement occasions, washout interval (for intra-rater), setting, and blinding procedures
- ICC model: Specify the exact model (e.g., "ICC(2,1), two-way random effects, absolute agreement, single measures") and justify the selection
- Results: ICC point estimate, 95% CI, F-statistic and p-value, and interpretation using a named benchmark scale
- Measurement error: SEM and MDC₅₅ in original measurement units
- Supplementary: Bland-Altman plot with 95% limits of agreement, mean difference between raters, and any evidence of proportional bias (correlation between mean and difference)
APA 7th edition does not prescribe a specific reliability statistic for continuous data but requires reporting the coefficient, CI, and a referenced interpretation benchmark for any reliability index used in a study.
Practical Tips for Postgraduate Researchers
Choose Your ICC Model Before Data Collection
Read the Koo and Li (2016) decision guide and commit to a model in your protocol. Pre-registering your ICC model (on ClinicalTrials.gov or OSF) protects you from reviewers who question the model choice and demonstrates methodological rigour from the start.
Plan Sample Size Using the Lower CI Bound
Determine the minimum ICC you need for your measurement tool to be clinically acceptable (e.g., ≥ 0.75), then calculate how many subjects you need for the lower 95% CI bound to exceed that threshold. The PASS software and R's pwr.icc function support this directly.
Train Raters Before — Not During — the Reliability Study
Any training or protocol calibration must happen before the data collection period starts. If raters are learning the protocol during the study, reliability will artificially improve over time, violating the assumption of stable measurement conditions across the whole sample.
Randomise the Order of Subject Measurement
If Rater A always measures patients before Rater B, systematic order effects (patient fatigue, practice effects) will confound the inter-rater comparison. Randomise whether Rater A or B measures first for each patient, and document the randomisation in your methods.
Include the SEM in Your Power Calculation
If the reliability study is a prerequisite for a future treatment trial, use the SEM from your reliability study to adjust the expected standard deviation in your sample size calculation. Measurement error expands the effective SD and reduces power — this is a commonly overlooked source of underpowering in clinical trials.
Report Raw Data in Supplementary Materials
Include the full measurement table (each subject × each rater) in your thesis appendix or journal supplementary materials. This enables readers to compute their own ICC, verify your analysis, or incorporate your data into meta-analyses of measurement reliability.
Frequently Asked Questions
What is the Intraclass Correlation Coefficient (ICC)? +
The ICC is a reliability statistic for continuous data that measures how consistently two or more raters (or the same rater on different occasions) produce the same numeric measurement when assessing the same subjects. Unlike Pearson's r, ICC is sensitive to both correlation and systematic bias between raters. It expresses the proportion of total measurement variance that is due to genuine between-subject differences, ranging from 0 (no reliability — all variance is error) to 1 (perfect reliability — all variance reflects true subject differences).
What is the difference between inter-rater and intra-rater reliability? +
Inter-rater reliability measures agreement between two or more different observers assessing the same subjects — e.g., two radiologists measuring tumour diameter. Intra-rater reliability measures how consistently a single observer produces the same measurement when re-assessing the same subjects after a washout period — e.g., the same physiotherapist re-measuring knee flexion one week later. ICC measures both; the choice of ICC model (and the interpretation) differs slightly between the two designs.
Which ICC type should I use in my study? +
For most clinical inter-rater studies where raters represent a sample of equivalent professionals and systematic bias matters clinically, use ICC(2,1): two-way random, absolute agreement, single measures. For intra-rater test-retest studies where the rater is a fixed element and you only care about consistency of rank ordering, use ICC(3,1): two-way mixed, consistency, single measures. Use average measures versions (2,k) or (3,k) only if the clinical protocol averages across k raters. Koo and Li (2016) provide a practical decision table.
What ICC value is considered good reliability? +
Using Koo and Li (2016): ICC < 0.50 = poor; 0.50–0.75 = moderate; 0.75–0.90 = good; > 0.90 = excellent. For clinical measurements used in individual patient decisions, ICC > 0.75 is the minimum, and > 0.90 is preferred for precision instruments. Always report the 95% CI and discuss the clinical implications of the lower bound, which is the most conservative estimate of reliability.
What is the difference between ICC and Pearson's r correlation? +
Pearson's r measures whether two variables move together linearly — it detects correlation but not systematic bias. Two raters who consistently differ by 20 units show r = 1.00 despite major absolute disagreement. ICC measures true agreement, capturing both correlation and systematic offset. For reliability studies, ICC is always the correct choice. Reporting Pearson's r as a reliability measure is a well-documented methodological error that inflates the apparent agreement.
When should I use ICC vs Cohen's Kappa? +
Use ICC for continuous or ratio-level numeric data (blood pressure, tumour size, pain VAS score). Use Cohen's Kappa for categorical data — nominal or ordinal groups (positive/negative, disease grade, classification label). For ordinal data with 4+ ordered levels, quadratic-weighted kappa and ICC(2,1) give very similar values and either is acceptable. The key rule: ICC requires a meaningful numeric scale; kappa works with any discrete categories.
What is SEM and why does it matter alongside ICC? +
The Standard Error of Measurement (SEM = SD_pooled × √(1 − ICC)) expresses measurement error in the original units of the scale (mmHg, degrees, mm). The Minimal Detectable Change (MDC₉₅ = SEM × 1.96 × √2) is the minimum real-world change that exceeds measurement noise with 95% confidence. A high ICC can coexist with a large SEM if subjects are very heterogeneous. Always report SEM and MDC alongside ICC so clinicians can judge whether the instrument is precise enough to detect the treatment effects they are investigating.
How do I calculate ICC in SPSS? +
Go to Analyze → Scale → Reliability Analysis. Add all rater (or session) columns to the Items box. Click Statistics → Intraclass correlation coefficient. Choose the Model (one-way random, two-way random, or two-way mixed) and Type (absolute agreement or consistency). Click OK. SPSS outputs single-measures and average-measures ICC with 95% CIs and the F-test. For two-rater inter-rater studies with generalisation intent, select: Two-Way Random → Absolute Agreement → Single Measures = ICC(2,1).
How many subjects do I need for an ICC study? +
A minimum of 30 subjects is a commonly cited lower bound, but 50 subjects is recommended by COSMIN guidelines for patient-reported outcome measure reliability studies. Formal sample size depends on the expected ICC and desired CI width — use the kappaSize or pwr package in R. Studies with very low expected ICC or narrow CI requirements may need 80–100+ subjects to achieve acceptable precision.
How do I report ICC in a thesis or journal paper? +
Report: the ICC model used and justification, the point estimate (2 decimal places), 95% CI, F-statistic, p-value, SEM, MDC₉₅, and verbal interpretation with a cited benchmark scale. Example: "Inter-rater reliability for knee extensor force measurement was excellent (ICC(2,1) = 0.922, 95% CI: 0.884–0.949, F(44,44) = 24.8, p < 0.001; SEM = 4.2 N; MDC₉₅ = 11.7 N), using Koo and Li (2016) benchmarks." Include a Bland-Altman plot in the supplementary materials.
Calculate ICC Online
Use StatClinic's ICC calculator to compute inter-rater and intra-rater reliability from your measurement data. Get ICC(2,1) and ICC(3,1) with 95% confidence intervals, SEM, and MDC — all formatted for COSMIN-compliant thesis and journal reporting.
Open ICC Calculator →