Agreement between raters for blinded scoring: Cohen's kappa, weighted kappa and the ICC

Quick answer. Match the statistic to the scores. For categories (present or absent, cell type), use Cohen’s kappa, which corrects the raw agreement for agreement expected by chance. For ordinal grades (histology 0–4), use a weighted kappa (quadratic weights), which counts a one-grade disagreement as smaller than a three-grade one. For measurements (lesion area, cell counts), use the intraclass correlation (ICC) for absolute agreement, which falls when one rater measures systematically higher. A correlation coefficient is not agreement: two raters can correlate almost perfectly while one scores consistently higher. Report the estimate with its 95% confidence interval and, for grades and measurements, the mean difference between raters, and agree a scoring guide before scoring the study.

Terms used below. Inter-rater agreement: how closely two people scoring the same samples give the same result. Cohen’s kappa: agreement on categories, corrected for chance; 1 is perfect, 0 is chance level, and a negative value is worse than chance. Weighted kappa: a kappa for ordered grades that gives partial credit to near misses; quadratic weights penalise a disagreement by the square of its size. Intraclass correlation (ICC): the share of the variation in measurements that comes from real differences between samples rather than between raters or measurement error. Absolute agreement ICC: counts systematic differences between raters as disagreement (written ICC(2,1) or ICC(A,1)). Consistency ICC: ignores a constant difference between raters (ICC(3,1) or ICC(C,1), a two-way mixed-effects model that treats these raters as the only ones of interest); use it only when the same rater will score every sample, so that an offset cannot become a difference between groups. Single rater: the ICC for one rater’s scores, the right form when each study sample will be scored by one person. Systematic difference: one rater scoring higher or lower on average. 95% confidence interval (CI): the range of values compatible with the data. Prevalence: how common each category is. Raw (exact) agreement: the share of samples given the same score by both raters. Average-measures ICC: the ICC for the mean of several raters’ scores (ICC(2,k)), the right form when study samples are scored by consensus or by averaging raters. Two-way random-effects model: an ICC model treating both samples and raters as drawn from larger populations, so the result applies to other raters like these. Interchangeable raters: raters whose scores could be swapped without changing results. Bland–Altman plot: the difference between two raters plotted against their mean, with limits of agreement (mean difference ± 1.96 standard deviations of the differences) showing the range of most disagreements. Fleiss’ kappa: a kappa for unordered categories and more than two raters.

Which statistic for your scores

Scores	Statistic	Also report
Two categories or unordered categories	Cohen’s kappa	Raw agreement; the 2 × 2 (or k × k) table
Ordered grades (0–4, mild/moderate/severe)	Weighted kappa, quadratic weights	Raw agreement; mean difference between raters
Measurements, raters interchangeable	ICC, absolute agreement, single rater (ICC(2,1))	Mean difference between raters; a Bland–Altman plot (correlation vs linear regression)
Measurements, a constant offset between raters does not matter (one rater will score every study sample)	ICC, consistency (ICC(3,1), two-way mixed effects)	State why the offset does not matter
More than two raters	Fleiss’ kappa for unordered categories; for ordinal grades and measurements, the ICC with all raters (single-rater or average form, matching how the study will be scored)	As above

Cohen described kappa as “directly interpretable as the proportion of joint judgments in which there is agreement, after chance agreement is excluded” (Cohen 1960), and generalised it to weighted kappa, which allows for “ratio-scaled degrees of disagreement (or agreement)” between categories (Cohen 1968); quadratic weights are the convention used here. Koo and Li recommend reporting the ICC form (model, type, and whether single or average measures are reported) with its 95% CI; for studies with at least 30 heterogeneous samples and at least 3 raters, they suggest reading benchmarks from the interval, where “values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 are indicative of poor, moderate, good, and excellent reliability” (Koo & Li 2016). Bland and Altman made the point about correlation: “r measures the strength of a relation between two variables, not the agreement between them” (Bland & Altman 1986).

Worked example: 40 slides scored by two raters

Constructed data: 40 slides, each given a histology grade (0–4) and a lesion-area measurement by two raters blind to group. By construction, rater B grades half a grade higher on average and measures areas 15% larger. The random seed was the first one tried; it was not selected. Stored code and data: scores.csv, make_data.py, analysis.py, analysis.R and the analysis output; analysis.py and analysis.R print the same output.

Grades (0–4)	Result
Exact agreement	47.5%
Cohen’s kappa (unweighted)	0.343 (95% CI 0.149 to 0.536, Cohen’s 1960 approximation)
Quadratic-weighted kappa	0.868 (95% CI 0.81 to 0.93, from psych::cohen.kappa; library_check.R, output)
Mean difference (B − A)	0.42 grades
Spearman correlation (not an agreement measure)	0.925

Lesion area	Result
Pearson correlation (not an agreement measure)	0.976
Mean difference (B − A)	0.298 mm²
Ratio B/A, with 95% limits of agreement (log scale)	1.146 (0.904 to 1.454)
ICC consistency, ICC(3,1)	0.974 (95% CI 0.951 to 0.986)
ICC absolute agreement, ICC(2,1)	0.951 (95% CI 0.693 to 0.984)
ICC absolute agreement, average of both raters, ICC(2,k)	0.975 (95% CI 0.818 to 0.992), from the stored cross-checks

For the grades, the raters agree exactly on fewer than half the slides, but most disagreements are by one grade, so the weighted kappa is high; the mean difference shows that most of those disagreements come from rater B grading higher. For the areas, the correlation and the consistency ICC are near 1, yet the raters are not interchangeable: rater B measures about 0.3 mm² more, and the absolute-agreement ICC’s interval reaches down to 0.69. If the two raters each scored different animals in the study, that offset would become a difference between groups. Because B’s excess grows with lesion size (about 15%), the differences are analysed as ratios on the log scale: B’s measurements were 1.146 times A’s, and 95% of slides fell between 0.90 and 1.45 times. The absolute-agreement interval is wide mainly because only two raters inform the between-rater term; Koo and Li suggest at least three raters for a reliability study. In 10,000 simulated agreement studies with the same grading model (simulation.py, the simulation output), the middle 95% of quadratic-weighted kappas ran from 0.68 to 0.93 with 20 slides, 0.74 to 0.91 with 40 and 0.78 to 0.89 with 80; for the unweighted kappa, from 0.13 to 0.67, 0.21 to 0.59 and 0.27 to 0.54.

# Install once: pip install pandas scipy scikit-learn pingouin; scores.csv in the working folder
# scikit-learn gives no CI for kappa: use R's psych::cohen.kappa (below) or the approximation in analysis.py
# pingouin's long format below takes any number of raters (one row per slide per rater)
import pandas as pd, pingouin as pg
from sklearn.metrics import cohen_kappa_score

d = pd.read_csv("scores.csv")                        # one row per slide: grade_a, grade_b, area_a, area_b
print(pd.crosstab(d.grade_a, d.grade_b), (d.grade_a == d.grade_b).mean())   # the table and raw agreement
print(cohen_kappa_score(d.grade_a, d.grade_b), cohen_kappa_score(d.grade_a, d.grade_b, weights="quadratic"))
long = d.melt(id_vars="slide", value_vars=["area_a", "area_b"], var_name="rater", value_name="area")
icc = pg.intraclass_corr(long, targets="slide", raters="rater", ratings="area")
print(icc[icc.Type.isin(["ICC(A,1)", "ICC(C,1)"])][["Type", "ICC", "CI95"]])   # A = absolute agreement, C = consistency

d <- read.csv("scores.csv")
psych::ICC(d[, c("area_a", "area_b")])$results[c("Single_random_raters", "Single_fixed_raters"), ]   # ICC2, ICC3 with CIs
psych::cohen.kappa(d[, c("grade_a", "grade_b")])     # unweighted and weighted (quadratic by default) kappa with CIs
# Bland-Altman on the log scale: r <- log(d$area_b / d$area_a); exp(mean(r) + c(-1.96, 0, 1.96) * sd(r))

Before you score the study

1. Write a scoring guide with reference images for each grade, and train raters on slides not used in the study.
2. Score blind to group, in random order (randomisation and blinding).
3. One section per animal in the agreement set. Several sections from one animal are not independent; use one per animal, or a model with animal as a level, so that the agreement is not overstated.
4. Plan the agreement study: at least 40 slides covering the whole range of grades (with 20 slides, the weighted kappa could plausibly lie anywhere from about 0.68 to 0.93 in our simulation) and three or more raters if you need a precise absolute-agreement ICC.
5. Look at the disagreements, not only the coefficient: a systematic difference can be corrected by retraining; random scatter needs a clearer guide.
6. Watch for skewed categories. When one category has high prevalence, kappa can be low despite high raw agreement, the paradox described by Feinstein and Cicchetti (Feinstein & Cicchetti 1990); report the table and raw agreement as well.
7. Treat benchmarks as rough. Landis and Koch called their kappa categories “clearly arbitrary” (Landis & Koch 1977); decide the agreement your study needs before you look.

Check what an AI assistant ran

1. A Pearson or Spearman correlation reported as agreement. Ask for kappa or the absolute-agreement ICC and the mean difference.
2. Unweighted kappa for ordinal grades. Ask for a weighted kappa as well.
3. An ICC with no form stated. Ask which model and type (agreement or consistency) were used, and whether single or average measures are reported.
4. A kappa with no confidence interval from a handful of slides. Ask for the interval.

Report it

Two raters blinded to treatment independently scored [40] slides. Agreement on histology grade (0–4) was assessed with quadratic-weighted kappa ([0.87], 95% CI [0.81] to [0.93]) and exact agreement ([47.5]%); rater B graded [0.42] grades higher on average. Agreement on lesion area was assessed with a two-way random-effects, absolute-agreement, single-rater ICC ([0.95], 95% CI [0.69] to [0.98]); rater B’s measurements were [1.15] times rater A’s (95% limits of agreement [0.90] to [1.45]). [Study samples were scored by one observer, so single-rater ICCs are reported / study samples were scored by both raters and averaged, so the average-measures ICC(2,k) is reported: [0.975], 95% CI [0.818] to [0.992].]

Replace the bracketed values with your own.

References

• Cohen J (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37–46. doi:10.1177/001316446002000104
• Cohen J (1968). Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin 70:213–220. doi:10.1037/h0026256
• Koo TK, Li MY (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine 15:155–163. doi:10.1016/j.jcm.2016.02.012
• Bland JM, Altman DG (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 327:307–310. doi:10.1016/S0140-6736(86)90837-8
• Feinstein AR, Cicchetti DV (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 43:543–549. doi:10.1016/0895-4356(90)90158-L
• Landis JR, Koch GG (1977). The measurement of observer agreement for categorical data. Biometrics 33:159–174. doi:10.2307/2529310