# Agreement between raters for blinded scoring: Cohen's kappa, weighted kappa and the ICC

> For two raters' categories use Cohen's kappa, for ordinal grades a weighted kappa, for measurements the intraclass correlation (ICC) for absolute agreement. Correlation is not agreement. Report the estimate with its confidence interval. Python and R code.

Canonical: https://www.licklider.ai/guides/inter-rater-agreement/
Section: Guides › Correlation, regression and survival
Author: Licklider · Published 2026-11-11 · Last reviewed 2026-11-11 (UTC)

> **Quick answer.** Match the statistic to the scores. For **categories** (present or absent, cell type), use **Cohen's kappa**, which corrects the raw agreement for agreement expected by chance. For **ordinal grades** (histology 0–4), use a **weighted kappa** (quadratic weights), which counts a one-grade disagreement as smaller than a three-grade one. For **measurements** (lesion area, cell counts), use the **intraclass correlation (ICC)** for **absolute agreement**, which falls when one rater measures systematically higher. **A correlation coefficient is not agreement**: two raters can correlate almost perfectly while one scores consistently higher. Report the estimate with its 95% confidence interval and, for grades and measurements, the mean difference between raters, and agree a scoring guide before scoring the study.

**Terms used below.** *Inter-rater agreement*: how closely two people scoring the same samples give the same result. *Cohen's kappa*: agreement on categories, corrected for chance; 1 is perfect, 0 is chance level, and a negative value is worse than chance. *Weighted kappa*: a kappa for ordered grades that gives partial credit to near misses; quadratic weights penalise a disagreement by the square of its size. *Intraclass correlation (ICC)*: the share of the variation in measurements that comes from real differences between samples rather than between raters or measurement error. *Absolute agreement ICC*: counts systematic differences between raters as disagreement (written ICC(2,1) or ICC(A,1)). *Consistency ICC*: ignores a constant difference between raters (ICC(3,1) or ICC(C,1), a two-way mixed-effects model that treats these raters as the only ones of interest); use it only when the same rater will score every sample, so that an offset cannot become a difference between groups. *Single rater*: the ICC for one rater's scores, the right form when each study sample will be scored by one person. *Systematic difference*: one rater scoring higher or lower on average. *95% confidence interval (CI)*: the range of values compatible with the data. *Prevalence*: how common each category is. *Raw (exact) agreement*: the share of samples given the same score by both raters. *Average-measures ICC*: the ICC for the mean of several raters' scores (ICC(2,k)), the right form when study samples are scored by consensus or by averaging raters. *Two-way random-effects model*: an ICC model treating both samples and raters as drawn from larger populations, so the result applies to other raters like these. *Interchangeable raters*: raters whose scores could be swapped without changing results. *Bland–Altman plot*: the difference between two raters plotted against their mean, with *limits of agreement* (mean difference ± 1.96 standard deviations of the differences) showing the range of most disagreements. *Fleiss' kappa*: a kappa for unordered categories and more than two raters.

## Which statistic for your scores

| Scores | Statistic | Also report |
| --- | --- | --- |
| Two categories or unordered categories | Cohen's kappa | Raw agreement; the 2 × 2 (or k × k) table |
| Ordered grades (0–4, mild/moderate/severe) | Weighted kappa, quadratic weights | Raw agreement; mean difference between raters |
| Measurements, raters interchangeable | ICC, absolute agreement, single rater (ICC(2,1)) | Mean difference between raters; a Bland–Altman plot ([correlation vs linear regression](/guides/correlation-vs-linear-regression/)) |
| Measurements, a constant offset between raters does not matter (one rater will score every study sample) | ICC, consistency (ICC(3,1), two-way mixed effects) | State why the offset does not matter |
| More than two raters | Fleiss' kappa for unordered categories; for ordinal grades and measurements, the ICC with all raters (single-rater or average form, matching how the study will be scored) | As above |

Cohen described kappa as "directly interpretable as the proportion of joint judgments in which there is agreement, after chance agreement is excluded" ([Cohen 1960](https://doi.org/10.1177/001316446002000104)), and generalised it to weighted kappa, which allows for "ratio-scaled degrees of disagreement (or agreement)" between categories ([Cohen 1968](https://doi.org/10.1037/h0026256)); quadratic weights are the convention used here. Koo and Li recommend reporting the ICC form (model, type, and whether single or average measures are reported) with its 95% CI; for studies with at least 30 heterogeneous samples and at least 3 raters, they suggest reading benchmarks from the interval, where "values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 are indicative of poor, moderate, good, and excellent reliability" ([Koo & Li 2016](https://doi.org/10.1016/j.jcm.2016.02.012)). Bland and Altman made the point about correlation: "r measures the strength of a relation between two variables, not the agreement between them" ([Bland & Altman 1986](https://doi.org/10.1016/S0140-6736(86)90837-8)).

## Worked example: 40 slides scored by two raters

Constructed data: 40 slides, each given a histology grade (0–4) and a lesion-area measurement by two raters blind to group. By construction, rater B grades half a grade higher on average and measures areas 15% larger. The random seed was the first one tried; it was not selected. Stored code and data: [scores.csv](/guides/inter-rater-agreement/scores.csv), [make_data.py](/guides/inter-rater-agreement/make_data.py), [analysis.py](/guides/inter-rater-agreement/analysis.py), [analysis.R](/guides/inter-rater-agreement/analysis.R) and [the analysis output](/guides/inter-rater-agreement/analysis_output.txt); analysis.py and analysis.R print the same output.

| Grades (0–4) | Result |
| --- | --- |
| Exact agreement | 47.5% |
| Cohen's kappa (unweighted) | 0.343 (95% CI 0.149 to 0.536, Cohen's 1960 approximation) |
| **Quadratic-weighted kappa** | **0.868** (95% CI 0.81 to 0.93, from `psych::cohen.kappa`; [library_check.R](/guides/inter-rater-agreement/library_check.R), [output](/guides/inter-rater-agreement/library_check_R_output.txt)) |
| Mean difference (B − A) | 0.42 grades |
| Spearman correlation (not an agreement measure) | 0.925 |

| Lesion area | Result |
| --- | --- |
| Pearson correlation (not an agreement measure) | 0.976 |
| Mean difference (B − A) | 0.298 mm² |
| Ratio B/A, with 95% limits of agreement (log scale) | 1.146 (0.904 to 1.454) |
| ICC consistency, ICC(3,1) | 0.974 (95% CI 0.951 to 0.986) |
| **ICC absolute agreement, ICC(2,1)** | **0.951 (95% CI 0.693 to 0.984)** |
| ICC absolute agreement, average of both raters, ICC(2,k) | 0.975 (95% CI 0.818 to 0.992), from the stored cross-checks |

For the grades, the raters agree exactly on fewer than half the slides, but most disagreements are by one grade, so the weighted kappa is high; the mean difference shows that most of those disagreements come from rater B grading higher. For the areas, the correlation and the consistency ICC are near 1, yet the raters are not interchangeable: rater B measures about 0.3 mm² more, and the absolute-agreement ICC's interval reaches down to 0.69. If the two raters each scored different animals in the study, that offset would become a difference between groups. Because B's excess grows with lesion size (about 15%), the differences are analysed as ratios on the log scale: B's measurements were 1.146 times A's, and 95% of slides fell between 0.90 and 1.45 times. The absolute-agreement interval is wide mainly because only two raters inform the between-rater term; Koo and Li suggest at least three raters for a reliability study. In 10,000 simulated agreement studies with the same grading model ([simulation.py](/guides/inter-rater-agreement/simulation.py), [the simulation output](/guides/inter-rater-agreement/simulation_output.txt)), the middle 95% of quadratic-weighted kappas ran from 0.68 to 0.93 with 20 slides, 0.74 to 0.91 with 40 and 0.78 to 0.89 with 80; for the unweighted kappa, from 0.13 to 0.67, 0.21 to 0.59 and 0.27 to 0.54.

```python
# Install once: pip install pandas scipy scikit-learn pingouin; scores.csv in the working folder
# scikit-learn gives no CI for kappa: use R's psych::cohen.kappa (below) or the approximation in analysis.py
# pingouin's long format below takes any number of raters (one row per slide per rater)
import pandas as pd, pingouin as pg
from sklearn.metrics import cohen_kappa_score

d = pd.read_csv("scores.csv")                        # one row per slide: grade_a, grade_b, area_a, area_b
print(pd.crosstab(d.grade_a, d.grade_b), (d.grade_a == d.grade_b).mean())   # the table and raw agreement
print(cohen_kappa_score(d.grade_a, d.grade_b), cohen_kappa_score(d.grade_a, d.grade_b, weights="quadratic"))
long = d.melt(id_vars="slide", value_vars=["area_a", "area_b"], var_name="rater", value_name="area")
icc = pg.intraclass_corr(long, targets="slide", raters="rater", ratings="area")
print(icc[icc.Type.isin(["ICC(A,1)", "ICC(C,1)"])][["Type", "ICC", "CI95"]])   # A = absolute agreement, C = consistency
```

```r
d <- read.csv("scores.csv")
psych::ICC(d[, c("area_a", "area_b")])$results[c("Single_random_raters", "Single_fixed_raters"), ]   # ICC2, ICC3 with CIs
psych::cohen.kappa(d[, c("grade_a", "grade_b")])     # unweighted and weighted (quadratic by default) kappa with CIs
# Bland-Altman on the log scale: r <- log(d$area_b / d$area_a); exp(mean(r) + c(-1.96, 0, 1.96) * sd(r))
```

## Before you score the study

1. **Write a scoring guide** with reference images for each grade, and train raters on slides not used in the study.
2. **Score blind to group**, in random order ([randomisation and blinding](/guides/randomisation-and-blinding-in-animal-studies/)).
3. **One section per animal in the agreement set.** Several sections from one animal are not independent; use one per animal, or a model with animal as a level, so that the agreement is not overstated.
4. **Plan the agreement study**: at least 40 slides covering the whole range of grades (with 20 slides, the weighted kappa could plausibly lie anywhere from about 0.68 to 0.93 in our simulation) and three or more raters if you need a precise absolute-agreement ICC.
5. **Look at the disagreements**, not only the coefficient: a systematic difference can be corrected by retraining; random scatter needs a clearer guide.
6. **Watch for skewed categories.** When one category has high prevalence, kappa can be low despite high raw agreement, the paradox described by Feinstein and Cicchetti ([Feinstein & Cicchetti 1990](https://doi.org/10.1016/0895-4356(90)90158-L)); report the table and raw agreement as well.
7. **Treat benchmarks as rough.** Landis and Koch called their kappa categories "clearly arbitrary" ([Landis & Koch 1977](https://doi.org/10.2307/2529310)); decide the agreement your study needs before you look.

## Check what an AI assistant ran

1. **A Pearson or Spearman correlation reported as agreement.** Ask for kappa or the absolute-agreement ICC and the mean difference.
2. **Unweighted kappa for ordinal grades.** Ask for a weighted kappa as well.
3. **An ICC with no form stated.** Ask which model and type (agreement or consistency) were used, and whether single or average measures are reported.
4. **A kappa with no confidence interval** from a handful of slides. Ask for the interval.

## Report it

> Two raters blinded to treatment independently scored [40] slides. Agreement on histology grade (0–4) was assessed with quadratic-weighted kappa ([0.87], 95% CI [0.81] to [0.93]) and exact agreement ([47.5]%); rater B graded [0.42] grades higher on average. Agreement on lesion area was assessed with a two-way random-effects, absolute-agreement, single-rater ICC ([0.95], 95% CI [0.69] to [0.98]); rater B's measurements were [1.15] times rater A's (95% limits of agreement [0.90] to [1.45]). [Study samples were scored by one observer, so single-rater ICCs are reported / study samples were scored by both raters and averaged, so the average-measures ICC(2,k) is reported: [0.975], 95% CI [0.818] to [0.992].]

Replace the bracketed values with your own.

## References

- Cohen J (1960). A coefficient of agreement for nominal scales. *Educational and Psychological Measurement* 20:37–46. [doi:10.1177/001316446002000104](https://doi.org/10.1177/001316446002000104)
- Cohen J (1968). Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. *Psychological Bulletin* 70:213–220. [doi:10.1037/h0026256](https://doi.org/10.1037/h0026256)
- Koo TK, Li MY (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. *Journal of Chiropractic Medicine* 15:155–163. [doi:10.1016/j.jcm.2016.02.012](https://doi.org/10.1016/j.jcm.2016.02.012)
- Bland JM, Altman DG (1986). Statistical methods for assessing agreement between two methods of clinical measurement. *The Lancet* 327:307–310. [doi:10.1016/S0140-6736(86)90837-8](https://doi.org/10.1016/S0140-6736(86)90837-8)
- Feinstein AR, Cicchetti DV (1990). High agreement but low kappa: I. The problems of two paradoxes. *Journal of Clinical Epidemiology* 43:543–549. [doi:10.1016/0895-4356(90)90158-L](https://doi.org/10.1016/0895-4356(90)90158-L)
- Landis JR, Koch GG (1977). The measurement of observer agreement for categorical data. *Biometrics* 33:159–174. [doi:10.2307/2529310](https://doi.org/10.2307/2529310)

## Related

- [Pearson vs Spearman](https://www.licklider.ai/guides/pearson-vs-spearman-correlation/)
- [Correlation vs linear regression](https://www.licklider.ai/guides/correlation-vs-linear-regression/)
- [Randomisation and blinding](https://www.licklider.ai/guides/randomisation-and-blinding-in-animal-studies/)
- [Clinical scores over time](https://www.licklider.ai/guides/clinical-scores-over-time/)

## How this page was made

- Written and published by Licklider. Recommendations cite peer-reviewed methods papers listed under References; cited passages were checked against the source text.
- The worked example uses constructed data. Its numbers were produced by stored code (Python 3.13 with SciPy 1.18.1; R 4.3.3, base R), and the two languages print identical output; stored cross-checks with scikit-learn 1.9.1, pingouin 0.7.0 and psych 2.4.1 give the same values and supply the weighted kappa's interval; the simulation was run in Python.
- Separate review passes checked the statistics, the citations and readability before publication.