ROC curve and AUC for a biomarker: what the AUC means, its confidence interval, and choosing a cut-off

Quick answer. The AUC (area under the ROC curve) is the chance that a randomly chosen diseased animal has a higher marker value than a randomly chosen healthy one: 0.5 is no better than a coin, and 1.0 is perfect separation. Report it with a 95% confidence interval and the numbers of diseased and healthy animals (terms are defined in the box below); with 15 of each, the interval is wide, and in our simulation the usual DeLong interval was even slightly too narrow. A cut-off chosen on the same data (for example by the Youden index) looks better there than it will in new samples, so report its sensitivity and specificity as provisional or check them in a separate validation set; a study of 15 animals per group is too small to recommend a cut-off for use (in our simulation, sensitivity + specificity at such a cut-off fell from 1.48 to 1.31 in new data). To compare two markers measured on the same animals, use DeLong’s test for correlated ROC curves.

Terms used below. ROC curve: sensitivity plotted against 1 − specificity for every possible cut-off. Sensitivity: the share of diseased animals at or above the cut-off. Specificity: the share of healthy animals below it. Cut-off: the marker value at or above which you call an animal positive (this page assumes higher values indicate disease; reverse the inequalities for markers that fall with disease). Youden index: sensitivity + specificity − 1, used to pick the cut-off where it is largest. DeLong method: a way to compute the AUC’s standard error and to compare correlated AUCs (DeLong et al. 1988). 95% confidence interval (CI): the range of values compatible with the data. Coverage: how often a 95% interval contains the true value in repeated studies. Bootstrap interval: an interval from resampling the animals many times and recomputing the AUC. Reference standard: the method that decides which animals truly have the disease, such as histology. Standard error: the uncertainty of an estimate, used to build its confidence interval. Ties: equal marker values in a diseased and a healthy animal; each such pair counts as half. Power: the chance of detecting a difference that is really there. Cross-validation: choosing the cut-off on part of the data and testing it on the rest, repeated over different splits.

What the AUC is, and what to report

Hanley and McNeil showed that the area under the ROC curve “represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject” (Hanley & McNeil 1982). That makes the AUC the same quantity as the Mann–Whitney statistic divided by the number of diseased–healthy pairs; a Mann–Whitney test comparing the two groups asks whether the AUC differs from 0.5, so the test and the AUC lead to the same conclusion.

Your question	Report	Method
How well does one marker separate diseased animals from healthy ones?	AUC with 95% CI; n diseased and n healthy; the ROC curve	DeLong CI; with small samples, tied (ordinal) marker values or an AUC near 1, add a bootstrap CI (ci.auc(r, method = "bootstrap") in R)
Is marker A better than marker B, both measured on the same animals?	Difference in AUC with CI and p	DeLong’s test for correlated ROC curves
The same, on different animals	Difference in AUC with p, and each AUC with its CI (pROC’s unpaired test returns no CI for the difference)	Unpaired comparison of AUCs (in R, pROC::roc.test(r, r2, method = "delong", paired = FALSE))
What cut-off should a lab use?	Cut-off with sensitivity and specificity, from a separate validation set or cross-validation	Youden index or a cost-based choice, decided in advance
Is the study reported completely?	Follow the STARD 2015 checklist	Bossuyt et al. 2015

STARD 2015 is “an updated list of 30 essential items that should be included in every report of a diagnostic accuracy study” (Bossuyt et al. 2015); it was written for human diagnostic studies, and we apply its items concerning participants, the reference standard and cut-offs to animal biomarker studies by analogy.

Worked example: one marker in 30 mice

Constructed data: a blood marker in 15 diseased and 15 healthy mice, one measurement per mouse; by construction the true AUC is about 0.74, and the random seed of the data generator (make_data.py) was chosen so that this dataset’s AUC is close to that value. Stored code and data: marker.csv, analysis.py, analysis.R and the analysis output; analysis.py and analysis.R print the same output. The R block below also adds a bootstrap interval and the plot.

Quantity	Value
AUC (95% CI, DeLong)	0.778 (0.602 to 0.954)
Youden cut-off, chosen on these data	3.35 ng/mL
Sensitivity and specificity at that cut-off (at or above = positive), on the same data	0.87 and 0.67

The interval runs from modest to excellent separation; halving its width takes roughly four times as many mice. The cut-off and its sensitivity and specificity were chosen to look best on these very data, so they are optimistic. In 10,000 simulated studies of this design (15 diseased and 15 healthy mice, log-normal marker, true AUC 0.74), the Youden cut-off’s sensitivity + specificity averaged 1.48 on the data used to choose it but 1.31 on a fresh study of the same size; the DeLong 95% interval contained the true AUC in 91.9% of studies rather than 95%, so at this sample size it is somewhat too narrow (simulation.py, the simulation output). Leeflang and colleagues make the same point: cut-offs chosen on the same data give optimistic estimates of accuracy (Leeflang et al. 2008).

import numpy as np, pandas as pd
from scipy import stats

d = pd.read_csv("marker.csv")                        # columns: mouse, disease (0/1), marker
pos, neg = d.marker[d.disease == 1].to_numpy(), d.marker[d.disease == 0].to_numpy()
psi = (pos[:, None] > neg[None, :]) + 0.5 * (pos[:, None] == neg[None, :])
auc = psi.mean()                                      # = sklearn.metrics.roc_auc_score(d.disease, d.marker)
se = np.sqrt(psi.mean(1).var(ddof=1) / len(pos) + psi.mean(0).var(ddof=1) / len(neg))   # DeLong
print(f"AUC = {auc:.3f}, 95% CI {auc - 1.96 * se:.3f} to {min(1, auc + 1.96 * se):.3f}")   # condensed; analysis.py also finds the Youden cut-off
from sklearn.metrics import roc_curve
import matplotlib.pyplot as plt
fpr, tpr, _ = roc_curve(d.disease, d.marker)          # the ROC curve
plt.plot(fpr, tpr); plt.plot([0, 1], [0, 1], "--"); plt.xlabel("1 - specificity"); plt.ylabel("sensitivity"); plt.savefig("roc.png")

library(pROC)
d <- read.csv("marker.csv")
r <- roc(d$disease, d$marker, levels = c(0, 1), direction = "<")
ci.auc(r, method = "delong")                         # AUC with 95% CI
set.seed(1); ci.auc(r, method = "bootstrap")        # bootstrap check: 0.60 to 0.94 here
plot(r)                                              # the ROC curve
# Two markers measured on the same mice (a second ROC object, r2):
# roc.test(r, r2, method = "delong", paired = TRUE); set the same direction for both

DeLong and colleagues developed their method for “areas under correlated ROC curves” (DeLong et al. 1988), which is why it is the standard test when two markers are measured on the same animals; in R, pROC::roc.test(r, r2, method = "delong", paired = TRUE) runs it.

Before you report an AUC

1. Fix the direction in advance. Say whether higher values indicate disease. pROC’s default direction = "auto" chooses it from the group medians, which turns an AUC below 0.5 into one above 0.5 and so biases AUCs upward; set direction yourself. SciPy and scikit-learn do not flip the direction, so an AUC below 0.5 there means the marker runs the other way.
2. One value per animal. Several samples from one animal are not independent; average them or use one per animal (what counts as n).
3. Small samples. With 15 per group, the DeLong interval was too narrow in our simulation (about 92% coverage); report the interval, add a bootstrap interval as a check (for these data, 0.60 to 0.94 with 2,000 resamples), and avoid strong claims from a single small study. A DeLong comparison of two markers on 15 + 15 animals has little power; a non-significant difference does not show they are equal.
4. Cut-offs. Choose the rule for the cut-off (Youden, a fixed sensitivity, or costs of errors) before looking, and estimate its sensitivity and specificity in a separate validation set or by cross-validation; with 15 animals per group, cross-validation mainly shows how much the cut-off moves between splits. Note whether the cut-off value itself counts as positive (“at or above”).
5. The reference standard. State how disease was defined and that the reference standard was assessed without knowledge of the marker value, as STARD asks.

Check what an AI assistant ran

1. An AUC with no confidence interval or no numbers of diseased and healthy animals. Add both.
2. Direction chosen automatically (direction = "auto" in pROC is the default). Set it from your hypothesis.
3. Sensitivity and specificity at a cut-off chosen on the same data presented as the test’s accuracy. Mark them as optimistic or validate them.
4. Two markers on the same animals compared with an unpaired test. Use DeLong’s paired test.

Report it

Methods:

The marker’s ability to distinguish diseased from healthy mice was summarised by the area under the ROC curve, with a 95% confidence interval by DeLong’s method (pROC [1.18.5], R [4.3.3]; or a DeLong standard error computed in Python [3.13] with SciPy [1.18.1]); higher values were specified in advance to indicate disease. One sample per mouse was analysed (n = [15] diseased, [15] healthy). The cut-off was chosen by the Youden index; its sensitivity and specificity are reported as estimates from the same data.

Results:

The AUC was [0.78] (95% CI [0.60] to [0.95]). At a cut-off of [3.35] ng/mL (values at or above it counted as positive), chosen on these data, sensitivity was [0.87] and specificity [0.67]; these estimates are likely to be lower in new samples.

Replace the bracketed values with your own.

References

• Hanley JA, McNeil BJ (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29–36. doi:10.1148/radiology.143.1.7063747
• DeLong ER, DeLong DM, Clarke-Pearson DL (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845. doi:10.2307/2531595
• Leeflang MMG, Moons KGM, Reitsma JB, Zwinderman AH (2008). Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clinical Chemistry 54:729–737. doi:10.1373/clinchem.2007.096032
• Bossuyt PM, Reitsma JB, Bruns DE, et al. (2015). STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ 351:h5527. doi:10.1136/bmj.h5527