What does the ROC-AUC score mathematically represent, and why is it threshold-independent?

The ROC (Receiver Operating Characteristic) curve plots the True Positive Rate (TPR/recall) against the False Positive Rate (FPR) as the classification decision threshold is varied from 0 to 1. The Area Under this Curve (AUC) has an elegant probabilistic interpretation: AUC equals the probability that a randomly chosen positive example receives a higher predicted score than a randomly chosen negative example: AUC = P(score(positive) > score(negative)).

This is why AUC is threshold-independent — it measures the model's ability to rank positive examples above negative examples across all possible thresholds simultaneously, rather than evaluating performance at one specific cutoff. A perfect classifier achieves AUC=1.0 (every positive ranked above every negative); random guessing achieves AUC=0.5 (equivalent to a coin flip ranking).

from sklearn.metrics import roc_auc_score, roc_curve
import numpy as np

y_true  = np.array([0, 0, 1, 1, 1, 0, 1])
y_score = np.array([0.1, 0.4, 0.35, 0.8, 0.65, 0.2, 0.9])

auc = roc_auc_score(y_true, y_score)
print(f'AUC: {auc:.3f}')

# Manual verification: count concordant pairs (Mann-Whitney U statistic)
pos_scores = y_score[y_true == 1]
neg_scores = y_score[y_true == 0]
concordant = sum(p > n for p in pos_scores for n in neg_scores)
total_pairs = len(pos_scores) * len(neg_scores)
print(f'Manual AUC: {concordant / total_pairs:.3f}')  # matches roc_auc_score

fpr, tpr, thresholds = roc_curve(y_true, y_score)
# Each point on the curve corresponds to a different threshold

Caution with imbalanced classes: AUC can be misleadingly optimistic on highly imbalanced datasets because the False Positive Rate denominator (total negatives) is large, making even a meaningful number of false positives look small. In such cases, Precision-Recall AUC is usually more informative.