What does it mean for a classifier's predicted probabilities to be 'well-calibrated', and why don't all models produce calibrated probabilities naturally?

A classifier is well-calibrated if, among all the examples it assigns a predicted probability of (say) 0.7 to belonging to the positive class, approximately 70% of them actually are positive. Mathematically, calibration requires P(y=1 | p̂(x)=p) ≈ p for all probability values p the model outputs. This is a stronger requirement than just having good ranking ability (which is what AUC measures) — a model can perfectly rank examples (always score true positives higher than true negatives) while being badly calibrated (e.g., consistently outputting 0.9 for examples that are only 60% likely to be positive).

Models trained by directly optimising a proper probabilistic loss (like logistic regression's cross-entropy) tend to be naturally well-calibrated, because the loss function itself rewards accurate probability estimates, not just correct rankings. Models like SVMs (which optimise margin, not probability) or unregularised tree ensembles can produce poorly calibrated scores even when their predictions and rankings are good, because their training objective never explicitly targets calibrated probability output.

from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression

svm = SVC(probability=True).fit(X_train, y_train)
logreg = LogisticRegression().fit(X_train, y_train)

for name, model in [('SVM', svm), ('LogReg', logreg)]:
    probs = model.predict_proba(X_test)[:, 1]
    frac_pos, mean_pred = calibration_curve(y_test, probs, n_bins=10)
    # Well-calibrated: frac_pos should closely track mean_pred
    print(f'{name}: predicted vs actual', list(zip(mean_pred, frac_pos)))

# Fix poor calibration with a calibration wrapper
calibrated_svm = CalibratedClassifierCV(svm, method='isotonic', cv=5)
calibrated_svm.fit(X_train, y_train)
calibrated_probs = calibrated_svm.predict_proba(X_test)[:, 1]