What does it mean for a classifier's predicted probabilities to be 'well-calibrated', and why don't all models produce calibrated probabilities naturally?

A classifier is well-calibrated if, among all the examples it assigns a predicted probability of (say) 0.7 to belonging to the positive class, approximately 70% of them actually are positive. Mathematically, calibration requires P(y=1 | p̂(x)=p) ≈ p for all probability values p the model outputs. This is a stronger requirement than just having good ranking ability (which is what AUC measures) — a model can perfectly rank examples (always score true positives higher than true negatives) while being badly calibrated (e.g., consistently outputting 0.9 for examples that are only 60% likely to be positive).

Models trained by directly optimising a proper probabilistic loss (like logistic regression's cross-entropy) tend to be naturally well-calibrated, because the loss function itself rewards accurate probability estimates, not just correct rankings. Models like SVMs (which optimise margin, not probability) or unregularised tree ensembles can produce poorly calibrated scores even when their predictions and rankings are good, because their training objective never explicitly targets calibrated probability output.

from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression

svm = SVC(probability=True).fit(X_train, y_train)
logreg = LogisticRegression().fit(X_train, y_train)

for name, model in [('SVM', svm), ('LogReg', logreg)]:
    probs = model.predict_proba(X_test)[:, 1]
    frac_pos, mean_pred = calibration_curve(y_test, probs, n_bins=10)
    # Well-calibrated: frac_pos should closely track mean_pred
    print(f'{name}: predicted vs actual', list(zip(mean_pred, frac_pos)))

# Fix poor calibration with a calibration wrapper
calibrated_svm = CalibratedClassifierCV(svm, method='isotonic', cv=5)
calibrated_svm.fit(X_train, y_train)
calibrated_probs = calibrated_svm.predict_proba(X_test)[:, 1]

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.