How does maximum likelihood estimation connect to the logistic regression cost function?

Logistic regression models the probability of a binary outcome using the sigmoid function: p(y=1|x) = σ(xᵀβ) = 1/(1+e^(-xᵀβ)). To fit β, we use maximum likelihood estimation (MLE) — finding the parameters that make the observed labels most probable under the model.

For a single example, the likelihood is p^y · (1-p)^(1-y) (this equals p when y=1 and 1-p when y=0 — a compact way to write both cases in one expression). Taking the log of the product of likelihoods across all n examples gives the log-likelihood, and negating it (since we minimise rather than maximise) yields the binary cross-entropy loss: L = -1/n Σ[yᵢ log(pᵢ) + (1-yᵢ) log(1-pᵢ)]. This is exactly the loss function scikit-learn's LogisticRegression minimises.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

model = LogisticRegression().fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]

# log_loss computes exactly the negative log-likelihood (cross-entropy)
loss = log_loss(y_test, probs)
print(f'Cross-entropy loss: {loss:.4f}')

# Manual implementation of the cross-entropy formula
def manual_log_loss(y_true, p_pred, eps=1e-15):
    p = np.clip(p_pred, eps, 1 - eps)  # avoid log(0)
    return -np.mean(y_true * np.log(p) + (1 - y_true) * np.log(1 - p))

print(manual_log_loss(y_test, probs))  # matches log_loss above

Why log instead of raw likelihood: multiplying many small probabilities together causes numerical underflow; taking the log converts the product into a sum, which is numerically stable and also turns the optimisation into a convex problem that gradient-based methods can solve reliably.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.