Python / Python Mathematical Intuition and Scikit Learn Interview Questions
Why is the decision boundary of standard logistic regression always a straight line (or hyperplane), mathematically?
Logistic regression predicts class 1 when p(y=1|x) = σ(wᵀx + b) ≥ 0.5. Since the sigmoid function σ is monotonically increasing and equals exactly 0.5 when its input is 0, this condition simplifies to wᵀx + b ≥ 0 — a linear inequality in x. The boundary where the model is exactly undecided (p=0.5) is therefore the set of points satisfying wᵀx + b = 0, which is precisely the equation of a hyperplane (a line in 2D, a plane in 3D, and so on).
This is mathematically guaranteed regardless of how the weights w are learned — the sigmoid transformation only reshapes the probability output, it never changes the fact that the underlying decision rule depends linearly on x. To capture non-linear decision boundaries, you must either engineer non-linear features (e.g. polynomial terms x², x₁x₂) before applying logistic regression, or switch to inherently non-linear models like kernel SVMs, trees, or neural networks.
from sklearn.linear_model import LogisticRegression from sklearn.preprocessing import PolynomialFeatures from sklearn.pipeline import make_pipeline from sklearn.datasets import make_circles X, y = make_circles(n_samples=300, noise=0.1, factor=0.4) # Plain logistic regression: linear boundary, FAILS on circular data plain_logreg = LogisticRegression().fit(X, y) print('Plain accuracy:', plain_logreg.score(X, y)) # poor, ~50% # Add polynomial features to create a non-linear boundary # in the ORIGINAL space (still linear in the TRANSFORMED space) poly_logreg = make_pipeline( PolynomialFeatures(degree=2, include_bias=False), LogisticRegression() ) poly_logreg.fit(X, y) print('Polynomial accuracy:', poly_logreg.score(X, y)) # much better # The model is STILL linear in the transformed feature space # (x1, x2, x1^2, x1*x2, x2^2), but the boundary curves in original space
More Related questions...
