Python / Python Mathematical Intuition and Scikit Learn Interview Questions
Why do you need to scale features before using gradient descent-based models or distance-based algorithms like KNN?
Feature scaling matters for two distinct mathematical reasons depending on the algorithm family. For gradient-based optimisation (logistic regression, SVM, neural networks), features with very different scales create an elongated, elliptical loss surface. Gradient descent then zig-zags inefficiently across the narrow dimension instead of taking a direct path to the minimum, slowing convergence dramatically. Scaling features to similar ranges makes the loss surface more circular, so gradient descent converges in far fewer iterations.
For distance-based algorithms (KNN, K-Means, SVM with RBF kernel), the Euclidean distance formula √Σ(xᵢ - yᵢ)² is dominated by whichever feature has the largest numeric range. A feature measured in thousands (like income) would completely swamp a feature measured in single digits (like age) when computing distances, even if age is more predictive.
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
# Without scaling: income (range ~50000) dominates age (range ~80)
X = [[25, 45000], [30, 52000], [45, 110000]]
# StandardScaler: (x - mean) / std -> mean 0, unit variance
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Always fit scaler on train, transform both train and test
pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
pipe.fit(X_train, y_train)
# MinMaxScaler: rescales to [0, 1] range — sensitive to outliers
mm = MinMaxScaler()
X_mm = mm.fit_transform(X)| Algorithm family | Sensitive to scale? | Why |
|---|---|---|
| Linear/Logistic Regression (gradient descent) | Yes | Elongated loss surface slows convergence |
| KNN, K-Means, SVM (RBF) | Yes | Distance metric dominated by large-range features |
| Decision Trees, Random Forest | No | Splits are based on feature thresholds, not magnitudes |
| Gradient Boosting (tree-based) | No | Same reason — split-based, scale-invariant |
Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!
Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.
Invest now!!! Get Free equity stock (US, UK only)!
Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.
The Robinhood app makes it easy to trade stocks, crypto and more.
Webull! Receive free stock by signing up using the link: Webull signup.
More Related questions...
