Python / Python Mathematical Intuition and Scikit Learn Interview Questions
1. Why does linear regression minimise the sum of squared errors instead of, say, absolute errors?
Ordinary least squares (OLS) minimises squared residuals for both mathematical and statistical reasons. Mathematically, the squared error loss L(β) = Σ(yᵢ - Xᵢβ)² is smooth and differentiable everywhere, so its minimum can be found analytically by setting the gradient to zero — this gives the clo...
2. Explain the mathematical intuition behind gradient descent and why the learning rate matters.
Gradient descent is an iterative optimisation algorithm that finds a local minimum of a differentiable function by repeatedly stepping in the direction of steepest descent — the negative gradient. The gradient ∇L(θ) points in the direction of steepest increase , so subtracting a scaled version of...
3. Why do you need to scale features before using gradient descent-based models or distance-based algorithms like KNN?
Feature scaling matters for two distinct mathematical reasons depending on the algorithm family. For gradient-based optimisation (logistic regression, SVM, neural networks), features with very different scales create an elongated, elliptical loss surface. Gradient descent then zig-zags inefficien...
4. Explain the bias-variance tradeoff mathematically and how it relates to model complexity.
The expected test error of a model can be decomposed into three components: E[(y - f̂(x))²] = Bias(f̂(x))² + Var(f̂(x)) + σ² , where σ² is irreducible noise. Bias measures how far the average prediction (over many training sets) is from the true function — high bias means the model is too simple ...
5. What is the mathematical difference between L1 (Lasso) and L2 (Ridge) regularization, and why does L1 produce sparse solutions?
Both methods add a penalty term to the loss function to discourage large coefficients. Ridge (L2) adds λΣβᵢ² ; Lasso (L1) adds λΣ|βᵢ| . The mathematical consequence of this difference is profound: L1 regularization can drive coefficients to exactly zero (producing sparse, interpretable models wit...
6. How does maximum likelihood estimation connect to the logistic regression cost function?
Logistic regression models the probability of a binary outcome using the sigmoid function: p(y=1|x) = σ(xᵀβ) = 1/(1+e^(-xᵀβ)) . To fit β, we use maximum likelihood estimation (MLE) — finding the parameters that make the observed labels most probable under the model. For a single example, the like...
7. How do decision trees decide which feature and threshold to split on? Explain Gini impurity and entropy.
At each node, a decision tree evaluates every feature and every possible threshold, and selects the split that produces the greatest reduction in impurity between the parent node and the weighted average impurity of the two child nodes. Two common impurity measures are Gini impurity and entropy ....
8. Why does a random forest reduce variance compared to a single decision tree, and what role does feature randomness play?
A random forest builds many decision trees, each trained on a bootstrap sample of the data (bagging), and averages their predictions. The variance of the average of n independent, identically distributed random variables each with variance σ² is σ²/n — averaging reduces variance proportionally to...
9. What is the mathematical intuition behind gradient boosting? How does it differ from random forests?
Gradient boosting builds an ensemble of weak learners (typically shallow decision trees) sequentially , where each new tree is trained to predict the negative gradient of the loss function with respect to the current ensemble's predictions — essentially, each tree learns to correct the errors (re...
10. Explain the mathematical foundation of PCA. What do eigenvectors and eigenvalues represent in this context?
Principal Component Analysis (PCA) finds a new orthogonal coordinate system where the axes (principal components) are ordered by the amount of variance they capture in the data. Mathematically, PCA computes the eigenvectors and eigenvalues of the data's covariance matrix Σ = (1/n)XᵀX (after cente...
11. What is the mathematical concept of the margin in Support Vector Machines, and why does maximizing it improve generalization?
A linear SVM finds the hyperplane wᵀx + b = 0 that separates two classes while maximising the margin — the distance between the hyperplane and the nearest points from either class. This distance for a normalised hyperplane is 2/‖w‖ , so maximising the margin is equivalent to minimising ‖w‖² subje...
12. What is the kernel trick in SVMs and why does it avoid explicitly computing high-dimensional feature mappings?
Many datasets are not linearly separable in their original feature space but become separable after mapping to a higher-dimensional space via some function φ(x) . Computing this mapping explicitly (especially for infinite-dimensional mappings like the RBF kernel implies) would be computationally ...
13. Why does K-Nearest Neighbors suffer from the curse of dimensionality, mathematically?
KNN relies on the assumption that nearby points in feature space share similar labels — its entire predictive power comes from local neighborhoods being meaningful. As the number of dimensions d increases, two related mathematical phenomena destroy this assumption. First, the volume of a hypersph...
14. What is the mathematical objective function K-Means optimises, and why can it converge to a local minimum?
K-Means seeks to partition n points into k clusters by minimising the within-cluster sum of squares (WCSS) , also called inertia: J = Σₖ Σ_{x ∈ Cₖ} ‖x - μₖ‖² , where μₖ is the centroid (mean) of cluster k. This is a non-convex combinatorial optimisation problem — finding the global optimum requir...
15. What is the statistical rationale behind k-fold cross-validation, and why are k=5 or k=10 commonly used?
Cross-validation estimates how well a model generalises to unseen data by repeatedly splitting the training data into a training fold and a validation fold, training on the former and evaluating on the latter, then averaging the results. K-fold CV divides data into k equal partitions, using each ...
16. What does the ROC-AUC score mathematically represent, and why is it threshold-independent?
The ROC (Receiver Operating Characteristic) curve plots the True Positive Rate (TPR/recall) against the False Positive Rate (FPR) as the classification decision threshold is varied from 0 to 1. The Area Under this Curve (AUC) has an elegant probabilistic interpretation: AUC equals the probability...
17. Explain the mathematical tradeoff between precision and recall, and why F1 score is the harmonic mean rather than the arithmetic mean.
Precision is TP/(TP+FP) — of everything predicted positive, what fraction was actually positive. Recall is TP/(TP+FN) — of everything that was actually positive, what fraction did the model find. Adjusting the classification threshold creates an inherent tradeoff: lowering the threshold to captur...
18. What is the 'naive' independence assumption in Naive Bayes, and why does it still work well in practice despite being unrealistic?
Naive Bayes applies Bayes' theorem to classify: P(y|x₁,...,xₙ) ∝ P(y)·P(x₁,...,xₙ|y) . Computing the joint likelihood P(x₁,...,xₙ|y) exactly would require modelling all interactions between features — infeasible with limited data. The 'naive' simplification assumes all features are conditionally ...
19. Why is a log transformation commonly applied to skewed numerical features before modeling, mathematically?
Many real-world quantities — income, population, word frequencies, prices — follow a right-skewed (long right tail) distribution, often approximately log-normal. The mathematical property of the logarithm that makes it useful here is that it compresses large values much more than small ones: log(...
20. What is multicollinearity, mathematically, and how does the Variance Inflation Factor (VIF) detect it?
Multicollinearity occurs when two or more predictor features are highly linearly correlated with each other. Mathematically, this means the design matrix X approaches rank deficiency — the columns become nearly linearly dependent, causing XᵀX to become nearly singular (its determinant approaches ...
21. Why must features be standardized before applying Ridge or Lasso regularization, mathematically?
Ridge and Lasso add a penalty proportional to coefficient magnitude — λΣβⱼ² or λΣ|βⱼ| respectively. The magnitude of a coefficient βⱼ is inversely related to the scale of its corresponding feature: if feature j is measured in millions (e.g. company revenue) its coefficient will naturally be tiny,...
22. What is the mathematical relationship between learning_rate and n_estimators in gradient boosting?
In gradient boosting, the final ensemble prediction is F(x) = F₀(x) + η · Σₘ hₘ(x) , where η is the learning rate (also called shrinkage) and the sum runs over n_estimators trees. The learning rate scales down the contribution of each individual tree. A smaller η means each tree contributes less ...
23. How does the softmax function generalize logistic regression to multiclass classification, mathematically?
Binary logistic regression uses the sigmoid function to convert a single linear score into a probability between 0 and 1. For k classes, the softmax function generalises this: given k linear scores (logits) z₁,...,zₖ , softmax computes p_i = e^{z_i} / Σⱼ e^{z_j} for each class i. This produces a ...
24. Why does fitting a scaler or transformer on the entire dataset (before train/test split) cause data leakage, mathematically?
Data leakage occurs when information from outside the training set improperly influences the model. If you fit a StandardScaler on the full dataset before splitting, the computed mean and standard deviation incorporate statistics from the test set. The scaled training data therefore implicitly co...
25. How does the class_weight parameter mathematically address class imbalance in scikit-learn classifiers?
When classes are imbalanced (e.g. 95% negative, 5% positive), a model trained with the standard loss function will naturally lean toward predicting the majority class, since doing so minimises average loss across the imbalanced training set even while completely ignoring the minority class. The c...
26. Why does using simple label encoding (integers) for nominal categorical features mislead most machine learning models, mathematically?
Label encoding assigns each category an arbitrary integer: e.g. Red=0, Green=1, Blue=2. The problem is that most models — linear regression, logistic regression, distance-based methods, and even many tree splitting algorithms that treat features as ordered — implicitly assume numeric features hav...
27. What is the difference between a single train/validation/test split and k-fold cross-validation for hyperparameter tuning, statistically?
A single validation split estimates a hyperparameter's performance using just one specific subset of data — this estimate has high variance because it depends entirely on which particular samples happened to land in the validation fold. If you tune hyperparameters against this single estimate, yo...
28. Why is PCA sensitive to feature scaling while decision tree feature importance is not, mathematically?
PCA's objective is to find directions of maximum variance in the data, computed from the covariance matrix. Variance is measured in squared units of the original feature, so a feature measured in large units (e.g. salary in dollars, variance in the millions) will dominate the covariance matrix an...
29. Why is the decision boundary of standard logistic regression always a straight line (or hyperplane), mathematically?
Logistic regression predicts class 1 when p(y=1|x) = σ(wᵀx + b) ≥ 0.5 . Since the sigmoid function σ is monotonically increasing and equals exactly 0.5 when its input is 0, this condition simplifies to wᵀx + b ≥ 0 — a linear inequality in x. The boundary where the model is exactly undecided (p=0....
30. Why can R-squared be a misleading metric for model comparison, and how does adjusted R-squared address this?
R-squared is defined as R² = 1 - SS_res/SS_tot , where SS_res is the sum of squared residuals and SS_tot is the total sum of squares (variance of y). It represents the proportion of variance in y explained by the model. The mathematical issue is that R² is monotonically non-decreasing as you add ...
31. Derive mathematically why bagging (bootstrap aggregating) reduces variance, and under what condition it does NOT help.
Suppose you have B independent models, each with the same variance σ² and the same expected prediction (no bias change from averaging). If the predictions were truly independent, the variance of their average would be Var(average) = σ²/B — variance shrinks proportionally to the number of models, ...
32. Why does convexity of the loss function matter for optimization algorithms like gradient descent, mathematically?
A function is convex if a line segment connecting any two points on its graph lies above (or on) the graph itself — equivalently, its second derivative (or Hessian, in multiple dimensions) is non-negative everywhere. The critical property of a convex function is that any local minimum is also the...
33. Mathematically, why does RobustScaler handle outliers better than StandardScaler?
StandardScaler transforms features using the mean and standard deviation: z = (x - μ)/σ . Both the mean and standard deviation are heavily influenced by extreme values — a single huge outlier can shift the mean substantially and dramatically inflate the standard deviation (since it involves squar...
34. What does it mean for a classifier's predicted probabilities to be 'well-calibrated', and why don't all models produce calibrated probabilities naturally?
A classifier is well-calibrated if, among all the examples it assigns a predicted probability of (say) 0.7 to belonging to the positive class, approximately 70% of them actually are positive. Mathematically, calibration requires P(y=1 | p̂(x)=p) ≈ p for all probability values p the model outputs....
35. Mathematically, why does stochastic gradient descent (SGD) scale to large datasets better than batch gradient descent?
Batch gradient descent computes the exact gradient of the loss using all n training examples before taking a single parameter update step: ∇L(θ) = (1/n)Σᵢ ∇Lᵢ(θ) . This requires O(n) computation per update — for datasets with millions of examples, even one update step becomes expensive, and you t...
36. Beyond scaling, why must feature selection methods also be included inside a cross-validation pipeline rather than applied beforehand?
Feature selection methods like SelectKBest choose features based on a statistical test (e.g. ANOVA F-value, mutual information) computed between each feature and the target across the available data. If you perform feature selection on the entire dataset before cross-validation, the selected feat...
