Python / Python Mathematical Intuition and Scikit Learn Interview Questions
1. Why does linear regression minimise the sum of squared errors instead of, say, absolute errors?
Ordinary least squares (OLS) minimises squared residuals for both mathematical and statistical reasons. Mathematically, the squared error loss L(β) = Σ(yᵢ - Xᵢβ)² is smooth and differentiable everywhere, so its minimum can be found analytically by setting the gradient to...
2. Explain the mathematical intuition behind gradient descent and why the learning rate matters.
Gradient descent is an iterative optimisation algorithm that finds a local minimum of a differentiable function by repeatedly stepping in the direction of steepest descent — the negative gradient. The gradient ∇L(θ) points in the direction of steepest increase ,...
3. Why do you need to scale features before using gradient descent-based models or distance-based algorithms like KNN?
Feature scaling matters for two distinct mathematical reasons depending on the algorithm family. For gradient-based optimisation (logistic regression, SVM, neural networks), features with very different scales create an elongated, elliptical loss surface.
4. Explain the bias-variance tradeoff mathematically and how it relates to model complexity.
The expected test error of a model can be decomposed into three components: E[(y - f̂(x))²] = Bias(f̂(x))² + Var(f̂(x)) + σ² , where σ² is irreducible noise. Bias measures how far the average prediction (over many training sets) is...
5. What is the mathematical difference between L1 (Lasso) and L2 (Ridge) regularization, and why does L1 produce sparse solutions?
Both methods add a penalty term to the loss function to discourage large coefficients. Ridge (L2) adds λΣβᵢ² ; Lasso (L1) adds λΣ|βᵢ| ....
6. How does maximum likelihood estimation connect to the logistic regression cost function?
Logistic regression models the probability of a binary outcome using the sigmoid function: p(y=1|x) = σ(xᵀβ) = 1/(1+e^(-xᵀβ)) . To fit β, we use maximum likelihood estimation (MLE) — finding the parameters that make the observed labels most probable under...
7. How do decision trees decide which feature and threshold to split on? Explain Gini impurity and entropy.
At each node, a decision tree evaluates every feature and every possible threshold, and selects the split that produces the greatest reduction in impurity between the parent node and the weighted average impurity of the two child nodes. Two common...
8. Why does a random forest reduce variance compared to a single decision tree, and what role does feature randomness play?
A random forest builds many decision trees, each trained on a bootstrap sample of the data (bagging), and averages their predictions. The variance of the average of n independent, identically distributed random variables each with variance σ² is σ²/n —...
9. What is the mathematical intuition behind gradient boosting? How does it differ from random forests?
Gradient boosting builds an ensemble of weak learners (typically shallow decision trees) sequentially , where each new tree is trained to predict the negative gradient of the loss function with respect to the current ensemble's predictions — essentially, each tree...
10. Explain the mathematical foundation of PCA. What do eigenvectors and eigenvalues represent in this context?
Principal Component Analysis (PCA) finds a new orthogonal coordinate system where the axes (principal components) are ordered by the amount of variance they capture in the data. Mathematically, PCA computes the eigenvectors and eigenvalues of the data's covariance matrix Σ...
11. What is the mathematical concept of the margin in Support Vector Machines, and why does maximizing it improve generalization?
A linear SVM finds the hyperplane wᵀx + b = 0 that separates two classes while maximising the margin — the distance between the hyperplane and the nearest points from either class. This distance for a normalised hyperplane is 2/‖w‖...
12. What is the kernel trick in SVMs and why does it avoid explicitly computing high-dimensional feature mappings?
Many datasets are not linearly separable in their original feature space but become separable after mapping to a higher-dimensional space via some function φ(x) . Computing this mapping explicitly (especially for infinite-dimensional mappings like the RBF kernel implies) would be...
13. Why does K-Nearest Neighbors suffer from the curse of dimensionality, mathematically?
KNN relies on the assumption that nearby points in feature space share similar labels — its entire predictive power comes from local neighborhoods being meaningful. As the number of dimensions d increases, two related mathematical phenomena destroy this assumption....
14. What is the mathematical objective function K-Means optimises, and why can it converge to a local minimum?
K-Means seeks to partition n points into k clusters by minimising the within-cluster sum of squares (WCSS) , also called inertia: J = Σₖ Σ_{x ∈ Cₖ} ‖x - μₖ‖² , where μₖ is the centroid (mean) of cluster k....
15. What is the statistical rationale behind k-fold cross-validation, and why are k=5 or k=10 commonly used?
Cross-validation estimates how well a model generalises to unseen data by repeatedly splitting the training data into a training fold and a validation fold, training on the former and evaluating on the latter, then averaging the results. K-fold CV divides...
16. What does the ROC-AUC score mathematically represent, and why is it threshold-independent?
The ROC (Receiver Operating Characteristic) curve plots the True Positive Rate (TPR/recall) against the False Positive Rate (FPR) as the classification decision threshold is varied from 0 to 1. The Area Under this Curve (AUC) has an elegant probabilistic interpretation:...
17. Explain the mathematical tradeoff between precision and recall, and why F1 score is the harmonic mean rather than the arithmetic mean.
Precision is TP/(TP+FP) — of everything predicted positive, what fraction was actually positive. Recall is TP/(TP+FN) — of everything that was actually positive, what fraction did the model find....
18. What is the 'naive' independence assumption in Naive Bayes, and why does it still work well in practice despite being unrealistic?
Naive Bayes applies Bayes' theorem to classify: P(y|x₁,,xₙ) ∝ P(y)·P(x₁,,xₙ|y) . Computing the joint likelihood P(x₁,,xₙ|y) exactly would require modelling all interactions between features — infeasible with limited data.
19. Why is a log transformation commonly applied to skewed numerical features before modeling, mathematically?
Many real-world quantities — income, population, word frequencies, prices — follow a right-skewed (long right tail) distribution, often approximately log-normal. The mathematical property of the logarithm that makes it useful here is that it compresses large values much more than...
20. What is multicollinearity, mathematically, and how does the Variance Inflation Factor (VIF) detect it?
Multicollinearity occurs when two or more predictor features are highly linearly correlated with each other. Mathematically, this means the design matrix X approaches rank deficiency — the columns become nearly linearly dependent, causing XᵀX to become nearly singular (its determinant...
21. Why must features be standardized before applying Ridge or Lasso regularization, mathematically?
Ridge and Lasso add a penalty proportional to coefficient magnitude — λΣβⱼ² or λΣ|βⱼ| respectively. The magnitude of a coefficient βⱼ is inversely related to the scale of its corresponding feature: if feature j is measured in millions (e.g....
22. What is the mathematical relationship between learning_rate and n_estimators in gradient boosting?
In gradient boosting, the final ensemble prediction is F(x) = F₀(x) + η · Σₘ hₘ(x) , where η is the learning rate (also called shrinkage) and the sum runs over n_estimators trees. The learning rate scales down the contribution...
23. How does the softmax function generalize logistic regression to multiclass classification, mathematically?
Binary logistic regression uses the sigmoid function to convert a single linear score into a probability between 0 and 1. For k classes, the softmax function generalises this: given k linear scores (logits) z₁,,zₖ , softmax computes p_i = e^{z_i}...
24. Why does fitting a scaler or transformer on the entire dataset (before train/test split) cause data leakage, mathematically?
Data leakage occurs when information from outside the training set improperly influences the model. If you fit a StandardScaler on the full dataset before splitting, the computed mean and standard deviation incorporate statistics from the test set....
25. How does the class_weight parameter mathematically address class imbalance in scikit-learn classifiers?
When classes are imbalanced (e.g. 95% negative, 5% positive), a model trained with the standard loss function will naturally lean toward predicting the majority class, since doing so minimises average loss across the imbalanced training set even while completely ignoring...
26. Why does using simple label encoding (integers) for nominal categorical features mislead most machine learning models, mathematically?
Label encoding assigns each category an arbitrary integer: e.g. Red=0, Green=1, Blue=2....
27. What is the difference between a single train/validation/test split and k-fold cross-validation for hyperparameter tuning, statistically?
A single validation split estimates a hyperparameter's performance using just one specific subset of data — this estimate has high variance because it depends entirely on which particular samples happened to land in the validation fold. If you tune hyperparameters...
28. Why is PCA sensitive to feature scaling while decision tree feature importance is not, mathematically?
PCA's objective is to find directions of maximum variance in the data, computed from the covariance matrix. Variance is measured in squared units of the original feature, so a feature measured in large units (e.g....
29. Why is the decision boundary of standard logistic regression always a straight line (or hyperplane), mathematically?
Logistic regression predicts class 1 when p(y=1|x) = σ(wᵀx + b) ≥ 0.5 . Since the sigmoid function σ is monotonically increasing and equals exactly 0.5 when its input is 0, this condition simplifies to wᵀx + b ≥ 0...
30. Why can R-squared be a misleading metric for model comparison, and how does adjusted R-squared address this?
R-squared is defined as R² = 1 - SS_res/SS_tot , where SS_res is the sum of squared residuals and SS_tot is the total sum of squares (variance of y). It represents the proportion of variance in y explained by the...
31. Derive mathematically why bagging (bootstrap aggregating) reduces variance, and under what condition it does NOT help.
Suppose you have B independent models, each with the same variance σ² and the same expected prediction (no bias change from averaging). If the predictions were truly independent, the variance of their average would be Var(average) = σ²/B — variance...
32. Why does convexity of the loss function matter for optimization algorithms like gradient descent, mathematically?
A function is convex if a line segment connecting any two points on its graph lies above (or on) the graph itself — equivalently, its second derivative (or Hessian, in multiple dimensions) is non-negative everywhere. The critical property of a...
33. Mathematically, why does RobustScaler handle outliers better than StandardScaler?
StandardScaler transforms features using the mean and standard deviation: z = (x - μ)/σ . Both the mean and standard deviation are heavily influenced by extreme values — a single huge outlier can shift the mean substantially and dramatically inflate...
34. What does it mean for a classifier's predicted probabilities to be 'well-calibrated', and why don't all models produce calibrated probabilities naturally?
A classifier is well-calibrated if, among all the examples it assigns a predicted probability of (say) 0.7 to belonging to the positive class, approximately 70% of them actually are positive. Mathematically, calibration requires P(y=1 | p̂(x)=p) ≈ p for all...
35. Mathematically, why does stochastic gradient descent (SGD) scale to large datasets better than batch gradient descent?
Batch gradient descent computes the exact gradient of the loss using all n training examples before taking a single parameter update step: ∇L(θ) = (1/n)Σᵢ ∇Lᵢ(θ) . This requires O(n) computation per update — for datasets with millions of examples,...
36. Beyond scaling, why must feature selection methods also be included inside a cross-validation pipeline rather than applied beforehand?
Feature selection methods like SelectKBest choose features based on a statistical test (e.g. ANOVA F-value, mutual information) computed between each feature and the target across the available data....
