AI / Essential Machine Learning Algorithms Interview questions

Linear regression is a supervised algorithm that predicts a continuous value by fitting a straight line (or a hyperplane, with many features) through the training data. The model looks like y = w1*x1 + w2*x2 + ... + b, and the weights are chosen to minimize the sum of squared errors between predictions and actual values.

It can be solved directly with the normal equation or iteratively with gradient descent. It is fast, easy to interpret, and a sensible baseline for problems like price or demand forecasting.

It assumes a roughly linear relationship, independent errors, constant error variance, and little multicollinearity between features.

Logistic regression is a supervised classification algorithm. It computes a weighted sum of the features, then squeezes it through the sigmoid function to get a probability between 0 and 1. A threshold, usually 0.5, turns that probability into a class label.

Training minimizes log loss (cross-entropy). Each coefficient can be read as the change in the log-odds of the positive class for a one-unit change in that feature, which makes the model easy to explain.

Despite the word 'regression' in its name, it is used for classification, such as spam detection or churn prediction.

A decision tree is a supervised model that makes predictions by asking a series of yes/no questions about the features. Internal nodes hold a test (for example age < 30), branches are the outcomes, and leaf nodes hold the final class or value.

The tree is built top-down. At each node the algorithm picks the split that most reduces impurity, measured by Gini or entropy for classification and variance for regression.

Trees are easy to read and need little preprocessing, but a deep tree memorizes the training set. Pruning or limiting max_depth keeps that in check.

KNN predicts the label of a new point by looking at the k closest training points and letting them decide. For classification it takes a majority vote; for regression it averages their target values.

It is a lazy learner: there is no real training step, the algorithm just stores the data and does the work at prediction time. Closeness is usually measured with Euclidean distance, so features should be scaled first.

KNN is simple and makes no assumption about the data shape, but prediction gets slow on large datasets and it struggles in very high dimensions.

k-means is an unsupervised algorithm that groups data into k clusters by minimizing the distance between points and their cluster center, called the centroid.

1. Pick k initial centroids.
2. Assign every point to its nearest centroid.
3. Update each centroid to the mean of its assigned points.
4. Repeat steps 2-3 until assignments stop changing.

You must choose k up front, and the result depends on the starting centroids, so libraries run it several times and keep the best run.

from sklearn.cluster import KMeans
model = KMeans(n_clusters=3, n_init=10, random_state=42)
labels = model.fit_predict(X)

Naive Bayes is a probabilistic classifier built on Bayes' theorem. For each class it multiplies the class prior by the likelihood of every feature, then picks the class with the highest posterior.

It is called naive because it assumes features are conditionally independent given the class. That is rarely true, but it makes the math cheap and needs very little training data.

• Gaussian - continuous features
• Multinomial - word counts in text
• Bernoulli - binary presence/absence features

It is a classic choice for spam filtering and document classification.

An SVM is a supervised algorithm that finds the decision boundary with the widest possible margin between two classes. The training points sitting closest to that boundary are the support vectors, and they alone define it.

The parameter C controls the trade-off: a high C punishes misclassification hard and gives a narrow margin, while a low C tolerates some errors for a wider, smoother margin.

With a kernel such as RBF, an SVM can separate data that is not linearly separable. It works well on medium-sized datasets but scales poorly to millions of rows.

A random forest is an ensemble of many decision trees whose predictions are combined: a majority vote for classification, an average for regression.

Two kinds of randomness make the trees different from each other. Each tree trains on a bootstrap sample of the rows, and at every split it only considers a random subset of the features.

Averaging many decorrelated trees cancels out their individual mistakes, so the forest overfits much less than a single deep tree. It also gives feature importance scores and out-of-bag error for free.

Gradient boosting builds a strong model by adding small trees one after another, where each new tree is trained to correct the errors of the ensemble so far.

Concretely, each tree fits the residuals (the negative gradient of the loss) of the current predictions. Its output is then added to the ensemble after being multiplied by a small learning rate, so no single tree dominates.

It usually beats random forests on tabular data when tuned well, but it trains sequentially and overfits if you use too many trees without early stopping.

PCA is an unsupervised technique that reduces the number of features while keeping as much variance as possible. It rotates the data onto new, mutually perpendicular axes called principal components.

The first component points in the direction of greatest variance, the second captures the most remaining variance at a right angle to the first, and so on. Keeping only the top few components compresses the data.

Features should be centered, and usually scaled, before PCA. Common uses are visualization, noise reduction, and speeding up downstream models.

Gradient descent is an optimization algorithm that minimizes a loss function by repeatedly stepping in the direction that decreases it fastest, which is opposite to the gradient.

The update rule is w = w - lr * gradient. The learning rate sets the step size: too small and training crawls, too large and the loss oscillates or diverges.

for _ in range(epochs):
    grad = (2 / n) * X.T @ (X @ w - y)   # MSE gradient
    w -= lr * grad

It powers linear models, logistic regression, and neural networks.

A neural network is a model made of layers of connected units called neurons. Each neuron takes a weighted sum of its inputs, adds a bias, and passes the result through an activation function.

Data flows forward through an input layer, one or more hidden layers, and an output layer. During training, backpropagation computes how much each weight contributed to the error, and an optimizer such as SGD or Adam adjusts the weights.

Networks can learn very complex non-linear patterns, but they need more data and compute than classical algorithms and are harder to interpret.

Hierarchical clustering builds a tree of clusters instead of a flat set. The common agglomerative version starts with every point as its own cluster and repeatedly merges the two closest clusters until only one remains.

How 'closest' is measured depends on the linkage: single (nearest members), complete (farthest members), average, or Ward (smallest increase in variance).

The result is drawn as a dendrogram. Cutting it at a chosen height gives the final clusters, so you do not need to fix k in advance. The cost is poor scaling, roughly quadratic or worse in the number of points.

DBSCAN is a density-based clustering algorithm. It groups together points that are packed closely and marks isolated points as noise.

It uses two parameters: eps, the neighborhood radius, and min_samples, the number of neighbors needed to call a point dense.

• Core point - has at least min_samples points within eps
• Border point - within eps of a core point but not dense itself
• Noise - neither of the above

Unlike k-means, it finds clusters of arbitrary shape and does not need k, but it struggles when clusters have very different densities.

Machine learning is usually split into four types based on the feedback the algorithm learns from.

Supervised learning covers classification and regression, while unsupervised learning covers clustering and dimensionality reduction.

Regularization adds a penalty on model complexity to the loss function so the model cannot fit noise in the training data. Its purpose is to reduce overfitting and improve performance on unseen data.

In linear models the penalty is on the size of the weights: L2 (ridge) shrinks all weights, while L1 (lasso) can push some to exactly zero. In neural networks, dropout and weight decay serve the same goal. Tree models use limits such as max_depth and min_samples_leaf.

A strength parameter (lambda or alpha) controls the trade-off. Too much regularization causes underfitting; too little leaves overfitting in place.

XGBoost (Extreme Gradient Boosting) is an optimized, regularized implementation of gradient-boosted decision trees, famous for winning many tabular-data competitions.

It improves on plain gradient boosting in a few concrete ways:

• Uses second-order gradient information when fitting each tree
• Adds L1/L2 regularization and a complexity penalty on leaves
• Finds splits in parallel and supports column and row subsampling
• Learns a default direction for missing values at every split

It also supports early stopping and custom objectives, which makes it practical for production use.

Bagging (bootstrap aggregating) trains several copies of the same model on different bootstrap samples - random draws from the training data with replacement - and combines their outputs by voting or averaging.

Because each model sees a slightly different dataset, their errors are partly independent and averaging reduces variance without raising bias much. It works best with unstable, high-variance learners such as deep decision trees.

About 37% of rows are left out of each bootstrap sample. These out-of-bag rows give a free validation estimate.

Q-learning is a model-free reinforcement learning algorithm that learns a table of Q-values, where Q(s, a) estimates the total future reward for taking action a in state s and acting well afterward.

Q[s, a] += alpha * (r + gamma * max(Q[s_next]) - Q[s, a])

alpha is the learning rate and gamma is the discount factor that decides how much future rewards matter. The agent usually follows an epsilon-greedy policy: mostly the best known action, occasionally a random one to keep exploring.

It is off-policy, since the update uses the best next action rather than the one actually taken.

An activation function adds non-linearity to a neuron's output. Without it, stacking layers would still collapse into a single linear transformation, and the network could never learn curved decision boundaries.

ReLU is the usual hidden-layer choice because it is cheap and does not saturate for positive inputs.

Both combine many weak models, but they attack different errors. Bagging trains models in parallel to cut variance, while boosting trains them sequentially to cut bias.

Pick bagging when your base model is unstable and overfits. Pick boosting when a simple model underfits and you can afford careful tuning.

Both penalize large weights, but the shape of the penalty changes the result. L1 adds the sum of absolute weights; L2 adds the sum of squared weights.

L1 gives sparse, compact models. L2 is smoother and more stable with collinear features. Elastic Net mixes both when you want sparsity and stability.

Log loss is the negative log-likelihood of the Bernoulli model that logistic regression assumes, and it is convex in the weights. That guarantees gradient descent reaches the global minimum.

Mean squared error on sigmoid outputs is not convex. When the sigmoid saturates, its slope is nearly zero, so a confidently wrong prediction produces a tiny gradient and learning stalls.

Log loss avoids this because the sigmoid derivative cancels out, leaving a clean gradient: (p - y) * x. It also punishes confident mistakes heavily, which pushes the model toward well-calibrated probabilities.

At each node the tree tries every feature and every candidate threshold, scores each split by how much it reduces impurity, and keeps the best one.

For classification, impurity is Gini (1 - sum(p_i^2)) or entropy (-sum(p_i * log2 p_i)). The gain is the parent impurity minus the weighted average impurity of the two children. For regression, the tree minimizes variance (MSE) instead.

The procedure is greedy: it picks the best split at the current node and never revisits earlier choices, so the final tree is not guaranteed to be globally optimal. It recurses until it hits a stopping rule such as max_depth, min_samples_leaf, or a pure node.

A single deep tree has low bias but very high variance. Averaging many trees lowers that variance, and the trees must be decorrelated for the averaging to work.

For B trees with variance sigma^2 and pairwise correlation rho, the ensemble variance is rho*sigma^2 + (1 - rho)*sigma^2 / B. More trees shrink the second term, but only lower correlation shrinks the first.

That is the reason for random feature subsets at each split: without them, a few strong features would dominate every tree and they would all look alike. Bootstrap sampling adds more diversity.

Adding more trees does not cause overfitting; the accuracy gain just flattens out.

A linear SVM cannot separate data like concentric circles. Mapping the data into a higher-dimensional space can make it separable, but computing that mapping explicitly is expensive or even infinite-dimensional.

The SVM's dual formulation only needs dot products between points. A kernel function K(x, z) returns the dot product in the high-dimensional space directly from the original inputs, so the mapping is never computed.

• Linear - no mapping
• Polynomial - captures feature interactions up to a degree
• RBF - similarity decays with distance; gamma sets how narrow each point's influence is

from sklearn.svm import SVC
clf = SVC(kernel='rbf', C=1.0, gamma='scale')

A large gamma gives a wiggly boundary that can overfit; a small gamma gives a smoother one.

Both algorithms are driven by distances or dot products. If one feature ranges from 0 to 100,000 (income) and another from 0 to 1 (a flag), the large feature dominates the distance and the small one is practically ignored.

For KNN this changes which neighbors are 'nearest'. For an RBF SVM it distorts the kernel; for a regularized linear SVM it makes the penalty uneven across features.

Use standardization (zero mean, unit variance) or min-max scaling. Fit the scaler on the training set only, then apply it to validation and test data to avoid leakage.

Tree-based models are the exception, since they split on thresholds and are unaffected by monotonic scaling.

Plain k-means picks starting centroids at random, which can drop two of them in the same natural cluster and leave another cluster uncovered. The algorithm then settles into a poor local minimum.

k-means++ spreads the starting points out:

1. Pick the first centroid uniformly at random.
2. For each point, compute its squared distance to the nearest chosen centroid.
3. Pick the next centroid with probability proportional to that squared distance.
4. Repeat until k centroids are chosen.

Far-away points are more likely to be picked, so the seeds start well separated. This usually gives lower final inertia and faster convergence, and it is the default in scikit-learn alongside multiple restarts (n_init).

There is no single correct answer, so practitioners combine a few methods and check them against domain knowledge.

Inertia always decreases as k grows, so it cannot be minimized blindly. Silhouette is often more reliable because it penalizes clusters that overlap. Also check cluster stability across random seeds.

k-means partitions data around centroids; DBSCAN grows clusters out of dense regions. That one design difference drives everything else.

Use k-means for compact, similar-sized groups and large data. Use DBSCAN when clusters have odd shapes or you need explicit outlier detection.

Gradient boosting performs gradient descent in function space: each tree is a step that moves the predictions toward lower loss.

flowchart TD
  A["Start with constant prediction F0"] --> B["Compute pseudo-residuals<br/>negative gradient of loss"]
  B --> C["Fit a small tree to the residuals"]
  C --> D["Set leaf values for best loss reduction"]
  D --> E["Update F = F + learning_rate * tree"]
  E --> F{More trees or<br/>early stop?}
  F -- continue --> B
  F -- stop --> G["Final model F"]

1. Initialize with a constant: the mean of y for squared error, or the log-odds for classification.
2. Compute residuals for every row from the current predictions.
3. Fit a shallow tree (depth 3-8) to those residuals.
4. Add the tree, scaled by the learning rate, to the running prediction.
5. Repeat for the chosen number of trees, or stop when validation loss stops improving.

A smaller learning rate needs more trees but usually generalizes better.

All three are gradient-boosted tree libraries. They differ mainly in how they grow trees, handle categories, and trade off speed.

Reach for LightGBM on large datasets, CatBoost when you have many high-cardinality categorical columns, and XGBoost when you want maximum ecosystem support.

Classification only needs the correct class to have the highest posterior, not accurate probabilities. The independence assumption distorts the probability values, but the ranking of classes is often still right.

Dependencies between features can also partially cancel out, and with only a few parameters per class the model has low variance, which matters when data is limited. Text is a good example: word counts are correlated, yet Naive Bayes still classifies spam well.

The weakness is that its probabilities are poorly calibrated, often pushed toward 0 or 1, and strongly correlated features get counted twice. Use Laplace smoothing so an unseen feature value does not zero out the entire product.

A model's expected error splits into bias squared + variance + irreducible noise. Bias is error from overly simple assumptions; variance is error from being too sensitive to the particular training sample.

Making a model more flexible lowers bias but raises variance. The goal is the sweet spot where validation error is lowest, which you find with cross-validation and learning curves.

PCA finds the directions along which the data varies most by decomposing its covariance structure.

flowchart LR
  A["Raw data X"] --> B["Center and scale features"]
  B --> C["Covariance matrix or SVD"]
  C --> D["Eigenvectors and eigenvalues"]
  D --> E["Sort by eigenvalue, keep top k"]
  E --> F["Project data onto k components"]

1. Center each feature (and usually scale it).
2. Compute the covariance matrix, or apply SVD directly to the centered data.
3. Extract eigenvectors (component directions) and eigenvalues (variance along each).
4. Keep the top k components, often enough to explain 90-95% of total variance.
5. Multiply the data by those eigenvectors to get the reduced representation.

The explained_variance_ratio_ attribute tells you how much information each component keeps. scikit-learn uses SVD under the hood because it is more numerically stable than forming the covariance matrix.

Both reduce dimensions, but PCA is a general-purpose linear tool while t-SNE is mainly for visualization.

In t-SNE plots, distances between far-apart clusters and cluster sizes are not meaningful. A common workflow is to run PCA down to 30-50 dimensions first, then t-SNE for the final plot.

Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule from the output layer back to the input.

flowchart LR
  A["Input batch"] --> B["Forward pass<br/>compute activations"]
  B --> C["Compute loss"]
  C --> D["Backward pass<br/>chain rule layer by layer"]
  D --> E["Gradients for all weights"]
  E --> F["Optimizer updates weights"]
  F --> A

1. Forward pass - compute and cache every layer's activations.
2. Loss - compare the output with the target.
3. Backward pass - start with dLoss/dOutput and multiply by each layer's local derivative moving backward, reusing cached activations.
4. Update - the optimizer (SGD, Adam) changes weights using these gradients.

Reusing intermediate results makes the backward pass cost about the same as one forward pass, which is what makes training deep networks feasible.

They differ in how many samples are used to compute each weight update.

Mini-batch is the practical default in deep learning because matrix operations on a batch run efficiently on GPUs and the gradient noise is moderate.

The noise in SGD is not purely bad: it can help the optimizer jump out of shallow local minima and saddle points, though it needs a decaying learning rate to settle near the minimum.

Adam combines momentum with per-parameter adaptive learning rates, so it makes steady progress even when gradients are noisy or scaled very differently across parameters.

• First moment m: moving average of gradients, which smooths direction like momentum
• Second moment v: moving average of squared gradients, used to scale each parameter's step
• Update: w -= lr * m_hat / (sqrt(v_hat) + eps)

Parameters with consistently large gradients get smaller steps, and rarely updated ones get larger steps. Bias correction fixes the fact that m and v start at zero and would otherwise be underestimated early on.

Adam needs little learning-rate tuning, but plain SGD with momentum sometimes generalizes better on image tasks, so it is not always the winner at the end of training.

In deep networks, backpropagation multiplies many local derivatives. If each is below 1, as with sigmoid (max slope 0.25) and tanh in their saturated regions, the gradient shrinks exponentially toward early layers. Those layers barely learn.

Common fixes:

• ReLU or variants (Leaky ReLU, GELU), which have slope 1 for active units
• Proper initialization such as He or Xavier
• Batch normalization to keep activations in a healthy range
• Residual connections that give gradients a shortcut path
• LSTM/GRU gates for recurrent networks

The opposite issue, exploding gradients, is handled with gradient clipping.

Multicollinearity means predictors are highly correlated, so the model cannot tell which one deserves the credit. Predictions may stay fine, but coefficients become unstable with huge standard errors and flipping signs.

Detecting it: check the correlation matrix, compute the Variance Inflation Factor (VIF) for each feature (above 5-10 is a warning), or look at a large condition number.

Handling it:

• Drop or merge redundant features
• Combine correlated features into one (ratio, average)
• Apply PCA to get uncorrelated components
• Use ridge regression, which stabilizes coefficients by shrinking them

If you only care about prediction, mild multicollinearity is often acceptable. It matters most when you interpret coefficients.

A discriminative model learns the boundary or P(y|x) directly. A generative model learns how the data is produced, P(x, y) = P(x|y) P(y), then uses Bayes' rule to classify.

If you only need predictions and have plenty of labeled data, a discriminative model is usually the better pick.

Start by fixing the metric. On 99:1 data, a model that always predicts the majority class scores 99% accuracy, so use precision, recall, F1, or PR-AUC instead.

Then choose a technique at the data, algorithm, or decision level:

Apply resampling only inside the training fold. Resampling before the split leaks synthetic copies into validation and inflates scores.

Choose a tree when the relationship between features and target is non-linear or involves interactions that logistic regression would need hand-crafted features to capture.

A single tree is unstable, so in practice you often move to a random forest or boosted trees. Trees also need no feature scaling, while regularized logistic regression does.

Increasing k smooths the decision boundary: bias goes up and variance goes down.

• k = 1 - boundary follows every point, including noise, so the model overfits (low bias, high variance).
• Moderate k - neighbors vote, so single noisy points matter less.
• k close to n - every prediction approaches the overall majority class, so the model underfits.

Prediction also gets slightly slower with larger k. Pick k with cross-validation, and use an odd k for binary problems to avoid tie votes. Distance weighting is another option, letting closer neighbors count more.

AdaBoost starts with every sample weighted equally (1/N) and uses each round to push attention onto the examples the last weak learner got wrong.

1. Train a weak learner (often a depth-1 stump) on the weighted data.
2. Compute its weighted error e.
3. Compute its vote weight alpha = 0.5 * ln((1 - e) / e).
4. Multiply weights of misclassified samples by exp(alpha) and correct ones by exp(-alpha).
5. Normalize the weights so they sum to 1 and repeat.

The final prediction is a weighted vote of all learners. A learner with lower error gets a larger alpha. AdaBoost is sensitive to noisy labels, since mislabeled points keep gaining weight.

A Gaussian Mixture Model assumes the data comes from several Gaussians, but you do not know which point came from which. EM solves this chicken-and-egg problem by alternating between guessing memberships and refitting parameters.

flowchart TD
  A["Initialize means, covariances, weights"] --> B["E-step: compute responsibilities<br/>probability of each component per point"]
  B --> C["M-step: update means, covariances, mixing weights<br/>using responsibilities"]
  C --> D{Log-likelihood<br/>converged?}
  D -- no --> B
  D -- yes --> E["Final GMM"]

• E-step - for each point, compute the posterior probability (responsibility) that each Gaussian produced it.
• M-step - recompute each Gaussian's mean, covariance, and mixing weight as responsibility-weighted statistics.

Each iteration never decreases the log-likelihood, but EM can land in a local optimum, so you run several initializations (k-means is a common starting point). Compared with k-means, GMM gives soft assignments and elliptical clusters.

Both learn action values, but they bootstrap from different next actions. Q-learning uses the best possible next action (max Q(s', a')); SARSA uses the action the agent actually takes next (Q(s', a')).

The classic cliff-walking example shows it: Q-learning hugs the cliff edge because the optimal path is shortest, while SARSA stays further away since it accounts for random exploratory steps that might fall off.

Neither wins everywhere. Gradient boosting usually reaches higher accuracy on tabular data when tuned, while random forest is more forgiving and easier to get right.

Use random forest as a quick, strong baseline. Move to gradient boosting when you need the last few points of accuracy and have time to tune.

Self-attention lets every token look at every other token in the sequence and decide how much each one matters when building its own representation.

flowchart LR
  X["Token embeddings"] --> Q["Queries Q"]
  X --> K["Keys K"]
  X --> V["Values V"]
  Q --> S["Scores = Q K^T / sqrt d_k"]
  K --> S
  S --> W["Softmax weights"]
  W --> O["Weighted sum of V"]
  V --> O

1. Project each token into a query, key, and value vector using learned matrices.
2. Score every query against every key with a dot product.
3. Divide by sqrt(d_k) so large dot products do not push softmax into saturation.
4. Apply softmax to get attention weights, then take the weighted sum of the values.

Multi-head attention runs several of these in parallel so different heads can capture different relationships. The cost grows quadratically with sequence length, which is why long contexts are expensive.