Explain the mathematical intuition behind gradient descent and why the learning rate matters.

Gradient descent is an iterative optimisation algorithm that finds a local minimum of a differentiable function by repeatedly stepping in the direction of steepest descent — the negative gradient. The gradient ∇L(θ) points in the direction of steepest increase, so subtracting a scaled version of it moves the parameters toward lower loss: θ_{t+1} = θ_t - η∇L(θ_t), where η is the learning rate.

The learning rate controls the step size. If η is too small, convergence is correct but painfully slow, requiring many iterations to reach the minimum. If η is too large, the update can overshoot the minimum and oscillate or even diverge — the loss increases instead of decreasing, because the linear approximation the gradient provides is only locally accurate.

import numpy as np

def gradient_descent(X, y, lr=0.01, n_iters=1000):
    n, d = X.shape
    theta = np.zeros(d)
    for _ in range(n_iters):
        predictions = X @ theta
        error = predictions - y
        gradient = (2 / n) * X.T @ error   # gradient of MSE w.r.t. theta
        theta -= lr * gradient             # step toward lower loss
    return theta

# Demonstrating learning rate sensitivity
X = np.column_stack([np.ones(100), np.linspace(0, 10, 100)])
y = 3 + 2 * X[:, 1] + np.random.randn(100) * 0.5

theta_good = gradient_descent(X, y, lr=0.01)   # converges smoothly
theta_bad  = gradient_descent(X, y, lr=0.9)    # likely diverges or oscillates
print(theta_good)

Why this matters in scikit-learn: estimators like SGDRegressor and SGDClassifier expose learning_rate and eta0 parameters precisely because of this trade-off. Adaptive schedules (e.g. learning_rate='adaptive') reduce the step size over time, balancing fast initial progress against fine-grained convergence near the minimum.