Explain the mathematical intuition behind gradient descent and why the learning rate matters.

Gradient descent is an iterative optimisation algorithm that finds a local minimum of a differentiable function by repeatedly stepping in the direction of steepest descent — the negative gradient. The gradient ∇L(θ) points in the direction of steepest increase, so subtracting a scaled version of it moves the parameters toward lower loss: θ_{t+1} = θ_t - η∇L(θ_t), where η is the learning rate.

The learning rate controls the step size. If η is too small, convergence is correct but painfully slow, requiring many iterations to reach the minimum. If η is too large, the update can overshoot the minimum and oscillate or even diverge — the loss increases instead of decreasing, because the linear approximation the gradient provides is only locally accurate.

import numpy as np

def gradient_descent(X, y, lr=0.01, n_iters=1000):
    n, d = X.shape
    theta = np.zeros(d)
    for _ in range(n_iters):
        predictions = X @ theta
        error = predictions - y
        gradient = (2 / n) * X.T @ error   # gradient of MSE w.r.t. theta
        theta -= lr * gradient             # step toward lower loss
    return theta

# Demonstrating learning rate sensitivity
X = np.column_stack([np.ones(100), np.linspace(0, 10, 100)])
y = 3 + 2 * X[:, 1] + np.random.randn(100) * 0.5

theta_good = gradient_descent(X, y, lr=0.01)   # converges smoothly
theta_bad  = gradient_descent(X, y, lr=0.9)    # likely diverges or oscillates
print(theta_good)

Why this matters in scikit-learn: estimators like SGDRegressor and SGDClassifier expose learning_rate and eta0 parameters precisely because of this trade-off. Adaptive schedules (e.g. learning_rate='adaptive') reduce the step size over time, balancing fast initial progress against fine-grained convergence near the minimum.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.