What is the mathematical intuition behind gradient boosting? How does it differ from random forests?

Gradient boosting builds an ensemble of weak learners (typically shallow decision trees) sequentially, where each new tree is trained to predict the negative gradient of the loss function with respect to the current ensemble's predictions — essentially, each tree learns to correct the errors (residuals, for squared loss) of all previous trees combined. The final prediction is F(x) = F₀(x) + η·Σ hₘ(x), where each hₘ is a tree fit to the current residuals and η is a shrinkage/learning rate.

This is fundamentally different from random forests, which build trees independently and in parallel on bootstrap samples and average them to reduce variance. Gradient boosting builds trees sequentially and dependently — each tree depends on all previous ones — primarily to reduce bias by progressively fitting the parts of the function the ensemble has gotten wrong so far.

from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier

# Conceptual implementation of gradient boosting for regression (squared loss)
import numpy as np
from sklearn.tree import DecisionTreeRegressor

def simple_gradient_boost(X, y, n_estimators=50, lr=0.1, max_depth=3):
    F = np.zeros(len(y))             # initial prediction: all zeros
    trees = []
    for m in range(n_estimators):
        residuals = y - F            # negative gradient for squared loss
        tree = DecisionTreeRegressor(max_depth=max_depth)
        tree.fit(X, residuals)       # fit tree to the residuals
        F += lr * tree.predict(X)    # update ensemble prediction
        trees.append(tree)
    return trees, F

# scikit-learn's production implementation
gb = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,   # shrinkage — smaller values need more trees but generalise better
    max_depth=3,
)
gb.fit(X_train, y_train)

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.