What is the mathematical relationship between learning_rate and n_estimators in gradient boosting?

In gradient boosting, the final ensemble prediction is F(x) = F₀(x) + η · Σₘ hₘ(x), where η is the learning rate (also called shrinkage) and the sum runs over n_estimators trees. The learning rate scales down the contribution of each individual tree. A smaller η means each tree contributes less to the final prediction, so more trees (larger n_estimators) are needed to reach the same total predictive capacity — there is a direct multiplicative tradeoff between the two parameters.

The reason smaller learning rates with more estimators usually generalise better, despite requiring more compute, is regularisation through gradual fitting: taking many small steps allows the ensemble to average out noise in individual trees' residual-fitting, similar to how a smaller step size in gradient descent finds a more precise minimum. A large learning rate with few trees can overfit aggressively to the training residuals in just a few steps.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

configs = [
    {'learning_rate': 0.3, 'n_estimators': 50},    # fast, fewer trees
    {'learning_rate': 0.1, 'n_estimators': 150},   # balanced
    {'learning_rate': 0.01, 'n_estimators': 1500}, # slow, many trees
]

for cfg in configs:
    gb = GradientBoostingClassifier(**cfg, max_depth=3, random_state=42)
    scores = cross_val_score(gb, X, y, cv=5)
    print(f"lr={cfg['learning_rate']}, n_est={cfg['n_estimators']}: "
          f"{scores.mean():.3f}")
    # Smaller lr + more trees often generalizes slightly better,
    # at the cost of significantly longer training time

# Practical rule of thumb: lower the learning rate, increase n_estimators
# proportionally, and use early stopping (validation_fraction,
# n_iter_no_change) to find the right number of trees automatically
gb_early_stop = GradientBoostingClassifier(
    learning_rate=0.05,
    n_estimators=1000,
    validation_fraction=0.1,
    n_iter_no_change=10,   # stop if no improvement for 10 rounds
)

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.