Why does a random forest reduce variance compared to a single decision tree, and what role does feature randomness play?

A random forest builds many decision trees, each trained on a bootstrap sample of the data (bagging), and averages their predictions. The variance of the average of n independent, identically distributed random variables each with variance σ² is σ²/n — averaging reduces variance proportionally to the number of estimators, provided the trees are independent.

In practice, trees trained on bootstrap samples of the same dataset are correlated, not independent, because they share much of the same underlying data. The variance of the average of n correlated variables with pairwise correlation ρ is ρσ² + (1-ρ)σ²/n — as n grows large, this approaches ρσ², not zero. This is why random forests also randomly restrict the features considered at each split (max_features): this decorrelates the trees from each other, reducing ρ and allowing variance reduction to continue benefiting from larger n.

from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score

single_tree = DecisionTreeClassifier(max_depth=None)
forest      = RandomForestClassifier(
    n_estimators=200,
    max_features='sqrt',   # randomly consider sqrt(n_features) per split
    bootstrap=True,        # sample with replacement
)

tree_scores   = cross_val_score(single_tree, X, y, cv=5)
forest_scores = cross_val_score(forest, X, y, cv=5)

print(f'Single tree: {tree_scores.mean():.3f} (+/- {tree_scores.std():.3f})')
print(f'Forest:      {forest_scores.mean():.3f} (+/- {forest_scores.std():.3f})')
# Forest typically has similar mean but MUCH lower std (variance)

Out-of-bag (OOB) error: because each tree is trained on roughly 63% of samples (bootstrap sampling), the remaining ~37% can be used to validate that specific tree — giving a free, built-in validation estimate without needing a separate holdout set. Set oob_score=True to access this.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.