Why does a random forest reduce variance compared to a single decision tree, and what role does feature randomness play?

A random forest builds many decision trees, each trained on a bootstrap sample of the data (bagging), and averages their predictions. The variance of the average of n independent, identically distributed random variables each with variance σ² is σ²/n — averaging reduces variance proportionally to the number of estimators, provided the trees are independent.

In practice, trees trained on bootstrap samples of the same dataset are correlated, not independent, because they share much of the same underlying data. The variance of the average of n correlated variables with pairwise correlation ρ is ρσ² + (1-ρ)σ²/n — as n grows large, this approaches ρσ², not zero. This is why random forests also randomly restrict the features considered at each split (max_features): this decorrelates the trees from each other, reducing ρ and allowing variance reduction to continue benefiting from larger n.

from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score

single_tree = DecisionTreeClassifier(max_depth=None)
forest      = RandomForestClassifier(
    n_estimators=200,
    max_features='sqrt',   # randomly consider sqrt(n_features) per split
    bootstrap=True,        # sample with replacement
)

tree_scores   = cross_val_score(single_tree, X, y, cv=5)
forest_scores = cross_val_score(forest, X, y, cv=5)

print(f'Single tree: {tree_scores.mean():.3f} (+/- {tree_scores.std():.3f})')
print(f'Forest:      {forest_scores.mean():.3f} (+/- {forest_scores.std():.3f})')
# Forest typically has similar mean but MUCH lower std (variance)

Out-of-bag (OOB) error: because each tree is trained on roughly 63% of samples (bootstrap sampling), the remaining ~37% can be used to validate that specific tree — giving a free, built-in validation estimate without needing a separate holdout set. Set oob_score=True to access this.