What is the statistical rationale behind k-fold cross-validation, and why are k=5 or k=10 commonly used?

Cross-validation estimates how well a model generalises to unseen data by repeatedly splitting the training data into a training fold and a validation fold, training on the former and evaluating on the latter, then averaging the results. K-fold CV divides data into k equal partitions, using each partition once as validation while training on the remaining k-1 folds, giving k separate performance estimates that are then averaged.

This addresses a fundamental statistical tension: using more folds (larger k) means each training set is larger and closer to using all the data, which reduces bias in the performance estimate, but the k estimates become more correlated with each other (since training sets overlap heavily), which can increase variance of the final averaged estimate. The extreme case, k=n (leave-one-out CV), has very low bias but high variance and is computationally expensive. Empirically, k=5 or k=10 has been found to offer a good bias-variance balance for the estimate itself, while remaining computationally tractable.

from sklearn.model_selection import KFold, cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, random_state=42)

# Standard k-fold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=kf)
print(f'Mean: {scores.mean():.3f}, Std: {scores.std():.3f}')

# StratifiedKFold preserves class proportions in each fold —
# CRITICAL for imbalanced classification problems
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
stratified_scores = cross_val_score(model, X, y, cv=skf)

# Leave-one-out: k=n, very low bias but high variance, expensive
from sklearn.model_selection import LeaveOneOut
# loo_scores = cross_val_score(model, X, y, cv=LeaveOneOut())  # slow!