Explain the bias-variance tradeoff mathematically and how it relates to model complexity.

The expected test error of a model can be decomposed into three components: E[(y - f̂(x))²] = Bias(f̂(x))² + Var(f̂(x)) + σ², where σ² is irreducible noise. Bias measures how far the average prediction (over many training sets) is from the true function — high bias means the model is too simple to capture the underlying pattern (underfitting). Variance measures how much the prediction would change if trained on a different sample — high variance means the model is overly sensitive to the specific training data (overfitting).

As model complexity increases (more polynomial terms, deeper trees, more parameters), bias decreases because the model can fit more intricate patterns, but variance increases because the model has more freedom to fit noise in the training data. The goal is to find the complexity level that minimises the sum, not either component alone.

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

# Demonstrating bias-variance via polynomial degree
degrees = [1, 4, 15]  # underfit, good fit, overfit
for d in degrees:
    pipe = make_pipeline(
        PolynomialFeatures(degree=d),
        Ridge(alpha=0.1)
    )
    scores = cross_val_score(pipe, X, y, cv=5, scoring='neg_mean_squared_error')
    print(f'Degree {d}: CV MSE = {-scores.mean():.3f} (+/- {scores.std():.3f})')
    # degree=1: high bias, low variance (underfit)
    # degree=4: balanced
    # degree=15: low bias, high variance (overfit — CV scores vary widely)

Practical signal: high variance shows up as a large gap between training and validation error (model memorises training data); high bias shows up as both training and validation error being similarly poor (model can't even fit the training data well). Regularisation (Ridge, Lasso) deliberately introduces a small amount of bias to substantially reduce variance.