What is multicollinearity, mathematically, and how does the Variance Inflation Factor (VIF) detect it?

Multicollinearity occurs when two or more predictor features are highly linearly correlated with each other. Mathematically, this means the design matrix X approaches rank deficiency — the columns become nearly linearly dependent, causing XᵀX to become nearly singular (its determinant approaches zero). Since OLS requires inverting XᵀX, near-singularity makes (XᵀX)⁻¹ numerically unstable: small changes in the data produce wildly different coefficient estimates, and standard errors of the coefficients inflate dramatically.

The Variance Inflation Factor for feature j is computed by regressing feature j against all other features and measuring VIF_j = 1/(1-R²_j), where R²_j is the R-squared of that auxiliary regression. If feature j is well-predicted by the others (high R²_j), VIF is large — a VIF of 10 means the variance of that coefficient's estimate is 10 times what it would be if the feature were uncorrelated with the others. A common rule of thumb flags VIF > 5 or 10 as concerning.

from statsmodels.stats.outliers_influence import variance_inflation_factor
import pandas as pd
import numpy as np

def calculate_vif(X_df):
    vif_data = pd.DataFrame()
    vif_data['feature'] = X_df.columns
    vif_data['VIF'] = [
        variance_inflation_factor(X_df.values, i)
        for i in range(X_df.shape[1])
    ]
    return vif_data

print(calculate_vif(X_train_df))
# feature      VIF
# age          1.2
# income       8.7   <- concerning, correlated with other features
# debt_ratio   9.1   <- concerning

# Mitigation: use Ridge regression, which handles multicollinearity
# gracefully by shrinking correlated coefficients together
from sklearn.linear_model import Ridge
ridge = Ridge(alpha=1.0).fit(X_train, y_train)