Why is PCA sensitive to feature scaling while decision tree feature importance is not, mathematically?

PCA's objective is to find directions of maximum variance in the data, computed from the covariance matrix. Variance is measured in squared units of the original feature, so a feature measured in large units (e.g. salary in dollars, variance in the millions) will dominate the covariance matrix and consequently the principal components, regardless of whether that feature is actually more informative than a feature measured in small units (e.g. age in years, variance in the tens). This makes PCA fundamentally scale-dependent.

Decision trees, by contrast, choose splits based on threshold comparisons (feature ≤ t) and evaluate the resulting impurity reduction — neither the comparison nor the impurity calculation depends on the numeric scale of the feature, only its relative ordering and how well a split separates classes/reduces variance. Multiplying a feature by 1000 doesn't change which split point achieves the best separation, so tree-based feature importance (computed from total impurity reduction attributable to a feature across all trees/splits) is naturally scale-invariant.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler

# Demonstrating PCA's scale sensitivity
X_unscaled = np.column_stack([
    np.random.randn(100) * 1,        # small variance feature
    np.random.randn(100) * 1000,     # huge variance feature (different units)
])

pca_unscaled = PCA(n_components=2).fit(X_unscaled)
print(pca_unscaled.explained_variance_ratio_)
# Almost entirely dominated by the large-variance feature!

pca_scaled = PCA(n_components=2).fit(StandardScaler().fit_transform(X_unscaled))
print(pca_scaled.explained_variance_ratio_)
# Closer to 50/50 — reflects each feature's TRUE informativeness

# Tree-based feature importance is scale-invariant — no scaling needed
rf = RandomForestClassifier(n_estimators=100).fit(X_unscaled, y)
print(rf.feature_importances_)  # unaffected by the artificial scale difference