2.5 When Log-Scale Transformation Is Appropriate

2.5 When Log-Scale Transformation Is Appropriate

Distribution Transformations for Skewed Features

Many real-world features (e.g., pricing, income, click counts, server latencies) follow long-tailed, right-skewed distributions. Transforming these features toward a bell-shaped Gaussian distribution stabilizes variance, improves linear model convergence, and satisfies homoscedasticity assumptions.

Raw Right-Skewed:       [||||||||||||| |  |   |     |       ]  --> Long right tail
Log-Transformed ln(x):  [   ||| |||||||||||||||| |||   ]       --> Bell-shaped normal

Common Transformation Mathematical Formulations

PySpark and scikit-learn Transformation Implementations

# PySpark Log-Plus-One Transformation
from pyspark.sql.functions import log1p, expm1

df_transformed = df.withColumn("log_revenue", log1p("raw_revenue"))

# Scikit-learn PowerTransformer for Box-Cox and Yeo-Johnson
from sklearn.preprocessing import PowerTransformer
import numpy as np

# Yeo-Johnson handles zero and negative values
pt_yj = PowerTransformer(method="yeo-johnson", standardize=True)
X_train_yj = pt_yj.fit_transform(X_train)
X_test_yj = pt_yj.transform(X_test)  # Fit on train, transform on test!

---

The Decision Rule

Apply a log transform when all of the following hold:

1. The feature is strictly positive (or non-negative, using log1p).
2. Its distribution is right-skewed with a long tail — the mean sits well above the median, and the maximum is orders of magnitude beyond the 75th percentile.
3. The quantity is naturally multiplicative: a change from $1,000 to $2,000 means what a change from $100,000 to $200,000 means. Prices, incomes, revenues, population counts, page views, session durations, and API latencies all behave this way.
4. The downstream model is sensitive to scale and to leverage — linear and logistic regression, regularised linear models, k-NN, SVM, K-Means, or a neural network.

Skip it when any of the following hold:

Exam framing: "a strictly positive, heavily right-skewed feature feeding a linear model" is the canonical yes. "A right-skewed feature feeding XGBoost" is the canonical no — a monotonic transform cannot change a tree's split points.

Choosing Among the Power Transforms

from pyspark.sql import functions as F

# Feature transform: log1p tolerates the zeros in a revenue column
df = df.withColumn("log_revenue", F.log1p("raw_revenue"))

Transforming the Target Is a Different Decision

Logging a feature rescales an input. Logging the target changes the loss the model minimises: squared error on $\ln(y+1)$ penalises relative error, so a $50k miss on a $1M house counts the same as a $500 miss on a $10k one. That is often exactly what a business wants — and it is also why a log-target model is not directly comparable to a raw-target model until predictions are inverted.

Two consequences follow, and both are examinable:

1. Every prediction must be inverted with np.expm1 (or F.expm1) before it is reported or compared in original units.
2. Metrics computed on the log scale are log-scale metrics (effectively RMSLE), not dollars. Section 3.13 covers this in full.

---

Target Variable Transformation & Prediction Inversion

When a continuous target variable $y$ is heavily right-skewed, training a regression model on the log-transformed target $\tilde{y} = \ln(y + 1)$ frequently achieves superior optimization stability.

However, the model will output predictions on the transformed log scale ($\hat{\tilde{y}}$). Before computing business metrics (such as Mean Absolute Error in dollars) or reporting results to stakeholders, predictions must be inverted back to the original domain:

$$\hat{y}_{raw} = \exp(\hat{\tilde{y}}) - 1$$

import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error

# 1. Transform target during training
y_train_log = np.log1p(y_train)
model.fit(X_train, y_train_log)

# 2. Predict on test set (outputs log-scale predictions)
y_pred_log = model.predict(X_test)

# 3. Invert predictions back to original domain
y_pred_original = np.expm1(y_pred_log)

# 4. Evaluate true performance against actual original target
rmse_dollars = np.sqrt(mean_squared_error(y_test, y_pred_original))
mae_dollars = mean_absolute_error(y_test, y_pred_original)
print(f"Test RMSE: ${rmse_dollars:,.2f}, MAE: ${mae_dollars:,.2f}")