2.5 When Log-Scale Transformation Is Appropriate
Key Takeaways
- A log transform is appropriate for strictly positive, right-skewed, multiplicative quantities — prices, incomes, counts, durations, and latencies.
- `log1p` ($\ln(x+1)$) is the standard choice when the feature contains true zeros, because $\ln(0)$ is undefined.
- A log transform is inappropriate for features with negative values (use Yeo-Johnson), for already-symmetric features, and for features consumed only by tree models, which are invariant to it.
- Logging the *target* changes what the model optimises: errors become proportional rather than absolute, so predictions must be inverted with `expm1` before reporting business metrics.
- A log transform compresses large values, which shrinks the leverage of extreme observations — often a better response to skew than deleting rows.
2.5 When Log-Scale Transformation Is Appropriate
Distribution Transformations for Skewed Features
Many real-world features (e.g., pricing, income, click counts, server latencies) follow long-tailed, right-skewed distributions. Transforming these features toward a bell-shaped Gaussian distribution stabilizes variance, improves linear model convergence, and satisfies homoscedasticity assumptions.
Raw Right-Skewed: [||||||||||||| | | | | ] --> Long right tail
Log-Transformed ln(x): [ ||| |||||||||||||||| ||| ] --> Bell-shaped normal
Common Transformation Mathematical Formulations
| Transformation | Formula | Domain / Constraints | Primary Use Case |
|---|---|---|---|
| Natural Log | $y = \ln(x)$ | $x > 0$ strictly | Compresses severe right skew when all values are strictly positive. |
Log-Plus-One (log1p) | $y = \ln(x + 1)$ | $x \ge 0$ | Standard choice for non-negative data containing true zeros (e.g., counts, charges). Prevents $\ln(0) = -\infty$. |
| Square Root | $y = \sqrt{x}$ | $x \ge 0$ | Moderate compression of right skew; milder effect than log transformation. |
| Box-Cox | $y^{(\lambda)} = \begin{cases} \frac{x^\lambda - 1}{\lambda} & \text{if } \lambda \ne 0 \ \ln(x) & \text{if } \lambda = 0 \end{cases}$ | $x > 0$ strictly | Optimizes parameter $\lambda$ via maximum likelihood to maximize normality. |
| Yeo-Johnson | Generalization of Box-Cox | Supports all real numbers ($x < 0, x = 0, x > 0$) | Stabilizes variance and normalizes features containing negative, zero, and positive values. |
PySpark and scikit-learn Transformation Implementations
# PySpark Log-Plus-One Transformation
from pyspark.sql.functions import log1p, expm1
df_transformed = df.withColumn("log_revenue", log1p("raw_revenue"))
# Scikit-learn PowerTransformer for Box-Cox and Yeo-Johnson
from sklearn.preprocessing import PowerTransformer
import numpy as np
# Yeo-Johnson handles zero and negative values
pt_yj = PowerTransformer(method="yeo-johnson", standardize=True)
X_train_yj = pt_yj.fit_transform(X_train)
X_test_yj = pt_yj.transform(X_test) # Fit on train, transform on test!
The Decision Rule
Apply a log transform when all of the following hold:
- The feature is strictly positive (or non-negative, using
log1p). - Its distribution is right-skewed with a long tail — the mean sits well above the median, and the maximum is orders of magnitude beyond the 75th percentile.
- The quantity is naturally multiplicative: a change from $1,000 to $2,000 means what a change from $100,000 to $200,000 means. Prices, incomes, revenues, population counts, page views, session durations, and API latencies all behave this way.
- The downstream model is sensitive to scale and to leverage — linear and logistic regression, regularised linear models, k-NN, SVM, K-Means, or a neural network.
Skip it when any of the following hold:
| Situation | Why a log transform is wrong | Better option |
|---|---|---|
| The feature contains negative values | $\ln(x)$ is undefined for $x \le 0$ | Yeo-Johnson |
| The feature contains true zeros | $\ln(0) = -\infty$ | log1p — $\ln(x+1)$ maps 0 to 0 |
| The distribution is already roughly symmetric | The transform introduces left skew | Leave it, or standardise |
| The model is a decision tree or tree ensemble | Splits depend on rank order, and log is monotonic, so the splits are unchanged | No transform needed |
| The feature is bounded and narrow (e.g. a 1–5 rating) | Nothing to compress | No transform needed |
Exam framing: "a strictly positive, heavily right-skewed feature feeding a linear model" is the canonical yes. "A right-skewed feature feeding XGBoost" is the canonical no — a monotonic transform cannot change a tree's split points.
Choosing Among the Power Transforms
| Transform | Domain | Strength | Notes |
|---|---|---|---|
| $\ln(x)$ | $x > 0$ | Strong compression | Fails on zeros |
$\ln(x+1)$ (log1p) | $x \ge 0$ | Strong compression | The default for counts and currency |
| $\sqrt{x}$ | $x \ge 0$ | Mild compression | Good for moderate skew, e.g. Poisson-like counts |
| Box-Cox | $x > 0$ | Fits $\lambda$ by maximum likelihood | Chooses the strength for you; still fails on zeros |
| Yeo-Johnson | All reals | Fits $\lambda$; handles negatives and zeros | The general-purpose choice when the sign is mixed |
from pyspark.sql import functions as F
# Feature transform: log1p tolerates the zeros in a revenue column
df = df.withColumn("log_revenue", F.log1p("raw_revenue"))
Transforming the Target Is a Different Decision
Logging a feature rescales an input. Logging the target changes the loss the model minimises: squared error on $\ln(y+1)$ penalises relative error, so a $50k miss on a $1M house counts the same as a $500 miss on a $10k one. That is often exactly what a business wants — and it is also why a log-target model is not directly comparable to a raw-target model until predictions are inverted.
Two consequences follow, and both are examinable:
- Every prediction must be inverted with
np.expm1(orF.expm1) before it is reported or compared in original units. - Metrics computed on the log scale are log-scale metrics (effectively RMSLE), not dollars. Section 3.13 covers this in full.
Target Variable Transformation & Prediction Inversion
When a continuous target variable $y$ is heavily right-skewed, training a regression model on the log-transformed target $\tilde{y} = \ln(y + 1)$ frequently achieves superior optimization stability.
However, the model will output predictions on the transformed log scale ($\hat{\tilde{y}}$). Before computing business metrics (such as Mean Absolute Error in dollars) or reporting results to stakeholders, predictions must be inverted back to the original domain:
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error
# 1. Transform target during training
y_train_log = np.log1p(y_train)
model.fit(X_train, y_train_log)
# 2. Predict on test set (outputs log-scale predictions)
y_pred_log = model.predict(X_test)
# 3. Invert predictions back to original domain
y_pred_original = np.expm1(y_pred_log)
# 4. Evaluate true performance against actual original target
rmse_dollars = np.sqrt(mean_squared_error(y_test, y_pred_original))
mae_dollars = mean_absolute_error(y_test, y_pred_original)
print(f"Test RMSE: ${rmse_dollars:,.2f}, MAE: ${mae_dollars:,.2f}")
A continuous feature representing website visit duration contains valid zero values (representing immediate bounces) along with heavily right-skewed positive durations. Which transformation should the data scientist apply to normalize the feature without generating undefined mathematical errors?
A regression model is trained to predict real estate sales prices using a log-transformed target variable y_train_log = np.log1p(y_train). During test set evaluation, the model outputs a predicted value of y_hat_log = 12.5. How must this prediction be processed to determine the estimated price in original dollars?
A right-skewed transaction_amount feature will be fed to an XGBoost classifier. A colleague proposes applying log1p to normalise it. What is the most accurate assessment?