2.2 Creating Visualizations for Categorical and Continuous Features

2.2 Creating Visualizations for Categorical and Continuous Features

Summary statistics tell you the moments of a distribution; a plot tells you its shape. Two columns can share a mean and standard deviation while one is unimodal and the other bimodal — a difference that changes which model and which preprocessing are appropriate, and which no summary table will reveal.

---

Interactive Notebook Visualizations with display()

The native Databricks display() function renders any PySpark DataFrame, Spark SQL query result, or pandas DataFrame into an interactive rich table. From the display() output, users can click the + (Add Visualization) button to launch the Databricks Visualization Editor.

# Render interactive table in Databricks notebook
df_features = spark.sql("""
    SELECT 
        customer_id,
        tenure_months,
        monthly_charges,
        total_charges,
        contract_type,
        payment_method,
        churn_label
    FROM ml_catalog.silver_prep.churn_features
""")

display(df_features)

Supported Built-in Visualization Types

Databricks notebook visualizations execute aggregations at the Spark engine level or aggregate client-side depending on sample size:

1. Bar Charts & Grouped / Stacked Bars: Ideal for comparing category counts, mean target rates across categorical segments (e.g., churn rate by contract_type), and categorical distributions.
2. Histograms & Frequency Density Plots: Essential for inspecting numerical distributions, identifying multi-modal clusters, and spotting severe right or left skewness.
3. Box Plots (Whisker Plots): Displays the five-number summary (minimum, lower fence $Q1 - 1.5 \times IQR$, median, upper fence $Q3 + 1.5 \times IQR$, maximum) alongside individual outlier points. Critical for comparing numerical distributions across target classes.
4. Scatter Plots: Visualizes bivariate relationships between two continuous features (e.g., monthly_charges vs. total_charges) to detect non-linear dependencies, clustering, or heteroscedasticity.
5. Line & Area Charts: Used primarily for time-series feature trends, seasonality detection, and monitoring feature value drift over time.
6. Pivot Tables & Heatmaps: Facilitates two-dimensional cross-tabulations (e.g., average churn rate broken down simultaneously by payment_method and contract_type).

---

Matching the Chart to the Question

Reading each one

• Histogram — bin count matters. Too few bins hide bimodality; too many turn the distribution into noise. A long right tail with a spike at zero usually means a count or currency feature, which points at a log-plus-one transform (Section 2.5).
• Box plot — the box spans Q1 to Q3, the line is the median, and the whiskers extend to Tukey's 1.5 × IQR fences with points beyond drawn individually. Those points are exactly the observations Section 2.4's IQR rule flags.
• Bar chart of a categorical — look for a level that dominates (near-constant column), a very long tail of rare levels (a one-hot encoding hazard), and case or whitespace variants of the same value ("Male", "male", " M").
• Grouped box plot — if the boxes for churned and retained customers overlap almost completely, that feature carries little univariate signal for the target.

Aggregate in Spark, Then Plot

The notebook chart layer works from the rows display() returns, so a plot built directly on a billion-row DataFrame is drawn from a bounded slice of it — which misrepresents the true distribution. Push the aggregation into Spark first:

from pyspark.sql import functions as F

# Correct: Spark computes the aggregate, the chart plots a small result
churn_by_contract = (
    df.groupBy("contract_type")
      .agg(F.count("*").alias("customers"),
           F.avg("churn_label").alias("churn_rate"))
      .orderBy(F.desc("customers"))
)
display(churn_by_contract)   # bar chart: contract_type on X, churn_rate on Y

# Histogram of a continuous feature: bucket in Spark, then plot the counts
bucketed = (
    df.withColumn("charge_bucket", (F.col("monthly_charges") / 10).cast("int") * 10)
      .groupBy("charge_bucket").count().orderBy("charge_bucket")
)
display(bucketed)

For an unaggregated scatter of a huge table, sample deliberately rather than relying on truncation: display(df.sample(fraction=0.01, seed=42)).

When to Leave the Built-In Charts

Built-in visualizations cover the common cases without moving data. Reach for Matplotlib or Seaborn on a sampled pandas DataFrame when you need a correlation heatmap with annotations, a pair plot, overlaid density curves, or any figure you intend to log to MLflow with mlflow.log_figure:

import seaborn as sns, matplotlib.pyplot as plt

pdf = df.select("tenure_months", "monthly_charges", "churn_label").sample(0.05, seed=42).toPandas()
fig, ax = plt.subplots(figsize=(7, 5))
sns.boxplot(data=pdf, x="churn_label", y="monthly_charges", ax=ax)
mlflow.log_figure(fig, "eda/charges_by_churn.png")

The rule is unchanged from every other Spark operation: reduce first, collect second. toPandas() on an unsampled large table is the fastest way to an out-of-memory driver.