2.1 Computing Summary Statistics on a Spark DataFrame

2.1 Computing Summary Statistics on a Spark DataFrame

Exploratory Data Analysis (EDA) is the indispensable first phase of any production machine learning workflow on the Databricks Lakehouse. In a Lakehouse architecture, data scientists interact directly with massive raw, bronze, or silver Delta Lake tables using Apache Spark. Because Lakehouse datasets frequently scale to millions or billions of rows across distributed clusters, traditional single-node EDA tools (such as native pandas or matplotlib) cannot operate directly without memory bottlenecks or aggressive downsampling. Databricks resolves this challenge by providing distributed data profiling and interactive visualization utilities natively inside Databricks notebooks.

Rigorous EDA achieves several mission-critical objectives before a single algorithm is trained:

• Data Quality & Integrity Verification: Identifying missing values, unexpected null clusters, sentinel values (such as -999 or "N/A"), and data type mismatches.
• Target Variable Diagnostics: Assessing class balance for classification problems and evaluating distribution skewness, kurtosis, and multi-modality for regression targets.
• Feature Distribution & Variance Analysis: Uncovering constant or near-zero variance features that contribute no predictive signal, as well as extreme outliers.
• Feature Interactions & Collinearity: Detecting multi-collinearity among predictor variables and uncovering non-linear relationships with the target label.
• Temporal & Spatial Partitioning Patterns: Verifying that temporal distributions are consistent across time to prevent future data leakage.

---

Automated Data Profiling with dbutils.data.summarize()

Databricks provides a powerful built-in utility specifically engineered for rapid tabular profiling: dbutils.data.summarize(). When executed against an Apache Spark DataFrame, Databricks leverages the distributed Spark Catalyst optimizer to compute aggregate statistics and renders an interactive, visual data profile directly in the notebook output cell.

# Load feature dataset from Unity Catalog Silver table
df = spark.table("ml_catalog.feature_store.customer_churn_features")

# Generate an automatic interactive profile
dbutils.data.summarize(df)

# Comparison with native Spark DataFrame summary methods
# df.summary() returns a Spark DataFrame containing standard statistics
summary_df = df.summary("count", "mean", "stddev", "min", "25%", "50%", "75%", "max")
display(summary_df)

Metrics Computed by dbutils.data.summarize()

The dbutils.data.summarize() command automatically categorizes columns by data type and generates distinct statistical profiles:

dbutils.data.summarize() vs. df.describe() vs. df.summary()

Understanding the operational differences between these profiling approaches is frequently tested on the Databricks Machine Learning Associate exam:

• df.describe(*cols): A standard PySpark DataFrame method that calculates basic descriptive statistics: count, mean, stddev, min, and max. It does not calculate percentiles (such as the median or interquartile range).
• df.summary(*statistics): An extended PySpark DataFrame method that calculates count, mean, stddev, min, 25%, 50%, 75%, and max by default, or allows custom percentile specifications (e.g., df.summary("count", "5%", "95%")). It returns a tabular Spark DataFrame.
• dbutils.data.summarize(df): A Databricks-specific UI command that executes distributed computations across the cluster and renders a comprehensive visual dashboard with interactive charts, null indicators, and distribution shapes without requiring manual charting.

---

Choosing Between the Three Commands

# describe(): no percentiles at all
display(df.describe("tenure_months", "monthly_charges"))

# summary(): quartiles by default, or name exactly what you want
display(df.summary())
display(df.summary("count", "mean", "5%", "50%", "95%"))

Both describe() and summary() return a DataFrame, not printed output. A cell that ends in df.summary() shows nothing useful until it is wrapped in display() or .show() — a detail that appears in exam code snippets.

Behaviour worth remembering

• describe() and summary() operate on numeric and string columns; string columns get count, min, and max (lexicographic) but no mean or stddev.
• Percentiles are computed with an approximate algorithm so the work stays distributed. For an exact quantile use df.stat.approxQuantile(col, probs, 0.0), which is exact at a relative error of 0 but far more expensive.
• Neither method reports null counts directly; the count row shows non-null counts, so a column whose count is lower than the DataFrame's row count has nulls. This is the reason dbutils.data.summarize exists — it reports missingness explicitly.

Reading the Profile for Defects

A summary table is only useful if you know what to look for: