9.2 Data Sources, SaveModes, and Partitioning

Data ingestion and data persistence represent the boundaries of any Apache Spark application. Spark provides a unified, format-agnostic abstraction for reading data from external storage into DataFrames via the DataFrameReader interface, and writing DataFrames back to storage via the DataFrameWriter interface. This section covers the read/write APIs, storage format trade-offs, write modes, and physical layout optimization techniques including directory partitioning and table bucketing.

---

1. The Unified Data Source API

Apache Spark standardizes all data ingestion and persistence through a consistent, fluent builder pattern:

spark.read.format(...).option(...).load(...)   -->  DataFrameReader
df.write.format(...).mode(...).option(...).save(...)  -->  DataFrameWriter

The DataFrameReader Interface

# Generic DataFrameReader syntax
df = spark.read \
    .format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .option("sep", ",") \
    .load("dbfs:/data/raw_transactions.csv")

# Equivalent format-specific shortcut syntax
df = spark.read.csv(
    "dbfs:/data/raw_transactions.csv",
    header=True,
    inferSchema=True,
    sep=","
)

The DataFrameWriter Interface

# Generic DataFrameWriter syntax
df.write \
    .format("parquet") \
    .mode("overwrite") \
    .option("compression", "snappy") \
    .save("dbfs:/data/processed_transactions.parquet")

# Equivalent format-specific shortcut syntax
df.write.mode("overwrite").parquet("dbfs:/data/processed_transactions.parquet")

---

2. Storage Formats: Row-Oriented, Columnar, and Delta Lake

Choosing the appropriate file format fundamentally dictates storage footprint, read latency, and cluster compute costs.

Why Parquet is the Analytical Gold Standard

Parquet is Apache Spark's default storage format (spark.sql.sources.default=parquet). Unlike CSV or JSON, which store data row-by-row as plain text, Parquet stores data in binary columnar format:

1. Column Pruning: When a query executes df.select("user_id", "amount"), Spark's Parquet reader reads only the byte ranges corresponding to those two columns from disk or cloud storage. All other columns are completely bypassed, drastically reducing network and disk I/O.
2. Predicate Pushdown: Parquet files organize records into horizontal blocks called Row Groups (typically 128 MB each). The Parquet file footer stores metadata statistics—specifically minimum and maximum values—for every column chunk within each row group. When a query contains a filter like .filter("amount > 1000"), Spark inspects the footer metadata first. If a row group's maximum amount is 450, Spark skips reading the entire 128 MB row group entirely.
3. Efficient Encodings: Because columns contain homogeneous data types, Parquet applies dictionary encoding, bit packing, and run-length encoding, often reducing size relative to text; actual ratios depend on data, encoding and codec.

---

3. SaveModes: Controlling Destination Write Behavior

When writing a DataFrame to a destination path or table that already exists, the SaveMode controls how Spark handles the conflict. Specify the mode via df.write.mode(mode_string):

# Example of setting SaveMode
df.write.mode("append").parquet("dbfs:/mnt/datalake/events")

---

4. Physical Storage Layout: Partitioning vs. Bucketing

Organizing physical files on disk to match common query filter and join patterns is the most effective way to eliminate unnecessary I/O.

Directory Partitioning (partitionBy)

df.write.partitionBy(*cols) divides output data into physical subdirectories based on the distinct values of the specified partitioning columns:

# Partitioning transactions by year and month
df.write \
    .partitionBy("year", "month") \
    .parquet("dbfs:/data/transactions")

This creates a nested directory structure on the filesystem:

dbfs:/data/transactions/
├── year=2026/
│   ├── month=09/
│   │   ├── part-00000.snappy.parquet
│   │   └── part-00001.snappy.parquet
│   └── month=10/
│       ├── part-00000.snappy.parquet
│       └── part-00001.snappy.parquet

When downstream queries run df.filter("year = 2026 AND month = 10"), Spark performs partition pruning, reading only files within the year=2026/month=10/ directory and ignoring all other folders.

Table Bucketing (bucketBy and sortBy)

For high-cardinality columns where directory partitioning is unsuitable, use Bucketing:

# Bucketing data by customer_id into 16 hash buckets
df.write \
    .bucketBy(16, "customer_id") \
    .sortBy("transaction_date") \
    .saveAsTable("bucketed_transactions")

Key rules and benefits of bucketing:

1. Potential Join Exchange Reuse: When two tables are bucketed on the same join key with the same number of buckets (or multiples thereof), Spark may avoid a join shuffle when compatible bucketing is recognized and enabled; sort requirements and provider support still apply. Bucket count need not equal physical file count. It can perform a SortMergeJoin with fewer exchanges. Each bucket from Table A joins directly with the corresponding bucket from Table B locally.
2. Persistent Table Requirement: bucketBy() works only with saveAsTable(). It cannot be used with raw file path writes (.save("path")) because bucketing metadata must be tracked in the Spark metastore/catalog.