8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log

Delta Lake is the core storage technology powering the Azure Databricks Lakehouse architecture. Traditional cloud data lakes built on raw formats (such as raw CSV, JSON, or vanilla Apache Parquet) suffer from significant operational limitations: lack of ACID transactions, inability to handle concurrent append and update operations, painful metadata bottlenecks during directory listings on cloud object storage, and frequent data corruption from failed pipeline jobs.

Delta Lake resolves these challenges by coupling standard Apache Parquet payload files with a structured, transactionally guaranteed metadata layer housed in the _delta_log/ directory. Understanding this underlying storage architecture is essential for designing resilient data engineering pipelines and excelling on the DP-750 exam.

---

1. Delta Lake Physical Storage Architecture

A Delta Lake table stored in Azure Data Lake Storage Gen2 (ADLS Gen2) consists of two distinct components residing in the table root directory:

abfss://container@storageaccount.dfs.core.windows.net/tables/sales/
├── _delta_log/
│   ├── 00000000000000000000.json
│   ├── 00000000000000000001.json
│   ├── ...
│   ├── 00000000000000000010.json
│   ├── 00000000000000000010.checkpoint.parquet
│   └── _last_checkpoint
├── part-00000-4b2e1f2a-c000.snappy.parquet
├── part-00001-9c3f2e1d-c000.snappy.parquet
└── part-00002-7a1b4c8e-c000.snappy.parquet

1. Parquet Payload Data Files

• Delta Lake persists raw table rows in standard, immutable Apache Parquet format compressed with Snappy by default (or zstd).
• Parquet is a columnar binary format providing efficient column pruning, vectorized decompression, and dictionary encoding.
• Parquet files in Delta Lake are append-only and immutable. Updating or deleting a record never modifies an existing Parquet file in place; instead, new Parquet files are written and the transaction log updates pointers to reference the new files while marking the old files as logically removed.

2. The _delta_log/ Transaction Log

• The _delta_log/ subdirectory resides at the root of the Delta table and acts as the single source of truth for table state, metadata, and data file references.
• Every modification to the table (such as an INSERT, UPDATE, DELETE, MERGE, OPTIMIZE, or schema change) constitutes a discrete, atomic transaction that generates a new JSON commit log file.
• Commit files follow a strict zero-padded 20-digit numerical naming convention starting at 00000000000000000000.json.

---

2. Anatomy of the Transaction Log: Commit Actions

Within each JSON commit file, Delta Lake records an array of atomic metadata actions. If a transaction succeeds, all of its actions are written to the JSON file; if a transaction fails, no commit file is created (guaranteeing atomicity).

+-------------------------------------------------------------------------+
|                   DELTA TRANSACTION LOG COMMIT ACTIONS                  |
+-------------------------------------------------------------------------+
|  1. add            | Registers a newly created Parquet file + stats     |
|  2. remove         | Marks an existing Parquet file as logically deleted|
|  3. metaData       | Records schema, partition columns, table properties|
|  4. protocol       | Defines minimum reader/writer protocol versions    |
|  5. setTransaction | Tracks streaming transaction IDs for idempotency   |
|  6. commitInfo     | Audit info: user, timestamp, operation type, params|
+-------------------------------------------------------------------------+

Detailed Breakdown of Core Actions

1. add Action

Registers a new Parquet file as part of the active table state. It includes file metadata and embedded column statistics:

{
  "add": {
    "path": "part-00000-4b2e1f2a-c000.snappy.parquet",
    "partitionValues": {},
    "size": 15482910,
    "modificationTime": 1724673600000,
    "dataChange": true,
    "stats": "{\"numRecords\":100000,\"minValues\":{\"id\":1,\"order_date\":\"2026-08-01\"},\"maxValues\":{\"id\":100000,\"order_date\":\"2026-08-26\"},\"nullCount\":{\"id\":0,\"customer_id\":12}}"
  }
}

2. remove Action

Logically deletes a file from the active state without physically removing it from ADLS Gen2 storage:

{
  "remove": {
    "path": "part-00000-old-file-guid.snappy.parquet",
    "deletionTimestamp": 1724673600000,
    "dataChange": true,
    "extendedFileMetadata": true,
    "partitionValues": {},
    "size": 14205800
  }
}

3. metaData Action

Defines the table schema, partition columns, table name, and configuration properties:

{
  "metaData": {
    "id": "c8b321a4-9e32-4d2a-89a1-5d93b91a7420",
    "format": {"provider": "parquet", "options": {}},
    "schemaString": "{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":false},{\"name\":\"amount\",\"type\":\"double\",\"nullable\":true}]}",
    "partitionColumns": [],
    "configuration": {"delta.enableChangeDataFeed": "true"},
    "createdTime": 1724673500000
  }
}

4. commitInfo Action

Provides provenance and auditability for table history (viewable via DESCRIBE HISTORY table_name):

{
  "commitInfo": {
    "timestamp": 1724673600000,
    "operation": "MERGE",
    "operationParameters": {"predicate": "(target.id = source.id)"},
    "job": {"jobId": "89421", "jobName": "Daily_Ingestion_Job"},
    "engineInfo": "Databricks-Runtime/15.4.x-photon-scala2.12",
    "operationMetrics": {"numTargetRowsInserted": "500", "numTargetRowsUpdated": "1200"}
  }
}

---

3. State Reconstruction & Checkpoint Compaction

When a Spark cluster queries a Delta table, the Spark driver must reconstruct the latest snapshot of the table. A naive log replay would require reading every single JSON commit file from version 0 to version $N$, which would introduce crippling latency for tables with thousands of commits.

The 10-Commit Checkpoint Mechanism

To guarantee fast snapshot reconstruction, Delta Lake creates a checkpoint file every 10 commits:

1. Trigger: At commit 10 (and every subsequent multiple of 10: 20, 30, 40...), the writer engine reads commits 00000000000000000000.json through 00000000000000000010.json and computes the exact set of active Parquet files by canceling out matched add and remove actions.
2. Parquet Checkpoint: The driver writes this resolved state into a single compacted Parquet file: 00000000000000000010.checkpoint.parquet.
3. _last_checkpoint Pointer: Delta Lake updates a lightweight pointer file named _last_checkpoint in _delta_log/ containing JSON metadata pointing to the latest checkpoint version:

{"version":10,"size":245,"sizeInBytes":14280,"numOfAddFiles":182}

Snapshots at Version 14 Reconstruction:
1. Read `_last_checkpoint` -> Identifies Version 10 Checkpoint.
2. Read `00000000000000000010.checkpoint.parquet` (Loads baseline state of 182 active files).
3. Replay only `00000000000000000011.json` through `00000000000000000014.json`.
4. Result: Table snapshot constructed in milliseconds by scanning only 5 files instead of 15!

Exam Tip: The DP-750 exam tests your knowledge of how checkpoint files improve query planning. Checkpoints prevent driver out-of-memory errors and eliminate slow cloud directory listing bottlenecks by consolidating historical commit logs every 10 transactions.

---

4. Metadata-Based Data Skipping

When writing new Parquet files, Delta Lake automatically collects and embeds column statistics into the stats field of the add action:

• numRecords: Total row count in the file.
• minValues: Minimum value for each column in the file.
• maxValues: Maximum value for each column in the file.
• nullCount: Count of NULL entries per column.

How File Skipping Operates During Query Planning

When an analytical query executes with a filter predicate (e.g., WHERE order_date >= '2026-08-20' AND customer_id = 4501), the Spark optimizer evaluates the query predicate against the minValues and maxValues stored in the transaction log.

Query Predicate: WHERE order_date >= '2026-08-20'

File A Stats: min = '2026-08-01', max = '2026-08-15' --> SKIPPED (Max < 2026-08-20)
File B Stats: min = '2026-08-10', max = '2026-08-25' --> READ    (Overlap exists)
File C Stats: min = '2026-08-21', max = '2026-08-26' --> READ    (Min >= 2026-08-20)

Because statistics are stored directly in the _delta_log metadata, Spark can discard 90%+ of Parquet files before reading a single byte of data from storage, dramatically slashing I/O cost and latency.

Configuration of Indexed Columns

By default, Delta Lake collects statistics for the first 32 columns defined in the table schema. For wide tables (e.g., 100+ columns), if frequently filtered columns reside beyond column 32, you can adjust this limit via table properties:

ALTER TABLE sales_delta_table 
SET TBLPROPERTIES ('delta.dataSkippingNumIndexedCols' = '64');

---

5. Schema Enforcement vs. Schema Evolution

Delta Lake provides robust governance mechanisms to balance data quality integrity against changing business schemas.

+-------------------------------------------------------------------------+
|                 SCHEMA ENFORCEMENT VS. SCHEMA EVOLUTION                 |
+-------------------------------------------------------------------------+
|  SCHEMA ENFORCEMENT (DEFAULT)       | SCHEMA EVOLUTION (EXPLICIT)       |
|  - Rejects incoming bad data        | - Dynamically adds new columns    |
|  - Throws AnalysisException         | - Enabled via mergeSchema option  |
|  - Prevents schema pollution        | - Safely handles evolving sources |
+-------------------------------------------------------------------------+

1. Schema Enforcement (Schema Validation)

• Default Behavior: Delta Lake validates every write operation against the existing table schema registered in metaData.
• If an incoming DataFrame contains columns not present in the target schema, or if column data types are incompatible (e.g., attempting to write a StringType into a LongType column), Delta Lake aborts the transaction immediately and throws an AnalysisException.
• Benefit: Guarantees downstream Gold tables and BI dashboards never encounter unexpected schema corruptions.

2. Schema Evolution (mergeSchema)

When schema changes are intentional (such as upstream API additions), Delta Lake allows seamless schema evolution:

In PySpark DataFrame Writes:

# Explicitly allow schema evolution during append or overwrite
df_incoming.write \
    .format("delta") \
    .mode("append") \
    .option("mergeSchema", "true") \
    .saveAsTable("bronze_telemetry")

In Spark SQL Session Configuration:

-- Enable automatic schema merging for all subsequent SQL operations
SET spark.databricks.delta.schema.autoMerge.enabled = true;

In MERGE INTO Upsert Statements:

MERGE WITH SCHEMA EVOLUTION INTO target_table AS t
USING source_updates AS s
ON t.user_id = s.user_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Rules of Delta Schema Evolution

• Allowed Operations: Adding new nullable columns, adding nested struct fields, and widening types (e.g., ByteType $\rightarrow$ ShortType $\rightarrow$ IntegerType $\rightarrow$ LongType under Type Widening).
• Disallowed Operations: Changing incompatible data types (e.g., StringType to DoubleType), dropping existing columns, or renaming columns (unless Column Mapping is explicitly enabled via delta.columnMapping.mode = 'name').