8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

DP-750 Exam Focus: The blueprint bullet is "Choose a data table format, such as Parquet, Delta, CSV, JSON, or Iceberg." The exam is not asking you to like Delta - it is asking you to say which format satisfies a stated requirement, and to know when Iceberg is the answer.

---

1. Two Different Kinds of "Format"

The bullet mixes two categories, and conflating them is the most common error:

• File formats - CSV, JSON, Parquet, Avro, ORC. These describe how bytes are laid out in one file. They carry no transaction log, so concurrent writers can corrupt a read, and there is no versioning.
• Table formats - Delta Lake and Apache Iceberg. These add a metadata layer over Parquet files that provides ACID transactions, schema evolution, time travel, and statistics for file skipping.

---

2. Where Each Format Belongs

The trap. Scenarios describing "we keep the bronze layer as compressed JSON to save conversion cost" are describing a defect, not an optimization: no ACID guarantees, no time travel, no file skipping, and every downstream read re-parses text. Ingest into Delta at the bronze boundary.

---

3. Apache Iceberg in Unity Catalog

Azure Databricks supports Iceberg tables that use the Apache Parquet file format and versions 1, 2, and 3 of the Iceberg specification. Iceberg maintains atomicity and consistency by writing new metadata files for each table change, and the Iceberg catalog is the layer that returns current metadata when a table is loaded.

Two distinct kinds exist, and the exam distinguishes them sharply:

Managed Iceberg Tables (Unity Catalog owns them)

Requirements:

• A workspace with Unity Catalog enabled
• Databricks Runtime 16.4 LTS or above
• A workspace with serverless compute enabled - Databricks uses serverless to maintain Iceberg table metadata, so serverless must have network connectivity to the backing storage account
• Predictive optimization enabled for table maintenance; managed Iceberg tables can only be created when it is on

What you get in return: Unity Catalog handles lifecycle tasks such as snapshot expiration and file compaction, liquid clustering works, predictive optimization automates maintenance, materialized views support incremental refresh, and streaming tables support incremental loading from Kafka and cloud object storage.

-- A managed Iceberg table is created like any other Unity Catalog table
CREATE TABLE prod_retail.gold.orders_iceberg (
  order_id   BIGINT,
  store_id   INT,
  order_ts   TIMESTAMP,
  net_amount DECIMAL(18,2)
) USING ICEBERG;

Foreign Iceberg Tables (another catalog owns them)

A foreign Iceberg table is managed by a catalog outside Unity Catalog - AWS Glue, a Hive metastore, or Snowflake Horizon Catalog. The external catalog holds the current metadata, and Azure Databricks uses Lakehouse Federation (Section 3.5) to retrieve that metadata and read the table from object storage. Foreign Iceberg tables are read-only and have limited platform support.

Access From External Iceberg Engines

Every Iceberg table in Unity Catalog is reachable through the Iceberg REST Catalog API, which supports read and write from external engines such as Apache Spark, Flink, Trino, and Kafka. The REST Catalog supports credential vending, delivering temporary credentials so an external engine can reach the underlying storage without a long-lived key. Credential vending is not supported on workspaces that use default storage.

---

4. Iceberg Limitations Worth Memorizing

• Iceberg tables support only the Apache Parquet file format.
• For Iceberg v2, position deletes and equality deletes are not supported; Azure Databricks supports Iceberg v3 deletion vectors for row-level deletion instead (Section 8.3).
• Branching and tagging are not supported. Only the main branch is accessible when reading foreign Iceberg tables.
• Partition evolution works on managed Iceberg tables only when driven from external Iceberg engines through the REST Catalog, not from Databricks SQL. Foreign Iceberg tables do not support it at all.
• Partitioning by expression transforms (years(), months(), days(), hours(), bucket()) is not supported for managed Iceberg tables, and partitioning by BINARY is unsupported generally.
• Views are not accessible from external Iceberg engines.
• Unsupported data types: UUID, Fixed(L), TIME, and nested STRUCT with required fields.
• Managed Iceberg tables cannot use Delta Lake generated columns, constraints, or collation, because Apache Iceberg does not implement them. All managed Iceberg tables use Zstd compression and the codec cannot be changed.
• Foreign Iceberg tables support time travel only for snapshots previously read in Azure Databricks.

Constraints matter more than they look. Section 9.5 builds data quality on CHECK constraints and NOT NULL invariants. Those are Delta Lake features. Choosing managed Iceberg for a silver table gives up the constraint layer - a real architectural trade-off, not a footnote.

---

5. The Decision Table

---

6. Exam Traps

• "Use Parquet for the silver layer for performance" is wrong. Parquet without a table format has no ACID guarantees, no MERGE, and no time travel.
• Managed Iceberg is not a runtime toggle. It needs UC, DBR 16.4 LTS or above, serverless compute, and predictive optimization.
• Foreign Iceberg tables cannot be written to. They are read-only, like every other federated object.
• Iceberg v2 row-level deletes are not the answer on Databricks. Deletion vectors under Iceberg v3 are.