5.5 Apache Iceberg, S3 Tables & Open Table Operations

5.5 Apache Iceberg, S3 Tables & Open Table Operations

Parquet, ORC, and Avro define file encodings. They do not by themselves make a collection of files behave like a database table. Apache Iceberg is an open table format that adds table metadata, snapshots, atomic commits, evolution, and maintenance over files in object storage.

Iceberg metadata model

An Iceberg table points to a current metadata file. That metadata references snapshots, manifest lists, and manifests that identify the exact data and delete files belonging to the table.

A query plans from metadata rather than listing every S3 object. It can prune manifests and files using statistics, which is important at data-lake scale.

Atomic commits and concurrency

An Iceberg writer produces new files and metadata, then atomically swaps the catalog pointer from the old metadata version to the new one. Readers continue seeing a consistent old or new snapshot rather than a half-written table.

Iceberg uses optimistic concurrency. Two writers can work from the same base snapshot, but a conflicting commit is detected when the pointer changes. The losing writer should refresh, validate whether its operation can still apply, and retry. Blindly overwriting the metadata location can lose another writer's changes.

This pattern supports append, overwrite, merge, update, and delete behavior when the engine and catalog implement the operation. ACID at the table level does not create a transaction across several unrelated tables; multi-table publication still needs orchestration or a higher-level manifest.

Schema and partition evolution

Iceberg assigns stable IDs to fields. Renaming a column does not have to look like dropping one field and adding an unrelated field, and compatible schema changes can be applied without rewriting all existing files. Type compatibility still matters; a narrowing conversion can lose data.

Partition evolution lets a table change its partition specification for future writes. Old files retain their former layout while queries use table metadata to plan both versions. This is different from Hive-style partition paths, where the query often depends directly on folder names. Iceberg supports transforms such as day, month, bucket, and truncate so consumers filter logical columns rather than constructing physical prefixes.

Snapshots, time travel, and rollback

Every successful commit creates a snapshot. A query engine can read a table as of a snapshot or timestamp for debugging, audit, or recovery. A rollback changes the current table state to a prior snapshot; it does not necessarily delete newer files immediately.

Snapshot expiration is an operational decision. Retaining every snapshot forever increases metadata and storage. Expiring too aggressively removes the ability to time travel and can make delayed jobs unable to find their base state. Align retention with recovery, legal-hold, and audit requirements before cleanup.

File maintenance

Frequent streaming writes create small files. Compaction rewrites them into fewer larger files, reducing query planning and S3 request overhead. It must preserve table semantics, apply relevant deletes, and commit a new snapshot. Maintenance competes with application writers, so both paths need conflict retry.

Other maintenance includes snapshot expiration and removal of unreferenced files. Never delete objects by scanning an Iceberg prefix with a generic script: files not referenced by the current snapshot can still be referenced by an older retained snapshot.

Amazon S3 Tables

S3 table buckets are an S3 bucket type for tables. Every S3 table uses Apache Iceberg. S3 manages maintenance operations including compaction and snapshot management, and table-bucket maintenance can remove unreferenced files. Automatic compaction is enabled by default and uses a configurable target file size; current defaults and allowed ranges belong in service configuration, not hard-coded application assumptions.

Integrating a table bucket with the AWS Glue Data Catalog makes the tables discoverable to services such as Athena and Redshift. Open-source engines can use the S3 Tables Catalog integration for Iceberg. Table-bucket permissions use S3 Tables actions and resource types, which differ from granting GetObject on an ordinary general-purpose bucket.

Choose self-managed Iceberg in a general-purpose bucket when you need engine-specific catalog or maintenance control. Choose S3 Tables when the supported integrations fit and managed table maintenance reduces operational burden.

Exam traps

Iceberg is not a file compression codec. A crawler finding Parquet files does not automatically create an Iceberg transaction log. Partition evolution is not the same as renaming S3 folders. And time travel depends on retained snapshots and files; a lifecycle rule that independently deletes table objects can corrupt the table.