Prev Next

BigData / Apache Iceberg Interview questions

Explain the lifecycle of a write operation (commit) in Apache Iceberg?

A write to an Iceberg table moves through a defined sequence: preparing new data (and, for merge-on-read, delete) files, building new manifest and manifest list metadata describing the resulting state, and finally an atomic commit that swaps the catalog's pointer to a brand-new metadata file.

flowchart TD A[Writer begins operation, reads current snapshot as base] --> B[New/changed rows written to new data files] B --> C{Merge-on-read delete needed?} C -- Yes --> D[Delete files written for removed/changed rows] C -- No --> E[Copy-on-write: affected files fully rewritten instead] D --> F[New manifest files created/reused referencing all files] E --> F F --> G[New manifest list created for this operation] G --> H[New table metadata JSON written, referencing new snapshot] H --> I[Atomic commit: catalog pointer swapped to new metadata file] I --> J{Commit succeeded - base snapshot still current?} J -- Yes --> K[New snapshot now current; readers see complete new state] J -- No, conflict --> L[Commit rejected; writer retries against new current snapshot]

Everything up through building the new manifest list and metadata file happens without touching the catalog at all — it's only the final step, the atomic pointer swap, that actually makes the new state visible to other readers and writers, which is exactly what gives Iceberg its all-or-nothing commit guarantee: any failure before that final atomic swap simply leaves the table's current state completely untouched.

This design also means a failed or aborted write leaves no partial trace in the table's visible state: orphaned data files from an incomplete write might exist in storage, but since no committed snapshot ever references them, they're invisible to any query and can be safely cleaned up later by a maintenance operation without affecting the table's correctness.

What is the only step that actually makes a new write visible to other readers?
What happens to data files from a failed, never-committed write?

More Related questions...

What is Apache Iceberg? What is the purpose of Apache Iceberg? What is a table format, and how does it differ from a file format? What are the key features of Apache Iceberg? What is the architecture of an Iceberg table? What is a snapshot in Apache Iceberg? What is a manifest file? What is a manifest list? What is the table metadata file? What is an Iceberg catalog? What is hidden partitioning? What are partition transforms in Iceberg? What is schema evolution in Iceberg? What is time travel in Apache Iceberg? Which query engines support Apache Iceberg? What file formats does Iceberg use to store data? How do you create an Iceberg table? What is the difference between Iceberg and a Hive table? What are field IDs in Iceberg, and why do they matter? What is ACID compliance in the context of Apache Iceberg? What is the difference between Apache Iceberg and Delta Lake? What is the difference between Apache Iceberg and Apache Hudi? Explain how hidden partitioning differs from Hive-style partitioning? What is a lakehouse, and how does Iceberg enable it? What is partition evolution, and how does it work internally? What is the difference between copy-on-write and merge-on-read in Iceberg? What are positional deletes versus equality deletes? Explain the internal working of Iceberg's snapshot isolation mechanism? What is the REST catalog, and why has it become important? What is the difference between a Hive catalog and a REST catalog? Explain the execution flow of a query against an Iceberg table? How does Iceberg achieve schema evolution without rewriting data? Explain the internal working of manifest-level partition pruning? What is compaction in Iceberg, and why is it needed? How do you perform time travel queries in Iceberg? What is the difference between a snapshot rollback and time travel? Explain how Iceberg handles concurrent writes? What is the role of sequence numbers in Iceberg snapshots? Explain the lifecycle of a write operation (commit) in Apache Iceberg? What are branches and tags in Apache Iceberg? How does Iceberg support upserts via MERGE INTO? What is the small file problem, and how does Iceberg address it? Explain the internal working of column-level statistics in manifest files? What is the difference between Iceberg V1, V2, and V3 table specs? How do you migrate an existing Hive table to Iceberg? What are deletion vectors, and how do they improve on positional delete files? Explain how Iceberg integrates with Apache Spark for reading and writing? What is metadata table querying in Iceberg? How do you troubleshoot slow query planning on a large Iceberg table? Explain the execution flow of building a streaming lakehouse pipeline with Iceberg and Flink?
Show more question and Answers...

Web

Comments & Discussions