BigData / Apache Iceberg Interview questions
1. What is Apache Iceberg?
Apache Iceberg is an open-source table format for large, analytical datasets stored in a data lake, adding database-like structure — schema, partitioning, snapshots, ACID transactions — on top of files (typically Parquet) sitting in object storage like S3, GCS, or HDFS. It originated ...
2. What is the purpose of Apache Iceberg?
Apache Iceberg exists to bring reliable, database-like semantics — ACID transactions, consistent snapshots, safe schema and partition changes — to data stored as plain files in a data lake, solving problems that plagued earlier approaches like Hive tables, which tracked table state th...
3. What is a table format, and how does it differ from a file format?
A file format (like Parquet, ORC, or Avro) defines how data is encoded and compressed within a single physical file — how rows and columns are laid out on disk, what compression is used, how to read a specific file efficiently. A table format sits one layer above that: it defines how a coll...
4. What are the key features of Apache Iceberg?
Iceberg combines a specific set of capabilities aimed at bringing warehouse-like reliability to data lake tables, several of which are difficult or impossible to achieve with older, Hive-style table management. Feature What it Provides ACID transactions Atomic, isolated commits via snapshot-based...
5. What is the architecture of an Iceberg table?
An Iceberg table is organized as a tree-shaped hierarchy of metadata layers, each narrowing down from the table as a whole to the specific physical data files a query actually needs to read. flowchart TD A[Catalog: table name to metadata pointer] --> B[Table Metadata File - JSON] B --> C[Snapshot...
6. What is a snapshot in Apache Iceberg?
A snapshot represents the complete, immutable state of a table at one specific point in committed history — every snapshot has a unique ID, a parent snapshot ID, a timestamp, a summary of what changed, and a pointer to the manifest list describing exactly which data files belong to the tabl...
7. What is a manifest file?
A manifest file is an Avro file that lists the actual data files belonging to a specific portion of a snapshot, with one record per data file, including that file's physical path, format, and per-column statistics like value counts, null counts, and min/max bounds. Because a manifest carries min/...
8. What is a manifest list?
A manifest list is an Avro file, one per snapshot, containing one record per manifest file belonging to that snapshot, along with a summary of each manifest's own partition-level bounds — effectively an index one level above the manifest files themselves. This summary is what enables manife...
9. What is the table metadata file?
The table metadata file is a JSON document that serves as the entry point describing a table's full current state: its schema (with stable column IDs, not just names), all partition specs (current and historical), sort order, the list of all snapshots, and a pointer to which snapshot is currently...
10. What is an Iceberg catalog?
A catalog is the component that maps a table name to the location of its current metadata file, serving as the single source of truth for "what is the current state of this table" that every reader and writer consults before doing anything else. Catalog Type Notes Hive Metastore Uses the existing...
11. What is hidden partitioning?
Hidden partitioning is Iceberg's approach to partitioning where the partition values are derived automatically from a table's actual data columns, without exposing separate, artificial partition columns that users have to know about and explicitly filter on in their queries. CREATE TABLE events (...
12. What are partition transforms in Iceberg?
A partition transform is a function applied to a source column's value to derive the actual partition value a row is grouped under — letting a table be partitioned by a meaningful, derived grouping (like the day or month of a timestamp) rather than requiring a raw, pre-computed partition co...
13. What is schema evolution in Iceberg?
Schema evolution is Iceberg's ability to add, drop, rename, reorder, or widen the type of columns in a table's schema without needing to rewrite any of the table's existing data files, since Iceberg tracks a schema's column identity by a stable internal field ID rather than by column name or phys...
14. What is time travel in Apache Iceberg?
Time travel is the ability to query an Iceberg table as it existed at a past point in its committed history, by specifying either a specific snapshot ID or a timestamp, rather than only ever being able to query the table's current, latest state. SELECT * FROM events FOR SYSTEM_TIME AS OF '2026-01...
15. Which query engines support Apache Iceberg?
Because Iceberg is an open specification rather than a single vendor's proprietary format, a broad range of independently-developed query and processing engines support reading and writing Iceberg tables, which is a major part of its appeal for teams wanting to avoid being locked into one specifi...
16. What file formats does Iceberg use to store data?
Iceberg's data files — the actual rows and columns of table data — are most commonly stored as Parquet, though Iceberg also supports ORC and Avro as alternative underlying file formats, since the table format specification is deliberately decoupled from any single physical file format...
17. How do you create an Iceberg table?
Creating an Iceberg table typically means running a standard SQL CREATE TABLE statement through an Iceberg-aware engine like Spark or Trino, specifying the table's schema and, optionally, its initial partition spec using Iceberg's partition transforms. CREATE TABLE catalog .db.events ( id BIGINT,...
18. What is the difference between Iceberg and a Hive table?
Both organize data files in a data lake into a queryable table, but they differ fundamentally in how table state is tracked, which cascades into very different reliability and flexibility characteristics. Hive Table Iceberg Table Table state inferred by listing directories/partitions. Table state...
19. What are field IDs in Iceberg, and why do they matter?
A field ID is a permanent, stable integer identifier assigned to every column (and nested field) in an Iceberg table's schema when it's first created, and this ID — not the column's name or physical position — is what Iceberg actually uses internally to identify a field across schema ...
20. What is ACID compliance in the context of Apache Iceberg?
ACID (Atomicity, Consistency, Isolation, Durability) compliance in Iceberg means that every write to a table — however large or complex — either fully succeeds and becomes visible as a complete, consistent new snapshot, or fully fails and leaves the table exactly as it was before, wit...
21. What is the difference between Apache Iceberg and Delta Lake?
Both are open table formats bringing warehouse-like reliability to data lakes, but they differ in metadata architecture and their degree of coupling to a specific engine, stemming from their different origins — Iceberg from Netflix, Delta Lake from Databricks. Apache Iceberg Delta Lake Tree...
22. What is the difference between Apache Iceberg and Apache Hudi?
Both are open table formats, but Hudi — originating at Uber — was built from the ground up around high-frequency upserts and streaming ingestion, while Iceberg's design historically prioritized simplicity and broad analytical query performance, leading to different strengths. Apache I...
23. Explain how hidden partitioning differs from Hive-style partitioning?
Hive-style partitioning requires a physically separate partition column (often derived and stored by an ETL job ahead of time, like event_date ) baked directly into the directory structure, and users must explicitly filter on that partition column in their queries to get efficient pruning. -- Hiv...
24. What is a lakehouse, and how does Iceberg enable it?
A lakehouse is an architectural pattern combining a data lake's low-cost, open, scalable storage (typically cloud object storage) with a data warehouse's reliability guarantees — ACID transactions, schema enforcement, consistent snapshots — that were historically only available in a p...
25. What is partition evolution, and how does it work internally?
Partition evolution is the ability to change an Iceberg table's partitioning strategy going forward — for example, switching from monthly to daily partitioning as data volume grows — without rewriting any of the table's existing data files, since only new data written after the change...
26. What is the difference between copy-on-write and merge-on-read in Iceberg?
These are two different strategies for handling updates and deletes in Iceberg, trading off write cost against read cost, and the right choice depends on a table's specific ratio of writes to reads. Copy-on-Write (CoW) Merge-on-Read (MoR) Affected data files are entirely rewritten on update/delet...
27. What are positional deletes versus equality deletes?
Both are the two kinds of delete files Iceberg's merge-on-read strategy can write to record removed rows without rewriting the underlying data file, differing in exactly how they identify which rows are deleted. Positional Deletes Equality Deletes Identifies deleted rows by exact file path + row ...
28. Explain the internal working of Iceberg's snapshot isolation mechanism?
Snapshot isolation in Iceberg comes from the combination of immutable snapshots and atomic catalog pointer swaps: a reader that begins a query captures the current snapshot at that moment and continues reading against exactly that snapshot's manifest list and files for the entire duration of the ...
29. What is the REST catalog, and why has it become important?
The REST catalog is a standardized HTTP API specification for Iceberg catalog operations — listing tables, resolving a table's current metadata location, committing new snapshots — letting any client speak one common protocol to interact with a catalog, regardless of what's actually i...
30. What is the difference between a Hive catalog and a REST catalog?
Both serve the same fundamental role — mapping a table name to its current metadata location — but they differ in what technology backs that mapping and how tightly coupled a client needs to be to that specific backing technology. Hive Catalog REST Catalog Backed by an existing Hive M...
31. Explain the execution flow of a query against an Iceberg table?
Running a query against an Iceberg table moves through a defined sequence of metadata resolution steps before any actual data file is ever read, progressively narrowing down exactly which bytes need to be scanned. flowchart TD A[Query submitted with filter predicate] --> B[Catalog resolves curren...
32. How does Iceberg achieve schema evolution without rewriting data?
Schema evolution avoids rewriting data because Iceberg never actually relies on a data file's own embedded schema being identical to the table's current schema — instead, every read reconciles an older file's schema against the current one using stable field IDs, filling in appropriate defa...
33. Explain the internal working of manifest-level partition pruning?
Manifest-level pruning eliminates entire manifest files from consideration during query planning, using only the summary statistics stored in the manifest list, without ever opening the eliminated manifests to inspect their individual data file entries. When Iceberg writes a manifest file, it com...
34. What is compaction in Iceberg, and why is it needed?
Compaction is the maintenance operation of rewriting many small data files into fewer, larger ones, addressing the "small files problem" that naturally accumulates from frequent small writes — streaming ingestion, small batch jobs, or merge-on-read delete files — each of which tends t...
35. How do you perform time travel queries in Iceberg?
Time travel queries use engine-specific SQL syntax to anchor a query to a past snapshot, either by its unique numeric ID or by specifying a timestamp, which Iceberg resolves to whichever snapshot was current as of that moment. -- By snapshot ID SELECT * FROM db.events VERSION AS OF 89347598 ; -- ...
36. What is the difference between a snapshot rollback and time travel?
Both involve accessing a past state of the table, but they differ in whether that past state becomes the table's new current, writable state, or is only being read temporarily without changing what the table's current pointer refers to. Time Travel Rollback Read-only query against a past snapshot...
37. Explain how Iceberg handles concurrent writes?
Iceberg uses optimistic concurrency control for writes: multiple writers can prepare their changes (new data files, new manifests) simultaneously without any locking, and conflicts are only detected and resolved at the final commit step, when a writer attempts to atomically swap the catalog's poi...
38. What is the role of sequence numbers in Iceberg snapshots?
Every snapshot in Iceberg carries a monotonically increasing sequence number, distinct from its snapshot ID, used specifically to determine the relative ordering of data and delete files — which matters most when reconciling merge-on-read delete files against the data files they're meant to...
39. Explain the lifecycle of a write operation (commit) in Apache Iceberg?
A write to an Iceberg table moves through a defined sequence: preparing new data (and, for merge-on-read, delete) files, building new manifest and manifest list metadata describing the resulting state, and finally an atomic commit that swaps the catalog's pointer to a brand-new metadata file. flo...
40. What are branches and tags in Apache Iceberg?
Branches and tags are named references to specific points in a table's snapshot history, giving human-readable, stable names to otherwise opaque numeric snapshot IDs, and differing in whether that reference can continue accumulating new snapshots of its own. ALTER TABLE events CREATE BRANCH audit...
41. How does Iceberg support upserts via MERGE INTO?
Iceberg supports standard SQL MERGE INTO syntax for upserts — conditionally inserting new rows or updating existing ones based on a join condition against a source dataset — letting engines like Spark express complex conditional write logic in familiar SQL rather than requiring separa...
42. What is the small file problem, and how does Iceberg address it?
The small file problem refers to the performance degradation that occurs when a table accumulates a very large number of small data files — commonly from streaming ingestion writing frequent small batches, or from merge-on-read delete files — rather than a smaller number of appropriat...
43. Explain the internal working of column-level statistics in manifest files?
Each data file entry within a manifest file carries per-column statistics computed at write time — value counts, null counts, and, most importantly for query pruning, the minimum and maximum value observed for that column within that specific file. flowchart TD A[Data file written] --> B[Wr...
44. What is the difference between Iceberg V1, V2, and V3 table specs?
Iceberg's table format specification has evolved through versioned revisions, each adding capabilities while remaining a well-defined, backward-compatible evolution rather than a breaking rewrite — a table's spec version is recorded in its metadata, and engines check this version to know wh...
45. How do you migrate an existing Hive table to Iceberg?
Iceberg provides migration procedures specifically for converting an existing Hive (or other legacy) table into an Iceberg table, generally offering two approaches: an in-place migration that reuses existing data files without rewriting them, and a full snapshot/copy migration that creates an ent...
46. What are deletion vectors, and how do they improve on positional delete files?
Deletion vectors are a more efficient representation for tracking row-level deletes than the original positional delete file approach — rather than a full Avro (or Parquet) file per set of deletes with per-row entries, a deletion vector uses a compact, bitmap-like binary structure to record...
47. Explain how Iceberg integrates with Apache Spark for reading and writing?
Spark integrates with Iceberg through a dedicated catalog plugin and SQL extensions, letting standard Spark SQL and DataFrame operations transparently read from and write to Iceberg tables once the appropriate catalog is configured, without needing Iceberg-specific application code for basic oper...
48. What is metadata table querying in Iceberg?
Iceberg exposes a table's own internal metadata — snapshots, manifests, data files, history — as queryable "metadata tables," accessed by appending a suffix to the table name, letting an analyst or engineer inspect a table's structure and history using ordinary SQL rather than needing...
49. How do you troubleshoot slow query planning on a large Iceberg table?
Slow query planning on a large Iceberg table generally traces back to a handful of recurring causes, and working through them systematically — usually starting with the metadata tables covered earlier — is faster than guessing blindly at a fix. Check for a small-files problem: query t...
50. Explain the execution flow of building a streaming lakehouse pipeline with Iceberg and Flink?
A streaming pipeline using Flink and Iceberg continuously ingests events, periodically committing them as new snapshots, while downstream analytical queries read the same table concurrently — combining low-latency ingestion with the same consistent, ACID-compliant table other batch and inte...
