BigData / Apache Iceberg Interview questions
How does Iceberg support upserts via MERGE INTO?
Iceberg supports standard SQL MERGE INTO syntax for upserts — conditionally inserting new rows or updating existing ones based on a join condition against a source dataset — letting engines like Spark express complex conditional write logic in familiar SQL rather than requiring separate, hand-coded insert and update operations.
MERGE INTO events t USING updates s ON t.id = s.id WHEN MATCHED THEN UPDATE SET t.data = s.data WHEN NOT MATCHED THEN INSERT (id, event_time, data) VALUES (s.id, s.event_time, s.data);
Under the hood, executing a MERGE INTO resolves to either a copy-on-write or merge-on-read execution path depending on the table's configured write strategy: under copy-on-write, matched data files are fully rewritten with the updated/inserted rows incorporated; under merge-on-read, the matched rows are recorded as delete files (for the old values) alongside new data files (for the updated/inserted values), deferring reconciliation to read time.
Because MERGE INTO is standard, portable SQL rather than an Iceberg-specific extension, the same upsert logic generally works consistently across the different engines that support Iceberg (Spark, Trino, Flink SQL), which matters for teams that want their upsert pipelines to remain engine-agnostic rather than tightly coupled to one specific tool's proprietary upsert API.
More Related questions...