BigData / Apache Iceberg Interview questions
How does Iceberg achieve schema evolution without rewriting data?
Schema evolution avoids rewriting data because Iceberg never actually relies on a data file's own embedded schema being identical to the table's current schema — instead, every read reconciles an older file's schema against the current one using stable field IDs, filling in appropriate defaults (typically null) for any column that didn't exist when that file was written.
Because the reconciliation happens per-file at read time based on field IDs (rather than requiring every file to physically conform to one single, current schema), adding a column is purely a metadata operation: the new field ID is registered in the table's current schema, and any file lacking that ID is simply treated as having null for it, with no need to touch the file itself.
The same field-ID-based mechanism handles renames (updating which name maps to an existing ID, with no data change) and compatible type widening (like int to long), while genuinely incompatible changes — changing a column's type in an unsafe, non-widening way — are either disallowed or require an explicit, deliberate data migration, since no metadata-only trick can safely reinterpret data that was physically encoded under a fundamentally different, incompatible type.
More Related questions...