BigData / Apache Iceberg Interview questions
Explain the internal working of column-level statistics in manifest files?
Each data file entry within a manifest file carries per-column statistics computed at write time — value counts, null counts, and, most importantly for query pruning, the minimum and maximum value observed for that column within that specific file.
At query planning time, the engine compares a filter's predicate against each candidate file's recorded min/max bounds for the relevant column: if a file's maximum value for price is provably below a filter's WHERE price > 1000 threshold, that file cannot possibly contain any matching rows and can be safely skipped without ever opening it — the file-level equivalent of the same pruning logic manifest lists apply one level higher, at the manifest granularity.
Because these statistics are computed once at write time and simply read from the manifest at query time (rather than requiring the engine to open every data file just to check whether it could possibly match), this pruning is extremely cheap relative to actually scanning file contents, which is why well-organized tables (sorted or clustered on frequently-filtered columns) benefit so much from Iceberg's statistics-driven pruning — tighter, more meaningful min/max ranges per file directly translate into more files being safely skipped.
More Related questions...