What is the small file problem, and how does Iceberg address it?

The small file problem refers to the performance degradation that occurs when a table accumulates a very large number of small data files — commonly from streaming ingestion writing frequent small batches, or from merge-on-read delete files — rather than a smaller number of appropriately-sized ones.

Small files hurt performance on two fronts: query planning has more manifest entries to evaluate and prune through, and query execution has more individual file-open operations to perform, each carrying its own fixed overhead that doesn't shrink proportionally just because the file itself is small — the cumulative effect can mean reading the same total volume of data takes meaningfully longer from many small files than from fewer large ones.

Iceberg addresses this primarily through compaction (the rewrite_data_files procedure), which consolidates many small files into fewer, appropriately-sized ones as a routine maintenance operation, and through configurable target file sizes on write, which can reduce how aggressively small files accumulate in the first place; some engines and platforms also support automatic, scheduled compaction, removing the need for a team to manually trigger this maintenance on a regular basis.