What is compaction in Iceberg, and why is it needed?

Compaction is the maintenance operation of rewriting many small data files into fewer, larger ones, addressing the "small files problem" that naturally accumulates from frequent small writes — streaming ingestion, small batch jobs, or merge-on-read delete files — each of which tends to produce its own comparatively small file.

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  options => map('target-file-size-bytes', '536870912')
);

A table with a very large number of small files carries real overhead: query planning has more manifest entries to evaluate, and the actual scan phase has more file-open operations to perform (each with its own fixed overhead), both of which drag down performance compared to reading the same total amount of data from a smaller number of appropriately-sized files.

Compaction procedures (commonly invoked via a stored procedure call like the example above, in engines that support it) rewrite qualifying small files into new, larger files sized toward a configured target, and commit that rewrite as a new snapshot — the old small files remain referenced by older snapshots until those are eventually expired, but new queries against the current snapshot benefit immediately from the more efficient, consolidated file layout.