BigData / Apache Hudi Interview Questions
Why do MoR tables need periodic compaction?
Because MoR writes updates to append-only Avro log files instead of merging them into Parquet immediately, those log files keep growing with every batch of updates to a given file slice if nothing ever merges them back.
Left unchecked, this creates two problems: snapshot queries get progressively slower, since more and more log data must be merged with the base file at read time; and the file slice itself becomes harder to manage operationally. Periodic compaction resets this by folding accumulated logs into a fresh base file version, which is why compaction scheduling (how often, and how much log data to allow before compacting) is one of the main tuning levers for a MoR table.
Without compaction, MoR log files would:
Compaction scheduling is a key tuning lever because it controls:
More Related questions...
