BigData / Apache Hudi Interview Questions
Explain the internal working of the LSM-tree-based timeline in Hudi 1.x?
The original timeline stored each instant as its own small file directly in the table's .hoodie directory on object storage. That worked fine for tables with dozens or hundreds of commits, but at thousands of commits it turned into a small-file listing problem of its own — just for the metadata layer.
Hudi 1.x re-architects the timeline using an LSM (Log-Structured Merge) tree design: instead of one file per instant, new instant metadata is appended to a small active log, which periodically gets compacted and merged into progressively larger, sorted metadata files — the same fundamental strategy databases like Cassandra or RocksDB use for write-heavy key-value storage. The result is far fewer, larger metadata files to list and read, which significantly improves the performance of any timeline operation (finding the latest instant, scanning history for an incremental query, and so on) on tables with a long commit history.
More Related questions...
