BigData / Apache Hudi Interview Questions
Why does Hudi need indexing for upserts?
An upsert has to answer one question fast for every incoming record: "which existing file group, if any, already holds this record's key?" Without an index, Hudi would have to scan the entire table (or at least the entire partition) on every single upsert batch to find that out, which doesn't scale.
Indexing solves this by maintaining a fast mapping from record key (and optionally partition path) to file group, so incoming records can be "tagged" with their target file location in close to constant time instead of a full scan. This tagging step happens before the actual write, and it's the single biggest factor in how fast Hudi's upserts run at scale.
Without an index, an upsert would need to:
The indexing step "tags" incoming records with their:
More Related questions...
