BigData / Apache Hudi Interview Questions
Explain the internal working of Hudi's incremental query mechanism?
Incremental queries work because the timeline already records, instant by instant, exactly which files were touched by each commit — so "what changed between commit A and commit B" is a metadata lookup, not a data scan.
flowchart TD
A[User specifies begin/end commit time] --> B[Read timeline metadata for instants in that range]
B --> C[Collect the list of files written/updated by those instants]
C --> D[Read only those specific files, not the full table]
D --> E[Return changed records to the consumer]
Given a begin and end commit time, Hudi reads the timeline's metadata for every instant in that range and collects the specific files each one touched — base files for CoW, log blocks for MoR. It then reads only those files rather than the entire table, and returns the resulting records to the caller. This is why an incremental query's cost tracks the amount of change rather than total table size: the timeline metadata already tells Hudi precisely where to look, without any full scan.
An incremental query determines what changed by reading:
This is why an incremental query's cost scales with:
More Related questions...
