BigData / Apache Hudi Interview Questions
1. What is Apache Hudi?
Apache Hudi is an open-source data lakehouse platform built around a high-performance table format that brings database-like capabilities — record-level inserts, updates, and deletes — to data stored on cloud object storage such as S3, GCS, and Azure Blob Storage. It was originally bu...
2. What does the acronym Hudi stand for?
Hudi stands for Hadoop Upserts Deletes and Incrementals . The name captures its three founding goals: bringing upsert and delete operations to Hadoop-compatible storage, and enabling incremental processing instead of full-table batch rewrites. The project's original internal codename at Uber was ...
3. What are the key features of Apache Hudi?
A few core capabilities come up repeatedly in interviews because they're what distinguish Hudi from a plain Parquet-on-S3 data lake. Fast, pluggable upserts and deletes — record-level mutations instead of rewriting whole partitions. ACID transactions — snapshot isolation and non-block...
4. What is a data lakehouse, and how does Hudi fit into that model?
A data lakehouse combines the low-cost, scalable storage of a data lake (files sitting on object storage) with the transactional reliability, schema management, and query performance traditionally associated with a data warehouse. Hudi fits into this model as the table format layer sitting direct...
5. What are the two table types supported by Hudi?
Hudi supports exactly two table types, chosen when a table is created and fixed for the lifetime of that table : Copy-on-Write (CoW) and Merge-on-Read (MoR) . Both support the same upsert/delete API and the same indexing and table services; what differs is when the cost of merging an update into ...
6. What is Copy-on-Write (CoW) in Hudi?
In a Copy-on-Write table, every update is merged into the affected columnar (Parquet) file immediately at write time , producing a brand-new version of that file rather than editing it in place. The result is a table made up entirely of plain Parquet files, so any engine that can read Parquet rea...
7. What is Merge-on-Read (MoR) in Hudi?
In a Merge-on-Read table, updates aren't merged into the base Parquet file right away. Instead they're appended to small row-based Avro log files attached to that file group, and the merge is deferred until read time or until a background compaction runs. This makes writes much cheaper — ap...
8. What is a HoodieKey in Hudi?
A HoodieKey is the combination of a record key and a partition path that uniquely identifies a single record within a Hudi table. It's the primary mechanism Hudi uses to decide whether an incoming row is a brand-new insert or an update to an existing row. The record key can be a single column or ...
9. What is the Hudi timeline?
The timeline is an ordered log of every action ever performed on a Hudi table — commits, delta commits, cleans, compactions, clusterings, rollbacks, and savepoints — each recorded as an instant with a monotonically increasing timestamp. Each instant moves through states — reques...
10. What are the main write operations supported by Hudi?
Hudi exposes a small set of write operations that cover most ingestion patterns: insert — adds new records without checking for duplicates against existing data. upsert — the default operation; inserts new records and updates existing ones based on the HoodieKey. bulk_insert — a...
11. What is HoodieStreamer (formerly DeltaStreamer)?
HoodieStreamer — called DeltaStreamer before Hudi 1.x renamed it — is Hudi's built-in ingestion tool for continuously or one-time moving data from an external source into a Hudi table. It can pull from sources like Kafka, DFS files, or JDBC, apply an optional transformer and schema (i...
12. What is compaction in Hudi?
Compaction is a background table service, specific to Merge-on-Read tables, that merges the row-based Avro log files accumulated for a file slice into its Parquet base file, producing a new, fully-merged columnar version. Without compaction, a MoR table's log files would keep growing indefinitely...
13. What is clustering in Hudi?
Clustering is a table service that reorganizes existing data files to improve layout and query performance, without changing the table's logical content — no records are added, removed, or modified. Typical clustering work includes merging many small files into fewer larger ones, and sortin...
14. What is the purpose of the cleaner service in Hudi?
The cleaner is a background table service that removes older file versions that are no longer needed once they fall outside the table's configured retention policy, reclaiming storage that would otherwise grow unbounded. Because CoW writes a whole new file version on every update, and MoR periodi...
15. List the query types supported by Apache Hudi?
Hudi offers three distinct query types, and knowing when each applies is a common interview probe: Snapshot query — returns the latest committed, fully-merged state of the table. Works on both CoW and MoR tables. Incremental query — returns only the records that changed between two sp...
16. Define the Hudi Metadata Table?
The Metadata Table (MDT) is a Hudi-managed, internal key-value table — backed by the HFile format — that stores metadata needed for table operations directly inside the table itself, instead of relying on repeatedly listing files from cloud storage. It holds things like file listings,...
17. Describe the history and origin of Apache Hudi?
Hudi was developed at Uber in 2016 under the internal codename "Hoodie," built to solve a real scaling problem: Uber's largest Spark jobs were using over 1,000 executors to rewrite entire datasets just to absorb upstream inserts, updates, and deletes. Uber open-sourced Hudi in 2017 , then donated...
18. What is a payload class in Hudi?
A payload class defines the exact merge logic Hudi applies when an incoming record shares a HoodieKey with an existing record — in other words, it decides which values win when an update is applied. The default, OverwriteWithLatestAvroPayload , simply keeps whichever record has the latest v...
19. Why does Hudi need indexing for upserts?
An upsert has to answer one question fast for every incoming record: "which existing file group, if any, already holds this record's key?" Without an index, Hudi would have to scan the entire table (or at least the entire partition) on every single upsert batch to find that out, which doesn't sca...
20. How does the Bloom index work in Hudi?
The Bloom index stores a bloom filter — a probabilistic, space-efficient data structure — along with min/max record-key range stats for each data file, historically in the file's footer and now more efficiently in the Metadata Table's dedicated bloom filter partition. To tag an incomi...
21. What is the difference between a global index and a partition-level index?
A partition-level (local) index assumes the record's partition path is already known and only checks files within that specific partition for a key match — fast, but it breaks if a record's partition value can change between updates. Partition-level index Global index Only scans files withi...
22. How does the Record-Level Index (RLI) improve on the Bloom index?
The Record-Level Index (RLI) , added in Hudi 0.14.0, stores a direct record key → file group mapping inside the Metadata Table, backed by the HFile format, giving O(1) key lookups instead of the range-pruning-plus-probabilistic-check approach the Bloom index uses. Because it's a global index...
23. When should you choose Copy-on-Write over Merge-on-Read?
CoW is the better fit when a workload is read-heavy and update volume is moderate — dashboards, BI queries, or any consumer that wants native Parquet read speed with zero merge overhead at query time. It also simplifies operations: there's no compaction service to schedule and tune, since e...
24. When would you choose Merge-on-Read over Copy-on-Write?
MoR is the right call for write-heavy or high-frequency workloads — think streaming CDC ingestion from Kafka/Debezium where updates arrive continuously and need to land with low latency. Because updates go to cheap append-only log files rather than rewriting Parquet immediately, ingestion l...
25. What is the difference between snapshot and incremental queries?
A snapshot query answers "what does the table look like right now (or as of a given commit)?" — it reads the full, latest merged state. Snapshot query Incremental query Returns the full current (or point-in-time) table state. Returns only records changed between two specified commits. Used ...
26. How do you perform a rollback in Hudi?
A rollback reverts an inflight or failed commit, undoing any partial data it wrote so the table returns to its last known-good, fully-completed state. Identify the problematic instant on the timeline (its state will show as inflight or requested rather than completed). Trigger the rollback, eithe...
27. How does a savepoint differ from a rollback in Hudi?
A savepoint marks a specific completed commit as protected from the cleaner service, guaranteeing the data and metadata needed to restore the table to that exact point in time won't be garbage-collected — even if it would normally fall outside the retention window. A rollback , by contrast,...
28. Why do MoR tables need periodic compaction?
Because MoR writes updates to append-only Avro log files instead of merging them into Parquet immediately, those log files keep growing with every batch of updates to a given file slice if nothing ever merges them back. Left unchecked, this creates two problems: snapshot queries get progressively...
29. How does Hudi achieve schema evolution?
Hudi tracks a table's schema over time and supports common evolution patterns — adding nullable columns, widening a column's type, and reordering fields — without requiring a full table rewrite, by relying on Avro-compatible schema resolution under the hood. Because each commit on the...
30. What is the difference between synchronous and asynchronous table services?
Hudi's table services (compaction, clustering, cleaning, indexing) can run in two modes relative to the ingestion job. Synchronous (inline) Asynchronous Runs as part of the same write job, blocking the next write until done. Runs as a separate job/thread, decoupled from the write path. Simpler op...
31. How can you optimize small file management in Hudi?
Small files are one of the most common Hudi performance complaints, and there are several complementary levers to address it. Auto file sizing — configure Hudi's target file size so writers pack records into fewer, appropriately-sized files instead of many tiny ones. Clustering — sche...
32. How do you troubleshoot slow upserts in a Hudi table?
Slow upserts almost always trace back to one of a few common culprits, so it's worth checking them in order. Check the index type — a Bloom or Simple index scanning many files under heavy random-update patterns is far slower than a Record-Level Index at scale. Look for small-file sprawl &md...
33. What is the difference between optimistic and non-blocking concurrency control in Hudi?
Hudi's original concurrency model is Optimistic Concurrency Control (OCC) : multiple writers can attempt writes concurrently, but conflicts (two writers touching the same file group) are detected at commit time, and one of the conflicting writers fails and must retry — typically coordinated...
34. When should you use a bucket index instead of a Bloom index?
A bucket index assigns each record to a file group deterministically by hashing its record key into a fixed number of buckets, so no lookup against stored index data is needed at all — the target file is computed directly from the key. This makes bucket indexing extremely fast for upserts, ...
35. How does the Column Stats index differ from the Bloom filter index?
The Bloom filter index is built to answer point-lookup questions during upserts — "does this specific record key exist in this file?" — and lives in the Metadata Table's bloom filter partition. The Column Stats index serves a different purpose: it stores per-file min/max (and other) s...
36. Explain the internal working of an upsert operation in Hudi?
An upsert moves through three distinct internal stages, and being able to name each one is a strong signal in an interview. flowchart LR A[Incoming records] --> B[Index lookup: tag records with target file group] B --> C{New or existing key?} C -- New --> D[Route to insert path] C -- Existing -->...
37. Explain the execution flow of a Merge-on-Read compaction?
Compaction on a MoR table follows a two-phase schedule-then-execute flow, which is deliberately split so scheduling can be cheap and frequent while the heavier execution work can run separately. sequenceDiagram participant W as Writer / Streamer participant T as Timeline participant Comp as Compa...
38. Explain the lifecycle of a Hudi commit on the timeline?
Every write action on a Hudi table — not just user commits, but compactions, clusterings, and cleans too — passes through the same three-state lifecycle on the timeline. flowchart LR A[requested: action is planned] --> B[inflight: action is executing] B --> C[completed: action is dura...
39. How does Hudi's Non-Blocking Concurrency Control work internally?
Non-Blocking Concurrency Control (NBCC), introduced in Hudi 1.0, is designed for the specific case where multiple writers — for example, a streaming ingestion job and an async compaction/clustering job — need to touch the same file group at roughly the same time, something the older O...
40. Explain the internal working of the multi-modal indexing subsystem in Hudi?
Hudi's multi-modal index is the framework that lets several different index types — files, bloom filters, column stats, record-level index, secondary index, expression index, and more — all live as separate partitions within the single Metadata Table, rather than as one-off, index-spe...
41. How do you design a partitioning strategy for a very large Hudi table?
Partitioning in Hudi determines both physical file layout and, for partition-level indexes, how upsert lookups are scoped — so it's worth deliberately designing rather than defaulting to whatever column happens to be handy. Pick a partition column whose value rarely or never changes per rec...
42. Explain the internal working of Hudi's incremental query mechanism?
Incremental queries work because the timeline already records, instant by instant, exactly which files were touched by each commit — so "what changed between commit A and commit B" is a metadata lookup, not a data scan. flowchart TD A[User specifies begin/end commit time] --> B[Read timelin...
43. How does Hudi ensure ACID guarantees on cloud object storage?
Cloud object stores like S3 don't natively offer multi-file atomic transactions, so Hudi has to build ACID guarantees itself on top of storage primitives it can rely on — mainly atomic single-file writes/renames and the timeline's own ordering. Atomicity — a commit's data files are al...
44. Explain the internal working of the LSM-tree-based timeline in Hudi 1.x?
The original timeline stored each instant as its own small file directly in the table's .hoodie directory on object storage. That worked fine for tables with dozens or hundreds of commits, but at thousands of commits it turned into a small-file listing problem of its own — just for the meta...
45. How do you set up change data capture ingestion into Hudi?
CDC ingestion typically flows from a source database through a capture tool into Kafka, and then into Hudi via HoodieStreamer, so the setup spans a few distinct pieces. Capture changes at the source — a tool like Debezium reads the source database's change log and publishes insert/update/de...
46. Which is better for high-frequency upserts: Apache Hudi or Apache Iceberg?
Both are mature open table formats, and the honest answer depends on what "better" is optimized for — the two projects made different bets from day one. Hudi's case: it was purpose-built at Uber specifically for high-frequency, record-level upserts and CDC ingestion. Its Merge-on-Read table...
47. Explain the internal working of Hudi's file group and file slice model?
A file group is the fundamental storage unit within a partition, identified by a unique fileId . Every record in the table maps to exactly one file group via the index, and that mapping is what makes fast upserts possible — incoming records already know which file group to update. flowchart...
48. How do you migrate an existing Parquet-based data lake to Apache Hudi?
Hudi's bootstrap feature is purpose-built for this migration, letting you bring existing Parquet data under Hudi's management without necessarily rewriting every byte immediately. Choose a bootstrap mode — metadata-only bootstrap keeps the original Parquet files as-is and just generates Hud...
49. Explain the internal working of partial updates in Hudi 1.x?
Traditionally, an update record supplied to Hudi needed to contain the record's full set of columns, even if only one field actually changed — the payload class would merge the "new" full record against the "old" full record. That's wasteful for CDC sources that only emit the columns that a...
50. How do you tune Hudi for trillion-record-scale upsert workloads?
At trillion-record scale, the bottlenecks shift from "does it work" to specific, well-understood pressure points, and production deployments (Uber's own engineering being the most public example) converge on a similar tuning playbook. Enable and rely on the Metadata Table for all file listing, av...
