BigData / Apache Hudi Interview Questions
How can you optimize small file management in Hudi?
Small files are one of the most common Hudi performance complaints, and there are several complementary levers to address it.
- Auto file sizing — configure Hudi's target file size so writers pack records into fewer, appropriately-sized files instead of many tiny ones.
- Clustering — schedule it regularly to merge accumulated small files into larger, better-organized ones after the fact.
- Tune bulk_insert parallelism — for initial loads, too much parallelism relative to data volume creates many small output files.
- Right-size ingestion batch intervals — very frequent, very small micro-batches naturally produce more small files than slightly larger, less frequent ones.
Which table service directly merges accumulated small files?
Overly high bulk_insert parallelism relative to data volume tends to:
More Related questions...
