BigData / Apache Parquet Interview Questions
What is small file problem in Parquet-based data lakes and how do you solve it?
The small file problem occurs when a Parquet dataset accumulates thousands of tiny files (often from streaming writes or over-partitioning). Each file requires a separate HDFS/S3 metadata operation and a separate footer read, causing significant overhead.
Effects:
- Slow query planning — the driver/namenode must enumerate and open many files.
- Inefficient Parquet statistics — small row groups give poor skipping benefits.
- High object-store API costs (S3 LIST + GET per file).
Solutions:
- Compaction job — periodically coalesce small files into larger ones. In Spark:
spark.read.parquet(path).coalesce(N).write.mode("overwrite").parquet(path) - Delta Lake / Iceberg OPTIMIZE —
OPTIMIZE my_table;
- Streaming micro-batch tuning — increase trigger interval or use
maxFilesPerTriggerto reduce write frequency. - Hudi MOR → COW compaction — compact merge-on-read log files into base Parquet files.
What is the primary overhead caused by too many small Parquet files in a data lake?
Which Spark transformation consolidates many small partitions into fewer output files?
More Related questions...
