BigData / Apache Parquet Interview Questions
What is partitioning in Parquet and how does it improve query performance?
Partitioning organises files into a directory hierarchy based on column values, following the Hive partition layout:
s3://bucket/events/ date=2026-01-01/ part-0000.parquet date=2026-01-02/ part-0001.parquet region=US/ ...
When a query filters on a partition column, the engine lists only matching subdirectories and skips all others — this is called partition pruning. No Parquet files in the skipped partitions are opened at all.
Example savings: a table with 3 years of daily data (1,095 partitions) filtering on a single date skips 99.9% of files before any Parquet-level statistics are consulted.
Trade-offs:
- Partition on columns with moderate cardinality (dates, regions, categories) — not user IDs (too many small files).
- Avoid over-partitioning: thousands of tiny files hurt throughput more than they help with pruning.
What is the performance optimisation called when a query skips entire directory partitions based on a filter column value?
Why should you avoid partitioning a Parquet dataset by a high-cardinality column like user_id?
More Related questions...
