BigData / Apache Parquet Interview Questions
What are Bloom Filters in Parquet and when should you use them?
A Bloom Filter is a probabilistic data structure that answers "is this value possibly in this row group?" with zero false negatives. Parquet 1.12+ supports per-column Bloom filters stored in the file footer.
They complement min/max statistics for high-cardinality columns where the min–max range spans the full value space (e.g., UUIDs, email addresses). Without a Bloom filter, the engine must open every row group; with one, it can skip groups that definitely do not contain the queried value.
Enable in PySpark/Spark:
spark.conf.set("parquet.bloom.filter.enabled#user_id", "true") spark.conf.set("parquet.bloom.filter.expected.ndv#user_id", "1000000")
When to use:
- Equality filters (
WHERE uuid = '...') on high-cardinality string/UUID columns. - Not useful for range queries (use sorted data + min/max for that).
Trade-off: adds size to the footer; tune expected.ndv (number of distinct values) to control false-positive rate vs. filter size.
For which query pattern are Parquet Bloom filters most beneficial?
What does a Parquet Bloom filter guarantee about its results?
More Related questions...
