BigData / Apache Parquet Interview Questions
How does Parquet handle compression and encoding?
Parquet uses two complementary techniques to minimise storage:
Encoding — applied first, at the column level, to exploit data patterns:
- Dictionary Encoding — replaces repeated values with integer codes. Ideal for low-cardinality columns (e.g., status, country).
- Run-Length Encoding (RLE) — collapses consecutive identical values into (value, count) pairs.
- Delta Encoding — stores differences between successive integers; great for timestamps and monotonic IDs.
- Bit-packing — packs small integers into fewer bits.
Compression codec — applied after encoding to the byte stream:
- Snappy (default in Spark) — fast, moderate ratio (~2–3×).
- GZIP/Zlib — higher ratio (~4–6×) but slower CPU.
- ZSTD — best balance; recommended for most new deployments.
- LZO, LZ4, Brotli — specialised use cases.
Encoding and codec are configured independently per column, so hot columns can use fast codecs while archive columns use ZSTD for maximum compression.
Which Parquet encoding is best suited for a low-cardinality column like country or status?
Which compression codec offers the best balance of ratio and speed for modern Parquet deployments?
More Related questions...
