BigData / Apache Parquet Interview Questions
What is the Vectorized Reader in Spark and how does it improve Parquet performance?
The Vectorized Parquet Reader (introduced in Spark 2.0) reads a batch of rows at once directly into an in-memory columnar format (ColumnarBatch) rather than converting each row individually to a JVM object. This avoids object creation overhead and allows the JVM's JIT compiler to apply SIMD-style optimisations.
Key benefits:
- Reduces per-row CPU overhead significantly.
- Enables whole-stage code generation to operate on batches.
- Speeds up filters and projections applied directly to column vectors.
Enable or verify via Spark config:
spark.conf.set("spark.sql.parquet.enableVectorizedReader", "true") # default true
For complex nested schemas (e.g., deeply nested structs/maps), the vectorized reader may fall back to row-by-row mode automatically.
What memory format does the Spark Vectorized Parquet Reader output rows into?
When does Spark's vectorized Parquet reader fall back to row-by-row mode?
More Related questions...
