BigData / Apache Parquet Interview Questions
1. What is Apache Parquet and why is it used?
Apache Parquet is an open-source, columnar storage file format designed for the Hadoop ecosystem and modern big-data platforms. Unlike row-based formats (CSV, JSON), Parquet stores each column's data contiguously on disk. This layout allows query engines to read only the columns they need, dramat...
2. What are the advantages of Parquet over CSV?
CSV is simple but untyped and inefficient at scale. Parquet improves on it in every dimension that matters for analytics: Aspect CSV Parquet Storage layout Row-based Columnar Compression Low (mixed types per row) High (homogeneous per column) Schema None (inferred) Embedded in file Partial reads ...
3. How are Parquet files structured? (Row Groups, Column Chunks, Pages)?
Parquet organises data in a three-level hierarchy: Row Group — a horizontal slice of the dataset, typically 128 MB–1 GB of data. Each row group contains one column chunk per column. Column Chunk — all values for a single column within a row group. This is the unit of compression and encoding. Pag...
4. What is Schema Evolution in Parquet?
Schema evolution is the ability to change a Parquet dataset's schema over time without rewriting existing files. Parquet natively supports: Adding columns — new columns appear as null in older files when read together with newer files. Renaming columns — supported via field IDs (used in formats l...
5. What is Column Pruning and Projection Pushdown in Parquet?
Column pruning (also called projection pushdown ) is an optimisation where the query engine reads only the columns referenced in the query, ignoring all other column chunks on disk. Because Parquet stores each column separately, skipping unrequested columns costs nothing beyond reading the footer...
6. When would you choose Avro over Parquet?
Parquet and Avro serve different access patterns. The table below summarises when to prefer each: Criterion Choose Parquet Choose Avro Query pattern OLAP — aggregate few columns over many rows Row-based access — read/write entire records Streaming Batch / micro-batch Streaming (Kafka, Flink event...
7. How does Parquet handle compression and encoding?
Parquet uses two complementary techniques to minimise storage: Encoding — applied first, at the column level, to exploit data patterns: Dictionary Encoding — replaces repeated values with integer codes. Ideal for low-cardinality columns (e.g., status, country). Run-Length Encoding (RLE) — collaps...
8. What is the Vectorized Reader in Spark and how does it improve Parquet performance?
The Vectorized Parquet Reader (introduced in Spark 2.0) reads a batch of rows at once directly into an in-memory columnar format ( ColumnarBatch ) rather than converting each row individually to a JVM object. This avoids object creation overhead and allows the JVM's JIT compiler to apply SIMD-sty...
9. How do you handle schema mismatches when merging multiple Parquet files?
When a dataset is composed of Parquet files written at different times (possibly with different schemas), you have several options: 1. Spark mergeSchema — the simplest approach; Spark unions all schemas and fills missing columns with null : df = spark.read.option("mergeSchema", "true").parquet("s...
10. If a Spark query on Parquet is slow, what optimisation steps would you take?
Diagnosing and tuning slow Parquet queries in Spark follows a layered approach: Check partitioning — ensure the table is partitioned on high-cardinality filter columns (e.g., date , region ). Without partitions, Spark scans all files. df.write.partitionBy("date", "region").parquet("s3://path/") V...
11. How do you load Parquet files into Snowflake?
Snowflake can query and load Parquet files staged in cloud storage using a two-step approach: Step 1 — Stage the files (S3, Azure Blob, or GCS external stage): CREATE OR REPLACE STAGE my_stage URL = 's3://my-bucket/parquet-data/' CREDENTIALS = (AWS_KEY_ID='...' AWS_SECRET_KEY='...'); Step 2a — In...
12. What are the supported data types in Parquet?
Parquet defines primitive types (physical storage) and logical types (semantic meaning layered on top). Primitive types: BOOLEAN, INT32, INT64, INT96 (legacy timestamps), FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY. Common logical types (annotations on primitives): Logical Type Physical Mappi...
13. How do you read and write Parquet files in PySpark?
Spark provides first-class Parquet support via the DataFrameReader and DataFrameWriter APIs. Read: # Read a single file or directory of Parquet files df = spark.read.parquet("s3://my-bucket/events/") # With options df = (spark.read .option("mergeSchema", "true") .parquet("hdfs:///datalake/transac...
14. How do you read and write Parquet files in Python with PyArrow?
PyArrow provides a low-level Parquet library that is fast, pure Python-friendly, and interoperates with Pandas: Write: import pyarrow as pa import pyarrow.parquet as pq table = pa . Table . from_pandas(df) pq . write_table(table, "output.parquet" , compression = "zstd" ) Read: table = pq.read_tab...
15. What is partitioning in Parquet and how does it improve query performance?
Partitioning organises files into a directory hierarchy based on column values, following the Hive partition layout : s3://bucket/events/ date=2026-01-01/ part-0000.parquet date=2026-01-02/ part-0001.parquet region=US/ ... When a query filters on a partition column, the engine lists only matching...
16. What are Bloom Filters in Parquet and when should you use them?
A Bloom Filter is a probabilistic data structure that answers "is this value possibly in this row group?" with zero false negatives. Parquet 1.12+ supports per-column Bloom filters stored in the file footer. They complement min/max statistics for high-cardinality columns where the min–max range s...
17. What is the difference between Parquet, ORC, and Avro?
All three are Apache-ecosystem formats but optimised for different workloads: Feature Parquet ORC Avro Layout Columnar Columnar Row-based Ecosystem fit Spark, Presto, Hive, cloud lakes Hive-native; Spark Kafka, Flink, Hadoop MR Compression Excellent (ZSTD, Snappy) Excellent (ZLIB, Snappy, ZSTD) G...
18. What is Z-ordering (Z-order clustering) and how does it help Parquet queries?
Z-ordering is a multi-dimensional data-skipping technique that physically co-locates related rows across multiple filter columns within Parquet files. It maps multiple column values to a single Z-order curve value and sorts data along that curve. Problem it solves: Standard partitioning and sorti...
19. What is Apache Iceberg and how does it use Parquet?
Apache Iceberg is a high-performance open table format for huge analytic datasets, designed to replace traditional Hive table management. Iceberg stores actual data in Parquet (or ORC/Avro) files and adds a metadata layer on top. Iceberg adds to raw Parquet: ACID transactions — snapshot isolation...
20. How does DuckDB query Parquet files and what makes it fast?
DuckDB can query Parquet files directly without loading them into a database, using SQL: -- Query a local Parquet file SELECT region, SUM (revenue) FROM read_parquet( 'events.parquet' ) GROUP BY region; -- Glob pattern for multiple files / partitioned directory SELECT * FROM read_parquet( 's3://b...
21. What is the Parquet file footer and why does the reader fetch it first?
The Parquet file footer is a serialised Thrift structure at the end of every Parquet file. It contains: The full file schema (field names, types, nesting). Per-row-group metadata: byte offsets, compressed/uncompressed sizes, row counts. Per-column-chunk statistics: min value, max value, null coun...
22. How does Parquet support nested data (structs, lists, maps)?
Parquet uses the Dremel encoding (Google's paper, 2010) to represent arbitrarily nested data in a flat columnar layout. Two extra per-value integers are stored alongside each column's data: Definition level — how many optional fields in the path are actually defined (non-null). Encodes null posit...
23. What is small file problem in Parquet-based data lakes and how do you solve it?
The small file problem occurs when a Parquet dataset accumulates thousands of tiny files (often from streaming writes or over-partitioning). Each file requires a separate HDFS/S3 metadata operation and a separate footer read, causing significant overhead. Effects: Slow query planning — the driver...
24. What is the difference between repartition and coalesce when writing Parquet files?
Both control the number of output Parquet files, but they work differently: Aspect repartition(N) coalesce(N) Shuffle Full shuffle — all data redistributed across N partitions No full shuffle — merges existing partitions locally Output Exactly N evenly-sized partitions Up to N partitions; may be ...
25. How does AWS Athena query Parquet files in S3?
AWS Athena is a serverless interactive query service that uses Presto/Trino under the hood. It reads Parquet files directly from S3 using the Parquet columnar reader. Steps to query Parquet in Athena: Define a Glue Data Catalog table pointing to the S3 prefix: CREATE EXTERNAL TABLE events ( user_...
26. What is predicate pushdown in Parquet and how does it work end-to-end?
Predicate pushdown is the process of evaluating filter conditions as early as possible — before data reaches the compute layer — using metadata stored inside Parquet files. End-to-end flow for WHERE amount > 1000 : Reader fetches the Parquet footer and reads per-row-group statistics: min_amount ,...
27. What are best practices for writing Parquet files in production?
Producing high-quality Parquet files that perform well at query time requires attention at write time: Target 128 MB–512 MB row groups — too small wastes footer reads; too large makes predicate skipping coarse. Sort data before writing on filter columns — tight min/max ranges per row group dramat...
28. How does Google BigQuery use Parquet-style columnar storage internally?
BigQuery uses Capacitor , its proprietary columnar format, which shares the same fundamental principles as Parquet: columnar layout, aggressive encoding, and embedded statistics. When you export from BigQuery or import into it, Parquet is the preferred external format. Exporting BigQuery tables t...
29. What is Delta Lake and how does it extend Parquet for ACID transactions?
Delta Lake is an open-source storage layer built by Databricks that adds transactional guarantees on top of Parquet files stored in object storage. The core idea: all changes (inserts, updates, deletes) are written as immutable Parquet files and tracked via a JSON transaction log (the _delta_log ...
30. How do you perform upserts (MERGE INTO) on Parquet-based tables in Delta Lake?
Raw Parquet files are immutable — you cannot update individual rows. Delta Lake adds MERGE INTO support, which implements upserts by reading affected Parquet files, rewriting them with changes, and recording the transaction: from delta.tables import DeltaTable delta_table = DeltaTable . forPath(s...
