BigData / Apache Parquet Interview Questions
How do you handle schema mismatches when merging multiple Parquet files?
When a dataset is composed of Parquet files written at different times (possibly with different schemas), you have several options:
1. Spark mergeSchema — the simplest approach; Spark unions all schemas and fills missing columns with null:
df = spark.read.option("mergeSchema", "true").parquet("s3://datalake/events/")
2. AWS Glue Schema Registry — version schemas centrally; consumers register against a schema version and handle evolution rules explicitly.
3. Table formats (Iceberg / Delta / Hudi) — track schema history at the table level; ALTER TABLE ADD COLUMN is applied transactionally without rewriting files.
4. Manual reconciliation — use df.schema to inspect each file's schema, build a unified schema, then apply df.select(unified_cols) with lit(None).cast() for missing columns before union.
Always validate after merging: check for unexpected nulls or type coercions that widen types silently.
More Related questions...
