BigData / Apache Parquet Interview Questions
What is the difference between repartition and coalesce when writing Parquet files?
Both control the number of output Parquet files, but they work differently:
| Aspect | repartition(N) | coalesce(N) |
|---|---|---|
| Shuffle | Full shuffle — all data redistributed across N partitions | No full shuffle — merges existing partitions locally |
| Output | Exactly N evenly-sized partitions | Up to N partitions; may be uneven |
| Direction | Can increase or decrease partitions | Can only decrease partitions |
| Cost | Higher — network + disk I/O | Lower — local merge |
| When to use | Need even file sizes; before a wide join | Reducing many small files cheaply after a filter |
For writing right-sized Parquet files, a common pattern is:
# After a heavy filter that leaves few rows, coalesce is cheaper df.filter(df.date == "2026-01-01").coalesce(4).write.parquet(output_path)
Which Spark operation can both increase and decrease the number of partitions by performing a full shuffle?
After a heavy filter that leaves data very skewed across partitions, which is the cheaper way to reduce the number of output Parquet files?
More Related questions...
