BigData / Apache Parquet Interview Questions
How do you perform upserts (MERGE INTO) on Parquet-based tables in Delta Lake?
Raw Parquet files are immutable — you cannot update individual rows. Delta Lake adds MERGE INTO support, which implements upserts by reading affected Parquet files, rewriting them with changes, and recording the transaction:
from delta.tables import DeltaTable delta_table = DeltaTable.forPath(spark, "/delta/customers") delta_table.alias("target").merge( updates_df.alias("source"), "target.customer_id = source.customer_id" ).whenMatchedUpdateAll( ).whenNotMatchedInsertAll( ).execute()
Or in SQL:
MERGE INTO customers AS target USING updates AS source ON target.customer_id = source.customer_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *;
Under the hood: Delta identifies which Parquet files contain matching rows, rewrites only those files with updated values, and records the old files as removed and new files as added in the transaction log. Non-matching files are untouched.
Why can't you update individual rows directly in raw Parquet files?
In a Delta Lake MERGE INTO operation, what happens to the original Parquet files that contained matched rows?
More Related questions...
