BigData / Apache Parquet Interview Questions
What is Delta Lake and how does it extend Parquet for ACID transactions?
Delta Lake is an open-source storage layer built by Databricks that adds transactional guarantees on top of Parquet files stored in object storage. The core idea: all changes (inserts, updates, deletes) are written as immutable Parquet files and tracked via a JSON transaction log (the _delta_log directory).
ACID properties in Delta Lake:
- Atomicity — a transaction either fully commits (log entry added) or is rolled back (log entry absent).
- Consistency — schema enforcement prevents corrupt writes.
- Isolation — Optimistic Concurrency Control (OCC) detects conflicts between concurrent writers.
- Durability — log + data files in object storage are highly durable.
Data files remain Parquet; Delta adds a _delta_log/ with JSON commit files that record which Parquet files are added or removed in each transaction. Periodic checkpoint files (Parquet snapshots of the log) speed up log replay.
# Spark Delta write with schema enforcement df.write.format("delta").mode("append").save("/delta/events") # Time travel spark.read.format("delta").option("versionAsOf", 5).load("/delta/events")
Where does Delta Lake store its transaction log alongside Parquet data files?
Which concurrency control mechanism does Delta Lake use to resolve conflicts between concurrent writers?
More Related questions...
