Why does Paimon combine a lake format with an LSM-tree structure?

Traditional data lake formats are built around immutable, append-only files, which makes them cheap to scan but expensive to update — a single-row change historically meant rewriting whole files. Traditional databases handle updates well but weren't designed to scale to lake-sized batch and streaming workloads or to be queried directly by engines like Spark and Trino.

Paimon's answer is to borrow the LSM tree from database storage engines and put it inside the lake format itself: new records land in memory, get sorted, and flush as small sorted runs instead of rewriting existing files. Compaction periodically merges sorted runs in the background, folding in updates and deletes without blocking new writes.

The result is a table that streaming jobs can update at high throughput and low latency, while batch engines still read it as an ordinary, file-based lake table — letting the same table serve both a Flink upsert job and a Spark analytical query without duplicating data into two systems.