Why does DuckLake store metadata in a database instead of files, unlike Iceberg/Delta Lake?

Iceberg and Delta Lake track table metadata, schema versions, snapshot history, file listings, through a layered structure of small JSON and Avro files sitting in the same object storage as the actual data. This design works, and is proven at large scale, but coordinating consistent, concurrent updates to that metadata (multiple writers committing at once, for example) requires careful, sometimes intricate protocol design layered on top of what object storage itself natively guarantees.

DuckLake's core bet is that this coordination problem is exactly what SQL databases have already solved extremely well, transactional consistency, concurrent access control, efficient small updates, so rather than re-solving it at the object-storage-file layer, DuckLake stores its metadata directly as rows in an actual SQL database (which could be DuckDB itself, PostgreSQL, MySQL, or another supported backend), and lets that database's existing transactional machinery handle metadata consistency.

The practical upshot is a simpler mental model and implementation for the metadata layer specifically: instead of a bespoke, file-based protocol for tracking lakehouse state, DuckLake reuses well-understood, battle-tested database transaction semantics, while the actual bulk data still lives as ordinary Parquet files in object storage, keeping the parts of the architecture that already work well (columnar file storage) largely unchanged.