What is the Lance columnar format?

Lance is the open-source columnar file and table format that LanceDB is built on — conceptually similar in role to Parquet, but specifically optimized for the access patterns AI and machine learning workloads need, particularly fast random access to individual rows or vectors rather than only efficient full-column scans.

Because it's a columnar format, operations like adding a new column don't require rewriting every existing row, and reading only a subset of columns (say, just the vector column for a similarity search) can skip the data for columns that aren't needed, similar to how other columnar analytics formats behave.

Lance is designed to work directly on top of object storage like S3, GCS, or Azure Blob Storage, and it supports vector indexing, versioning, and schema evolution as first-class capabilities of the format itself, which is what lets LanceDB build a full-featured vector database on top of it rather than needing to bolt those capabilities on separately.