What is the role of object storage (S3/GCS) in LanceDB's architecture?

Object storage serves as the durable, scalable source of truth for a LanceDB table's data and version history, letting compute (the application process running queries) remain stateless and decoupled from where the actual data physically lives — a foundational pattern often called separating storage and compute.

Because Lance is designed for efficient random access directly against object storage, LanceDB doesn't need to download an entire dataset locally before querying it; it can read just the specific files (or even specific byte ranges within files) needed to answer a given query, which is what makes querying data that lives on S3 or GCS practical rather than prohibitively slow.

This architecture is also what enables background maintenance work — like index building or compaction — to run without competing for resources with latency-sensitive user queries: since object storage is the single shared source of truth, multiple separate compute processes (one running maintenance, others serving queries) can all interact with the same underlying data safely, coordinated through the version/manifest commit protocol rather than needing to share memory or a single running process.