Database / LanceDB Interview questions
What is the difference between IVF_PQ and HNSW indexing in LanceDB?
Both are ANN index types LanceDB supports for vector columns, but they take fundamentally different approaches to narrowing a search space, which leads to different trade-offs in memory usage, build time, and query speed.
| IVF-PQ | HNSW |
| Cluster-based partitioning plus vector compression. | Graph-based: a multi-layer navigable small-world graph. |
| Lower memory footprint due to PQ compression. | Typically higher memory footprint, storing full or lightly-compressed vectors. |
| Well suited to very large, disk-based datasets. | Often faster query latency when data fits comfortably in memory. |
| Good default for cost-sensitive, large-scale storage. | Good choice when top-tier query latency matters more than memory cost. |
HNSW builds a layered graph where each vector is a node connected to its approximate nearest neighbors; a query traverses the graph greedily from an entry point, descending through layers to quickly converge on a close neighborhood without ever needing to partition the space into clusters the way IVF does.
The practical choice often comes down to scale and budget: IVF-PQ's compression makes it more economical at the scale of hundreds of millions to billions of vectors on disk, while HNSW's graph traversal can offer lower query latency when the dataset and its memory footprint are more moderate.
More Related questions...
