What is the IVF_PQ index in LanceDB?

IVF-PQ (Inverted File index with Product Quantization) is one of LanceDB's supported ANN index types for vector columns, combining two complementary techniques to make approximate similarity search fast even over very large datasets stored on disk.

The IVF part partitions the vector space into a number of clusters (using a technique like k-means) during index build time; at query time, instead of scanning every vector, the search first identifies which cluster(s) the query vector is closest to and only searches within those, dramatically shrinking the candidate set.

The PQ part compresses each vector into a much smaller quantized representation, which reduces both the memory/disk footprint of the index and the cost of computing distances between the query and candidate vectors, at the cost of some precision in the resulting distance estimates — a trade-off that's usually well worth it for the storage and speed gains at large scale.