How do you monitor and troubleshoot slow vector search queries in LanceDB?

Diagnosing slow vector queries generally starts with confirming the basics — is an appropriate index actually present and being used — before moving on to more nuanced tuning of index parameters and query structure.

1. Check whether an ANN index exists on the vector column: without one, LanceDB falls back to an exact, brute-force scan, which is fine for small tables but scales poorly as row count grows.
2. Inspect the query plan: using explain_plan() (built on DataFusion's own plan introspection) shows whether filters are being pushed down and whether the vector index is actually being used as expected.
3. Tune index parameters: for IVF-PQ, increasing nprobe improves recall at the cost of latency, while too few clusters relative to dataset size can leave individual clusters too large to search quickly.
4. Check for excessive file fragmentation: a table that hasn't been compacted in a while, after many small writes, can suffer from unnecessary per-file overhead; running optimize() often helps.
5. Verify scalar indexes exist on frequently filtered columns: an unindexed filter combined with prefiltering can force scanning far more candidates than necessary before the vector search even begins.

A useful general habit is measuring before tuning: comparing query latency with and without a specific index or parameter change on representative data, rather than guessing, since the right fix (a missing index versus a fragmented table versus an under-tuned nprobe) can look similar from the outside but requires a different remedy.