How do you optimize a LanceDB table for query performance through compaction?

Compaction consolidates many small data files (accumulated from repeated small writes over a table's version history) into fewer, larger files, which improves both scan performance and vector index quality, since a fragmented table with many tiny files has more overhead per query than a well-consolidated one.

table.optimize()  # runs compaction and other maintenance tasks

Under the hood, frequent small add() calls each create their own new data file; over time, a table that receives many small incremental writes accumulates a large number of small files, and reading from many small files carries more per-file overhead (open/seek costs, metadata lookups) than reading the equivalent data from fewer, larger files.

The optimize() operation, run periodically (either manually or on a schedule), performs several related maintenance tasks together: compacting small files into larger ones, pruning old versions past a retention window to reclaim disk space, and optimizing the vector index to account for data added since the index was last built or updated — all of which keep both scan and vector search performance from degrading as a table accumulates history through normal use.