What is the role of DataFusion in LanceDB's query execution?

Apache DataFusion is a Rust-based, extensible query execution engine that LanceDB uses under the hood to plan and execute queries — particularly SQL-style filtering and more complex query expressions — rather than LanceDB implementing its own query planner and execution logic from scratch.

DataFusion's role is essentially the "SQL brain" of the system: it takes a logical description of what a query needs (a filter condition, a projection of specific columns, a limit) and produces an optimized physical execution plan, including decisions like pushing a filter down to use a scalar index rather than filtering after fetching every row.

Using an existing, actively-developed engine like DataFusion rather than building a custom query planner lets LanceDB benefit from DataFusion's ongoing optimizer improvements and its native Arrow integration, keeping the whole pipeline — storage format, in-memory representation, and query execution — consistently built around the same Arrow-based foundation.