How do you optimize a DuckDB query against a large Parquet dataset?

Optimization for large Parquet queries in DuckDB mostly comes down to helping the engine prune as much unnecessary I/O as possible and avoiding unnecessary intermediate data materialization.

• Select only needed columns - avoid SELECT * when only a few columns are actually needed, since columnar pruning only helps for columns the query doesn't reference at all.
• Push filters as early and specifically as possible - filters on columns the data is naturally sorted or partitioned by (like a date field in a partitioned dataset) enable much more effective row-group and file-level pruning.
• Use partitioned directory layouts where applicable - a dataset organized into directories like year=2026/month=01/ lets DuckDB skip entire directories (and their files) based on a query's filter, before even opening individual Parquet files.
• Check EXPLAIN ANALYZE for unexpected large intermediate results - a join or aggregation producing a far larger intermediate result than expected often points to a filter that isn't being applied as early as it could be, or a join condition that's less selective than assumed.
• Ensure sufficient memory and thread availability - DuckDB's performance scales with both, and a resource-constrained environment can force unnecessary spilling to disk that slows an otherwise well-optimized query.

The general principle across all of these is the same one that drives DuckDB's pruning-based performance more broadly: the fastest way to process data is to avoid reading it at all whenever a query's own logic makes that provably safe, so optimization is largely about giving the engine enough information (through selective filters, good data layout, and correct statistics) to make that determination as often as possible.