When should you use DuckDB instead of a distributed system like Spark?

Spark and similar distributed engines exist specifically to scale processing across many machines when a single node's memory and CPU genuinely aren't enough to handle the data volume or computation involved. DuckDB, by contrast, is a single-node engine, extremely fast and efficient within that constraint, but not designed to coordinate work across a cluster of machines.

DuckDB tends to be the better fit when a dataset fits comfortably (or with some out-of-core spilling) on a single reasonably capable machine, whether that's a laptop, a workstation, or a single cloud VM, and the actual query complexity doesn't require distributed shuffling across nodes. In those cases, DuckDB's lack of cluster coordination overhead, network shuffling, and distributed job scheduling often means it significantly outperforms a Spark job doing the equivalent work, simply because it isn't paying the overhead of a distributed system for a problem that didn't need to be distributed in the first place.

Spark (or a similar distributed engine) becomes the right choice once data volume or computational demands genuinely exceed what a single machine can handle, hundreds of terabytes or more, or workloads requiring the fault tolerance and horizontal scalability a distributed cluster provides, since DuckDB has no native mechanism for spreading work across multiple physical machines the way Spark's architecture is built around.