When would you choose GPU-accelerated indexes (like CAGRA) over CPU-based indexes?

GPU-accelerated indexes leverage a GPU's massive parallelism for both building and searching an index, which can dramatically outperform CPU-based indexes for the specific workloads GPUs excel at: very high query throughput requirements, very large-scale index builds, and dense, high-dimensional vectors where the parallel computation genuinely dominates the workload.

The trade-off is straightforward: GPU indexes require actual GPU hardware to be provisioned and kept available, which adds infrastructure cost and complexity compared to CPU-only deployment, and that cost is only worthwhile when the throughput or index-build-time gains genuinely matter for the application's requirements. A modest-scale application with relaxed latency requirements gains little from GPU acceleration relative to the added infrastructure cost and complexity, while a high-throughput, latency-sensitive production search system serving millions of queries can find that cost well justified by the resulting performance.

A reasonable decision process is to first establish whether a CPU-based index (HNSW being the common strong baseline) actually falls short of the application's throughput or latency requirements at the expected scale, and only then evaluate whether GPU acceleration is worth its added operational cost, rather than reaching for GPU indexes by default regardless of whether the workload genuinely needs that level of performance.