What are the performance tuning strategies for a Spring AI RAG application at scale?

When a RAG application moves from prototype to production load, several bottlenecks emerge. Addressing them requires tuning at the ingestion layer, retrieval layer, LLM call layer, and infrastructure layer.

Ingestion layer: Run chunking and embedding in parallel using a thread pool or Spring Batch. Batch embedding requests — most providers accept up to 100 texts per API call. Cache the result of ingestion so unchanged documents are not re-embedded on restarts.

Retrieval layer: Use HNSW indexes on PgVector or equivalent ANN indexes on other stores. Tune topK conservatively — fetching 10 chunks when 3 would suffice inflates prompt size and increases LLM cost. Add a reranker step (a cross-encoder model) to reorder retrieved chunks by relevance before truncating to the top 3 for the prompt.

LLM call layer: Cache responses to identical or near-identical prompts using a semantic cache backed by a VectorStore. If the cosine similarity between a new query and a cached query embedding exceeds a threshold, return the cached answer rather than calling the LLM. This can reduce API cost by 30-70% for FAQ-style workloads.

Parallel and async calls: For workflows that need multiple independent LLM calls (e.g. analysing several documents separately), use Flux merging or virtual threads to fire calls concurrently rather than sequentially.

Model selection: Use the cheapest model that meets quality requirements for each step. Metadata extraction during ingestion can use a cheap model; the final answer generation uses the flagship model. This is called model routing or cascading.