Why doesn't increasing similarity_top_k always improve answer quality?

It's tempting to assume retrieving more Nodes can only help, since the correct one is more likely to be included, but in practice a higher similarity_top_k comes with real costs that can offset or even reverse that benefit.

First, every additional Node retrieved is rarely perfectly relevant; padding the candidate set with marginally related text increases noise, and LLMs are known to struggle with attending correctly to relevant information buried in the middle of a long context, a pattern often called the "lost in the middle" effect. Second, more Nodes mean a larger prompt, raising both token cost and latency for no guaranteed benefit. Third, if the response mode is refine, each additional Node means another sequential LLM call, and each refinement step introduces a small chance of the model drifting from or diluting a previously correct answer.

This is why in practice a moderate similarity_top_k paired with a reranker to select the truly best few Nodes from a larger candidate pool tends to outperform simply cranking similarity_top_k up on its own.