How can you optimize token usage and cost in a large-scale LlamaIndex deployment?

Cost in a LlamaIndex application comes mainly from embedding calls and LLM calls, so optimization means reducing unnecessary calls on both fronts without hurting answer quality too much.

1. Cache aggressively. Use the IngestionPipeline's cache so unchanged documents aren't re-embedded, and consider caching LLM responses for repeated queries.
2. Tune similarity_top_k and chunk_size. Retrieving fewer, better-sized Nodes reduces tokens sent to the LLM per query.
3. Prefer compact over refine for response synthesis when possible, since it needs far fewer sequential LLM calls than refine for the same Nodes.
4. Add a reranker to cut a larger initial candidate set down to only the most relevant few before synthesis, rather than sending everything retrieved to the LLM.
5. Use cheaper models for cheaper sub-tasks, such as a smaller model for query routing or sub-question generation, reserving the most capable model for final answer synthesis.
6. Batch and parallelize with async where the cost is really latency rather than token spend, since concurrent calls reduce wall-clock time even if total token cost stays the same.

In practice, teams usually get the biggest win from combining a reasonable similarity_top_k with a reranker and compact synthesis, since that trims both the number of tokens and the number of LLM calls at once.