AI / Cache Augmented Generation (CAG) Interview questions

Cache-Augmented Generation (CAG) is a way of grounding an LLM in your own knowledge without a retrieval step. You load the whole knowledge base into the model's context ahead of time, run it through the model once, and keep the resulting key-value (KV) cache.

Every later question reuses that stored cache. The model only has to process the new question tokens, so there is no vector search, no chunk selection and no re-reading of the documents.

It works when the knowledge base is small and stable enough to fit inside the model's context window. The idea was popularised by the December 2024 paper Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks.

CAG targets the cost and fragility of the retrieval step in a RAG pipeline. In RAG, every query pays for embedding, vector search, optional reranking and a fresh prefill of the retrieved chunks.

It also inherits retrieval mistakes. If chunking splits an answer in half, or the top-k cutoff misses the one document that mattered, the model never sees the evidence and answers from incomplete context.

• Latency: no retrieval hop and no re-reading of documents.
• Accuracy: the model sees the full corpus, so nothing is left out by a retriever.
• Complexity: no vector database, embedding model or chunking strategy to tune.

In self-attention, every token produces a key and a value vector at each layer. When the model generates the next token, it needs the keys and values of all previous tokens.

Instead of recomputing them every step, the model stores them in the KV cache and only computes keys and values for the newest token. This turns generation from quadratic recomputation into a cheap append-and-attend step.

The price is memory: the cache grows linearly with the number of tokens. CAG simply takes this existing mechanism and applies it to the knowledge documents, so the expensive part is done once.

The CAG paper describes three phases:

1. Knowledge preloading: documents are formatted into one prompt, passed through the model, and the resulting KV cache is saved.
2. Inference: the user's question is appended to the cached context, only the new tokens are processed, and the answer is generated using the cache.
3. Cache reset: the question and answer tokens are trimmed off so the cache returns to its clean preloaded state, ready for the next query.

Phase 1 happens offline or at startup. Phases 2 and 3 repeat for every request.

Knowledge preloading is the one-time step where you turn your documents into a reusable KV cache. You gather the documents, clean them, and concatenate them into a single prompt, usually after an instruction such as "answer only from the documents below".

You then run one forward pass (the prefill) over that prompt and save the past_key_values the model produced. That cache represents everything the model "knows" about your corpus for this deployment.

Document order and formatting matter, because any later change to the prefix invalidates the cache from that point onwards.

RAG grounds an LLM by fetching relevant text at query time. Documents are split into chunks, embedded, and stored in a vector database. When a question arrives, it is embedded too, and the closest chunks are pulled back.

1. Chunk and embed the corpus (offline).
2. Embed the user query.
3. Run similarity search and take the top-k chunks, optionally reranking.
4. Insert the chunks into the prompt and generate the answer.

RAG scales to corpora far larger than the context window and is easy to update. CAG is the alternative that skips steps 2 to 4 by keeping the knowledge in context.

The context window is the maximum number of tokens a model can handle in one request, counting the prompt, any cached knowledge, and the tokens it generates. Modern models range from about 8K up to 1M tokens or more.

It is the hard ceiling for CAG. If your documents, the question and the expected answer do not fit together, the model cannot use them. Leave headroom: a corpus using 100% of the window leaves no room to answer.

A rough rule is that one token is about 0.75 English words, so a 128K window holds roughly 90,000 words of source material.

Prefix caching reuses the KV cache for the leading tokens that two prompts have in common. It works because of causal attention: the key and value of a token depend only on the tokens before it, so an identical prefix always yields identical cached values.

If a new prompt starts with the same 50,000 tokens as an earlier one, the server loads the stored cache and only processes the different tail. The first token that differs breaks the match, and everything after it must be recomputed.

This is the serving-side mechanism that makes CAG efficient, and it is built into engines like vLLM and SGLang.

Prompt caching is the hosted version of prefix caching. The provider stores the processed prefix of your prompt for a short time and charges less to read it back, while also lowering latency.

Anthropic lets you mark cache breakpoints with cache_control, with a default lifetime of about 5 minutes and an optional longer one. OpenAI applies caching automatically to long prompts, and Google Gemini offers both implicit and explicit context caching.

Minimum lengths, lifetimes and discounts differ by provider and change over time, so check the current documentation. In effect it is a time-limited, provider-managed form of CAG.

• Lower latency: no retrieval hop, and the documents are never re-processed per request.
• Fewer retrieval errors: the model has the complete corpus, so no relevant chunk is dropped by a ranker.
• Better multi-document reasoning: information spread across files is all visible at once.
• Simpler stack: no vector database, chunking rules or reranker to maintain.
• Consistent behaviour: the same context gives more predictable answers than varying top-k results.

The CAG paper reported lower generation time than RAG on its benchmarks, with the gap widening as the reference text grew.

• Size cap: the corpus must fit in the context window with room to spare.
• Memory cost: KV caches are large, often several gigabytes for long contexts.
• Staleness: any document change means rebuilding the cache.
• Long-context weakness: models can overlook facts buried in the middle of huge prompts.
• No per-user filtering: everything in the cache is visible to the model for every request.

None of these are fatal for a stable, bounded corpus, which is why CAG suits some jobs and not others.

CAG fits knowledge that is bounded, stable and queried often. The same expensive prefill is paid once and amortised across many questions.

A quick test: if all the text fits in about half of your model's window and changes less than daily, CAG is worth prototyping.

After each query, the cache holds the preloaded documents plus the question and the generated answer tokens. Cache reset removes that extra tail so the cache again contains only the original knowledge.

In Hugging Face Transformers this is done by remembering the original length and cropping back to it:

origin_len = cache.get_seq_length()   # after preloading
# ... answer a question ...
cache.crop(origin_len)                 # reset

Without it, memory keeps growing and one user's question can leak into the next user's context.

The reference paper is Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks by Brian J. Chan and colleagues, published on arXiv in December 2024.

It argues that with long-context models, preloading a small, manageable corpus and caching its KV states can match or beat RAG without retrieval errors or latency. The authors compared CAG and RAG on question-answering benchmarks, namely SQuAD and HotPotQA, using a long-context Llama 3.1 model.

The paper also introduced the three-phase view: preload, inference and cache reset.

• Knowledge corpus and preprocessor: cleans, orders and formats documents into the prefix.
• Long-context LLM: a model whose window comfortably fits the corpus.
• KV cache store: GPU memory, CPU memory or disk holding the precomputed cache.
• Inference wrapper: loads the cache, appends the question, generates the answer.
• Reset and refresh logic: trims per-query tokens and rebuilds the cache when sources change.
• Monitoring: tracks hit rate, TTFT and memory use.

Notice what is missing compared with RAG: no embedding model, vector index or retriever.

Time to first token is the delay between sending a request and receiving the first generated token. It is dominated by the prefill stage, where the model processes the whole prompt.

Prefill cost grows with prompt length, so a 100K-token prompt can take many seconds on a single GPU. CAG attacks this directly: the document tokens are already in the cache, so only the short question is prefilled.

This is why TTFT is the headline metric when comparing CAG to RAG or to plain long-context prompting.

A hit means the incoming prompt starts with exactly the tokens already cached. The engine loads the stored keys and values and prefills only the new tail.

A miss means the prompts diverge somewhere inside the cached prefix. Everything from that point on must be recomputed, which can be most of the prompt.

The pattern is: build the prefix, prefill it once into a DynamicCache, then reuse that cache for each question and crop it afterwards.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache

name = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(
    name, torch_dtype=torch.bfloat16, device_map="auto")

docs = open("handbook.txt").read()
prefix = f"Answer only from the documents below.\n\n{docs}\n\n"
prefix_ids = tok(prefix, return_tensors="pt").input_ids.to(model.device)

cache = DynamicCache()
with torch.no_grad():
    model(input_ids=prefix_ids, past_key_values=cache, use_cache=True)
origin_len = cache.get_seq_length()

def ask(question):
    q = tok(f"Question: {question}\nAnswer:", return_tensors="pt").input_ids.to(model.device)
    full = torch.cat([prefix_ids, q], dim=-1)
    out = model.generate(full, past_key_values=cache, max_new_tokens=200)
    cache.crop(origin_len)                      # cache reset
    return tok.decode(out[0, full.shape[-1]:], skip_special_tokens=True)

APIs shift slightly between Transformers versions, so verify against the release you use.

A hybrid keeps the most important, stable knowledge in a precomputed cache and uses retrieval only for what does not fit or changes often.

For example, a support assistant might cache the product manual and policy pages, then retrieve fresh release notes or a customer's ticket history and append those after the cached prefix. The cached part stays valid because the retrieved text is added at the end.

You get CAG's speed and full-context reasoning on the core material, and RAG's reach on the long tail.

1. Collect the FAQ and support policies, remove duplicates, and keep a clear structure (title, then answer) for each entry.
2. Estimate tokens. Around 300 entries at roughly 150 tokens each is about 45K tokens, comfortable inside a 128K window.
3. Build the prefix: instructions first, documents in a fixed order, no timestamps.
4. Preload once at deployment and keep the cache in GPU or CPU memory.
5. Serve each question by appending it, generating the answer, then cropping the cache.
6. Rebuild the cache from a CI job whenever the FAQ content hash changes.

Add an instruction to say "I don't know" when the answer is not in the FAQ, otherwise the bot may guess.

The core difference is when and how knowledge reaches the model. RAG fetches relevant chunks per query. CAG preloads the whole corpus once and reuses its KV cache.

Neither is universally better. CAG trades scale and freshness for speed and simplicity.

A RAG request pays for several steps before generation starts: embedding the query, a vector search, often a reranker, and then a prefill over the retrieved chunks. Each step adds time, and the chunks are re-encoded on every request.

In CAG the documents were encoded once during preloading. A request only prefills the short question, then decodes. There is no network hop to a vector store and no reranking.

One honest nuance: with a very long cache, each decoded token attends over more keys, so per-token decode speed can be slightly lower. The gain comes from skipping retrieval and re-prefill, which are usually the bigger costs.

Retrieval errors come from the selection step: poor chunk boundaries, an embedding that misjudges similarity, a top-k limit that drops the second document needed, or a reranker that demotes the right passage.

CAG has no selection step. The model sees every document, so a multi-hop question can combine facts from different files without a retriever having to guess both beforehand.

The failure mode does not vanish, it moves. Instead of the retriever missing a fact, the model can fail to attend to it in a very long prompt. So CAG removes retrieval errors, not reading errors.

It grows linearly with the number of tokens. Each token stores one key and one value vector in every layer:

bytes per token = 2 x layers x KV heads x head dim x bytes per value

For Llama 3.1 8B (32 layers, 8 KV heads, head dim 128, FP16) that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or 128 KB per token.

That is on top of the model weights, so long CAG contexts can need as much memory for the cache as for the model itself.

1. Read the model config: layers, number of KV heads and head dimension.
2. Compute bytes per token with the formula, using 2 bytes for FP16 or BF16.
3. Multiply by the number of cached tokens.
4. Add room for the question, the answer and prefill activations.
5. Multiply by the number of independent cache copies you keep in memory.

def kv_bytes(layers, kv_heads, head_dim, tokens, bytes_per=2):
    return 2 * layers * kv_heads * head_dim * tokens * bytes_per

# Llama 3.1 70B, 128K tokens, FP16
print(kv_bytes(80, 8, 128, 128_000) / 1e9)   # about 42 GB

The 70B model needs roughly 320 KB per token, so a full 128K cache is around 40 GB before weights. Sharing one prefix across requests, as vLLM does, keeps the number of copies at one.

A token's key and value are not a fixed property of its text. They depend on every earlier token (through causal attention in all layers above the first) and on its position (for example RoPE position encoding).

If you compute a chunk's cache on its own and later place it after a different set of documents, its keys and values no longer match what the model would have produced in that context. The result is degraded answers, since cross-document attention is missing.

That is why exact reuse only works for an identical prefix. Techniques such as CacheBlend try to approximate non-prefix reuse by selectively recomputing parts of the cache.

• The corpus fits in the window with comfortable headroom (a common target is half or less).
• Knowledge is stable, changing weekly or slower rather than hourly.
• Query volume is high against the same documents, so preloading is amortised.
• Questions need cross-document reasoning that a top-k retriever might split apart.
• Latency matters and you want to avoid a retrieval hop.
• The team is small and a vector stack would be extra operational weight.

If most of these hold, prototype CAG first. If two or three fail, RAG or a hybrid is safer.

• The corpus is larger than the context window, even after cleaning and deduplication.
• Data changes constantly, such as tickets, prices or news, and rebuilding a cache each time is wasteful.
• Per-user permissions must be enforced at document level.
• You need chunk-level citations or an auditable record of exactly which passages were supplied.
• Your model has a short window or limited GPU memory.

RAG also costs less per query when each question needs only a few hundred tokens of the corpus and you pay per input token.

Fine-tuning changes the model's weights so knowledge is baked in. CAG leaves the weights alone and supplies knowledge as context with a precomputed cache.

Fine-tuning is better for style, format and behaviour. CAG is better for facts that must stay accurate and updatable.

Both put the full documents in the prompt, so answer quality is similar. The difference is what happens on every call.

Plain long-context prompting re-processes all document tokens on each request. CAG processes them once and reuses the stored keys and values.

Take a 100K-token corpus and 1,000 queries a day. Without caching you prefill about 100 million document tokens a day. With CAG you prefill 100K once, plus the short questions.

So CAG is not a new way to read documents, it is a way to stop paying repeatedly for reading them.

vLLM stores KV memory in fixed-size blocks (PagedAttention). With automatic prefix caching, each full block is identified by a hash of its tokens together with the hash of the block before it.

1. A new request is split into blocks and hashed.
2. Hashes are matched against blocks already in GPU memory.
3. Matching blocks are shared by reference, and only the remaining tail is prefilled.
4. Unused blocks are evicted with a least-recently-used policy when memory runs low.

from vllm import LLM
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct",
          enable_prefix_caching=True, max_model_len=65536)

Recent vLLM versions enable this by default, but the flag shows the intent. Because the hash chains through previous blocks, any early change breaks every later block.

SGLang keeps KV caches in a radix tree keyed by token sequences. A shared system prompt becomes the trunk, document sets become branches, and individual questions become leaves.

flowchart TD
  R[Root] --> S["System prompt"]
  S --> A["Handbook A tokens"]
  S --> B["Handbook B tokens"]
  A --> Q1["Question 1"]
  A --> Q2["Question 2"]
  B --> Q3["Question 3"]

A new request walks the tree to find the longest matching prefix and reuses that node's cache. Leaves that are no longer referenced are evicted by LRU, and the scheduler can group requests that share prefixes.

It suits CAG deployments where several corpora or few-shot setups share a common beginning.

Put everything that never changes first, and everything that changes last. Since a cache is a prefix match, the stable part must come before any variable part.

• Never put timestamps, request IDs or user names near the top.
• Serialise JSON with sorted keys and consistent whitespace.
• Keep tool definitions identical between calls.
• Place cache breakpoints right after the documents.

Provider prompt caching is CAG delivered as a service. You send the same long prefix, the provider reuses the KV state it computed earlier, and you pay a reduced rate for those cached tokens.

The differences are control and lifetime. You cannot download or inspect the cache, it expires after a short time unless refreshed, and writing it may cost more than a normal request. You also have to mark where the cache should end.

client.messages.create(
    model="claude-sonnet-5-5", max_tokens=500,
    system=[
      {"type": "text", "text": "Answer only from the handbook."},
      {"type": "text", "text": HANDBOOK,
       "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": question}])

Check usage.cache_read_input_tokens in the response to confirm hits. Exact pricing and lifetimes change, so verify them in the provider docs.

1. Hash the sources. Keep a content hash per document and the whole corpus so you know when the cache is stale.
2. Rebuild on change. A CI job or scheduler re-prefills the prefix when the hash changes.
3. Swap versions safely. Build the new cache beside the old one, then switch traffic, so users never hit a half-built cache.
4. Order by volatility. Put rarely changing documents first and fast-changing ones last, so an edit only invalidates the tail.
5. Append urgent facts. New documents added at the end leave the existing prefix valid.

If updates arrive faster than a rebuild can finish, that is a signal to move the fast-changing part to RAG.

The prompt cannot be built. The request fails or the oldest tokens are truncated, and either way the model works with incomplete knowledge. You need to reduce or split the problem.

Most teams end up with a hybrid, because the long tail rarely deserves a slot in the cache.

Studies of long-context models (Liu et al., 2023) found that accuracy is highest when the relevant passage sits at the start or end of the prompt and dips when it is buried in the middle.

CAG makes this more likely, because the corpus is large and the answer can be anywhere in it. Removing the retriever does not help if the model skims past the passage.

• Put the most important documents first or last.
• Keep the corpus lean, since every extra page dilutes attention.
• Label each document with a clear ID and title so the model can locate it.
• Ask the model to quote the supporting passage before answering.
• Test with needle-in-a-haystack checks at different depths.

Newer models are noticeably better here, but you should measure your own model rather than assume.

1. Log cached token counts per request (for example cache_read_input_tokens or the engine's hit metrics).
2. Diff two prompts that should match and find the first differing character.
3. Look for dynamic text near the top: timestamps, request IDs, user names, random few-shot order.
4. Check serialisation: unsorted JSON keys, trailing whitespace and changing delimiters all alter tokens.
5. Check parameters: a changed model version, tool list or system setting can invalidate the prefix.
6. Check expiry: sparse traffic may let a time-limited cache lapse, so a periodic keep-alive may help.
7. Check minimum length: prefixes shorter than the provider's threshold are not cached.
8. Check routing: with several replicas, send repeat prefixes to the same one.

Most real problems turn out to be step 3 or step 4.

KV cache quantization stores keys and values in lower precision, such as FP8 or INT8 instead of FP16. Fewer bytes per value means a smaller cache, so a longer corpus fits or more caches fit on the same GPU.

The cost is a small risk of accuracy loss, which tends to grow at lower bit widths and on tasks that need exact recall. Research methods such as KIVI and KVQuant push towards 2-4 bit caches, and serving engines such as vLLM expose a KV cache dtype option.

Always re-run your evaluation set after switching precision.

In standard multi-head attention (MHA) every query head has its own key and value head. Grouped-query attention (GQA) lets several query heads share one KV head, and multi-query attention (MQA) shares a single one across all heads.

Since the cache stores only keys and values, fewer KV heads means a proportionally smaller cache.

At 128K tokens, the MHA cache would be around 64 GB, while the GQA one is 16 GB. That is why GQA models are the practical choice for CAG.

Persisting a cache avoids repeating the prefill after a restart or when scaling out. You serialise the cache tensors, store them with metadata, and load them into GPU memory at startup.

legacy = cache.to_legacy_cache()                 # tuple of (key, value) per layer
torch.save({"kv": legacy, "meta": {"model": name, "corpus_sha": sha}},
           "kv_handbook_v12.pt")

blob = torch.load("kv_handbook_v12.pt", weights_only=True)
cache = DynamicCache.from_legacy_cache(blob["kv"])

• Match exactly: store model name, dtype, tokenizer and corpus hash, and refuse to load on a mismatch.
• Mind the size: a 128K cache for an 8B model is about 16 GB, so load time depends on disk and PCIe bandwidth.
• Stay safe: never load pickled files from untrusted sources.

Helper names change between Transformers versions, so verify against your installed release.

Cost depends on how you run the model. Build a per-query model for each side rather than assuming CAG is cheaper.

• CAG: one-time prefill, cache storage (memory), plus question and output tokens each query.
• RAG: embedding and vector DB costs, plus the retrieved chunks as input tokens each query.

Take a hosted API with a 100K-token corpus. With provider caching, cached reads might be billed at about 10% of the normal input price, so each query costs the equivalent of roughly 10K full-price tokens. A RAG query that retrieves five 500-token chunks pays for about 2.5K full-price tokens.

So on an API, CAG may not win on token spend, but it can win on latency and answer quality. On self-hosted GPUs the cost is cache memory and throughput, so measure both. Use real prices from your provider.

The design question is what belongs in the cache, what is retrieved, and how they are combined without breaking the cache.

flowchart LR
  Q["User question"] --> R{Router}
  R -->|Covered by core knowledge| C["Cached core prefix"]
  R -->|Needs fresh or long-tail data| S[Retriever]
  S --> T["Top-k chunks appended after cache"]
  C --> L[LLM]
  T --> L
  L --> A[Answer]

1. Core tier: stable, high-traffic documents go into the cached prefix.
2. Retrieval tier: volatile or rarely used documents live in a vector store.
3. Append, never insert: retrieved text goes after the prefix so the cache stays valid.
4. Budget tokens: reserve window space for retrieved chunks, question and answer.
5. Deduplicate: drop retrieved chunks already present in the core cache.

Measure each tier separately, otherwise you cannot tell which one is helping.

A CAG cache holds knowledge for the model, and the model can repeat anything in its context. So access control must happen before content enters a cache, not by instructing the model to hide things.

• One cache per permission tier: for example public, employee and finance caches, with users routed to the highest tier they may see.
• Never mix restricted and open documents in one shared prefix.
• Use RAG for per-user data, where document-level filters are natural.
• Isolate tenants: do not share prefix caches across customers, because shared caches can leak information through timing differences.
• Log which cache version served each answer for audits.

A prompt that says "do not reveal salaries" is a guideline, not a security control.

sequenceDiagram
  participant U as User
  participant S as Serving layer
  participant K as Cache manager
  participant M as LLM
  U->>S: Question
  S->>K: Load cache for corpus version
  K-->>S: KV cache (preloaded prefix)
  S->>M: Question tokens + cache
  M-->>S: Prefill question, decode answer
  S->>K: Crop cache to original length
  S-->>U: Answer

1. The serving layer receives the question and picks the cache matching the current corpus version.
2. The cache manager returns the preloaded KV state, or builds it on a miss.
3. Only the question tokens are prefilled. Their keys and values attach to the existing cache.
4. The model decodes the answer token by token, attending over the cache plus the new tokens.
5. The cache is cropped back to its original length so the next request starts clean.
6. The answer is returned along with metrics such as TTFT and cached token count.

Compare both systems on the same model, corpus and questions, and measure quality, speed and cost together.

Build the test set from three groups: single-hop questions (SQuAD style), multi-hop questions (HotPotQA style) and questions the corpus cannot answer. Add needle-in-a-haystack probes at different positions to catch long-context weaknesses.

Run each system repeatedly under realistic load, because cold-start and warm-cache numbers differ widely.

Each request leaves its question and generated answer inside the cache. Without a reset these tokens pile up in front of the next request.

• Memory growth: the cache gets larger every query until the GPU runs out of memory.
• Window overflow: eventually the cache exceeds the model's context window and requests fail or get truncated.
• Cross-query contamination: the model may answer from earlier questions, and one user's data can surface in another user's response.
• Slower decoding: attention scans an ever longer cache.

origin_len = cache.get_seq_length()
try:
    answer = generate(question, cache)
finally:
    cache.crop(origin_len)     # always reset, even on errors

If you do want multi-turn memory, keep a separate per-session cache rather than growing the shared one.

RAG systems retrieve different chunk combinations, so exact prefix reuse rarely applies. Reusing each chunk's independently computed cache is fast but skips cross-attention between chunks, which hurts answer quality.

CacheBlend (Yao et al.) precomputes the KV cache of each chunk separately. At query time it concatenates them, then selectively recomputes the KV values of a small fraction of tokens whose values would change most because of the new context, layer by layer.

The aim is a cache close to what a full prefill would give, at a much lower cost, so TTFT drops with little quality loss. It is a research technique that shows how CAG-style reuse can extend beyond one fixed prefix.

For a pure CAG system with a fixed prefix you do not need it. It matters when you mix cached chunks dynamically.

Putting documents in context reduces hallucination but does not remove it. The model may still guess when the answer is missing, or blend two conflicting documents.

• Instruct grounding: answer only from the documents, and say "not found" otherwise.
• Label documents with IDs and titles, and ask for citations to those IDs.
• Require quotes: have the model extract the supporting sentence before answering.
• Clean the corpus: remove duplicates and outdated versions that contradict each other.
• Lower the temperature for factual question answering.
• Test with traps: unanswerable and near-miss questions, and track the refusal rate.

A second verification pass can catch unsupported claims, at the price of extra latency.

1. Estimate first: weights plus KV cache (2 x layers x KV heads x head dim x tokens x bytes) plus working memory. Compare it with GPU capacity.
2. Look for duplicate copies: a deep copy of the cache per request multiplies memory. Share one prefix and crop instead.
3. Check prefill peaks: processing a very long prompt in one pass needs large activation memory. Use FlashAttention or chunked prefill.
4. Shrink the cache: trim and deduplicate the corpus, or quantize the KV cache to FP8.
5. Choose a GQA model with fewer KV heads.
6. Spread the load: use tensor parallelism across GPUs, or offload cold caches to CPU memory or disk.
7. Limit concurrency: lower the batch size or the maximum model length.

Most surprises come from step 2 or step 3, because the cache itself is predictable but copies and prefill peaks are not.