AI / AI System Design Interview questions

AI system design is the practice of planning the full architecture around a machine learning model so it can serve real users reliably, quickly and at an acceptable cost. The model is only one box; the rest is data, serving, monitoring and safety.

In interviews you are usually expected to cover:

• data collection, cleaning and storage
• choosing or training a model (hosted API vs self-hosted)
• the serving layer, with latency and throughput targets
• retrieval, caching and orchestration for LLM apps
• evaluation, monitoring and feedback loops
• guardrails, privacy and cost control

It differs from classic system design because outputs are probabilistic, quality is hard to capture in one metric, and inference can be expensive.

A typical LLM application is made of these parts:

1. Client / UI that captures input and streams output.
2. API gateway for authentication, rate limits and routing.
3. Orchestrator that builds prompts, calls tools and manages conversation state.
4. Retriever (embedding model plus vector or search index) for private knowledge.
5. LLM, either a hosted API or a self-hosted inference server.
6. Guardrails for input and output checks.
7. Observability and evaluation for logs, traces, cost and quality metrics.

Caches and a conversation store usually sit beside the orchestrator. Not every app needs all of them; a simple summarizer may only need a gateway, a prompt and a model.

Retrieval-Augmented Generation (RAG) fetches relevant documents at query time and places them in the prompt, so the LLM answers from that evidence instead of relying only on its training data.

Basic flow:

1. Split documents into chunks and embed them (offline).
2. Store the vectors in an index.
3. At query time, embed the question and retrieve the top-k chunks.
4. Insert the chunks into the prompt and generate an answer, ideally with citations.

RAG reduces hallucination on private or fresh data and lets you update knowledge without retraining. Quality depends heavily on chunking and retrieval, so a weak retriever gives weak answers no matter how strong the model is.

A vector database stores embedding vectors and finds the ones closest to a query vector, usually with an approximate nearest neighbour (ANN) index such as HNSW or IVF.

Beyond similarity search it typically offers metadata filtering, upserts and deletes, sharding and replication. Examples include Pinecone, Weaviate, Milvus, Qdrant, and extensions such as pgvector for PostgreSQL.

Similarity is measured with cosine similarity, dot product or Euclidean distance. You reach for one when keyword matching misses meaning, for example when "refund policy" should match "how to get my money back". For small datasets, an in-memory index or pgvector is often enough.

Embeddings are dense numeric vectors that represent text, images or other data so that similar meaning lands close together in vector space.

An embedding model turns "reset my password" and "I forgot my login" into vectors with high cosine similarity, even though the two sentences share few words.

Common uses are semantic search, RAG retrieval, clustering, deduplication, recommendations and classification. Things to watch:

• Use the same model for indexing and querying.
• Dimension (for example 384 to 3072) affects storage and speed.
• Changing the model means re-embedding the whole corpus.

A token is the unit of text an LLM reads and writes. It can be a word, part of a word, punctuation or whitespace, depending on the tokenizer. As a rough rule, English averages about 4 characters, or 0.75 words, per token.

Tokens matter in system design because they drive:

• Cost: APIs usually bill per input and output token.
• Latency: output is generated one token at a time.
• Limits: the context window and max output are counted in tokens.

Code, non-English text and numbers often use more tokens per character, so measure with the model's own tokenizer rather than guessing.

The context window is the maximum number of tokens a model can consider in one request. It covers the system prompt, history, retrieved documents, the user message and the generated output together.

If the total goes over the limit, the request fails or older content must be dropped. Even within the limit, very long prompts raise cost and latency, and models can miss details buried in the middle.

Common ways to stay inside the window: summarize older turns, retrieve only the top chunks, trim tool output, and cache stable prefixes. A bigger window does not remove the need for retrieval; it only changes the trade-off.

Model serving is the infrastructure that loads a trained model and answers prediction requests in production, usually behind an API.

A serving stack handles loading the model onto GPUs or CPUs, request batching, autoscaling, versioning, health checks and metrics. For LLMs, specialized engines such as vLLM, TensorRT-LLM or Hugging Face TGI add features like continuous batching and paged KV cache. For classic models, TensorFlow Serving, TorchServe, Triton and KServe are common choices.

The goals are low latency, high throughput and predictable cost, while still allowing new model versions to roll out safely.

A system prompt is the instruction block sent with every request that defines the assistant's role, tone, rules and output format before the user speaks.

It typically contains the persona, allowed and disallowed behaviour, tool descriptions, formatting rules and sometimes few-shot examples. Because it is sent each time, a long system prompt adds cost and latency, which makes it a good candidate for prompt caching.

Treat it like code: version it, test changes against an evaluation set, and never put secrets in it, since users can often coax a model into revealing it.

Chunking is splitting source documents into smaller pieces before embedding them for retrieval. Retrieval returns whole chunks, so their size and boundaries decide what the model actually sees.

Common strategies:

• Fixed size with overlap (for example 300-500 tokens, 10-20% overlap).
• Recursive / structure-aware, splitting on headings, paragraphs and sentences.
• Semantic, splitting where the topic shifts.

Chunks that are too small lose context; chunks that are too large dilute relevance and waste prompt tokens. Keep metadata such as title, section and source URL with each chunk for filtering and citations.

Guardrails are checks placed around an LLM to keep inputs and outputs safe, on-topic and within policy.

On the input side: prompt-injection detection, PII redaction, topic filters and length limits. On the output side: toxicity and policy classifiers, schema validation for structured output, grounding checks against retrieved sources, and blocking of leaked secrets. Tool calls also need permission checks.

Guardrails can be rules, regex, small classifier models or a second LLM acting as a judge. They add latency and can cause false refusals, so measure both the bad content they block and the good content they wrongly block.

A standard ML pipeline has these stages:

1. Data ingestion from databases, logs or streams.
2. Data validation and cleaning, including schema and drift checks.
3. Feature engineering and storage, often in a feature store.
4. Training with experiment tracking.
5. Evaluation on held-out data against a baseline.
6. Registration and deployment through a model registry and CI/CD.
7. Serving, online or batch.
8. Monitoring and retraining, triggered by drift or a schedule.

For LLM apps, training is often replaced by prompt design, retrieval setup and evaluation, but the lifecycle keeps the same shape.

An AI agent is a system where an LLM decides which actions to take, calls tools to carry them out, observes the results and repeats until the goal is met.

Unlike a fixed workflow, where code defines each step, an agent chooses its own next step. The core parts are the model, a set of tools (search, code execution, APIs), memory or state, and a loop with stop conditions.

Agents suit open-ended tasks but cost more, run slower and are harder to test. If the steps are known in advance, a plain workflow is usually the better choice.

Function calling, also called tool use, lets a model return a structured request to run a named function with arguments, instead of plain text. Your code executes the function and sends the result back so the model can continue.

1. Describe each tool with a name, description and JSON schema for its arguments.
2. The model replies with a tool call.
3. Your application validates the arguments, runs the tool and returns the output.
4. The model writes the final answer using that output.

{"name": "get_order_status",
 "description": "Look up an order by ID",
 "parameters": {"type": "object",
   "properties": {"order_id": {"type": "string"}},
   "required": ["order_id"]}}

The model never executes anything itself. Always validate arguments and enforce permissions in your own code.

Model drift is the decline in a model's performance over time because the world changes after training.

Two main types:

• Data drift: the input distribution changes, such as new slang or a new customer segment.
• Concept drift: the relationship between inputs and the correct output changes, such as fraud patterns evolving.

For LLM apps, drift also appears when knowledge goes stale, when a provider updates a model version, or when user behaviour shifts. Monitor input statistics, output quality and business metrics, and retrain, re-index or update prompts when thresholds are crossed.

Latency is how long one request takes; throughput is how many requests or tokens the system handles per second. They often pull in opposite directions.

For LLMs, latency splits into time to first token (TTFT), driven by prompt size and queueing, and inter-token latency, driven by decoding speed. Throughput is usually measured in tokens per second per GPU.

RAG is preferred for company knowledge because facts change often and need to be traceable. Updating a document in the index takes minutes; retraining a model takes hours or days and still may not recall the fact reliably.

• Freshness: re-index changed files instead of retraining.
• Citations: you can show which source supported the answer.
• Access control: filter retrieval by the user's permissions, which is impossible once data is baked into weights.
• Cost: no GPU training run for every update.
• Deletion: removing a document removes that knowledge.

Fine-tuning teaches style and behaviour better than it teaches facts.

Choose fine-tuning when the problem is about how the model behaves, not what it knows.

• A consistent output format, tone or brand voice that long prompts fail to enforce.
• A narrow, high-volume task where a smaller fine-tuned model can replace a large one and cut cost and latency.
• Domain-specific language or labels, such as medical coding, where examples are plentiful.
• Prompts that have grown huge just to carry instructions and examples.

You need a few hundred to a few thousand good examples, an evaluation set, and a plan to retrain when the base model changes. Often the best answer combines both: fine-tune for behaviour, retrieve for facts.

During generation, each new token needs the attention keys and values of all earlier tokens. The KV cache stores those values after they are computed once, so the model does not recompute the whole prefix for every new token.

Without it, the work per token would grow with the full sequence length. With it, each step only processes the newest token and reads the cached keys and values.

The trade-off is memory. The cache grows with sequence length, batch size, layers and heads, and it often becomes the limit on how many concurrent requests a GPU can hold. Paged attention, grouped-query attention and KV quantization exist to shrink or manage it.

A GPU is underused when it handles one request at a time, because decoding is limited by memory bandwidth rather than compute. Loading the model weights for a single sequence wastes that bandwidth. Batching processes several sequences together, so the same weight read serves many users.

The result is much higher tokens per second and lower cost per request. The price is some waiting while a batch forms and, if the batch is large, slower individual responses.

Static batching waits for the slowest request in the batch. Modern servers use continuous (in-flight) batching, which adds and removes requests at every step so GPUs stay busy. You tune max batch size and queue timeout against your latency target.

Vector search matches meaning but can miss exact terms such as product codes, error IDs, names or rare acronyms. Keyword search (BM25) nails exact terms but misses paraphrases. Hybrid search runs both and merges the results.

A common merge method is Reciprocal Rank Fusion (RRF), which combines ranks from each list without needing comparable scores. A reranker often follows.

Example: the query "error E4031 on login" needs the exact code from BM25 and the login context from vectors.

First-stage retrieval is fast but coarse. It compares a query vector with chunk vectors that were embedded separately, so it can return near-misses. A reranker is a cross-encoder that reads the query and each candidate together and scores true relevance.

Typical pattern: retrieve the top 50-100 cheaply, rerank them, then pass the best 5-10 to the LLM.

Benefits are higher precision, less noise in the prompt, lower token cost and fewer hallucinations. The cost is extra latency (tens to a few hundred milliseconds) and compute, so rerank only a limited candidate set. Skip it if retrieval is already accurate or latency is tight.

LLMs generate text token by token. Without streaming, the user waits for the whole answer. With streaming, tokens are sent as they are produced, so the first words appear after the time to first token instead of after total generation time.

Total time does not change, but the interface feels responsive and users can start reading, or cancel early.

Common transports are Server-Sent Events (SSE) and WebSockets. Design notes:

• Disable proxy buffering so chunks are not held back.
• Run output guardrails on partial text or in sliding windows.
• Cancel generation when the client disconnects, to save GPU.
• Send usage and final metadata in a closing event.

There is no universal number, so you choose it by testing against your own content and questions.

1. Start with 300-500 tokens and 10-20% overlap for prose.
2. Prefer structure: split on headings, sections or functions rather than at arbitrary characters.
3. Build a small evaluation set of real questions with known source passages.
4. Measure recall@k and answer quality for a few sizes (for example 200, 400, 800).
5. Pick the best, then consider parent-child retrieval: embed small chunks but return their larger parent for context.

FAQs and short records need small chunks; legal or technical manuals often need larger ones. Also check that the top-k chunks fit comfortably in the context budget.

A semantic cache stores past prompts as embeddings together with their answers. A new query is embedded and compared with cached ones; if similarity passes a threshold, the stored answer is returned without calling the LLM.

This turns a multi-second, paid call into a millisecond vector lookup for repeated or near-duplicate questions, which is common in support and FAQ bots. It differs from exact-match caching and from provider-side prompt caching, which only reuses a repeated prefix.

Risks and controls:

• A threshold set too low returns wrong answers for different questions.
• Scope the cache by tenant, language and permission to avoid leaking data.
• Set a TTL and invalidate when source documents change.
• Skip caching for personalized or time-sensitive queries.

Both are ANN index types, but they organize vectors differently.

HNSW builds a layered graph. Search starts at a sparse top layer and walks greedily toward the query, descending to denser layers. IVF clusters vectors into partitions with k-means, and a query probes only the nearest few clusters.

Pick HNSW for low latency and high recall when memory allows. Pick IVF-PQ for very large collections where memory is the constraint.

Evaluate retrieval and generation separately, then end to end, because failures in each look the same to a user.

Retrieval metrics (need labeled question-to-passage pairs): recall@k, precision@k, MRR, nDCG.

Generation metrics:

• Faithfulness / groundedness: is every claim supported by the retrieved context?
• Answer relevance: does it address the question?
• Correctness against a reference answer.

Use an LLM-as-judge for scale, calibrated with human review on a sample. Build a golden set from real queries, run it in CI whenever prompts, chunking or models change, and track online signals such as thumbs-down rate, escalations and citation clicks.

Prompt injection happens when untrusted text, from a user, web page or document, contains instructions that override your intended behaviour. There is no single fix, so you layer defences.

1. Separate instructions from data with clear delimiters, and tell the model to treat retrieved text as content only.
2. Least privilege for tools: read-only by default, scoped credentials, no open shell.
3. Human approval for sensitive actions such as payments or sending email.
4. Input and output filtering with classifiers for known attack patterns and for data exfiltration, such as links that embed secrets.
5. Isolation: do not let one component read untrusted content, hold sensitive data and send data out all at once.
6. Red-team regularly and log every tool call.

Use model routing when your traffic mixes easy and hard requests and sending everything to the largest model wastes money.

A router picks a model per request using rules (task type, length, tenant tier), a lightweight classifier, or a cheap model's confidence. Simple extraction goes to a small model; multi-step reasoning goes to a larger one. A fallback can escalate when the small model's answer fails validation.

Gains: lower average cost and latency with similar quality. Costs: router errors, more models to evaluate and monitor, and inconsistent tone across models. Skip it if volume is low or one model already meets your cost target.

Request-count limits alone do not work for LLMs, because one request can cost 100 tokens and another 100,000. Limit by tokens as well as requests.

• Use a token bucket or sliding window per API key, user and tenant, tracked in a shared store such as Redis.
• Estimate input tokens before the call and reconcile with actual usage afterwards.
• Cap max output tokens per request.
• Add concurrency limits to protect GPUs from long-running requests.
• Use priority queues so interactive traffic beats batch jobs.

Return HTTP 429 with a Retry-After header, and have clients retry with exponential backoff and jitter. Also enforce daily or monthly spend budgets per tenant to stop runaway agent loops.

The difference is when predictions are computed.

Choose online when the result depends on live context. Choose batch when results can be precomputed and looked up. Many systems mix both, for example precomputing candidate items nightly and re-ranking them online. Batch APIs from LLM providers are often noticeably cheaper for non-urgent work.

Monitor three layers, because ground-truth labels often arrive late or never.

1. Inputs: compare current feature or embedding distributions with a training baseline using PSI, the KS test or KL divergence.
2. Outputs: track prediction distribution, confidence, refusal rate, response length and format-error rate.
3. Outcomes: when labels arrive, track accuracy, plus proxy signals such as thumbs-down rate, escalations and retries.

For LLM apps, run a fixed golden set on a schedule and alert on score drops, which also catches silent provider model updates. Alert on sustained shifts rather than single spikes, and wire alerts to a runbook: investigate, refresh data, re-index, update prompts or retrain.

Quantization stores model weights, and sometimes activations or the KV cache, in lower precision, for example INT8 or 4-bit instead of FP16.

• Memory: a 70B-parameter model needs about 140 GB at FP16 but roughly 35-40 GB at 4-bit, so it may fit on fewer GPUs.
• Speed: decoding is memory-bandwidth bound, so moving fewer bytes per token raises tokens per second.
• Cost: fewer or cheaper GPUs per replica.

The trade-off is a small quality loss that grows at lower bit widths and varies by task. Methods such as GPTQ, AWQ and FP8 try to limit it. Always evaluate the quantized model on your own tasks before rollout.

An agent runs a loop of think, act and observe until it can answer or hits a limit.

1. Receive the goal and load state, tools and memory.
2. The LLM decides: answer, or call a tool.
3. The orchestrator validates and runs the tool call.
4. The result is appended to the context as an observation.
5. Repeat from step 2.
6. Stop on a final answer, step limit, token budget, timeout or error.

flowchart TD
A["User goal"] --> B["LLM reasons"]
B --> C{Tool call needed?}
C -- Yes --> D["Validate and run tool"]
D --> E["Add result to context"]
E --> B
C -- No --> F["Final answer"]

Hard limits matter. Without a maximum step count and token budget, a confused agent can loop forever and run up a large bill. Log every step so failures can be debugged.

A production RAG request passes through several stages, each with its own latency and failure modes.

1. Auth and rate limit at the gateway; identify the user and tenant.
2. Query understanding: rewrite follow-up questions using chat history, and optionally expand or decompose them.
3. Retrieve: embed the query, run hybrid search with permission filters, take the top 50.
4. Rerank and keep the best 5-8 chunks.
5. Assemble the prompt within the token budget: system prompt, history, chunks with source IDs, question.
6. Generate with streaming, asking for citations.
7. Post-process: output guardrails, citation check, and logging of trace, tokens and cost.

flowchart LR
U[User] --> G[Gateway]
G --> Q["Query rewrite"]
Q --> R["Hybrid retrieval + ACL filter"]
R --> K[Reranker]
K --> P["Prompt builder"]
P --> L[LLM]
L --> O["Output guardrails"]
O --> U

A typical budget is 50-150 ms for retrieval, 100-200 ms for reranking, and about a second or less to the first token. Cache embeddings and frequent answers, and when retrieval confidence is low, reply that the information was not found instead of letting the model guess.

An LLM gateway is the single entry point between your applications and one or more model providers. Design it around isolation, control and resilience.

• Identity and quotas: per-tenant API keys, token-based rate limits and spend budgets.
• Routing: model selection, provider failover, regional routing for data residency.
• Caching: exact and semantic caches, partitioned by tenant.
• Safety: shared input and output guardrails, PII redaction, per-tenant policies.
• Observability: request IDs, traces, token and cost attribution per tenant, audit logs.
• Resilience: timeouts, retries with backoff, circuit breakers, queueing for bursts.

flowchart LR
A[Apps] --> B["Gateway: auth and quota"]
B --> C[Cache]
B --> D[Guardrails]
D --> E[Router]
E --> F["Provider A"]
E --> G["Provider B"]
E --> H["Self-hosted model"]
B --> I[(Usage and audit logs)]

Keep the gateway stateless and scale it horizontally, with counters in Redis and logs on a stream. Beware the noisy neighbour: one tenant's burst should queue or be throttled, not starve the others, so use per-tenant concurrency caps and priority lanes.

Attack cost in order of impact, and measure cost per successful task rather than per call.

1. Right-size the model: route easy tasks to smaller models, or fine-tune a small one for narrow work.
2. Trim tokens: shorter prompts, fewer retrieved chunks, summarized history, capped max output.
3. Cache: provider prompt caching for stable prefixes, plus exact and semantic response caches.
4. Batch non-urgent work through batch APIs, often at a discount.
5. Self-hosting tactics: quantization, continuous batching, speculative decoding, autoscaling on queue depth, spot instances.
6. Control loops: step limits and budgets for agents.

Set per-feature cost dashboards and alerts first; you cannot cut what you cannot attribute.

Split TTFT into stages and measure each before changing anything.

1. Network and gateway: check TLS, region distance and proxy buffering.
2. Queueing: if the server is saturated, requests wait. Check queue depth and GPU utilization, then scale out or limit concurrency.
3. Prompt size: prefill cost grows with input length. Look for bloated system prompts, too many chunks or long history, and trim, summarize or enable prefix caching.
4. Pre-model work: retrieval, reranking, guardrails and tool calls often add more than the model does. Check traces and parallelize independent steps.
5. Model and hardware: cold starts, wrong batch settings, or a model that is oversized for the task.

Look at percentiles (p50, p95, p99) rather than averages, since tail latency usually comes from queueing or very long prompts. Fix the biggest slice first, then re-measure.

At 100M items, raw vectors alone are large: 100M x 768 dimensions x 4 bytes is about 300 GB, so storage and index choice drive the design.

1. Ingestion: stream documents through a queue, chunk, embed in GPU batches and write to the index with metadata. Make it idempotent and replayable.
2. Index: shard across nodes; use HNSW if memory allows, or IVF-PQ and disk-based indexes to compress. Quantizing vectors to int8 or binary cuts memory 4x or more.
3. Query path: embed the query (cache hot ones), run ANN search per shard in parallel, merge the top-k, then rerank a small set.
4. Filtering: store metadata beside vectors for tenant and permission filters, and test filtered recall, since filters can hurt ANN quality.
5. Updates: handle upserts and deletes, with periodic compaction or re-indexing.
6. Replication for availability and read scaling.

Track recall against exact search on a sample, p95 latency and index freshness lag. Plan for re-embedding when the model changes, using a blue/green index.

It depends on the API. Most return an error, some truncate silently, and some wrappers drop the oldest messages. None of these is a good user experience, so manage the budget yourself before each call.

• Sliding window: keep the system prompt and the last N turns.
• Summarization: compress older turns into a running summary and keep recent turns verbatim.
• Retrieval memory: store old turns or facts in a vector store and retrieve those relevant to the new message.
• Trim bulky content: truncate large tool outputs or documents, keeping only what matters.
• Reserve output space: subtract max output tokens from the available input budget.

Count tokens with the model's tokenizer before sending. Summaries can lose details, so pin critical facts, such as user preferences or IDs, in a structured state block that is never summarized away.

Split memory into types, because each has different storage and retrieval needs.

Write path: after a session, an extraction step (LLM or rules) pulls out durable facts, deduplicates them against existing memories and stores them with source, time and confidence. Not every message deserves saving.

Read path: at the start of a turn, retrieve by relevance, recency and importance, then inject only a few short items into the prompt.

Hard problems are stale or contradictory memories (use updates and expiry), privacy (let users view, edit and delete memories, scoped per user), and memory poisoning from untrusted content. Evaluate by testing whether the agent recalls and correctly applies facts weeks later.

First find out whether the right context reached the model.

1. Inspect the trace: for a bad answer, log the query, rewritten query, retrieved chunks and final prompt.
2. Retrieval miss: the answer is not in the chunks. Fix chunking, add hybrid search, query rewriting or a reranker, or improve the source data.
3. Retrieval hit, generation miss: the answer is present but ignored or distorted. Tighten the prompt ("answer only from the context, otherwise say you don't know"), remove irrelevant chunks, lower temperature, and place key chunks where the model attends well.
4. Contradictory or stale sources: deduplicate, and add dates and versions as metadata.
5. Add verification: require citations, run a groundedness check, and refuse when confidence is low.

Confirm each fix on a golden set, tracking faithfulness and recall@k, so you do not trade one failure for another.

Static batching groups requests, runs them together and returns only when the longest one finishes. Since output lengths vary, short requests finish early and their GPU slots sit idle.

Continuous (in-flight) batching schedules at the level of a single decoding step instead of a whole request.

1. The scheduler holds a running batch and a waiting queue.
2. Each iteration, the model generates one token for every sequence in the running batch.
3. Sequences that hit an end token or max length leave immediately and free their KV cache blocks.
4. Waiting requests are admitted into the free slots, subject to available KV memory.
5. New requests run a prefill pass over their prompt, then join decoding.

It works well with paged attention, which allocates KV cache in fixed-size blocks, avoiding fragmentation and making admit and evict cheap. Some engines add chunked prefill so long prompts do not stall decoding for others. The outcome is much higher throughput at similar per-token latency, and the scheduler policy controls fairness and tail latency.

Assume the model can be wrong or manipulated, and put the safety in the system rather than in the prompt.

1. Least privilege: each tool gets narrowly scoped credentials, read-only unless writes are required.
2. Schema validation: reject calls with bad or unexpected arguments, and allowlist values such as table names or domains.
3. Sandboxing: run code in containers or microVMs with no default network access, CPU and memory limits and a time limit.
4. Risk tiers: low-risk actions run automatically; high-impact ones (payments, deletes, external emails) need human approval.
5. Idempotency and undo: use idempotency keys and soft deletes so retries are safe and mistakes are reversible.
6. Budgets: cap steps, spend and tool calls per run.
7. Audit: log every call with arguments and results, tied to the user.

Keep untrusted content (web pages, emails) from directly driving high-risk tools, since that is the usual prompt-injection path.

Treat a model change like any risky release: test offline, then expose it gradually with a quick way back.

1. Offline evaluation: run the golden set and regression tests, and compare quality, latency and cost with the current model.
2. Shadow mode: send real traffic to the new model in parallel and record outputs, but show users only the old result.
3. Canary: route 1-5% of traffic, per user or tenant so the experience stays consistent, and watch error rate, latency, guardrail hits, thumbs-down and cost.
4. A/B test if you need statistical evidence on business metrics.
5. Ramp up in steps, with automatic rollback on threshold breaches.

flowchart LR
A["Offline eval"] --> B["Shadow traffic"]
B --> C["Canary 1-5%"]
C --> D["Ramp 25-50%"]
D --> E["100%"]
C -- metrics regress --> R[Rollback]
D -- metrics regress --> R

Pin model versions instead of using floating aliases, keep the previous version warm for fast rollback, and re-run prompt tests, since prompts tuned for one model can behave differently on another.

Catalogues have millions of items, so scoring every one with a heavy model per request is impossible. The standard design splits the work into funnel stages.

1. Candidate generation (retrieval): fast methods reduce millions of items to a few hundred. Common sources are a two-tower model with ANN search, collaborative filtering, trending lists and recently viewed items.
2. Ranking: a heavier model (gradient boosted trees or a deep network) scores each candidate using user, item and context features such as time, device and history, predicting click, watch or purchase probability.
3. Re-ranking: apply business rules, diversity, freshness, deduplication and fairness constraints.

flowchart LR
A["Millions of items"] --> B["Candidate generation ~500"]
B --> C["Ranking model ~50"]
C --> D["Re-rank rules"]
D --> E["Shown to user"]

Features come from a feature store, with the same definitions online and offline to avoid train-serve skew. Offline you measure AUC or nDCG; online you A/B test. Watch for feedback loops, where the system only shows what it already predicts users like, and add exploration to counter that.

Neither always wins. Both split a model that is too large for one GPU, but they cut it differently and stress the interconnect differently.

For interactive serving, start with tensor parallelism inside a single node, since it keeps latency low. If the model still does not fit, add pipeline parallelism across nodes. For very large Mixture-of-Experts models, expert parallelism is also common.

Use the smallest degree of parallelism that fits, because every extra GPU in a group adds communication overhead, and consider quantization to avoid splitting at all.

The goal is to know, before release, whether a change to a prompt, model, retrieval step or tool made things better or worse.

1. Golden dataset: collect real queries, edge cases and past failures, with expected answers or rubrics. Version it, and keep a held-out part you never tune on.
2. Graders: use deterministic checks where possible (exact match, JSON schema, regex, unit tests for code). For open-ended output, use an LLM judge with a clear rubric, calibrated against human labels.
3. Metrics: task success, faithfulness, format validity, refusal rate, latency, tokens and cost.
4. Runner: execute in CI on each pull request, repeat samples to absorb randomness, and compare with the baseline using a threshold or significance test.
5. Online loop: feed thumbs-down and escalations back into the dataset.

Watch for judge bias, such as favouring longer answers or its own model's output, and for overfitting to the golden set. Review a sample of judged outputs by hand regularly.

Assume the model API will sometimes be slow, rate limited or down, and design for graceful degradation.

1. Timeouts and retries: set tight timeouts, retry only transient errors (429, 5xx) with exponential backoff and jitter, and cap attempts.
2. Circuit breaker: stop calling a failing provider for a short time so requests do not pile up.
3. Provider and model fallback: fail over to a secondary provider or a smaller model, after confirming your prompts are tested on it.
4. Cached or static response for common requests, or a clear "try again" message.
5. Output validation: if the JSON or schema is invalid, re-ask once with the error, then fall back.
6. Queue non-urgent work for later instead of failing.

Make writes idempotent so retries cannot double-execute actions. Alert on the fallback rate, since a quiet rise means the primary path is degrading, and run failure drills to confirm the fallback really works.

Start with a single agent and add more only when you can show a gain, because every extra agent adds cost, latency and failure modes.

Multi-agent designs help when a task splits into independent parts (parallel research), when one prompt would hold too many tools and instructions, or when you want separation of duties, such as a planner, a worker and a reviewer. A common pattern is an orchestrator that delegates to sub-agents and receives short summaries back, which also protects the main context window.

Risks include error compounding, repeated work, unclear ownership of state and runaway loops. Define clear roles, structured handoff messages, shared state with a single owner, and global budgets. Compare both designs on your evaluation set before committing.