AI / AI evals (evaluations) Interview questions

An AI eval is a repeatable test that measures how well a model, or a system built on a model, performs a specific job. You feed it a set of inputs, collect the outputs, and score them against a definition of good.

The score can come from code (exact match, regex, unit tests), from a human reviewer, or from another model acting as a judge. What matters is that the same eval can be rerun after every prompt, model, or pipeline change, so you can see whether things improved or got worse.

Think of it as the unit-test suite for non-deterministic software, except the assertions are often graded rather than strictly pass or fail.

The purpose of evals is to replace gut feeling with evidence whenever you change something in an LLM application. Without them, a prompt tweak that fixes one complaint can quietly break three other behaviors, and nobody notices until users do.

Evals do a few concrete jobs:

• Compare models, prompts, and retrieval settings on the same data
• Catch regressions before a release
• Show which failure categories are worth fixing first
• Give product and engineering one shared definition of quality

Teams that build evals early tend to iterate faster, because every change gets a quick verdict instead of a debate.

Evals are usually grouped by who or what does the scoring. Each type trades off cost, speed, and nuance.

You can also split evals by timing: offline evals run on a fixed dataset before release, while online evals use live traffic through A/B tests and sampled monitoring.

A golden dataset is a curated set of inputs paired with verified expected outputs or labels. Everyone agrees these answers are right, so they act as the ground truth you score against.

A good one usually contains:

• Representative real user queries
• Edge cases and tricky inputs
• Past production failures you never want to repeat

Start small, even 50 to 100 examples, and have domain experts review the labels. Keep it versioned so scores stay comparable over time.

LLM-as-a-judge means using a second language model to grade the outputs of the first one. The judge gets the output, a rubric, and often a reference answer, then returns a score or a verdict.

Rate the answer from 1 to 5 for factual accuracy.
Reference: {reference}
Answer: {answer}
Return JSON: {"score": <int>, "reason": "<one sentence>"}

It scales far better than human review and handles open-ended text that code cannot grade. The catch is that judges have biases, so you should check their scores against human labels before trusting them.

When an LLM outputs a label (spam or not, intent category, safe or unsafe), you can score it with standard classification metrics.

Accuracy misleads on imbalanced data. If 95% of emails are not spam, a model that never flags spam scores 95% accuracy and is useless.

Exact match marks an output correct only if it equals the reference string. In practice you normalize both sides first, by lowercasing, trimming whitespace, and stripping punctuation.

def norm(s):
    return " ".join(s.lower().strip().split())

def exact_match(pred, ref):
    return norm(pred) == norm(ref)

It works well for short factual answers, classification labels, and numeric results pulled out with a regex. It breaks down on free-form text, where many different phrasings are equally correct.

BLEU (Bilingual Evaluation Understudy) measures how many n-grams in the model output also appear in one or more reference texts. It combines n-gram precision for n = 1 to 4 with a brevity penalty that punishes outputs much shorter than the reference.

It was built for machine translation, and scores run from 0 to 1 (or 0 to 100). A higher score means more word-sequence overlap with the references.

It only sees surface overlap, so it is a rough proxy for quality rather than a measure of meaning.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) scores a generated text by its overlap with a reference, with an emphasis on recall. It is the standard automatic metric for summarization.

Like BLEU, it rewards matching words, not matching meaning. A faithful paraphrase can score low.

A benchmark is a public, standardized dataset plus a fixed scoring protocol, so different models can be compared on the same task. Well-known examples are MMLU for broad knowledge, GSM8K for grade-school math, HumanEval for code generation, and SWE-bench for real software issues.

Benchmarks are great for picking candidate models and tracking industry progress. They are poor stand-ins for your product, because your users, data, and failure modes differ. Treat benchmark scores as a shortlist filter, then run your own evals to decide.

MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark covering 57 subjects, from elementary math and history to law and medicine. Each question has four options, and the metric is plain accuracy.

Because there are four choices, random guessing lands near 25%. It was a useful yardstick for general knowledge, but top models now score so high that it separates them poorly, which is why harder variants such as MMLU-Pro appeared.

SWE-bench tests whether a model can fix real software bugs. Each task is a GitHub issue from a popular Python repository, and the model must produce a patch.

Grading is done by running tests. A patch counts as resolved only if the previously failing tests now pass and the tests that already passed still do. That makes it execution-based rather than opinion-based.

SWE-bench Verified is a subset of 500 tasks that humans checked to make sure the issue is clear and the tests are fair.

A hallucination is output that states something not supported by the source material or by reality, yet sounds confident. Evals usually separate two kinds:

• Faithfulness failures: the answer contradicts or goes beyond the provided context
• Factuality failures: the answer is wrong against world knowledge

A common way to measure it is to extract individual claims from the answer, then check each one against the context or a trusted source. The hallucination rate is the share of claims that fail.

A rubric is a written scoring guide. It lists the criteria you care about and says exactly what each score level looks like, so graders (human or LLM) apply the same standard.

Good criteria are specific and observable, such as 'cites a source for each claim', rather than vague phrases like 'is helpful'.

pass@k is the probability that at least one of k sampled solutions passes the tests. It comes from code-generation evals such as HumanEval, where a problem counts as solved if any of the k attempts is correct.

To reduce variance, you generate n samples per problem (n larger than k), count the c that pass, and use the unbiased estimator:

from math import comb

def pass_at_k(n, c, k):
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

pass@1 is the single-attempt success rate. Higher k always gives a higher or equal number.

Human evals use people to rate or compare model outputs against written guidelines. Common formats are 1-to-5 ratings, pairwise preference ('which answer is better'), and error annotation where reviewers tag specific mistakes.

People catch nuance that metrics miss, such as tone, subtle factual slips, and whether an answer actually helps. The downsides are cost, speed, and rater inconsistency.

That is why most teams use humans sparingly: to create gold labels and to calibrate automated graders, not to score every run.

A regression eval reruns a fixed suite of cases after a change to confirm that behavior that used to work still works. The question it answers is simple: did this update break anything?

The best regression cases come from bugs you already fixed and complaints you already handled. Each one is a promise you made to users.

Wire the suite into CI so a prompt edit, model upgrade, or retrieval change cannot merge if scores fall below the baseline.

An eval harness is the framework that runs your dataset through a model, applies graders, and aggregates the results into a report. It saves you from rewriting the same loop for every project.

Most harnesses have four parts:

1. A dataset loader
2. A runner that calls the model or application
3. Graders that score each output
4. An aggregator that reports metrics and failures

Open-source options include EleutherAI's lm-evaluation-harness, OpenAI Evals, Inspect, and promptfoo.

Contamination happens when benchmark test items leak into a model's training data. The model may then have memorized the answers, so its score overstates real ability.

Leaks are easy to cause. Benchmarks get posted on public sites, copied into blogs, and scraped into web-scale training corpora.

Common defenses are private held-out test sets, freshly written or time-split questions created after the training cutoff, and canary strings that let teams filter benchmark data out of training.

Red teaming is adversarial testing: people (or automated attackers) deliberately try to make a model misbehave. Typical targets are jailbreaks that bypass safety rules, prompt injection, leaking private data, and producing harmful content.

It can be manual, where experts get creative, or automated, where one model generates thousands of attack variations. Every successful attack should be added to a regression set so the same hole stays closed.

Unlike a normal eval, which measures typical quality, red teaming hunts for worst-case behavior.

Offline evals run before release on a fixed dataset. They are controlled, repeatable, and cheap enough to run on every change. Online evals measure behavior on live traffic, so they reflect what real users actually do.

Use both. Offline evals gate releases, and online signals tell you when your offline dataset has stopped representing reality.

The reason is scale and cost. A human reviewer might grade a few hundred outputs a day. A judge model grades thousands in minutes, at a fraction of the price, and it does so on every commit.

It also gives more consistent grading across a single run than a rotating pool of tired reviewers, and it can handle open-ended text that regex and exact match cannot.

It does not replace humans entirely. Keep a small human-labeled sample to measure whether the judge agrees with people, and recheck it whenever you change the judge prompt or model.

In pairwise comparison, many judges lean toward the answer in a certain slot, often the first one, regardless of quality. Swap the order of two identical answers and the verdict can flip, which means part of your score is noise from layout.

The standard fix is to run each comparison twice with the order swapped and only count verdicts that agree. Disagreements become ties or get routed to a human.

v1 = judge(a, b)   # a first
v2 = judge(b, a)   # b first
winner = v1 if v1 == flip(v2) else "tie"

Randomizing positions across the dataset also stops the bias from favoring one system consistently.

Precision@k asks how many of the top-k retrieved documents are actually relevant. Recall@k asks how many of all the relevant documents showed up in the top k.

In RAG, recall usually matters first, because the generator cannot use a document that was never retrieved. Precision matters for cost and distraction, since irrelevant chunks eat context and can mislead the model. Rank-aware metrics like MRR and nDCG add credit for placing relevant items near the top.

Evaluate the retriever and the generator separately, then the whole pipeline. If you only grade the final answer, you cannot tell whether a bad answer came from bad retrieval or bad generation.

1. Retrieval: recall@k and MRR against labeled relevant documents
2. Context relevance: is the retrieved text on topic for the question?
3. Faithfulness: are the answer's claims supported by the retrieved context?
4. Answer relevance and correctness: does it address the question, and does it match the reference?
5. End to end: run a golden question set through the full pipeline

flowchart LR
  Q[Question] --> R[Retriever]
  R --> C["Retrieved context"]
  C --> G[Generator]
  G --> A[Answer]
  R -.-> E1["Eval: recall@k, MRR"]
  C -.-> E2["Eval: context relevance"]
  A -.-> E3["Eval: faithfulness, correctness"]

Faithfulness checks whether every claim in the answer is supported by the retrieved context. Answer relevance checks whether the answer actually addresses the user's question.

They fail independently, which is why RAG evals track both. A system can be perfectly grounded and still useless.

Use pairwise comparison when you are choosing between two versions, such as an old prompt and a new one. Judges, human or model, are far more reliable at saying 'A is better than B' than at assigning an absolute 1-to-5 score that drifts between graders.

Use absolute scoring when you need a pass/fail gate, want to track one system over time, or must check specific criteria such as 'contains no PII'.

Pairwise needs order-swapping to control position bias, and it scales with the number of candidates being compared. Absolute scores are easier to aggregate but need a tight rubric.

BLEU rewards surface overlap, not meaning. If a model answers correctly using different words than the reference, it scores low. If it copies reference phrasing but gets a fact wrong, it can score high.

LLM tasks like chat, reasoning, and open-ended writing have many equally valid answers, and a handful of references cannot cover them. BLEU also ignores factual accuracy, tone, and instruction following.

Studies on open-ended generation have repeatedly found weak correlation with human judgment. It still has a place for constrained tasks like translation, but not as a general quality score.

Raw percent agreement is misleading because raters will agree by chance. Cohen's kappa corrects for that for two raters:

kappa = (p_observed - p_chance) / (1 - p_chance)

Use Fleiss' kappa or Krippendorff's alpha for more than two raters or missing labels. As a rough guide, values above about 0.6 are considered substantial agreement, and values near 0 mean raters agree no better than chance.

Low agreement usually points at an ambiguous rubric, not lazy raters. Tighten the criteria, add examples, and relabel.

Production logs show what users really ask, which no brainstormed dataset matches. Turn them into an eval set in a few steps:

1. Sample logs, stratified by topic, user segment, and outcome
2. Redact PII and secrets before anything leaves the logging system
3. Prioritize failures: thumbs-down, escalations, retries, and low-confidence cases
4. Label expected outputs or pass/fail criteria with domain experts
5. Split into a dev set for iteration and a held-out test set
6. Version it and refresh regularly as usage shifts

Do not sample only failures. Keep plenty of normal successes so improvements for hard cases do not quietly hurt the common path.

If the dataset quietly changes, last month's 82% and today's 79% are not comparable. Versioning makes every score traceable to the exact data it was measured on.

Good practice:

• Store the dataset in git or a data-versioning tool, with a content hash
• Record the dataset version next to every eval run
• Add new cases in a new version instead of editing old ones in place
• Keep a changelog explaining why cases were added, relabeled, or removed

When a version bump changes the baseline, rerun the previous system on the new version so comparisons stay fair.

A model eval tests the raw model's ability on a task. A system eval tests the whole product pipeline that surrounds it.

A strong model can still produce a weak product if retrieval is poor or the prompt is badly designed. Model evals narrow the choices, and system evals make the ship decision.

pass@k asks whether at least one of k attempts succeeds. pass^k asks whether all k attempts succeed, which measures reliability. It was popularized by tau-bench for agents.

The gap can be dramatic. If an agent solves a task 80% of the time per attempt, and attempts are independent:

pass@k tells you what the model can do when it gets retries. pass^k tells you what a customer experiences if the task must work every time.

Check what the agent did, not just what it said. Score four layers:

1. Tool selection: did it pick the right tool?
2. Argument correctness: were the parameters valid and complete?
3. Outcome: did the final state match expectations, for example the right row in the database or the right ticket status?
4. Efficiency and recovery: number of steps, cost, and how it handled tool errors

Final-state checking is the most robust option. Compare the actual environment state to the expected one with code, because there is often more than one valid sequence of calls. Mock external tools so runs stay deterministic and safe.

Pick code-based graders whenever the answer can be verified mechanically. They are cheap, instant, deterministic, and have no judge bias.

• Valid JSON or schema conformance
• Generated code that must pass unit tests
• Numeric answers, IDs, dates, and labels
• Required or forbidden keywords, regex patterns, length limits

Reserve LLM graders for qualities code cannot capture, like tone, helpfulness, or whether a summary preserved the key points. A hybrid works well: run code checks first as a cheap filter, then send survivors to the LLM judge.

A benchmark saturates when top models score near the ceiling. Differences between them shrink to noise, so the benchmark stops telling you which model is actually better.

Another problem is label errors. If a few percent of answers in the dataset are wrong, scores plateau below 100% and the remaining headroom is partly just mistakes.

The usual response is a harder successor (MMLU-Pro followed MMLU, for example), a newly written benchmark, or private test sets. For your own evals, saturation means it is time to add harder cases, not to celebrate.

Do not pick a round number out of the air. Anchor the threshold on evidence:

1. Measure the baseline. Run the current system several times to learn its average and its run-to-run variance.
2. Weigh the risk. A medical-advice assistant needs a stricter bar than a casual brainstorming tool.
3. Set per-category minimums for critical cases like safety, so a strong average cannot hide a weak area.
4. Ratchet upward. Raise the bar as the system improves, never lower it quietly.

Keep the threshold above the noise. If normal variance is plus or minus 2 points, a gate set at 1 point of regression will trigger false alarms constantly.

LLM judges often prefer longer answers even when the extra text adds nothing. If you compare two systems and one is wordier, it can win purely on length, and you ship a model that just talks more.

Ways to control it:

• State in the rubric that length does not earn points and padding is penalized
• Compare length-matched outputs, or regress scores on length to see the effect
• Add conciseness as an explicit criterion
• Spot-check judge scores against human preferences on short versus long answers

Calibration means proving the judge agrees with people before you rely on it. Treat the judge prompt like a model you are tuning, with its own train and test split.

1. Have domain experts label 100 to 200 representative outputs using your rubric.
2. Split the labels into a calibration set for tuning and a held-out set for the final check.
3. Run the judge and compute agreement (Cohen's kappa) plus a confusion matrix per label.
4. Read the disagreements. Most trace back to a vague criterion or a missing example.
5. Revise the rubric, add few-shot examples, or split one broad criterion into narrower ones.
6. Repeat, then confirm agreement on the held-out set only once.

Watch for asymmetric errors. A judge that is too lenient on safety failures is far worse than one that is slightly strict. Recalibrate whenever you change the judge model or the rubric.

The idea is to run evals the same way you run tests: automatically on every change, cheapest checks first, with a clear gate before merge.

flowchart TD
  A["Pull request changes prompt, model, or retrieval"] --> B["Run fast code-based evals"]
  B --> C{Hard checks pass?}
  C -- No --> X["Block merge and report failing cases"]
  C -- Yes --> D["Run LLM-judge evals on sampled set"]
  D --> E["Compare with baseline using confidence intervals"]
  E --> F{Regression beyond noise?}
  F -- Yes --> X
  F -- No --> G[Merge]
  G --> H["Nightly full suite and online monitoring"]

Fast checks (schemas, forbidden content, unit tests) finish in seconds and catch obvious breakage. The costlier judge stage runs on a stratified sample for pull requests, and the full suite runs nightly.

The gate compares against a stored baseline with a tolerance for noise, and every run is saved with the model version, prompt hash, and dataset version so any score can be traced.

There is no single test. You combine several signals, since each one catches a different kind of leak.

A large drop on rephrased or newer items, with the same difficulty, is the strongest practical warning. Canary strings embedded in the benchmark help later filtering, but only work if developers honor them.

First find out where the variance comes from. Scores that jump a few points between identical runs usually have one of these causes:

Then measure it. Run the same eval 5 to 10 times and report mean and standard deviation. Set your regression threshold above that standard deviation, otherwise the gate will fire on noise and people will start ignoring it.

Most savings come from running fewer expensive calls, not from weakening the evals.

• Tier the suite: cheap code checks on every commit, a sampled judge run on pull requests, the full suite nightly
• Cache model outputs and judge verdicts keyed by a hash of input, prompt, and model, so unchanged cases are never recomputed
• Use a smaller judge for easy rubrics, but only after confirming it agrees with the stronger judge or with human labels
• Batch non-urgent runs through batch APIs, which are usually priced below real-time calls
• Stop early on hard failures, and drop near-duplicate cases
• Stratify samples so a small subset still covers every category

Always measure the effect of a cost cut. If a cheaper judge shifts scores or disagrees with humans more often, the savings are not real.

Single-turn scoring misses what makes conversations hard: memory, consistency, and recovering from misunderstandings. You need to score both the individual turns and the whole conversation.

Replaying recorded chats breaks as soon as your new model answers differently, because the next recorded user message no longer fits. The more robust approach is a user simulator: an LLM plays the user following a scripted goal and persona, and reacts to whatever the agent actually says.

• Per turn: correctness, tone, policy compliance
• Per conversation: was the goal achieved, in how many turns?
• Consistency: does the agent keep earlier constraints and avoid contradicting itself?
• Recovery: how does it handle corrections and vague requests?

Run several simulated conversations per scenario, since the same script can unfold differently each time, and report goal completion rate with a confidence interval.

An eval score is an estimate from a finite sample, so report how uncertain it is. For a pass rate with n independent items, the normal approximation is a quick start:

p = 0.80
n = 100
margin = 1.96 * (p * (1 - p) / n) ** 0.5   # about 0.078
# 80% +/- 7.8 points

So 80% on 100 items really means somewhere between about 72% and 88%. For skewed metrics or non-binary scores, use the bootstrap: resample items with replacement a thousand or more times, compute the metric each time, and take the 2.5th and 97.5th percentiles.

When comparing two systems on the same items, bootstrap the paired difference per item. This removes the shared item difficulty and gives a much tighter interval. If the interval for the difference includes zero, you have not shown an improvement.

Outcome evaluation checks only the final result. Trajectory evaluation inspects the steps the agent took, such as which tools it called and in what order.

A practical mix is to make outcome the main score and add trajectory constraints: forbidden actions, a step or cost budget, and required confirmations before destructive operations. That keeps flexibility while still catching dangerous behavior.

This is the classic sign that the metric no longer represents what users want. Start by assuming the eval is wrong, not the users.

1. Compare distributions. Does the eval dataset still look like current production traffic by topic, length, and difficulty?
2. Mine the unhappy sessions. Pull low-rated conversations and check what the eval would have scored them.
3. Check for judge bias. Did the change make answers longer or more confident, which judges tend to reward?
4. Look for overfitting. Were prompts tuned repeatedly against the same dev set? Check a fresh held-out set.
5. Find unmeasured dimensions such as latency, tone, refusals, or formatting that the change worsened.

Fix it by adding the failing sessions as new cases, adding the missing metrics, and refreshing the held-out set. Optimizing hard against a proxy eventually breaks it, a pattern known as Goodhart's law.

G-Eval is an LLM-judging method from a 2023 paper that makes scoring more structured and more fine-grained. It has three ideas:

1. You provide the task description and evaluation criteria.
2. The judge first generates its own evaluation steps through chain-of-thought reasoning.
3. A form-filling prompt then asks for a score, for example 1 to 5, on the output.

flowchart LR
  A["Task + criteria"] --> B["Judge writes evaluation steps"]
  B --> C["Form-filling prompt with output"]
  C --> D["Score token probabilities"]
  D --> E["Probability-weighted final score"]

The last step is what sets it apart. Instead of taking the single score the judge emits, it weights each possible score by its token probability. That produces a continuous value like 3.7, which separates close systems better than coarse integers and correlated better with human judgments in the original study.

The trade-off is that you need access to token probabilities, which not every API exposes.

The headline metric is attack success rate (ASR): the share of adversarial prompts that get the model to produce disallowed content. You measure it over a large, varied attack set rather than a few clever examples.

1. Build the attack set from curated jailbreak templates, public sets such as HarmBench and JailbreakBench, and your own red-team findings.
2. Expand it automatically. Use an attacker model to rewrite and iteratively refine prompts, and add variants such as other languages, encodings, and role-play framing.
3. Grade responses with a validated harm classifier or judge, calibrated on human labels.
4. Report ASR per harm category, not just one average.

Also measure over-refusal on benign but edgy prompts. A model that refuses everything has a perfect ASR and is useless, so safety and helpfulness must be reported together.

Start small and grounded in real behavior, then grow the system in the order that gives you the most signal for the least effort.

1. Define success with stakeholders: what must the product do, and what must it never do?
2. Collect 30 to 50 real or realistic examples and read the outputs yourself. Error analysis beats any metric at this stage.
3. Group the failures into categories and rank them by user impact.
4. Write a cheap grader per failure mode. Prefer code checks, and use LLM judges for subjective criteria.
5. Calibrate the judges against human labels before trusting them.
6. Automate in CI with a baseline and a noise-aware gate.
7. Add online monitoring and feed production failures back into the dataset.

flowchart LR
  A["Define success"] --> B["Error analysis on real examples"]
  B --> C["Graders per failure mode"]
  C --> D["CI gate"]
  D --> E["Online monitoring"]
  E --> F["New failures become eval cases"]
  F --> B

The key point is that it is a loop. Production surprises keep teaching you which cases the dataset was missing.