AI / AI Harness engineering Interview questions

AI harness engineering is the practice of designing everything around a language model that turns it into a dependable system: the loop that calls it, the tools it can use, the context it sees, the permissions it runs under, and the checks that judge its output.

The model supplies reasoning. The harness supplies control, memory, safety, and feedback. A common shorthand is agent = model + harness.

It matters because two teams using the same model often see very different results, and the gap usually traces back to harness choices such as how errors are reported to the model or how much history is kept.

An agent harness is the runtime software that wraps a model and lets it act on a task. It takes a goal, assembles the prompt, calls the model, parses any tool requests, runs those tools, feeds the results back, and repeats until the task is finished or a limit is hit.

Think of it as the "operating system" for the model. The model decides what to do next; the harness decides what is allowed, what the model gets to see, and when to stop.

A coding assistant that reads files, edits them, and runs tests is a typical example. The model never touches your disk directly. The harness performs each action on its behalf and reports the result.

Most harnesses are built from the same handful of parts, even when the names differ.

If one of these is missing, the gap shows up quickly: no sandbox means risk, no context manager means bloated prompts, no evals means guessing.

The agent loop is the repeating cycle at the center of a harness: send context to the model, read its response, run any requested tool, append the result, and call the model again.

flowchart TD
  A["Build prompt"] --> B["Call model"]
  B --> C{Tool call requested?}
  C -- Yes --> D["Run tool"]
  D --> E["Append result to context"]
  E --> A
  C -- No --> F["Return final answer"]

The loop ends when the model replies without a tool request, or when a safeguard such as a turn limit or budget cap fires. Everything else in the harness exists to make this loop safe, cheap, and useful.

A tool is a function the harness exposes to the model so it can act beyond generating text. Examples are reading a file, running a shell command, querying a database, or calling a web API.

The model never runs a tool itself. It emits a structured request naming the tool and its arguments, and the harness validates the request, executes it, and returns the output as the next message.

Each tool needs a clear name, a description of when to use it, and an input schema. Those three things are all the model has to go on when choosing between tools.

A tool schema is a machine-readable description of a tool: its name, a plain-language description, and the JSON Schema for its inputs. The harness sends it to the model and later uses it to validate the arguments the model returns.

{
  "name": "read_file",
  "description": "Read a UTF-8 text file and return its contents.",
  "input_schema": {
    "type": "object",
    "properties": {
      "path": { "type": "string", "description": "Relative file path" }
    },
    "required": ["path"]
  }
}

Validating against the schema before execution catches malformed calls early, and the error message can be fed back so the model corrects itself.

The system prompt sets the model's standing instructions for the whole run: its role, rules, tone, and how to use the available tools. It is the first and most stable part of every request.

In a harness it often does more than a chat prompt would. It states conventions such as "read a file before editing it", explains when to ask the user versus proceed, and points to project rules loaded from a file.

Because it sits at the front of every call, keeping it stable also helps prompt caching. Putting changing details like the current time inside it can break the cache on every turn.

A stop condition is a rule that ends the loop. Without one, an agent can keep calling tools indefinitely.

Harnesses usually combine several:

• Natural finish - the model answers without requesting a tool.
• Turn limit - a maximum number of model calls.
• Budget cap - a ceiling on tokens, cost, or wall-clock time.
• Explicit signal - a "done" tool or a failed verification gate.

When a limit fires, a good harness reports why it stopped and what was completed, rather than failing silently.

A sandbox is an isolated environment where the agent's actions run, so a mistake or a malicious instruction cannot damage the host machine or reach data it should not.

Typical isolation layers include containers or virtual machines, a restricted filesystem view, limited CPU and memory, and controlled network access. A common pattern is a throwaway workspace with the project copied in and outbound traffic blocked or allow-listed.

The sandbox is a hard boundary enforced by the operating system. That makes it more reliable than a prompt that merely asks the model to be careful.

Context management is deciding what the model sees on each call. The context window is finite, and every message, tool result, and file read competes for the same space.

The harness controls this by loading only what is needed, trimming bulky tool output, summarizing older turns, and pulling in documents on demand instead of upfront.

Done well, the model stays focused and costs stay low. Done badly, the prompt fills with stale logs, relevant details get buried, and quality drops well before the hard token limit is reached.

An evaluation harness is the test rig that runs a model or agent against a fixed set of tasks and scores the results. It gives you a repeatable number instead of a gut feeling.

It typically contains the task set, a runner that executes each task in a clean environment, and graders. Graders can be code (unit tests, exact match), rules, or another model acting as judge.

Use it before and after any change to the prompt, tools, or model version. If the score moves, you know the change mattered; if it does not, you avoided shipping a placebo.

A permission system decides which actions an agent may take automatically, which need approval, and which are never allowed.

A common layout uses three tiers. Read-only actions such as listing files run freely. Reversible writes may be allowed within a project folder. Destructive or external actions, like deleting data, pushing code, or sending email, ask a human first or are blocked.

Rules are usually expressed as allow and deny lists on tool names and arguments, for example "allow git status, deny rm -rf". The principle behind it is least privilege: give the agent only what the task needs.

A hook is a piece of your own code the harness runs automatically at a defined point in the agent's lifecycle, such as before a tool call, after a tool call, or when the agent finishes.

Because hooks are ordinary code, they are deterministic. They can block a dangerous command, run a formatter after every file edit, log an audit entry, or force a test run before the agent is allowed to say it is done.

That is the key difference from prompt instructions. A prompt requests a behavior; a hook guarantees it.

The Model Context Protocol is an open standard for connecting AI applications to external tools and data sources. Instead of writing a custom integration for every pairing of app and service, a service exposes an MCP server and any MCP-capable client can use it.

A server can offer tools (actions), resources (readable data), and prompts (reusable templates). The harness acts as the client: it discovers what the server offers, lists those tools to the model, and routes calls back to the server.

The practical benefit is reuse. One database or ticketing server works across many harnesses, and your harness gains new capabilities without new glue code.

Agent memory is any information the harness keeps outside a single model call so the agent can continue work or learn across sessions. The model itself remembers nothing between calls; it only sees what the harness sends.

The simplest form is the running conversation. Beyond that, harnesses persist notes to files, keep a task list, store summaries of earlier sessions, or index past material for retrieval.

A practical example is a project instructions file that is loaded at the start of every session, so conventions like the test command survive restarts.

Most agent failures fall into a few repeatable patterns. Knowing them tells you what to build guards for.

1. Infinite or looping behavior - repeating the same failing call.
2. Wrong tool or bad arguments - often from vague tool descriptions.
3. Context overflow or drift - losing the original goal in a long run.
4. Premature completion - claiming success without verifying.
5. Hallucinated results - inventing file contents or API responses.
6. Unsafe actions - destructive commands or prompt injection from tool output.
7. Compounding errors - one early mistake spreading through later steps.

The model is the trained network that predicts text from a prompt. The harness is the software around it that decides what the prompt contains, what actions are possible, and what happens with the output.

This split is why you can often raise reliability without touching the model at all.

A scratchpad gives the agent a place to write down its plan and progress outside the flow of tool output. The simplest version is a todo tool or a notes file the model can read and update.

The agent writes the steps at the start, marks each one as it finishes, and re-reads the list when the context grows long. That keeps the original goal in view and makes it harder to skip steps.

The harness can also surface the list back to the user as live progress, and persist it so a resumed run knows what is already done.

The best default is to return the error to the model as a normal tool result, clearly marked as an error, instead of crashing the run. Models are good at reading "file not found" and trying a different path.

A few rules make this work well:

• Write actionable messages: say what failed and what a valid call looks like.
• Separate transient errors (timeouts, rate limits), which the harness retries with backoff, from logic errors, which the model must fix.
• Validate arguments against the schema first so malformed calls never reach the tool.
• Cap consecutive failures so the agent cannot retry the same broken call forever.

Compaction shrinks the history when it nears the window limit while keeping what the task still needs. The harness asks the model (or a cheaper one) to summarize older turns, then replaces them with that summary and keeps the most recent turns verbatim.

A good summary preserves decisions made, file paths touched, open problems, and the next step. It drops raw logs and dead ends.

A lighter technique is clearing old tool results while leaving the model's own reasoning and the call records in place. Large outputs are usually the biggest space-eaters, so this alone buys a lot of room.

Compaction is lossy by nature, so critical facts are better stored in a notes file the harness reloads.

A subagent is a separate model run with its own fresh context, started by the main agent to handle one focused job. The main agent receives only the final answer, not the dozens of intermediate tool calls.

The main benefits are:

1. Context isolation - a search through 40 files does not clutter the main thread.
2. Parallelism - independent investigations can run at the same time.
3. Specialization - a subagent can have its own prompt and a narrower toolset.

The tradeoff is cost and coordination. More model calls use more tokens, and the subagent only knows what the parent writes in its brief, so a vague brief gives vague results.

Use a workflow when the steps are known in advance and the path rarely changes. Use an agent when the number and order of steps depend on what the model discovers along the way.

A sensible rule is to start with the simplest design that works, often a single model call, and add loop autonomy only when a fixed flow genuinely cannot cover the cases.

Prompt caching lets the provider reuse the processed prefix of a prompt across calls, so repeated content is cheaper and faster. It only works when the beginning of the prompt is byte-identical to a previous request.

That shapes the harness:

• Put stable content first: system prompt, tool definitions, project rules.
• Append new messages at the end; do not rewrite or reorder earlier history.
• Keep tool lists stable, since changing them mid-run can invalidate the cache.
• Put volatile data such as the current time later in the prompt, not in the prefix.

In long agent loops, where the same large prefix is sent on every turn, this can cut cost and latency substantially.

An idempotent operation gives the same end result no matter how many times it runs. This matters because agents retry: after a timeout, a crash, or a confused model turn, the same call may be issued twice.

If a "create invoice" tool is not idempotent, one retry can bill a customer twice. If it accepts an idempotency key, the second call is recognized and returns the original result.

Prefer tools shaped as "set status to paid" over "toggle status". For non-idempotent actions like sending email, add a confirmation step or a dedupe check.

A simple test: run the tool twice with the same input and confirm the second run changes nothing.

The description is the only guidance the model has for choosing a tool, so write it like a short instruction to a new colleague.

1. State what the tool does and when to use it versus similar tools.
2. Describe each parameter, including format and an example value.
3. Note limits, such as "returns at most 50 results".
4. Use clear names like search_orders_by_email rather than query2.
5. Return concise, readable output, not raw dumps.

When an agent picks the wrong tool, the first fix is usually to improve the description, not to add prompt text elsewhere.

Read the descriptions cold: if a new teammate could not pick the right tool from them, the model cannot either.

Short-term memory is what sits in the current context window: the conversation, tool results, and any loaded files. It disappears when the session ends or when the history is trimmed.

Long-term memory is stored outside the window and brought back when relevant, such as notes files, a database of past decisions, or a vector index of earlier documents.

The harness decides what gets promoted from short-term to long-term, and what gets loaded back.

Layer several guards, because any single one can be bypassed.

• Hard turn limit on model calls per run.
• Budget caps on tokens, dollars, and wall-clock time.
• Repeat detection: if the same tool call with the same arguments appears several times, interrupt with a message or stop.
• Progress checks: stop if several turns pass with no new files changed or no new information.
• Tool timeouts so one hung call cannot stall the loop.

When a guard triggers, stop gracefully with a summary and a clear reason so a human can pick up from there.

Log every guard trigger, because frequent hits usually point to a prompt or tool problem rather than a one-off.

Before running a risky action, the harness pauses the loop, shows the user exactly what the agent wants to do, and waits for approve, deny, or edit. Only then does it execute and continue.

The pause point is usually a permission check inside the tool layer. The pending call is shown with its real arguments, such as the full shell command, not a vague summary.

Good designs avoid approval fatigue. Safe, reversible actions run automatically, and users can approve a pattern for the session so they are not clicking through the same prompt fifty times. A denial should go back to the model as a result so it can choose another route.

Tool results go straight into the context window. One unbounded cat of a large log or a database query returning 100,000 rows can fill the window, raise cost, and push the real task out of focus.

The harness should cap output size and tell the model what was cut, for example: "Showing lines 1-200 of 8,431. Use the offset argument for more." That lets the model fetch the next page only if it needs it.

Other helpful options are filtering at the source (a grep instead of reading the whole file), returning summaries with an ID for details, and saving huge output to a file while returning only the path.

Start from real work, not invented puzzles. Pull 20 to 50 tasks from actual usage or bug reports, since a small set of realistic cases beats a large synthetic one.

1. Write each task with a clear success criterion that can be checked by code where possible, such as a passing test or a matching database state.
2. Include easy, typical, and hard cases, plus a few that should be refused or escalated.
3. Run every task in a clean environment so runs do not contaminate each other.
4. Keep failures: every production bug becomes a new eval case.

Grow the set over time, and keep a held-out portion so you do not overfit your prompts to the examples you tuned on.

Offline evals run on a fixed, known task set before release. They answer "did this change make things better or worse?" and are repeatable, so they work well as a gate in CI.

Online monitoring watches real traffic after release. It answers "what is actually happening for users?" and catches things your test set never imagined.

You need both. Online findings, such as a new failure pattern, feed back into the offline set.

Streaming means the harness reads the model's response as a series of events instead of waiting for the full message. Text appears for the user token by token, and tool call arguments arrive in fragments.

The harness accumulates those fragments, parses the arguments only once the call block is complete, and then runs the tool. Parsing early risks reading half-finished JSON.

Streaming improves perceived latency a lot, and it lets the UI show tool activity as it happens. It also needs careful handling of dropped connections: if a stream breaks mid-call, the harness should discard the partial tool call and retry rather than execute it.

Harness code needs to read the model's answer reliably. Free text forces fragile string parsing, while structured output, such as JSON conforming to a schema, can be consumed directly.

They help in three places: extracting fields for downstream code, producing routing decisions such as a category or next step, and forcing a tool call to match its input schema.

Still validate on receipt. Even with schema enforcement, values can be wrong in meaning, like a date in the wrong year, so add semantic checks and a retry path that returns the validation error to the model.

Many APIs support schema-constrained output or forced tool calls, which fixes the format while the content still needs checking.

Treat each run as a trace, with one span per model call and one per tool call. Record the inputs, outputs, latency, token counts, and any errors, all tied to a run ID.

The minimum useful record is: the exact prompt sent, the raw model response, each tool call with arguments and result, stop reason, and cost. Without the exact prompt, you cannot reproduce a bad run.

Redact secrets and personal data before storing. Then use traces to replay failures, compare runs before and after a change, and mine real failures for new eval cases.

Sampling is fine at high volume, but always keep full traces for failed or flagged runs.

The core rule: the model should never see a secret it does not need to see. Anything in the context can be echoed back, logged, or leaked by a prompt injection.

• Keep API keys in the harness or a secrets manager, and inject them into tool calls at execution time, so the model only names the tool.
• Give the sandbox short-lived, narrowly scoped credentials instead of long-lived admin keys.
• Scan tool output and traces for key patterns and redact them.
• Block the agent from reading credential files such as environment dumps or key stores.

Rotate keys regularly and revoke any that appear in a trace, since logs often outlive the run.

Prompt injection happens when untrusted content, such as a web page, an email, or a tool result, contains instructions the model mistakes for the user's. No prompt wording fully prevents it, so defense has to be built into the harness.

1. Treat external content as data. Label it clearly as untrusted and tell the model not to follow instructions found inside it. This lowers the risk but is not a guarantee.
2. Apply least privilege. An agent summarizing web pages should not hold write access to your repository or mail.
3. Break the dangerous combination. Risk peaks when one session has private data, untrusted input, and a way to send data out. Remove at least one of the three.
4. Gate sensitive actions. Sending data externally, spending money, or deleting something requires human approval showing the real arguments.
5. Restrict egress. Allow-list network destinations so stolen data has nowhere to go.
6. Isolate with subagents. Let a low-privilege subagent read untrusted content and return only a plain summary.

Then test it: keep a set of known injection payloads in your evals and run them after every change.

A single turn is one pass through the loop. The details vary, but the sequence is consistent.

sequenceDiagram
  participant H as Harness
  participant M as Model
  participant T as Tool
  H->>H: Assemble prompt (system, history, tools)
  H->>M: Send request
  M-->>H: Response with tool_use block
  H->>H: Validate args and check permissions
  H->>T: Execute tool
  T-->>H: Result or error
  H->>H: Truncate, redact, append to history
  H->>M: Next request

Step by step: the harness builds the prompt from the stable prefix plus recent history, calls the model, and reads the response. If it contains a tool request, the harness validates the arguments against the schema, runs the permission check, and executes the tool in the sandbox.

The result is then post-processed: truncated if large, scanned for secrets, and wrapped as a tool result message. Hooks may fire before and after execution. Finally the harness checks stop conditions (turn limit, budget, final answer) and either loops again or returns.

Assume the code the agent runs is untrusted, because it may come from a mistaken model or an injected instruction. Design for containment first, convenience second.

• Isolation boundary: use a container with a restricted profile, or a microVM for stronger separation. Run as a non-root user.
• Filesystem: mount only the project workspace read-write, and everything else read-only or absent. Never mount home directories or credential stores.
• Network: default deny. Allow-list package registries or specific APIs when truly required.
• Resources: cap CPU, memory, disk, process count, and run time so a runaway loop cannot take the host down.
• Lifecycle: make environments disposable and reset between tasks so state does not leak across runs.
• Secrets: pass only short-lived, scoped credentials.

Log every command and its output. A sandbox you cannot audit gives you containment but no visibility into what actually happened.

In an agent loop the whole context is resent on every turn, so cost grows with history length times the number of turns. Most savings come from keeping that context small and stable.

1. Cache the prefix. Keep system prompt and tool definitions stable and in front so cached reads apply.
2. Trim tool output. Paginate, filter at the source, and clear stale results.
3. Compact history before it grows large, preserving decisions and next steps.
4. Delegate noisy work to subagents so intermediate calls never enter the main context.
5. Route by difficulty. Use a smaller model for simple steps like classification and a stronger one for planning.
6. Load tools lazily. Hundreds of tool definitions in every prompt cost tokens even when unused.

Measure before and after on your eval set. A cheaper run that fails more often is not a saving.

Start with the trace, not the prompt. Read the last ten or so turns and look at what actually repeats.

1. Same call, same arguments? Check the tool result. An unhelpful error like "failed" gives the model nothing new, so it retries. Make the message specific.
2. Different calls, no progress? The goal may be unclear or unreachable. Check whether the success condition is verifiable and whether a needed tool or permission is missing.
3. Context drift? In long runs the original goal gets buried. Re-inject the task or a todo list and consider compaction.
4. Flip-flopping between two states? Two instructions probably conflict, such as "keep changes minimal" and "make all tests pass".

Fix the root cause, then add a guard: repeat detection, a progress check, and a turn cap. Finally, save the failing run as an eval case so the loop cannot come back unnoticed.

An orchestrator agent breaks a task into subtasks, hands each to a worker agent, and then combines the results. Workers run with their own context and a narrow toolset, and return concise findings.

flowchart TD
  U["User task"] --> O[Orchestrator]
  O --> W1["Worker: research A"]
  O --> W2["Worker: research B"]
  O --> W3["Worker: write tests"]
  W1 --> S[Synthesize]
  W2 --> S
  W3 --> S
  S --> R["Final answer"]

It fits problems where subtasks are independent and the breakdown is not known upfront, such as researching several sources at once.

The weak points are the handoffs. A worker only knows what the orchestrator writes in its brief, so briefs need the goal, constraints, and expected output format. Parallel workers editing the same files can also conflict, so give them separate areas or serialize writes.

Costs are higher than a single agent, so use it when the parallel speedup or context isolation clearly pays for itself.

The idea is to persist enough state that a fresh process can pick up where the old one stopped, instead of restarting from zero.

1. Checkpoint after each turn. Save the message history, tool results, todo list, and run metadata to durable storage.
2. Record tool intent before execution and the outcome after. On restart you can tell whether the last tool call finished.
3. Make tools idempotent or use idempotency keys, so replaying an uncertain step does not duplicate effects.
4. Keep side effects visible. Store IDs of created resources so the resumed run can find them.
5. Version the state format so older checkpoints still load after a harness update.

On resume, reload the checkpoint, check the last uncertain step, and continue the loop. Keep the original budgets so a crash does not reset your spend limits.

The same agent can pass a task once and fail it the next time, so a single run tells you little. Evaluate by repetition and by outcome.

• Run each task several times. Report the pass rate rather than one pass or fail.
• Use pass@k when one success among k attempts is enough, as in a tool that proposes several fixes. Use a stricter all-k-pass measure when reliability matters, such as customer-facing automation.
• Grade outcomes, not exact paths. Agents reach the same result via different steps, so check final state, passing tests, or database contents.
• Use model graders carefully. A judge model suits subjective qualities, but spot-check it against human labels.
• Track variance. A high average with wide swings is still a risky agent.

Neither wins everywhere. The tradeoff is between model decision-making cost and capability coverage.

A broad shell tool is flexible but hard to constrain. A narrow get_order_status tool is easy to secure and understand but covers only one case.

A practical approach is to start with a small set of well-described tools for the common actions and keep one general escape hatch behind approval. If the model keeps mis-choosing between near-duplicates, merge them. If it keeps building complex arguments badly, split them.

Models can request several tools in one response. Running them concurrently saves time, but only when they do not interfere.

1. Classify tools: mark read-only tools as safe to run in parallel. Tools that write or change shared state run one at a time.
2. Check dependencies. If one call needs another's output, the model should request it in a later turn, not the same one.
3. Avoid write conflicts. Two parallel edits to the same file can overwrite each other, so lock or serialize them.
4. Return results in a matching order, each tied to its call ID, so the model can pair request and result.
5. Isolate failures. One failed call should return an error result without cancelling the others.

Apply permission checks to each call individually, not to the batch as a whole.

A new model can behave differently even when your harness is unchanged: it may call tools more or less often, follow instructions more literally, or respond at a different length. Treat every upgrade as a change that needs testing.

1. Run your full eval set on old and new models with the same harness, and compare pass rate, cost, turns, and latency.
2. Diff the traces. Look at tool-call patterns, retries, and where runs end, not only final scores.
3. Audit old prompt workarounds. Instructions added to fix a weakness in the old model, like repeated "double-check" lines, can over-correct on a stronger one.
4. Roll out gradually. Send a small share of traffic to the new model and watch online metrics before switching fully.

Keep the old model selectable so you can roll back quickly.

Context assembly is the step where the harness builds the exact prompt for each model call. It is the most influential piece of harness code, because the model can only act on what it is shown.

A typical order, from most to least stable:

1. System prompt with role and rules.
2. Tool definitions, sometimes loaded only for tools likely to be needed.
3. Project memory such as an instructions file.
4. Conversation history, with older turns summarized or cleared.
5. Fresh retrieved material like files or search results for this step.
6. Volatile details such as the current time or working directory.

The assembler enforces a token budget. When the total exceeds it, the harness trims in a planned order: stale tool results first, then compaction of older turns. It never trims the system prompt or the latest user request.

Keeping the stable parts first and appending new content at the end also preserves the prompt cache from one call to the next.

Group actions by reversibility and blast radius, then attach a policy to each group.

Policy should be evaluated on the real arguments, not just the tool name. git push to a feature branch and to main are different risks.

Add safety nets that make approval less critical: work on branches or copies, take snapshots before destructive steps, and keep an audit log. Default to deny for anything unclassified, and let users widen rules deliberately.

Do not rely on the model saying "done". Build a verification gate that must pass before the harness accepts completion.

1. Programmatic checks first. Run tests, linters, type checkers, schema validation, or a script that inspects the resulting state. These are fast, cheap, and objective.
2. Feed failures back. If a check fails, return the output to the model and continue the loop with a retry limit.
3. Use a model-based review for fuzzy criteria such as tone or completeness, ideally with a separate reviewer that has a fresh context and a clear rubric.
4. Enforce with a hook. A completion hook that runs the checks guarantees they happen, rather than hoping the prompt triggers them.

Where a task cannot be verified automatically, say so, and route it to human review.

Models improve and change, so a harness tuned too tightly to one model's quirks becomes a liability.

• Keep the model behind an interface. Isolate provider calls so you can swap or route models without touching the loop.
• Prefer simple, general scaffolding. Heavy rules and rigid step-by-step flows can hold back a stronger model. Add structure only where evals show it helps.
• Tag every workaround with the failure it fixes, so you can test removing it on the next model.
• Rely on evals, not intuition. A solid eval set lets you re-validate quickly after each change.
• Use deterministic enforcement (hooks, sandbox, permissions) for safety, since these do not depend on model behavior.
• Keep prompts and tool descriptions in versioned config, not scattered in code.

The goal is a harness that gets better, not worse, when a more capable model is dropped in.