AI / Memory Augmented Generation (MAG) Interview questions

Memory-Augmented Generation (MAG) is a pattern where a language model is paired with an external memory it can write to and read from. That lets it use information that is neither in the current prompt nor in its trained weights.

The memory typically holds past conversations, user preferences, task outcomes and learned facts. Before each reply, the system pulls the relevant entries into the prompt. After the reply, it decides what new information is worth saving.

The model itself stays stateless. The persistence comes from the application layer around it, which is what creates the feeling of continuity across turns and sessions.

It gives an otherwise stateless model continuity. A plain LLM call knows nothing about earlier sessions, and its context window is finite, so anything outside the prompt is simply gone.

Memory augmentation solves three practical problems:

1. Personalization - remembering a user's name, stack, tone or constraints.
2. Long-running tasks - carrying decisions and progress across sessions or context resets.
3. Cost - retrieving a few relevant facts is cheaper than replaying a whole transcript every turn.

It also avoids retraining: new knowledge is a database write, not a fine-tuning job.

Most MAG designs borrow terms from cognitive science and split memory into four types.

Real systems usually mix them: working memory lives in the prompt, while the other three sit in external stores.

Working memory is whatever the model can see right now: the system prompt, recent messages, retrieved memories, tool outputs and any scratchpad notes inside the context window.

It is fast and fully visible to the model, but it is small and temporary. Once a message is trimmed or the session ends, it is gone unless something copied it to long-term storage.

Good agent design treats working memory as a limited budget and fills it deliberately, rather than dumping every stored item in.

Episodic memory records specific events the agent lived through, usually with a timestamp and some context: who was involved, what was tried, and what happened.

Think of it as a diary. An entry might say that on 3 March the user asked for a CSV export, the first attempt failed on encoding, and switching to UTF-8 fixed it.

It is useful for questions like "what did we try last time?" and for learning from past successes and failures. Episodes are often later summarized into semantic facts so they don't pile up forever.

Semantic memory stores general facts and preferences detached from the moment they were learned, such as "the user works in healthcare" or "the staging database is Postgres 15".

It is usually produced by extracting or summarizing episodes: the date and chat are dropped, the durable fact is kept. That makes entries short, cheap to retrieve and easy to update when a fact changes.

The trade-off is lost nuance. An extracted fact like "likes dark mode" no longer says when or why it was said.

Procedural memory holds how to do things: reusable skills, workflows, tool-call patterns or refined instructions, rather than facts about the world.

A well-known example is Voyager, a Minecraft agent that stored working code snippets in a skill library and retrieved them for new tasks. In everyday agents it can be a saved runbook, a prompt template that worked, or a learned rule like "always run the linter before committing".

Updating it often means rewriting instructions or adding a new skill entry.

Long-term memory is any information that outlives a single context window or session and is kept in an external store, such as a vector database, key-value store or graph.

It is not seen by the model until retrieved. A retrieval step searches it using the current query, picks the best entries and injects them into the prompt.

Because it is external, its size is bounded by storage and retrieval quality, not by the model's context length.

Typical contents are user profiles, past decisions and distilled lessons. Entries are usually tagged with timestamps and owners so retrieval can filter and rank them, and so they can be expired or deleted later.

Nearly every memory module reduces to four operations:

1. Write - decide what to save and store it with metadata.
2. Retrieve - find the entries relevant to the current input.
3. Update - merge or correct entries when new information overlaps or conflicts.
4. Forget - expire, archive or delete entries that are stale, low value or must be erased.

Many early prototypes only implement write and retrieve, which is why their memory gets noisy over time. Update and forget are what keep it healthy.

A memory store is the backing storage that holds saved memories. The right choice depends on how you need to query them.

Production systems often combine two or more, since similarity search alone is weak at exact matches and structured filters.

Pick based on the dominant query pattern, not fashion. A support bot that mostly looks up a customer's plan needs a plain table; a research assistant recalling loosely related notes needs embeddings.

The basic flow is embed, store, search. Each memory is turned into an embedding and saved with metadata. At query time you embed the current message and fetch the nearest neighbors.

def remember(text, user_id):
    vec = embed(text)
    db.upsert(id=new_id(), vector=vec,
              payload={"text": text, "user": user_id, "ts": now()})

def recall(query, user_id, k=5):
    return db.search(vector=embed(query), k=k,
                     filter={"user": user_id})

Always filter by user or tenant at query time, and keep timestamps in the payload so you can rank by recency later.

Tune k and add a similarity cutoff so weak matches never reach the prompt. Returning five poor matches is worse than returning none.

Memory summarization compresses older conversation turns into a shorter note so the context window doesn't overflow. Typically a model is asked to summarize the oldest messages, and the summary replaces them in the prompt.

It keeps cost and latency down and preserves the gist of long chats. The risk is information loss: a summary may drop an exact number or a user's exact wording, and summarizing a summary repeatedly makes this worse.

Many systems keep the raw text in an archive so details can still be retrieved later.

A write policy is the rule set that decides what gets saved, when, and in what form. Without one, an agent either stores everything and drowns in noise, or stores nothing useful.

Common policies include:

• Write after every turn, then deduplicate later.
• Write only when an extractor flags a durable fact or preference.
• Write when the user says "remember this".
• Write a summary at session end.

Good policies also set what must never be saved, such as passwords or payment details.

A sliding window keeps only the last N messages or tokens of the conversation and drops older ones as new ones arrive. It is the simplest form of short-term memory.

It is cheap, predictable and needs no extra infrastructure. The downside is hard forgetting: if the user mentioned their deadline 30 turns ago and the window holds 20, the model no longer knows it.

That is why windows are usually paired with a summary of dropped turns or a long-term store.

MemGPT is a research system (Packer et al., 2023) that treats an LLM's context like an operating system treats RAM. The model gets a small main context plus larger external storage, and it moves data between them itself using function calls.

The external side has recall storage (searchable conversation history) and archival storage (long-term notes and documents). When context fills up, the system warns the model, which can save key facts and evict old messages.

The idea lives on in the open-source Letta framework.

Mem0 is an open-source memory layer for AI applications that sits between your app and the LLM. It watches conversations, extracts salient facts, and stores them so they can be retrieved in later sessions.

A notable design choice is that, for each candidate fact, the system compares it against similar existing memories and chooses an operation: add, update, delete or do nothing. That keeps memory from filling with duplicates and contradictions.

It supports user-, session- and agent-level scoping so memories stay separated.

Knowledge graph memory stores information as entities and relationships - nodes and edges - instead of free-text chunks. For example: (Priya) -[works_at]-> (Acme), (Acme) -[uses]-> (Postgres).

It shines on questions that chain facts together, like "which database does Priya's employer use?", where plain similarity search might retrieve unrelated chunks.

The cost is extra work at write time: an LLM or parser must extract clean entities and relations, and mistakes there become wrong edges.

Systems in this space often pair the graph with embeddings on nodes and edges, so you find the starting entity by similarity and then walk the relationships from there.

Metadata turns a pile of text into something you can filter, rank and audit. Typical fields:

• user_id / tenant_id - ownership and isolation
• created_at, last_accessed_at - recency and decay
• importance - a score, often rated by the model
• type - episodic, semantic or procedural
• source - which conversation or document it came from
• confidence and expiry - how sure you are and when it should lapse

Add only fields you will actually query on. Every extra field is something you must keep consistent through updates and deletions.

Memory decay lowers the weight of old or unused memories over time, or removes them outright. Without it, stores grow forever and retrieval fills up with outdated, low-value entries.

Common approaches are an exponential recency decay in the ranking score, a time-to-live on entries, and pruning memories that are never retrieved. Some systems boost items each time they're accessed, similar to how human recall works.

Forgetting is a feature, not a bug - but important items need protection through high importance scores.

A memory embedding is a vector, usually hundreds or a few thousand numbers, produced by an embedding model to capture the meaning of a memory's text. Similar meanings land close together in vector space.

That is what makes "I'm vegetarian" retrievable when the user later asks "suggest a dinner recipe" - the two texts share no keywords but sit near each other.

Use the same embedding model for writing and querying. Mixing models gives meaningless distances.

Re-embedding becomes necessary if you ever switch models, so keep the original text next to the vector instead of storing vectors alone.

Both retrieve text into the prompt, but they retrieve different things and treat the store differently. RAG pulls from a mostly static knowledge corpus (docs, wikis, manuals). MAG pulls from a memory the system keeps writing to as it interacts.

In practice they overlap: a MAG system often uses RAG-style retrieval machinery for reading, and many agents use both at once.

Each turn follows the same cycle: retrieve, generate, write.

1. The user message arrives and is used (sometimes rewritten) as a search query.
2. The memory store returns the top-ranked entries.
3. The app builds a prompt from system instructions, retrieved memories and recent turns.
4. The model generates a reply.
5. A writer step extracts anything worth keeping and updates the store.

sequenceDiagram
  participant U as User
  participant A as App
  participant M as Memory store
  participant L as LLM
  U->>A: message
  A->>M: search with query
  M-->>A: top memories
  A->>L: prompt plus memories
  L-->>A: reply
  A->>M: write or update memories
  A-->>U: reply

The write step often runs asynchronously so it doesn't add latency to the reply.

A long context only helps within a single call. Four things still break:

• Persistence - the window resets with each new session.
• Cost and latency - re-sending huge histories every turn is expensive and slow.
• Attention quality - models tend to use information in the middle of very long prompts less reliably (the "lost in the middle" effect).
• Noise - dumping everything in means irrelevant text competes with the useful parts.

Memory selects what deserves to be in the window. Bigger windows make that budget larger; they don't remove the need for selection.

A useful rule of thumb: use the window for the active task, and use memory for everything the task might need to look up.

In the Generative Agents paper (Park et al., 2023), every observation is saved to a natural-language memory stream. When the agent needs context, each memory gets a score from three components:

• Recency - exponential decay since the memory was last accessed.
• Importance - the LLM rates how significant the event is on a 1-10 scale.
• Relevance - embedding similarity to the current situation.

Each is normalized and summed, and the top-scoring memories go into the prompt.

score = a1*recency + a2*importance + a3*relevance
top = sorted(memories, key=score, reverse=True)[:k]

The same paper adds reflection: periodically the agent synthesizes higher-level insights from recent memories and stores them back in the stream.

MemGPT splits memory into a small main context (what the model sees) and large external context (recall and archival storage), then lets the model manage movement between them through function calls.

The main context holds system instructions, a writable working-context block, and a FIFO queue of recent messages. When the queue nears its limit, the system signals memory pressure. The model then calls functions to save key facts to archival storage, search recall storage, or edit its own working context before old messages are evicted.

flowchart LR
  A["New message"] --> B["Main context"]
  B --> C{Near token limit?}
  C -- No --> D["Generate reply"]
  C -- Yes --> E["Model saves facts via function call"]
  E --> F["Evict old messages to recall storage"]
  F --> D

The takeaway is that the model itself decides what to page in and out, instead of fixed rules deciding for it.

Parametric memory is knowledge baked into model weights during training or fine-tuning. Non-parametric memory is knowledge kept outside the weights and looked up at inference time.

MAG is mainly non-parametric. That's why it suits fast-changing, personal or deletable information, while stable general knowledge stays in the weights.

A common split is to let weights carry language skills and broad world knowledge, while memory carries user-specific, time-sensitive or removable data. Model-editing techniques exist for weights, but they are still less predictable than deleting a record.

kNN-LM (Khandelwal et al., 2019) builds a datastore where each key is a hidden representation of a text prefix and each value is the token that followed it in the training data.

At inference, the model's current hidden state queries the datastore for its nearest neighbors. Their distances become a probability distribution over next tokens, which is interpolated with the normal language-model distribution:

p(next) = lam * p_knn(next) + (1 - lam) * p_lm(next)

It improved perplexity without retraining, especially for rare patterns and domain-specific text. Retrieval happens at every token, so it is heavier than prompt-level retrieval.

One practical drawback: for large corpora the datastore can hold billions of entries, so storage and search cost grow with the data. That makes it more of a research technique than a typical agent pattern.

RAG retrieves text and pastes it into the prompt, so the model reads it like any other input. Memorizing Transformers (Wu et al., 2022) retrieve at the attention level instead.

They cache past key-value pairs from earlier segments in an external memory and, in one chosen layer, do a kNN lookup for each query. The retrieved keys and values are attended to alongside local context, blended by a learned gate. The memory isn't trained by gradients, so it can be extended with new data.

Titans (Google, 2024) adds a neural memory module, a small deep network whose weights act as the memory, that keeps learning while the model is running.

Each new token is treated as a key-value association, and the memory takes a gradient step to reduce its recall error on it. The size of that gradient is the surprise signal: unexpected inputs cause bigger updates. A momentum term carries surprise across nearby tokens, and an adaptive forgetting gate decays old content so capacity isn't wasted.

The paper proposes three ways to combine it with attention: Memory as Context (MAC), Memory as Gate (MAG) and Memory as Layer (MAL).

Note the naming clash. In Titans, "MAG" means Memory as Gate; in this topic it means the broader Memory-Augmented Generation pattern.

Summarize when you need the gist and tokens are tight. Store raw when exact wording matters.

• Summarize: long chit-chat, brainstorming, status updates, older context that just needs orienting.
• Keep raw: numbers, code, names, legal or medical statements, commitments, and anything a user might quote back.

A solid hybrid is to keep recent turns verbatim, summarize older ones into the prompt, and archive the raw text in a searchable store. That way the summary keeps the prompt small, and retrieval can still recover exact details when a question needs them.

A quick test: if a wrong digit or reworded clause would cause a wrong answer or real harm, keep the original text.

Not everything said is worth keeping. A practical filter asks whether the information is durable, reusable and specific.

1. Durable - will it still be true next week? "I'm in Berlin this month" may not be.
2. Reusable - will it change a future answer? A preferred language does; a one-off typo fix doesn't.
3. Specific - is it concrete enough to act on?
4. Safe - is it free of secrets and sensitive data you shouldn't hold?

Teams implement this with an extraction prompt that returns candidate facts with an importance score, then drop anything below a threshold or already known. Explicit user instructions like "remember that..." should always pass.

After a turn or session, the conversation is sent to an LLM with instructions to output structured candidate memories, usually as JSON, instead of prose.

[{"fact": "Prefers TypeScript over JavaScript",
  "type": "semantic",
  "importance": 7,
  "ttl_days": null}]

The system then runs each candidate through checks: search for similar existing memories, decide add/update/ignore, and apply policy filters for sensitive data. Only surviving entries get embedded and stored.

Prompt wording matters a lot. Tell it to extract only stated facts, not guesses, and to return an empty list when nothing is worth saving.

Measure extractor precision and recall on a small set of conversations with known correct facts, since this step silently caps the quality of everything downstream.

If you simply append, retrieval may return both and the model gets confused: "lives in Austin" and "moved to Denver". Conflict handling has to be a deliberate step.

The usual flow is: search for similar memories, let a model or rule judge whether the new fact replaces, refines or coexists with the old one, then act.

• Replace - overwrite the old value when it is simply outdated.
• Invalidate - mark the old one expired with an end date, keeping history.
• Coexist - keep both if they aren't really in conflict (a user can like tea and coffee).
• Ask - if confidence is low, confirm with the user.

Temporal fields like valid-from and valid-to make this much easier to reason about.

Consolidation turns a growing pile of fine-grained memories into a smaller, cleaner set, similar to how sleep is thought to organize human memory.

A background job typically runs periodically and does four things:

1. Deduplicate near-identical entries using embedding similarity.
2. Merge related episodes into one semantic fact.
3. Generate reflections - higher-level insights across many events, as in Generative Agents.
4. Prune low-value or expired entries.

Run it offline so it doesn't hurt latency, keep the originals archived for a while, and log what was merged so mistakes can be reversed.

A common failure is over-merging, where two different facts about similar entities collapse into one. Sample merged entries regularly and check that each source fact can still be recovered from the result.

Choose a graph when the questions depend on relationships and multi-hop reasoning, and vectors when they depend on fuzzy semantic similarity.

Graphs also give cleaner updates: changing an employer is one edge edit, not hunting through paraphrased chunks. The costs are heavier extraction at write time and schema upkeep.

Many teams combine both: a vector index for recall and a graph for structured entity facts.

If you're unsure, begin with vectors plus structured metadata. Move to a graph once you see recurring multi-hop failures, because that's a clear sign relationships are doing real work.

Temporal graphs, as in Zep's Graphiti, attach time information to facts instead of treating the graph as a snapshot. Edges carry validity timestamps such as when a fact became true and when it stopped being true.

When a new fact contradicts an existing edge, the old edge is invalidated (given an end time), not deleted. The new edge starts from the new date.

This lets the system answer both "where does the user live now?" and "where did they live in March?", and it avoids returning outdated facts as if they were current. The price is added complexity in extraction, since the system must infer dates from natural language.

Retrieval then filters by time: current-state questions use only edges with no end date, while historical questions use edges that were valid at the time being asked about.

Evaluate the memory and the end answers separately, otherwise you can't tell whether failures came from retrieval or generation.

• Retrieval metrics - recall@k and precision@k against a labeled set of "which memory should have been fetched".
• Answer quality - accuracy on questions that need prior context, judged by humans or an LLM judge.
• Benchmarks - LoCoMo tests QA over very long multi-session conversations; LongMemEval covers information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention.
• Operational - latency, token cost per turn, memory growth.

Don't skip abstention: a good memory system says "I don't know" when nothing relevant was stored, instead of inventing.

Build a small in-house test set from real conversations too. Public benchmarks measure general memory skills, but only your own data reveals domain-specific failures such as product names or internal jargon.

Retrieved text is treated by the model as trusted context. If the wrong memory shows up, the model often uses it anyway, and the answer gets worse than with no memory at all.

Typical failure modes:

• Irrelevant but similar entries, such as another project's config, steer the answer off course.
• Stale facts are presented as current.
• Too many entries bury the useful one, and long prompts dilute attention.
• Wrong user data causes leaks.

Mitigate with similarity thresholds, reranking, small k, recency and importance weighting, and a prompt that tells the model memory may be incomplete or outdated.

Log every injected memory with its score so you can inspect bad answers afterward, and consider having the model cite which memory it relied on.

Start by finding where time goes: embedding the query, the search itself, reranking, or network hops. Then fix the biggest piece.

1. Approximate nearest neighbor indexes such as HNSW trade a little recall for much faster search.
2. Metadata pre-filtering by user or type shrinks the search space.
3. Cache query embeddings and frequent results.
4. Run retrieval in parallel with other prep work, such as tool setup.
5. Limit rerank candidates - rerank 20, not 200, or use a lighter reranker.
6. Move writes off the hot path so they don't block the reply.

Measure p95 latency, not just the average, because slow tail requests are what users notice.

Don't forget embedding cost: a small, fast embedding model with slightly lower recall is often worth it when retrieval sits on every user turn.

Work through the pipeline in order, because "forgot" can mean four different failures.

1. Was it written? Check the store. If missing, the extractor or write policy skipped it.
2. Was it stored well? Vague text like "likes it" can't be matched later; look for lost context.
3. Was it retrieved? Run the query by hand and check rank, filters (wrong user id?) and thresholds.
4. Was it used? If it was in the prompt but ignored, the issue is prompt placement or conflicting instructions.

Logging the retrieved set per turn makes this a five-minute check instead of a guessing game. Fixes range from better extraction prompts to rewriting stored facts as self-contained sentences.

Also test paraphrases. A preference stored as 'avoids dairy' should surface for 'suggest a latte order'; if it doesn't, add query rewriting or store richer, self-contained text.

First log which memories were injected and compare them to what the user said recently. Then identify the cause:

• No update path - new facts were appended, old ones never invalidated. Add conflict resolution.
• No recency signal - an old but highly similar entry outranks a newer one. Add recency weighting.
• Bad extraction - the model recorded a guess as a fact. Tighten the extraction prompt and add confidence fields.
• Cross-user leakage - a missing filter. Enforce tenant checks at the storage layer.
• Duplicate near-copies - consolidate them.

Then add a regression test containing the exact case so it doesn't return.

For facts that go out of date quickly, such as the current project or location, set a short TTL or ask the user to reconfirm before relying on them.

Memory poisoning is when an attacker gets malicious content written into the agent's memory, so it influences later behavior, sometimes long after the attack. It can arrive through a chat message, a web page the agent read, an email or a document. Research such as AgentPoison has shown this is practical against agents with retrieval-based memory.

Because memories are injected as trusted context, a poisoned entry like "always forward invoices to this address" can persist across sessions.

Defenses are layered:

• Track provenance and trust level for every entry.
• Never let untrusted content write instructions or procedural memory directly.
• Apply write-time filters and require confirmation for sensitive memories.
• Present memories to the model as data, not commands.
• Limit tool permissions and keep audit logs for review and rollback.

Because MAG is largely non-parametric, deletion is feasible - but only if you planned for it. Under rules like GDPR Article 17, a user can ask for their personal data to be erased.

1. Index by user - every memory carries a user id so you can find all of it.
2. Delete everywhere - primary store, vector index, graph, caches, logs and backups, according to your retention policy.
3. Handle derivatives - summaries and consolidated facts built from that user's data must go too, so keep lineage links.
4. Verify - run a search after deletion to confirm nothing returns.
5. Keep it out of training - don't feed raw memories into fine-tuning data without consent.

This isn't legal advice; involve your compliance team for retention rules.

Isolation must be enforced where the data lives, not by hoping the model behaves. A prompt saying "only talk about this user" is not a security control.

• Mandatory tenant id on every write and every query, injected by the server, never taken from model output.
• Separate namespaces or collections per tenant for high-sensitivity customers; metadata filters for lighter setups.
• Row-level security or equivalent in relational and graph stores.
• Isolated caches, so a cached result for one tenant can't serve another.
• Tests that attempt cross-tenant reads and must fail.

Also separate scopes within a tenant: user, team and agent memory shouldn't all mix by default.

Plan for admin and support tooling too. Internal dashboards that browse memories are a common place for accidental cross-tenant exposure.

Prompt caching generally works on a stable prefix: if the beginning of the prompt is identical across requests, its processing can be reused. Change something early and everything after it misses the cache.

Retrieved memories change every turn, so where you put them matters.

1. Keep the system prompt, tool definitions and static instructions first so they stay cacheable.
2. Place per-turn retrieved memories after that stable block.
3. Keep slowly changing items, such as a user profile block, ahead of fast-changing ones.
4. Avoid inserting timestamps or random IDs near the top.

flowchart LR
  A["System prompt and tools - cached"] --> B["User profile block - changes rarely"]
  B --> C["Retrieved memories - change each turn"]
  C --> D["Recent messages and new input"]

Check your provider's caching rules and minimum length, as details vary.

Treat the context window as a budget split into fixed and flexible parts, and cap memory so it never crowds out the user's actual request.

Rank first, then fill until the cap. Drop low-score entries instead of truncating mid-sentence. Shorter, self-contained memories pack better, so rewrite them at write time. Review average injected tokens per turn - if it creeps up, tighten k or thresholds.

Dynamic budgets work well: give memory more room on simple chat turns and less when the task already carries large documents or tool output.

Vector search finds meaning but can miss exact tokens; keyword search (BM25) finds exact tokens but misses paraphrases. Hybrid search runs both and merges the lists, often with reciprocal rank fusion.

That matters for memory, where users mention IDs, error codes, names and product versions that embeddings blur together.

A reranker then rescores the top candidates, typically 20-50, using a cross-encoder or an LLM that reads the query and each memory together. It's slower but much more precise than embedding distance.

If latency allows, retrieve more candidates than you need, rerank, then keep only the top few. Reranking is where most of the precision gain comes from.

flowchart LR
  Q[Query] --> V["Vector search"]
  Q --> K["Keyword search"]
  V --> F["Fuse ranked lists"]
  K --> F
  F --> R["Rerank top candidates"]
  R --> T["Top k memories to prompt"]

A memory isn't static; it moves through stages from creation to retirement.

1. Capture - raw interaction is observed.
2. Extract and filter - candidate facts are pulled out and checked against the write policy.
3. Dedupe and resolve - compared with existing entries; added, merged or used to update an old one.
4. Store - embedded and saved with metadata.
5. Retrieve and use - ranked, injected, and its access time refreshed.
6. Consolidate - merged into higher-level facts in background jobs.
7. Expire or delete - removed by decay, TTL, supersession or user request.

flowchart TD
  A[Capture] --> B["Extract and filter"]
  B --> C["Dedupe and resolve conflicts"]
  C --> D["Store with metadata"]
  D --> E["Retrieve and inject"]
  E --> F[Consolidate]
  F --> G["Expire or delete"]
  E --> D

Start from what the agent must remember and how long, then pick stores to match.

Key design decisions:

• Scope every read and write by customer id; support data is sensitive.
• Write a short summary and resolution at ticket close, and keep raw text archived.
• Invalidate old facts when the customer upgrades or changes setup.
• Escalate to a human instead of answering when retrieval confidence is low.
• Evaluate with replayed historical tickets and track repeat-contact rate.

Finally, give users and staff a way to correct memory, such as a visible profile page or a 'that's wrong' command. It fixes errors faster than any automated cleanup.

A production setup separates the read path (fast, on the request) from the write path (can be slower, asynchronous), with governance cutting across both.

Read path: authenticate and resolve tenant, rewrite the query if needed, run hybrid retrieval with tenant filters, rerank, apply recency and importance weighting, trim to the token budget, assemble the prompt with stable content first, and call the model.

Write path: after the response, a queue worker runs the extractor, filters for sensitive data, checks similar entries, and then adds, updates or invalidates. Embeddings are computed and indexes updated.

Background: consolidation, decay, reindexing, and metrics collection.

Governance: provenance, audit logs, deletion APIs, access control and evaluation dashboards.

flowchart TD
  U["User request"] --> A["Auth and tenant resolve"]
  A --> R["Hybrid retrieve and rerank"]
  R --> P["Budget and build prompt"]
  P --> L[LLM]
  L --> O[Response]
  L --> Q["Write queue"]
  Q --> X["Extract and filter"]
  X --> C["Resolve and upsert"]
  C --> S[(Memory stores)]
  S --> R
  S --> B["Consolidate and decay jobs"]

The main design principle: reads must be cheap and safe, writes must be careful, and every memory must be traceable and deletable.