Spring / Spring AI interview questions
What is semantic caching in Spring AI and how would you implement it?
Semantic caching is an optimisation where you cache LLM responses not by exact query string match but by semantic similarity — if a new question is semantically close enough to a previously answered one, return the cached answer rather than calling the LLM again. This is far more effective than a traditional string-equality cache for AI workloads where users phrase the same question in different ways.
Spring AI does not ship a built-in semantic cache, but the framework provides all the building blocks — a VectorStore, an EmbeddingModel, and the Advisor pattern — to build one cleanly as a custom RequestResponseAdvisor:
@Component public class SemanticCacheAdvisor implements RequestResponseAdvisor { private final VectorStore cacheStore; private final double threshold; public SemanticCacheAdvisor(VectorStore cacheStore) { this.cacheStore = cacheStore; this.threshold = 0.92; } @Override public AdvisedRequest adviseRequest(AdvisedRequest req, Map<String, Object> ctx) { List<Document> hits = cacheStore.similaritySearch( SearchRequest.query(req.userText()) .withTopK(1).withSimilarityThreshold(threshold)); if (!hits.isEmpty()) { ctx.put("cache_hit", hits.get(0).getMetadata().get("cached_answer")); } return req; } @Override public ChatClientResponse adviseResponse(ChatClientResponse resp, Map<String, Object> ctx) { if (!ctx.containsKey("cache_hit")) { // Store new answer in cache Document entry = new Document( (String) resp.chatResponse().getResult().getOutput().getContent(), Map.of("cached_answer", resp.chatResponse().getResult().getOutput().getContent())); cacheStore.add(List.of(entry)); } return resp; } }
The similarity threshold (0.9–0.95) is the key tunable: too low and semantically different questions share cached answers; too high and the cache hit rate drops to near zero. For time-sensitive data, add a TTL by storing a timestamp in metadata and invalidating on retrieval.
More Related questions...
