What is semantic caching in Spring AI and how would you implement it?

Semantic caching is an optimisation where you cache LLM responses not by exact query string match but by semantic similarity — if a new question is semantically close enough to a previously answered one, return the cached answer rather than calling the LLM again. This is far more effective than a traditional string-equality cache for AI workloads where users phrase the same question in different ways.

Spring AI does not ship a built-in semantic cache, but the framework provides all the building blocks — a VectorStore, an EmbeddingModel, and the Advisor pattern — to build one cleanly as a custom RequestResponseAdvisor:

@Component
public class SemanticCacheAdvisor implements RequestResponseAdvisor {

    private final VectorStore cacheStore;
    private final double threshold;

    public SemanticCacheAdvisor(VectorStore cacheStore) {
        this.cacheStore = cacheStore;
        this.threshold = 0.92;
    }

    @Override
    public AdvisedRequest adviseRequest(AdvisedRequest req, Map<String, Object> ctx) {
        List<Document> hits = cacheStore.similaritySearch(
            SearchRequest.query(req.userText())
                .withTopK(1).withSimilarityThreshold(threshold));
        if (!hits.isEmpty()) {
            ctx.put("cache_hit", hits.get(0).getMetadata().get("cached_answer"));
        }
        return req;
    }

    @Override
    public ChatClientResponse adviseResponse(ChatClientResponse resp, Map<String, Object> ctx) {
        if (!ctx.containsKey("cache_hit")) {
            // Store new answer in cache
            Document entry = new Document(
                (String) resp.chatResponse().getResult().getOutput().getContent(),
                Map.of("cached_answer", resp.chatResponse().getResult().getOutput().getContent()));
            cacheStore.add(List.of(entry));
        }
        return resp;
    }
}

The similarity threshold (0.9–0.95) is the key tunable: too low and semantically different questions share cached answers; too high and the cache hit rate drops to near zero. For time-sensitive data, add a TTL by storing a timestamp in metadata and invalidating on retrieval.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.