What is the Spring AI Evaluation framework and how do you use it?

The Spring AI Evaluation framework provides programmatic tools for assessing the quality of LLM responses — particularly RAG outputs — without manual human review on every run. This is important for catching prompt regressions and measuring retrieval quality as your system evolves.

Spring AI ships two built-in evaluators:

RelevancyEvaluator — judges whether the LLM's answer is relevant to the question asked. Internally it sends the question, the answer, and the retrieved context to another LLM call and asks it to score relevancy. Returns an EvaluationResponse with a boolean pass/fail and a score.

FactCheckingEvaluator — verifies that statements in the answer are grounded in the retrieved context documents. It flags hallucinations — claims that have no basis in the provided context.

@Test
void ragAnswerShouldBeRelevant() {
    // Generate an answer using your RAG pipeline
    String question = "What is Spring AI's default retry backoff?";
    ChatResponse response = ragService.answer(question);
    List<Document> context = ragService.lastRetrievedContext();

    EvaluationRequest evalRequest = new EvaluationRequest(
        question,
        context,
        response.getResult().getOutput().getContent()
    );

    EvaluationResponse evalResponse = new RelevancyEvaluator(
        ChatClient.builder(chatModel).build()
    ).evaluate(evalRequest);

    assertThat(evalResponse.isPass()).isTrue();
}

Evaluators make LLM calls themselves, so they add latency and cost to test runs. Run evaluation suites as part of a separate CI stage on a representative question set rather than inline with every unit test.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.