Spring / Spring AI interview questions
How do you stream responses from an LLM in Spring AI?
Streaming in Spring AI lets you consume LLM output token-by-token as a reactive Flux<String> (or Flux<ChatResponse> for full metadata) rather than waiting for the entire response to arrive. This is critical for chat UIs where users expect to see text appear progressively.
Replace .call() with .stream() in the ChatClient chain:
// Stream plain text tokens Flux<String> tokenStream = chatClient.prompt() .user("Write a short story about a Java developer.") .stream() .content(); // Consume in a WebFlux controller @GetMapping(value = "/story", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<String> story() { return chatClient.prompt() .user("Tell me a story") .stream() .content(); }
For full response metadata (finish reason, token usage per chunk) use .stream().chatResponse() which returns Flux<ChatResponse>. If you need to collect the complete text after streaming for post-processing, use the standard Project Reactor .collectList() or .reduce() operators.
When using the lower-level ChatModel interface directly, call chatModel.stream(prompt) which also returns Flux<ChatResponse>. Note that not every provider supports streaming — check the provider's documentation. OpenAI, Anthropic, and Ollama all support it; some Bedrock models do not.
More Related questions...
