How does streaming work in LangChain4j and when should you use it?

Streaming in LangChain4j allows the LLM's response to be delivered token-by-token as it is generated, rather than waiting for the entire response to be produced before returning anything to the caller. For user-facing chat interfaces, this dramatically improves perceived responsiveness — the user sees text appearing progressively instead of staring at a loading spinner for several seconds.

LangChain4j supports streaming through two mechanisms:

1. TokenStream (AI Services) — Declare the return type as TokenStream in your AI Services interface. The caller registers handlers for each token, completion, and errors:

interface StreamingAssistant {
    TokenStream chat(String message);
}

StreamingAssistant assistant = AiServices.builder(StreamingAssistant.class)
    .streamingChatLanguageModel(streamingModel) // note: streaming model
    .build();

assistant.chat("Explain quantum entanglement")
    .onNext(token -> System.out.print(token))
    .onComplete(response -> System.out.println("\nDone. Tokens used: " + response.tokenUsage()))
    .onError(Throwable::printStackTrace)
    .start();

2. Direct StreamingChatLanguageModel — Use the lower-level interface for custom streaming logic without AI Services.

For Spring Boot applications serving a web API, the streaming response is typically connected to an SSE (Server-Sent Events) endpoint or a WebSocket. Spring WebFlux's Flux<String> integrates naturally with LangChain4j's streaming by bridging the onNext callback to a reactive publisher.

Use streaming when: building conversational UIs, generating long-form content where early tokens are already useful, or when you need to display a typing indicator. Avoid streaming for batch jobs, automated pipelines, or API calls where the complete response is needed before any processing begins.