How does streaming work in LangChain4j and when should you use it?

Streaming in LangChain4j allows the LLM's response to be delivered token-by-token as it is generated, rather than waiting for the entire response to be produced before returning anything to the caller. For user-facing chat interfaces, this dramatically improves perceived responsiveness — the user sees text appearing progressively instead of staring at a loading spinner for several seconds.

LangChain4j supports streaming through two mechanisms:

1. TokenStream (AI Services) — Declare the return type as TokenStream in your AI Services interface. The caller registers handlers for each token, completion, and errors:

interface StreamingAssistant {
    TokenStream chat(String message);
}

StreamingAssistant assistant = AiServices.builder(StreamingAssistant.class)
    .streamingChatLanguageModel(streamingModel) // note: streaming model
    .build();

assistant.chat("Explain quantum entanglement")
    .onNext(token -> System.out.print(token))
    .onComplete(response -> System.out.println("\nDone. Tokens used: " + response.tokenUsage()))
    .onError(Throwable::printStackTrace)
    .start();

2. Direct StreamingChatLanguageModel — Use the lower-level interface for custom streaming logic without AI Services.

For Spring Boot applications serving a web API, the streaming response is typically connected to an SSE (Server-Sent Events) endpoint or a WebSocket. Spring WebFlux's Flux<String> integrates naturally with LangChain4j's streaming by bridging the onNext callback to a reactive publisher.

Use streaming when: building conversational UIs, generating long-form content where early tokens are already useful, or when you need to display a typing indicator. Avoid streaming for batch jobs, automated pipelines, or API calls where the complete response is needed before any processing begins.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.