How do you stream LLM responses token by token for a better user experience?

Without streaming, the user waits for the model to finish generating the entire response before seeing anything — for long outputs this can be 10–30 seconds of blank wait time. Streaming delivers each token to the user as it is generated, making the application feel dramatically more responsive. Both the OpenAI API and Hugging Face support streaming.

# ── OpenAI streaming with the Python SDK
from openai import OpenAI

client = OpenAI()

with client.chat.completions.stream(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Write a haiku about transformers.'}],
    max_tokens=100,
) as stream:
    for text in stream.text_stream:
        print(text, end='', flush=True)
print()  # newline after stream ends

# ── LangChain LCEL streaming
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

chain = (
    ChatPromptTemplate.from_template('Write a short poem about {topic}.')
    | ChatOpenAI(model='gpt-4o-mini', streaming=True)
    | StrOutputParser()
)

for chunk in chain.stream({'topic': 'neural networks'}):
    print(chunk, end='', flush=True)

# ── Hugging Face streaming
from transformers import pipeline, TextIteratorStreamer
from threading import Thread
import torch

pipe = pipeline('text-generation', model='gpt2', torch_dtype=torch.bfloat16)
streamer = TextIteratorStreamer(pipe.tokenizer, skip_prompt=True)

thread = Thread(target=pipe, kwargs={
    'text_inputs': 'Once upon a time',
    'max_new_tokens': 100,
    'streamer': streamer,
})
thread.start()
for token in streamer:
    print(token, end='', flush=True)
thread.join()