What are the Spring AI Chat Model options for controlling response determinism?

Response determinism in LLMs is primarily controlled through two inference parameters: temperature and top-p (nucleus sampling). Both are set via ChatOptions in Spring AI and work together to shape how randomly the model selects the next token at each step of generation.

Temperature scales the probability distribution over the vocabulary before sampling. A temperature of 0.0 makes the model almost always choose the single highest-probability token (near-deterministic, repetitive). A temperature of 1.0 samples from the raw distribution. Values above 1.0 flatten the distribution further, increasing creativity and randomness. For factual tasks (Q&A, code generation, data extraction) use 0.0–0.3. For creative tasks (writing, brainstorming) use 0.7–1.0.

Top-p restricts sampling to the smallest set of tokens whose cumulative probability exceeds p. A top-p of 0.9 means the model only considers tokens that together account for 90% of the probability mass, discarding long-tail unlikely tokens. Most practitioners either tune temperature alone and leave top-p at 1.0, or tune top-p alone and leave temperature at 1.0 — adjusting both simultaneously is rarely necessary and harder to reason about.

// Deterministic (code analysis, data extraction)
ChatOptions factual = ChatOptionsBuilder.builder()
    .withTemperature(0.1f).withTopP(1.0f).build();

// Creative (story, marketing copy)
ChatOptions creative = ChatOptionsBuilder.builder()
    .withTemperature(0.85f).withTopP(0.95f).build();

Note that even temperature 0 is not fully deterministic across all providers due to floating-point parallelism in GPU computations — you may see occasional token variation on identical inputs.