Shift note: temperature checks before production

Temperature changes the next-token choice

When a model writes a response, it repeatedly chooses the next token: a word, part of a word, or punctuation mark. Several possible tokens may fit. Sampling settings influence how the model chooses among them, which changes how predictable or varied the finished response feels.

Temperature adjusts that choice. Lower values concentrate selection on more likely tokens. Higher values spread probability more broadly, giving less likely tokens more opportunity. In everyday terms, lower temperature usually produces steadier wording, while higher temperature usually produces more variation.

Temperature is not a truthfulness dial. Lowering it does not make missing information appear or guarantee correct answers. Raising it does not automatically produce better ideas. It changes selection behavior, not the evidence available to the model.

Start with the task, not a favorite number

Choose settings around the behavior your application needs. Extracting an invoice total and brainstorming campaign names are different jobs, even if they use the same model.

• For extraction, classification, or tightly constrained replies, try a low temperature if the model supports it, then check accuracy and format compliance.
• For ordinary explanations or support drafts, begin with the model's default and adjust only when you observe a specific problem.
• For brainstorming or alternative phrasings, try a moderately higher temperature and judge whether the additional variety is useful rather than merely unusual.

On an API that supports these values, 0.2 can be a low-temperature experiment and 0.8 a more varied one. These are test points, not universal recommendations. Supported ranges, defaults, and effects differ across models. Some models restrict or reject sampling controls, so check the documentation for your chosen endpoint before sending them.

Top_p narrows the candidate pool differently

Top_p, also called nucleus sampling, limits selection to a group of likely next tokens whose combined probability reaches a chosen threshold. A top_p of 0.9 generally keeps the smallest leading group covering at least 90 percent of the probability mass. It does not mean the model will be correct 90 percent of the time.

Temperature reshapes probabilities; top_p limits the pool considered. Both affect variety, and their interaction can make troubleshooting harder. For an initial experiment, leave top_p at its documented default and change temperature alone. If you later test top_p, keep the other settings fixed so you can identify what helped.

Zero temperature still needs repeatability checks

A temperature of zero generally selects the highest-probability next token instead of sampling broadly. That often reduces variation, but it is not a promise of identical results. Backend implementation details, numerical differences, and model updates can still affect outputs.

If an API offers a seed, treat it as a repeatability aid within that provider's documented limits, not a guarantee. Record the model identifier, prompt, sampling settings, and relevant API options. For structured output, use schema enforcement where supported and validate the result in your application. Temperature alone cannot enforce a contract.

Run a small test before changing the live setting

Collect representative prompts, including awkward cases such as incomplete inputs and ambiguous requests. Run each prompt several times with the default settings, then repeat with one controlled adjustment.

• Measure whether extracted values and classifications are correct against examples you have checked.
• Check whether replies follow the required format and avoid unsupported claims.
• Compare repeated outputs for useful diversity, unwanted drift, and the editing effort they require.

Repeat this process when switching models. A setting that works well for Gemini 2.5 Flash should not be assumed to behave identically elsewhere.

Keep relay access separate from sampling policy

Relay365 is a pay-as-you-go relay desk for multiple model families, making it an option when you want to evaluate models through a shared access point. For OpenAI SDK clients, its API base is https://api.api365relay.com/v1. A shared client interface does not mean every model accepts every sampling parameter.

If that access pattern fits your workflow, create a Relay365 account and begin with a small evaluation set. Keep the production rule simple: use documented settings, change one variable at a time, and promote a configuration only after its outputs pass your checks.