Shift Note: Sampling Settings for the First Test Run

The first sampling change on a new integration should be small, recorded, and easy to reverse. Temperature affects how a model selects tokens, but it does not repair unclear instructions or verify facts. Follow this walkthrough to establish a baseline, test variation, and leave settings the next operator can understand.

Start with one task and a repeatable baseline

1. Choose a task you can judge consistently, such as extracting invoice fields, summarizing a support ticket, or drafting three subject lines. Keep the model, prompt, input, tools, and output limit fixed. Save the full request and response, excluding secrets and sensitive customer data.

Run several representative inputs before changing anything. Include an ordinary case, an ambiguous case, and a case with missing information. Your baseline should reveal whether the model follows instructions, not just whether one response reads well.

Confirm which sampling controls the model accepts

2. Check the selected endpoint's documentation before sending temperature or top_p. Supported parameters, defaults, and allowed ranges can differ across providers and models. Some endpoints restrict sampling controls or reject combinations. An OpenAI-compatible request format does not mean every model supports identical behavior.

If you evaluate Claude Haiku 4.5 or Gemini 2.5 Flash through a relay, confirm the accepted settings for that exact model and route. Keep unsupported parameters out of the request rather than assuming the relay will translate or ignore them.

Lower temperature for tighter variation, not guaranteed truth

3. Where supported, test a low temperature for extraction, classification, and other tasks where consistency matters. Temperature changes the relative likelihood of candidate tokens during sampling. Lower values generally concentrate selection on more likely tokens; higher values generally allow more variation.

For a first experiment, compare a supported low value such as 0.2 with the documented default. Treat those as trial settings, not universal recommendations. Temperature zero does not guarantee identical responses, valid JSON, or factual accuracy. Use schema constraints and validation when the endpoint supports them, and handle failures explicitly.

Increase variation only when the task benefits

4. For brainstorming or alternative wording, try a moderately higher temperature within the documented range. Compare the results against your baseline. Look for useful diversity rather than novelty alone: three genuinely different subject lines are more valuable than three differently worded versions of the same idea.

If higher temperature introduces unsupported claims, tighten the prompt and evaluate again. Sampling is not a substitute for grounding answers in supplied evidence.

Test top_p separately from temperature

5. Leave top_p at its documented default while testing temperature. Top_p, often called nucleus sampling, limits selection to a set of tokens whose cumulative probability reaches the chosen threshold. Lower values usually narrow that candidate set.

If you later test top_p, hold temperature steady and change only top_p. Moving both controls together makes it harder to explain an improvement or diagnose a regression. Unless the endpoint recommends otherwise, tune one sampling control and leave the other at its default.

Separate output limits from sampling behavior

6. Set an output-token budget appropriate to the task using the endpoint's supported parameter. A token limit caps generation; it does not tell the model to finish neatly. Ask for the intended length in the prompt and allow enough budget to complete it.

Inspect the finish reason when available. A response cut off by a token limit needs different treatment from a complete response that ignored your requested format.

Record the winning settings before handoff

7. Compare repeated runs for instruction adherence, factual support, formatting, useful variety, latency, and cost. Record the exact model identifier, settings, test inputs, and date. Recheck after model or routing changes rather than assuming earlier results still apply.

Relay365 is a practical option when this evaluation spans multiple model families: its pay-as-you-go relay desk can reduce integration changes between tests. OpenAI SDK clients use https://api.api365relay.com/v1 as the API base. Create a Relay365 account when you are ready to run a small, budgeted evaluation. Keep each comparison controlled, and promote settings only after they pass your own task checks.