Shift Note: Model API Retries Need Breathing Room

The shift starts with a familiar report: a model request failed, so the application sent it again immediately. Then again. Soon, one delayed answer has become a queue of repeated requests. Retries can help recover from temporary trouble, but they need spacing, limits, and a clear reason to happen.

A retry means sending another attempt after a request fails. Backoff means waiting before that attempt. Together, they give a busy service time to recover while keeping your application from adding unnecessary traffic.

Check the failure before sending another request

Some failures are reasonable candidates for another attempt. A connection interruption, a temporary server error, or a rate-limit response may clear after a pause. A rate limit means the service is restricting how much work you can send within its limits.

Other failures need a correction. An invalid API key, an unsupported parameter, or a request that exceeds the model’s input limit usually will not improve through repetition. Read the error details before deciding. A rate-limit-looking response can also reflect an exhausted allowance that requires action.

The same distinction applies whether your request targets Gemini 2.5 Flash or Claude Haiku 4.5. The model name is less important than the failure you received.

Increase the pause when trouble continues

Exponential backoff increases the waiting time after each failed attempt. As an illustrative starting point, an application might wait one second, then two seconds, then four seconds. These are example settings, not timing requirements or a promise that the next attempt will succeed.

Set a maximum pause so the waiting time cannot grow indefinitely. Also limit the number of attempts. Make the wording precise in your configuration: three total attempts means the original request plus two retries.

If the response provides a Retry-After instruction, use it to guide the next attempt. Do not shorten a requested wait just to squeeze another attempt into your schedule. If that wait exceeds your application’s deadline, stop or defer the job.

Add randomness so clients do not retry together

If many clients fail at once and all wait exactly two seconds, they can return together and create another burst. Jitter adds randomness to the delay so those attempts spread out.

One approach is to calculate a growing delay ceiling, then choose a random wait between zero and that ceiling. For example, with a four-second ceiling, different requests could wait different amounts within that range. Keep the ceiling capped, and still honor any server-directed minimum wait.

Give each request a final stopping time

An attempt timeout limits one attempt. An overall deadline limits the complete operation, including retries and waiting. You need both, because several individually acceptable attempts can still leave a user waiting too long.

• Set an attempt timeout appropriate to the response length and workload.
• Stop retrying when the attempt budget or overall deadline is exhausted.
• Cancel pending retries when the user cancels the operation.
• Check whether your SDK already retries before adding another retry layer.

That last check matters because nested retry loops multiply requests. If an outer loop makes three attempts and an inner layer makes three attempts each time, one user action can trigger up to nine requests.

Treat uncertain completion as a duplicate-work risk

A timeout tells you that your application did not receive a timely result. It does not prove the server stopped processing. Sending the request again may create additional work and potentially additional billable usage.

Use an idempotency mechanism only where the endpoint explicitly supports it. This mechanism lets a service recognize repeated attempts as the same operation. For tool calls that send messages, place orders, or change records, protect those actions separately against duplication.

Streaming needs special care. If some text has already reached the user, a fresh attempt may produce a different answer. Do not silently append it as though it continues the interrupted response.

Keep the relay endpoint in the retry plan

Relay365 is a useful option to consider when you want pay-as-you-go access to multiple model families through a relay desk. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. Create a Relay365 account if that setup fits your workflow, then check its endpoint behavior and client retry settings.

For each failed operation, record the error, attempt count, waiting time, and final outcome without logging sensitive prompts or credentials. The next shift should be able to tell whether retries restored service or simply repeated the same failure.