Duty Log: Model API Retries Ready in Ten Minutes

A model request times out, the worker immediately resends it, and several other workers do the same. A temporary slowdown becomes a second wave of traffic. The fix is not more persistence. It is a small, explicit retry policy that knows when to wait, when to stop, and when a second request might duplicate work.

This ten-minute quick start gives an interactive application a practical starting policy. Treat the suggested values as configuration choices, not provider guarantees.

Set one retry owner and one request deadline

Spend the first two minutes finding every layer that can retry: the SDK, your HTTP client, the application worker, and the job queue. Choose one owner for request-level retries. Otherwise, three application attempts can each trigger additional SDK attempts, multiplying traffic without making that multiplication obvious.

If your application owns the policy, configure the SDK to disable its automatic retries where supported. For a starting point, allow three attempts total, including the original request, within a 30-second overall deadline. Pick a different deadline if your workload needs it; long reasoning requests and background jobs may require substantially different budgets.

Use a monotonic clock to track elapsed time. Every sleep and network attempt must fit inside the remaining budget. A per-attempt timeout alone does not bound the entire operation.

Separate temporary failures from requests that need repair

Use the next two minutes to classify errors before adding any delay logic. Read the provider's error body as well as the HTTP status.

• Retry rate-limit responses when they indicate a temporary limit, but stop when the error indicates exhausted credit, a disabled account, or another condition that waiting will not resolve.
• Consider retrying transient server failures such as HTTP 502, 503, and 504, plus connection failures that occurred before the request was sent.
• Do not automatically retry authentication failures, permission failures, malformed requests, unsupported parameters, or context-length errors without changing the request or configuration.
• Treat read timeouts and broken connections after transmission as ambiguous, because the service may already have processed the request.

A retry of an ambiguous request can generate another answer and incur another charge. Do not assume an idempotency header makes generation requests safe unless the endpoint explicitly documents that behavior.

Add jitter without overruling server instructions

Spend three minutes implementing bounded exponential backoff with full jitter. For the first retry, choose a random delay between zero and one second. For the second, choose between zero and two seconds. For policies with more attempts, double the ceiling each time up to a configured maximum, such as eight seconds.

When a response supplies a valid Retry-After header, parse either its delay-in-seconds form or its HTTP-date form. Treat that value as a minimum wait; use the greater of that delay and your jittered backoff. You may add a small positive jitter to spread workers out further.

If the required wait leaves insufficient time for another useful attempt, stop rather than retry early. Never shorten the server's requested wait merely to fit a local backoff cap. Check cancellation before sleeping and again before sending.

Protect streamed output and watch the retry budget

Use the final three minutes for guardrails and a controlled test. Once streamed text has reached the user, do not silently restart generation and append the new response. It may repeat or contradict the partial answer. Show an interrupted state and offer an explicit retry instead.

For tool-calling workflows, deduplicate side effects separately. Retrying a model request must not automatically repeat a payment, message, or database write.

• Record the attempt count, error category, chosen delay, total elapsed time, and provider request identifier when available.
• Avoid logging API keys or full prompts by default, and apply your data-handling policy to diagnostic payloads.
• Test a temporary rate limit, a server error, an expired deadline, and an interrupted stream before enabling retries broadly.
• Limit concurrent requests and queued retries so an upstream slowdown cannot create an unbounded recovery surge.

Keep Relay365 traffic inside the same controls

Relay365 fits when you want pay-as-you-go access across model families through a relay desk. OpenAI SDK clients use https://api.api365relay.com/v1 as the API base. Keep the same bounded retry policy when testing a workload with Gemini 2.5 Flash, and verify the endpoint's actual error responses rather than assuming identical behavior across models.

Create a Relay365 account when you are ready to test this policy with your workload. The shift handoff is simple: one retry owner, a finite deadline, jittered waits, and no silent replay of user-visible work.