Duty Log: Retry Storm Contained, Backoff Kept

The request timed out, the queue grew, and every worker tried again together. That is the failure pattern this shift note is designed to prevent. For model APIs, retry policy is not just error handling: it controls queue pressure, user latency, and the risk of paying for work whose response never arrived.

The contenders are immediate retries, fixed delays, exponential backoff with jitter, and model fallback. They solve different problems. The winning production setup combines bounded backoff with explicit deadlines, then treats fallback as a separate product decision.

Immediate retries recover quickly and amplify congestion

An immediate retry has one advantage: it adds almost no deliberate waiting. It can recover from an isolated connection failure before a request reaches the service. Against fixed delays, it wins on recovery speed when the failure is genuinely brief and local.

It loses badly during throttling or overload. Hundreds of callers retrying immediately create additional traffic precisely when capacity is constrained. A timeout is also ambiguous: the upstream may have accepted the request and continued generating after the client stopped waiting.

Allow an immediate retry only for narrowly classified failures, within a small attempt budget. Do not assume a timeout means no work occurred, and do not assume a request identifier provides deduplication unless the service documents that behavior.

Fixed delays are predictable but synchronize workers

A fixed pause is easier to reason about than an expanding schedule. It can be adequate for a low-volume background task with generous latency tolerance. Compared with immediate retries, it gives the upstream breathing room.

The weakness appears when many callers receive the same error together. A shared delay makes them return together, producing another traffic spike. Adding randomness helps, but a fixed range still does not adapt well to persistent overload. Its operational simplicity is real; its congestion control is limited.

Jittered exponential backoff wins under shared pressure

Exponential backoff increases the waiting window after each failed attempt. Full jitter chooses a random delay between zero and the current capped window. This spreads retry traffic rather than scheduling another synchronized wave.

When the server supplies a valid Retry-After value, honor that guidance instead of choosing an earlier random retry. Any added jitter should preserve the indicated minimum wait. If that wait cannot fit within the remaining request deadline, stop rather than retry prematurely.

Use both an attempt limit and an elapsed-time limit. Before another attempt, reserve enough time for a useful response. A retry that starts just before the caller's deadline wastes capacity and may leave expensive work running after the user has gone.

Error classification matters more than the delay formula

• Retry rate-limit responses only when the relevant limit can plausibly recover; exhausted credit or a hard quota usually needs intervention.
• Consider retries for transient server errors and selected connection failures, while recognizing that the original request may already have executed.
• Do not automatically retry authentication failures, invalid inputs, or unsupported parameters without changing the underlying condition.
• After streaming output reaches the user, do not silently replay the request and splice a second generation into the first.

Inspect SDK retry defaults before adding application retries. Nested retry loops multiply attempts. Choose one retry owner, record actual upstream attempts, and enforce a shared retry budget alongside concurrency limits.

Model fallback changes the answer, not just availability

Fallback can help when one model remains unavailable and another is acceptable. It is not equivalent to retrying: output style, tool behavior, context limits, and price may change. A workflow using Claude Haiku 4.5 should validate its alternate before an incident, not discover compatibility during one.

Relay365 fits teams that want pay-as-you-go access to multiple model families through a relay. OpenAI clients use https://api.api365relay.com/v1. That common entry point can simplify integration, but it does not make retries safe or establish that fallback routes have independent capacity. Create a Relay365 account if that integration model suits your workload, then test failure handling before production.

Shift verdicts for the production retry policy

• Reject immediate retries as the default because they amplify shared failures.
• Keep fixed delays for simple, low-volume background work.
• Choose capped exponential backoff with jitter for transient failures, bounded by server guidance, attempts, and deadlines.
• Enable model fallback only after compatibility checks, with explicit cost limits and visibility into which model answered.