Shift note: 429s need queues, not faster retries

The API returns HTTP 429, and the application has a choice: send another request immediately, wait deliberately, or stop admitting work it cannot finish. These choices look similar in a retry helper. Under load, they produce very different incidents.

The useful head-to-head is not which approach eventually gets a successful response. It is which approach protects latency, preserves useful work, and avoids making an overloaded dependency busier.

Immediate retries lose to bounded backoff

An immediate retry is simple, but a rate limit is rarely fixed by asking again a few milliseconds later. When many workers receive 429 responses together, immediate retries turn one burst into several. The original requests compete with their own replacements.

Bounded exponential backoff is the stronger default. Increase the delay between attempts, add random jitter so workers do not wake together, and cap both attempts and total elapsed time. The request deadline should include waiting, not just network time.

If the response includes Retry-After, parse it as either a delay in seconds or an HTTP date. Do not retry before the indicated time. If that wait exceeds the caller's remaining deadline, stop rather than silently extending the operation. Without usable server guidance, apply your own bounded backoff policy.

Also inspect the error body. A short-window throttle and an exhausted spending allowance can both surface as 429, but only one is likely to improve after a brief pause. Provider-specific error codes matter.

Verdict: Immediate retries are the wrong default for 429 responses. Use bounded backoff with jitter for retryable throttling, and surface quota or billing problems instead of repeatedly sleeping on them.

Backoff alone loses to a bounded queue under sustained load

Backoff controls requests that have already failed. It does not control how many new requests enter the system. If arrivals keep exceeding capacity, sleeping workers become a hidden queue with poor visibility and unpredictable memory use.

An explicit, bounded queue makes that pressure visible. Pair it with concurrency limits and, where appropriate, request-rate and token-budget controls. Concurrency alone does not enforce a requests-per-minute or tokens-per-minute allowance.

• Set queue limits and expiration times so stale work does not consume capacity after its caller has left.
• Separate interactive requests from batch work so a large import does not block a user-facing action.
• Coordinate admission control across workers when they share the same upstream allowance.
• Reject excess work clearly when the queue is full instead of accepting a deadline you cannot meet.

Backoff still belongs here, but behind admission control. Watch queue age, 429 frequency, retry attempts, and completed logical operations. Counting every retry as fresh demand hides the amplification you need to diagnose.

Verdict: Backoff handles temporary rejection; a bounded queue handles sustained pressure. Production workloads usually need both, with explicit load shedding when capacity runs out.

Model fallback wins only when the task permits substitution

Fallback can avoid waiting when another approved model has available capacity. It also changes the execution environment. A request tested on Claude Haiku 4.5 should not automatically move to Gemini 2.5 Flash without checking tool behavior, output contracts, safety requirements, cost, and answer quality.

Do not assume different models have independent limits. They may share an account-level or gateway-level constraint. Likewise, changing routes does not establish whether an earlier operation had side effects. For tool-driven workflows, preserve application-level operation identifiers and deduplicate actions where necessary.

Relay365 fits teams that want pay-as-you-go access to multiple model families through a relay desk. Its API base is https://api.api365relay.com; OpenAI clients append /v1. That can simplify integrating approved alternatives, but it does not make rate limits disappear or guarantee spare capacity.

Verdict: Use fallback for prevalidated, substitutable tasks. Keep the original model queued, or return a clear failure, when correctness depends on its specific behavior.

Duty-desk decision for the next 429 burst

Choose bounded backoff over immediate retries. Choose admission control plus backoff over backoff alone. Choose fallback only after compatibility checks, not as an automatic escape hatch.

If a multi-model relay suits that design, create a Relay365 account and test your retry, queue, and fallback policies with a limited workload. The operational target is controlled degradation: less admitted work, bounded waiting, and honest failures rather than an expanding retry storm.