Shift log: cap in-flight calls, bound the waiting line

The shift starts with rising latency and a growing pile of model requests. Two controls are available: limit how many calls execute at once, or let excess work wait in a queue. They address different failure modes, and neither creates upstream capacity. The operational question is which control should take the first hit when demand exceeds what the system can finish.

Concurrency limits protect the active workload

A concurrency limit caps requests currently in flight. When all permits are occupied, the next request must wait somewhere or be rejected. That last detail matters: a semaphore with unlimited waiters is already an unbounded queue, even if nobody named it one.

The strongest argument for a limit is containment. It can protect application connections, memory, and upstream services from excessive simultaneous work. Its weakness is bluntness. A short generation and a long streaming response can each occupy one permit while consuming very different amounts of time and resources.

Head-to-head verdict: Concurrency limits win for immediate protection. They do not, by themselves, provide fair scheduling, durable job storage, or a useful waiting-time promise.

Request queues absorb bursts, not sustained overload

A queue separates arrival from execution. Workers pull jobs when capacity becomes available, allowing a brief spike to drain without launching everything simultaneously. Queues can also support priorities, durable storage, and controlled retries, depending on their implementation.

The trade-off is delayed failure. If arrivals keep exceeding completions, the backlog grows until storage, deadlines, or customer patience runs out. A durable queue preserves work; it does not make that work timely. For an interactive assistant, a response delivered after the user has abandoned the page may have little value.

Head-to-head verdict: Queues win for temporary bursts and deferrable work. They lose when used to conceal a persistent capacity shortage.

Interactive calls need a short waiting budget

For chat, editor assistance, and CLI interactions, start with an explicit concurrency cap and a small, bounded waiting area. Set an end-to-end deadline that includes queue time, connection setup, generation, and any retries. Do not reset that budget whenever a request changes stages.

• Reject work when the waiting budget or queue capacity is exhausted, and return a clear overload signal.
• Remove cancelled requests from the queue and propagate cancellation upstream where supported.
• For streaming calls, hold the permit until the upstream response finishes or is actually closed, rather than releasing it when headers arrive.

Interactive verdict: Favor admission control over a deep queue. A prompt rejection is often more actionable than an accepted request that silently waits beyond its deadline.

Batch jobs benefit from durable, bounded queues

Offline extraction, evaluation runs, and document processing can usually tolerate waiting. Here, a durable queue paired with a worker concurrency limit is the stronger design. Bound backlog by job count, estimated work, or both, and expire jobs that are no longer useful.

Use bounded retries with backoff and jitter for transient failures. Respect retry guidance when provided. Retrying an ambiguous timeout can duplicate upstream work or charges, so use idempotency facilities where supported and distinguish retryable transport failures from invalid requests.

Batch verdict: Let the queue absorb bursts, but let worker limits control execution. Queue depth without oldest-job age is an incomplete health signal.

Shared relay traffic still needs local admission control

Relay365 is worth considering when one application needs pay-as-you-go access across multiple model families. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. That can simplify integration, but it does not replace your scheduler.

For example, keep separate admission budgets for workloads using Claude Haiku 4.5 and Gemini 2.5 Flash until measurements justify sharing capacity. Verify applicable provider and account limits rather than assuming a common endpoint means a common capacity pool. Concurrency limits also differ from request-per-minute and token-rate limits; enforce the relevant budgets together.

If that integration model fits your workload, create a Relay365 account and test representative traffic before moving production demand.

The handoff verdict is a cap plus an explicit queue

Choose concurrency limits to protect execution capacity. Choose queues to preserve useful work through short bursts. In most production systems, combine them: bounded admission, bounded waiting, finite deadlines, and observable rejection. Track active calls, queue age, completion latency, cancellations, and retry volume. When the oldest queued request is already too late to help, stop accepting more waiting as a substitute for capacity.