Shift Note: Requests Waiting for a Free Slot

The duty desk sees a familiar pattern: one API request finishes normally, but a batch of twenty takes longer than expected. Some requests start immediately. Others wait, fail, or get retried. The model may be working normally while the application is sending more work than the available capacity can handle at once.

Two ideas explain much of this behavior: concurrency limits control how many requests can run together, and request queues hold work until there is room to start it.

Each active request occupies a slot

Think of concurrency as a desk with a fixed number of service positions. If your application allows four requests to run at once, those four occupy its available slots. A fifth request needs to wait or be rejected, depending on how the application handles overflow.

That number is just an example. Actual limits can exist in your application, a relay service, or an upstream provider. They may also differ by account or model. Raising your application's limit does not raise a provider's limit.

Request duration matters too. A short response releases its slot quickly. A long generation keeps that slot occupied longer. For a streaming response, your application will usually count the request as active until the stream finishes or closes, although provider accounting can differ.

A queue gives waiting work a place to stay

A queue stores requests that are ready to run but cannot start yet. When an active request finishes, the application can take another request from the queue and send it.

For example, suppose your application has four active slots and receives ten requests together. It could start four and queue six. As each active request finishes, another begins. This smooths a short burst without sending every request upstream immediately.

A queue needs boundaries. Without a maximum size or waiting deadline, it can accumulate work faster than the system can finish it. Users may give up while the application continues processing requests they no longer need. A full queue should produce a clear response, and expired or canceled work should be removed when possible.

Requests per minute measure a different constraint

A rate limit controls how much traffic is allowed over time. A concurrency limit controls how much work is active together. Both can apply to the same application.

You could stay below a requests-per-minute allowance while still exceeding a concurrency limit by sending everything in one burst. You could also keep concurrency low and exceed a rate limit by completing many short requests rapidly. Some services additionally limit token usage over time.

This is why reducing simultaneous requests can help with one bottleneck while leaving another unresolved. Check the actual error and available limit information before changing settings.

Queue time belongs in the response-time budget

From the user's perspective, waiting starts when they submit their request. Their total wait includes time in your queue, network travel, upstream processing, and response delivery.

Measure queue waiting time separately from the time spent on an active API call. If waiting grows while active calls remain steady, demand may be outrunning your chosen concurrency. Increasing concurrency might help if capacity is available; otherwise, it can produce more rejections.

• Set a maximum number of active requests that fits your application's workload and the documented service limits.
• Give queued requests a maximum waiting time that fits the user's overall deadline.
• Track active requests, queue length, waiting time, and failures so you can identify where delays begin.
• Retry eligible transient failures with bounded backoff and random jitter, respecting retry guidance when the service supplies it.

A retry creates more work. Immediate retries during congestion can make the queue longer, so use a retry budget and stop when the request's deadline has passed.

Keep the same controls when using a relay

Relay365 is a useful option when you want pay-as-you-go access to multiple model families through one relay desk. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. Your application still needs concurrency controls, queue deadlines, and error handling; a shared entry point does not imply identical capacity across models.

If that setup fits your workflow, create a Relay365 account and start with a small, measured workload. Tune concurrency using observed waiting times and documented limits, then expand as the results support it.