Timeouts for Long Streams: A Practical Walkthrough

A long model response needs a clear waiting policy, not simply an enormous timeout. Work through the steps below to identify which limit ends your request, configure the parts you control, and handle interruptions without losing useful output.

Separate Connection Waits From Streaming Pauses

1. Start by identifying your client's timeout types. In HTTPX, connection, read, write, and connection-pool waits have separate controls. Its read timeout limits how long the client waits for a chunk of data, rather than setting a deadline for the entire response. Check your own client's documentation before assuming its timeout behaves the same way. ([python-httpx.org](https://www.python-httpx.org/advanced/timeouts/?utm_source=openai))

Write down three application requirements: how long you will wait to connect, how much silence you will tolerate during generation, and how long the complete operation may run. Treat the last requirement as a separate overall deadline unless your client explicitly provides one.

Identify the Layer That Closes the Stream

2. Draw the request path, starting with your application and including any reverse proxy, hosting gateway, relay, and model endpoint. Record the settings you can inspect and mark unknown limits for investigation.

Do not assume every timeout measures total duration. For example, NGINX's proxy_read_timeout measures the interval between successive reads from the upstream server. A continuously active response and a silent response can therefore behave differently under the same setting. ([nginx.org](https://nginx.org/en/docs/http/ngx_http_proxy_module.html))

For one failing request, record the start time, first received data, last received data, exception, and any request identifier. Keep credentials and sensitive prompt content out of diagnostic logs.

Configure Explicit Client Limits for Your Test

3. Set timeout values deliberately rather than relying on unfamiliar defaults. If your application uses HTTPX, configure its connect and read limits separately so you can distinguish connection problems from gaps during streaming. HTTPX also exposes write and pool timeout settings. ([python-httpx.org](https://www.python-httpx.org/advanced/timeouts/?utm_source=openai))

For an initial experiment, you might choose a 10-second connection limit, a 120-second read limit, and a separate 10-minute application deadline. These are illustrative test settings, not provider guarantees or universal recommendations. Adjust them using observed behavior and your product's waiting budget.

Keep the overall deadline active while consuming the stream, not just while obtaining the initial response.

Check Whether Your Proxy Buffers Streaming Data

4. Inspect the proxy you control before increasing more timers. NGINX documents that enabled response buffering collects upstream data into buffers, while disabling it passes data onward as it arrives. Review proxy_buffering for the streaming route and test the effect rather than changing unrelated routes. ([nginx.org](https://nginx.org/en/docs/http/ngx_http_proxy_module.html))

Compare timestamps at your server and client. Use that comparison to investigate whether delays appear before your application receives data or while it forwards data downstream.

Make Interrupted Responses Safe to Handle

5. Preserve received text as partial output and show an explicit interrupted state when the request fails. Do not treat connection closure alone as proof of successful completion; check the completion signal defined by the endpoint you use.

Review automatic retries before adding your own retry loop. The official OpenAI Python SDK documents configurable retries and timeout exceptions, so inspect those settings when using that client. ([github.com](https://github.com/openai/openai-python?utm_source=openai))

Require an explicit recovery decision after partial output. Keep separate attempts distinguishable, and do not blindly repeat tool actions that change external state.

Test Your Relay Without Assuming Timeout Guarantees

6. If you need a pay-as-you-go relay desk across model families, Relay365 is worth evaluating. Use a supported model such as Gemini 2.5 Flash for a controlled streaming test, keeping the prompt and client settings consistent between runs.

Create a Relay365 account when you are ready to test that workflow. Confirm the applicable streaming behavior and limits before choosing production deadlines; a relay should not be treated as a substitute for application-level cancellation and recovery.

Verify Successful Completion and Controlled Failure

7. Finish with three tests: a short successful response, a longer successful stream, and a deliberately interrupted request. Verify that partial text remains visible, cancellation stops local processing, and retries do not duplicate output or actions. Keep the resulting settings and observations beside your integration configuration.