Shift Note: SSE Beats Polling for Token Delivery

The request stayed open; the answer arrived in pieces

A streaming model response is not a succession of independent HTTP requests. The client opens a request, the server sends response headers, and the response body remains open while output arrives incrementally. With server-sent events, or SSE, that body uses a text format designed to separate events. The operational question is whether this delivery pattern fits the application better than a buffered response, polling, or a WebSocket.

For a chat interface, SSE is often the practical choice: the server has new output to deliver, and the client mostly needs to receive it. But an open connection alone does not guarantee that users see output promptly. Parsers, proxies, and rendering code all affect the result.

SSE framing wins over treating network chunks as messages

An SSE response normally uses the text/event-stream content type. Events contain fields such as data, event, and id, with a blank line terminating each event. Multiple data lines within one event are joined with newline characters. Comment lines can serve as heartbeats without becoming application messages.

Network reads do not align with these boundaries. One read may contain several events, half an event, or part of a UTF-8 character. A client must decode incrementally, preserve incomplete text, and parse complete events. Splitting each received chunk and immediately treating it as a finished message is fragile.

Verdict: Parse SSE framing first, then parse the application payload. A data field may contain JSON, but SSE itself does not require JSON. Likewise, a completion marker such as [DONE] belongs to a particular API convention, not to the SSE standard.

SSE beats buffered responses for visible progress

A buffered response waits until the server has assembled the result before the client receives the complete body. This simplifies processing when the application needs one finished document. Streaming instead lets the interface display partial text while generation continues. It improves perceived responsiveness without necessarily reducing total generation time.

The trade-off is state management. Streamed output can end halfway through a sentence, and structured output may not become valid JSON until completion. Tool-call arguments can also arrive as fragments. Applications should assemble and validate these before acting on them.

Verdict: Choose SSE when early output helps a person or downstream consumer. Choose buffering when only the completed result is useful and simpler failure handling matters more than incremental display.

SSE beats polling when updates follow one active request

Polling repeatedly asks whether work has progressed. It is straightforward for durable background jobs, especially when clients disconnect and return later. Its update cadence, however, trades responsiveness against request overhead.

SSE pushes events over one open response. That avoids repeatedly checking for new tokens, but introduces connection-lifetime concerns. Reverse proxies may buffer output, and intermediaries may close idle connections. Heartbeats can help with idle periods, but they cannot override every infrastructure limit.

Verdict: Use SSE for a live generation session. Prefer a durable job API with polling or resumable event delivery when work must survive browser closures and extended disconnections.

SSE beats WebSockets until two-way traffic becomes central

WebSockets provide bidirectional messaging after an HTTP handshake. They fit sessions with frequent client control messages, interactive audio, or other continuous two-way traffic. SSE is narrower: events travel from server to client, while commands use separate requests.

Native browser EventSource supports automatic reconnection, but it does not offer arbitrary request headers or a POST body. Model APIs commonly need both authentication and a JSON request body, so browser integrations often use fetch and parse its response stream instead. That path needs its own reconnection logic. Retrying a generation can create duplicate work or charges; it is not automatically a continuation.

Verdict: Keep SSE for predominantly server-to-client output. Choose WebSockets when bidirectional interaction justifies the additional connection protocol.

The relay belongs in the same streaming acceptance test

Relay365 is a pay-as-you-go relay desk for multiple model families and works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. It fits teams seeking one integration point across providers. For a Gemini 2.5 Flash integration, test actual event shapes, completion signals, cancellation, and disconnect behavior rather than assuming identical streaming semantics across routes.

Create a Relay365 account if that shared integration fits your workflow, then verify streaming through your production proxy path. The final acceptance test is practical: output appears incrementally, incomplete streams are reported honestly, and retries never silently masquerade as continuation.