Fix: LLM streaming dies mid-response — ECONNRESET, incomplete SSE, truncated output

TL;DR — A stream that dies partway is almost never the model. It is an intermediary: an idle timeout counting silence between tokens, a proxy buffering a response that was designed not to be buffered, gzip applied to text/event-stream, or a load balancer capping response duration. Diagnose it by bypassing the proxy once — if the direct call completes, you have a middlebox problem, and the section below tells you which one.

The symptoms

They look different but share one cause family:

Error: aborted
Error: read ECONNRESET
# or: the stream simply stops, no error, response object incomplete
# or: nothing appears for 30s, then everything arrives at once

That last one is the tell for buffering: the data was never lost, it was held.

First: is it the intermediary?

One command settles it. Call the provider directly, bypassing your gateway/proxy:

curl -N https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","stream":true,"messages":[{"role":"user","content":"count slowly to 50"}]}'

• Completes cleanly → the provider is fine; it is your proxy/LB/runtime. Continue below.
• Also dies → network path or client-side timeout, not the middlebox.

-N matters: without it curl buffers too, and you will misdiagnose your own test.

The five causes

1. Idle timeout measured between bytes. The most common. Gateways count silence, and a model that is thinking — or a reasoning model in particular — can be silent longer than the default idle window before the first token. The request is not slow overall; it is quiet at the start. Raise the idle/read timeout, not just the total one.

2. Response buffering. Nginx buffers proxied responses by default, which collapses streaming into one delayed blob and can trip other timeouts. For SSE routes:

location /v1/ {
    proxy_pass              http://upstream;
    proxy_buffering         off;
    proxy_cache             off;
    proxy_read_timeout      600s;
    proxy_http_version      1.1;
    chunked_transfer_encoding on;
}

3. Compression applied to the event stream. gzip on text/event-stream makes chunks arrive in compression-sized blocks instead of token-sized ones, so the stream stutters or stalls. Exclude SSE from compression, and send X-Accel-Buffering: no from the origin so downstream nginx honours it too.

4. HTTP/1.0 or keep-alive mismatch. A proxy downgrading to HTTP/1.0 loses chunked transfer encoding, which SSE depends on. Force HTTP/1.1 upstream.

5. Load-balancer maximum response time. Some managed LBs and serverless platforms cap total response duration regardless of activity. A long generation hits the ceiling and the connection is cut with no error your code can catch. This is a platform limit, not a setting — the workaround is a shorter generation or a platform that supports long-lived responses.

Handling it in client code

Even with everything tuned, streams break. Treat a truncated stream as a normal event:

• Check finish_reason. If the stream ends without one, it was cut off — do not present the partial answer as complete.
• Do not blindly retry a long stream. You pay for the tokens already generated. For long outputs, checkpoint and continue rather than restarting.
• Set a client idle timeout shorter than the server's, so you detect a dead stream instead of hanging forever.

Prevention

• Keep streaming routes on their own proxy config; a single global proxy_buffering on is what silently breaks them.
• Alert on streams that end without a finish reason — that is the metric that catches this class of failure, and it is invisible in a plain error rate.
• If you are routing across providers, the timeout and buffering behaviour has to be right for the slowest model you route to, not the fastest — related: gateway timeout limits on LLM requests.