What is streaming?

What the wire actually carries

A streaming request opens a normal HTTPS connection with a header like Accept: text/event-stream and keeps it open. The server writes a series of Server-Sent Events, each a small text frame with an event: name and a data: JSON payload, separated by blank lines. The SSE format is defined in the HTML Living Standard and is a one-way channel from server to client on top of HTTP; there is no framing library required on your side, only a parser that splits on blank lines and decodes the JSON.

The event names are provider-specific. Anthropic's Messages API emits message_start, then repeated content_block_start, content_block_delta and content_block_stop for each block in the reply, then message_delta with the stop reason and usage, then message_stop. OpenAI's Responses API emits its own set, including response.output_text.delta for text chunks and terminal events that signal completion. The shape differs, the pattern is the same: partial state arrives incrementally and the client is responsible for reassembling it.

Why turn streaming on at all?

The headline reason is perceived latency. Time to first token is often a fraction of the total generation time, so a chat UI that renders the first delta feels responsive even when the full answer takes several seconds. Streaming does not make the model generate faster; it moves the wait from before the first character to after the last one.

The other reasons are operational. You can cancel a stream mid-generation when the user closes the tab or a downstream check fails, and stop paying for tokens you were going to throw away. You can pipe deltas into a moderation pass or a diff view as they arrive. And for long tool-using turns, you get to see which tool the model is about to call before the full arguments have been produced, which is useful for logging and for early aborts.

Streaming tool calls and structured output

Tool arguments and JSON outputs stream too, and that is where naive parsers fall over. Anthropic streams tool-use blocks as an input_json_delta series whose partial_json fragments must be concatenated across many events before the arguments object is valid JSON. OpenAI's function-call deltas behave the same way. If you try to json.loads a delta as it arrives you will get a syntax error every time; the contract is accumulate then parse, not parse then accumulate.

Structured output modes such as JSON schema responses use the same mechanism. Providers still guarantee the final assembled object matches the schema, but the intermediate frames are fragments of that object, not a stream of validated partials. A common bug is showing users the raw text stream when the model is emitting JSON, which produces a UI that flashes braces and quotes at them for a second before the real content appears.

What streaming does not change

Streaming is a transport choice, not a pricing or capability change. The billed input and output token counts are the same as a non-streaming call, the model does the same amount of work, and the usage figures still arrive, typically on the terminal event. Rate limits apply to the request, not to each SSE frame.

It also does not remove the need for retries. A streamed connection can drop halfway through, and there is no standard way for a client to resume from the middle of a generation: you have to reissue the request. For agent loops this matters because a half-received tool call is worse than none, so treat any stream that did not reach its terminal event as a failed turn and restart it.

POST /v1/messages   { "stream": true, ... }

event: message_start
data: {"type":"message_start","message":{"id":"msg_01","role":"assistant","model":"claude-sonnet-4-5","usage":{"input_tokens":42,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" server returned a 500 because"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" the connection pool was exhausted."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":12}}

event: message_stop
data: {"type":"message_stop"}