What is latency?

What LLM latency actually measures

Every hosted model call has two phases and each has its own clock. In prefill, the server runs one forward pass over your entire prompt to build the key value cache and produce the first output token. In decode, it generates every subsequent token one at a time, each depending on the last. The user sees prefill as the pause before anything appears and decode as the speed at which words then arrive.

Providers and inference stacks name these with the same two acronyms. Time to first token (TTFT) is the elapsed time from the request hitting the server to the first token going out; NVIDIA describes it as the metric that reflects the prefill phase and recommends keeping it at or below 250 ms for interactive chat. Inter-token latency (ITL), sometimes called time per output token, is the average gap between successive output tokens during decode. End-to-end latency for a reply of n tokens is roughly TTFT plus ITL times n, which is why a long answer feels slow even when the model started replying fast.

How is TTFT different from inter-token latency?

They are limited by different hardware and they respond to different tuning. TTFT is compute-bound: prefill processes all your input tokens in parallel, so it scales with prompt length and with how many FLOPs the accelerator has spare. Cutting the input by half tends to cut TTFT by roughly the same amount, and a shared prefix that has already been cached skips the prefill work entirely.

ITL is memory-bound: decoding one token requires reading the whole model weights and the growing key value cache from HBM, then doing a small amount of math. That is why input length barely affects ITL once the cache is warm, why cutting the output length is the biggest lever on end-to-end time, and why speculative decoding and smaller models help ITL more than they help TTFT. OpenAI's own guidance puts it plainly: cutting 50% of your output tokens can cut about 50% of the latency, while cutting 50% of the prompt often buys 1 to 5%.

What moves each number in production

Different levers hit different phases, so pick the one that hits your actual bottleneck. A rough map:

• Streaming: does not change TTFT or ITL at all, but hides ITL behind the reader's own reading speed, so perceived latency drops sharply for anything a user is watching.
• Prompt caching: skips prefill for the cached prefix, so TTFT falls in proportion to how much of the prompt was already seen; ITL is unchanged.
• Shorter outputs: the largest lever on end-to-end time, because ITL is paid per generated token. Structured outputs and stop sequences that end the reply sooner both count.
• Smaller or distilled models: lower ITL because there is less memory to move per token, at some quality cost.
• Reasoning or extended thinking modes: raise end-to-end latency sharply because the model spends output tokens on hidden reasoning before it answers.
• Region and network: adds a fixed floor to TTFT that no server-side tuning can remove.

Measure both numbers separately in production, not just wall-clock time. Tail behaviour is where budgets get blown, so track p50 and p95 for each, and alert on the tail rather than the median.

Latency, batching and streaming

Server-side batching improves throughput at the cost of TTFT. Continuous batching, the technique modern serving stacks use, packs many users' decode steps into one GPU pass, which raises tokens per second for the fleet but adds a queuing wait to each request's TTFT. This is the direct opposite of the batch inference product mode, where you accept hours of latency in exchange for a 50% price cut. The two share a name and share nothing else.

Streaming is the other lever that changes what latency feels like without changing what it is. Server-sent events start delivering tokens the moment decode produces them, so a 6-second reply with a 400 ms TTFT feels responsive, while the same reply buffered to the end feels like a 6-second wait. Streaming does nothing for a caller that has to see the whole answer before acting, such as a function-calling loop that needs the full JSON.

$ time curl -N https://api.anthropic.com/v1/messages \
    -H "x-api-key: $ANTHROPIC_API_KEY" \
    -H "anthropic-version: 2023-06-01" \
    -H "content-type: application/json" \
    -d '{
      "model": "claude-sonnet-4-5",
      "max_tokens": 200,
      "stream": true,
      "messages": [{"role": "user", "content": "Give me a two sentence summary of TCP."}]
    }'

event: message_start                        # t = 0.412 s   (TTFT)
data: {"type":"message_start", ...}

event: content_block_delta                  # t = 0.438 s
data: {"delta":{"type":"text_delta","text":"TCP"}}
event: content_block_delta                  # t = 0.462 s
data: {"delta":{"type":"text_delta","text":" is"}}
event: content_block_delta                  # t = 0.487 s
data: {"delta":{"type":"text_delta","text":" a"}}
...
event: message_stop                         # t = 2.914 s

real    0m2.914s
# TTFT       = 0.412 s
# tokens     = 98
# ITL        = (2.914 - 0.412) / 97 = 25.8 ms per token
# throughput = 98 / 2.914 = 33.6 tokens/s