What is prompt caching?

What actually gets cached?

The cache is not the model's reply. When a transformer processes a prompt it does a prefill pass that computes a key and value tensor at every layer for every token, and those tensors (the KV cache) are what the next-token step reads from. Prompt caching stores that prefill state for a stable token prefix on the provider's side, so a follow-up request that starts with the same tokens can skip the compute and jump straight to generating from the end of the prefix.

Because the match is on the exact token sequence, the cache is keyed on the prefix bytes plus the model version and a small set of request-shaping parameters. A different system prompt, a reordered tool schema, a whitespace change, a new model snapshot, or in some providers a different sampling setting, all produce a new key and a cold prefill. There is no fuzzy match: one token off the front and the hit rate goes to zero.

Where you mark the cache: Anthropic, OpenAI, Gemini

Anthropic is explicit. You set cache_control: {"type": "ephemeral"} on up to four content blocks in a Messages request (system, tools, or messages), and each marker defines a cache breakpoint that covers everything before it. The default time to live is five minutes; a one-hour cache is available as a longer-lived option with a higher write price. Minimum cacheable prefix is 1024 tokens on Sonnet and Opus and 2048 on Haiku, and the response's usage reports cache_creation_input_tokens and cache_read_input_tokens so you can see hits directly.

OpenAI is automatic. Any prompt of 1024 tokens or more is eligible; the router hashes the prefix in 128-token increments and reuses the longest match it can find. There is no API to enable or place a breakpoint, so the only lever is prompt structure: put the stable content (system prompt, tools, few-shot examples, documents) at the front, and the volatile content (the user's latest turn) at the end. Hits appear as usage.prompt_tokens_details.cached_tokens and are billed at a discount to fresh input tokens.

Google's Gemini splits the difference with an explicit resource. You POST to cachedContents with the tokens you want cached and a TTL, get back a cache name, and pass that name on later generateContent calls. The minimum token count is set per model (higher on Pro than on Flash), and you pay a storage rate per hour on top of the reduced input rate for cached tokens.

What breaks a cache hit?

The failure mode is almost always the same: something the writer thinks of as boilerplate has changed between calls. A dynamic date in the system prompt, a tool description that was regenerated in a different order, a user id spliced above the retrieved documents, or a model upgrade rolled out overnight will each invalidate the prefix. The fix is to lay the request out so everything volatile lives after everything cached, and to freeze the model to a dated snapshot rather than a floating alias while you are measuring hit rate.

TTL is the other reason hits disappear. Anthropic's default cache lasts five minutes from the last hit, so a low-traffic endpoint can warm the cache on one request and miss on the next. If you need the cache to survive gaps between users, either use the longer TTL option where the provider offers one, or run a small keep-warm request on a schedule that reads the same prefix.

What prompt caching costs, and what it does not do

Cache reads are cheaper than fresh input on every major provider, but writes are not free. Anthropic charges a premium over base input to create a cache entry, so a prefix that is written once and read many times pays off, and a prefix that is written once and read once loses money. The break-even is usually two or three hits per write, which is why caching helps chat sessions and evals more than one-shot completions.

It is also not a response cache. The model still runs generation on every call, so output token cost, latency for the generated portion, and non-determinism from sampling all behave exactly as they did without caching. If two requests share a prefix and the same seed and settings, the outputs can still differ. If you want identical replies for identical inputs, put a response cache in front of the API, keyed on the full request.

POST /v1/messages
{
  "model": "claude-sonnet-4-5",
  "system": [
    {
      "type": "text",
      "text": "You are a support assistant for Acme...\n<10k tokens of policy>",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "tools": [ ...tool schemas..., {"cache_control": {"type": "ephemeral"}} ],
  "messages": [
    {"role": "user", "content": "Why was my order refunded?"}
  ]
}

first call (cold):
  "usage": {
    "input_tokens": 42,
    "cache_creation_input_tokens": 10318,
    "cache_read_input_tokens": 0,
    "output_tokens": 87
  }

second call, same prefix, new user turn:
  "usage": {
    "input_tokens": 51,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 10318,
    "output_tokens": 64
  }