# batch inference

Also called: batch API, Message Batches API, async inference, offline inference

Batch inference is a provider mode where you submit many model requests together as one asynchronous job instead of calling the API one request at a time. The provider queues the batch, runs the requests in parallel behind the scenes, writes the results to a file you download when the job finishes, and bills each request at half the synchronous rate. Anthropic, OpenAI and Gemini all cap the completion window at 24 hours.

## What batch inference actually is

A batch is a list of requests you would otherwise have sent one at a time to the same synchronous endpoint. You wrap them in one job, tag each entry with a custom_id so you can line up the answers with your inputs, and hand the whole thing to the provider. The provider queues the job, fans the requests out across its own capacity when there is room, and writes every reply to an output file. Your code polls the job until it reports done, then downloads the file and matches replies back to inputs by custom_id.

The trade is latency for price and headroom. Batch usage is billed at 50% of the synchronous per-token rate on Anthropic, OpenAI and Gemini, and batch jobs run under a separate rate-limit pool from your live traffic, so a large offline run does not starve a chat product sharing the same key. In return you give up any control over when a specific request runs inside the window, and you cannot stream tokens back as they are produced.

## How you submit a batch: Anthropic, OpenAI, Gemini

Anthropic takes the requests inline. You POST /v1/messages/batches with a requests array where each entry has a custom_id and a params object shaped like a normal Messages call. A batch is capped at 100,000 requests or 256 MB, whichever comes first. Most batches finish in under an hour; anything not done at 24 hours expires. Results stay downloadable for 29 days, and stream: true is rejected inside a batch because results come back as a single file.

OpenAI takes the requests as a file. You upload a .jsonl where each line is a request with a custom_id, a method, a url and a body, then create a batch that points at the file id and names a completion_window of 24h. One batch holds up to 50,000 requests and 200 MB of input, all against a single model, and the supported endpoints include /v1/chat/completions, /v1/responses, /v1/embeddings, /v1/moderations and the image endpoints. Output line order is not guaranteed to match input order, so join on custom_id.

Gemini accepts either shape. Inline batches carry the requests directly and are capped at 20 MB of payload; larger jobs upload a JSONL through the File API, up to 2 GB. You create the batch with client.batches.create() (or the REST equivalent), the target turnaround is 24 hours, and result files stay available for six weeks before deletion. Batch mode works with generateContent, embeddings and image generation, and it composes with context caching.

## When does batch inference pay off?

The shape that fits is work you can queue: backfilling embeddings for a corpus you just loaded, running an offline eval suite over thousands of prompts, tagging or summarising a document set overnight, generating synthetic training data, or re-scoring last week's chat logs after a prompt change. In all of these the wall-clock wait does not matter to a user, the request volume is high enough that the 50% price cut is real money, and pushing the work off the live rate limit protects a product that shares the same account.

The shape that does not fit is anything a person is waiting on, anything that depends on another request in the same batch, and anything that needs to stream. There is no ordering guarantee between requests in a single batch, no priority lane inside the window, and no way to cancel a specific request once it is running. If a job absolutely must finish by a deadline, size it well under the daily quota and start it early, because expiry after 24 hours is a real failure mode when demand is high.

## Batch inference and prompt caching

Batching and prompt caching are the two big cost levers on the same API and they compose, with one wrinkle. When many requests in the batch share a long prefix (a system prompt, a tool schema, retrieval context that is the same across queries), caching the prefix once and reading it back on each request saves input tokens on top of the batch discount. Because a batch can take longer than the default cache lifetime, Anthropic's guidance is to use the one-hour cache duration rather than the five-minute default when the shared prefix matters.

Pricing still applies per request. A cache read inside a batch is billed at the cache-hit rate multiplied by the batch discount, and a cache write is billed at the write rate multiplied by the batch discount. The write is worth it when the shared prefix is read many times across the batch, which is the common case for evals and bulk extraction.

## Creating an Anthropic Message Batch and reading the results

POST https://api.anthropic.com/v1/messages/batches
{
  "requests": [
    {
      "custom_id": "ticket-8842",
      "params": {
        "model": "claude-sonnet-4-5",
        "max_tokens": 512,
        "messages": [
          {"role": "user", "content": "Classify: 'my card was charged twice'"}
        ]
      }
    },
    {
      "custom_id": "ticket-8843",
      "params": {
        "model": "claude-sonnet-4-5",
        "max_tokens": 512,
        "messages": [
          {"role": "user", "content": "Classify: 'how do I reset my password'"}
        ]
      }
    }
  ]
}

201 Created:
{
  "id": "msgbatch_01abc",
  "processing_status": "in_progress",
  "request_counts": {"processing": 2, "succeeded": 0, "errored": 0, "canceled": 0, "expired": 0},
  "expires_at": "2026-09-13T14:22:00Z"
}

# poll: GET /v1/messages/batches/msgbatch_01abc
# when processing_status == "ended", fetch results_url and stream the JSONL:

{"custom_id":"ticket-8842","result":{"type":"succeeded","message":{"content":[{"type":"text","text":"billing/duplicate_charge"}],"usage":{"input_tokens":32,"output_tokens":6}}}}
{"custom_id":"ticket-8843","result":{"type":"succeeded","message":{"content":[{"type":"text","text":"account/password_reset"}],"usage":{"input_tokens":33,"output_tokens":5}}}}

The batch returns 201 immediately with a status handle, not the answers. Poll until processing_status is ended, then stream the results file and join to your inputs on custom_id. Both requests are billed at 50% of the synchronous per-token rate.

## Frequently asked

Q: Is batch inference just running lots of parallel requests?
A: No. Parallel synchronous calls are billed at the standard rate, count against your live rate limit, and put back-pressure on the same product your users are hitting. A batch is one asynchronous job that runs against a separate quota, is billed at 50% of the standard per-token rate, and hands you back a results file once the whole job is done. You give up ordering, streaming and interactive tool loops in return.

Q: Can I use tool calling and structured outputs inside a batch?
A: Yes, on Anthropic and OpenAI. Each request in the batch is shaped exactly like a synchronous call to the same endpoint, so tool schemas, strict mode and JSON Schema response formats all work the same way. What you cannot do is chain a tool result into a follow-up request within the same batch: each entry runs independently, so agent loops that depend on the model's previous reply still need synchronous calls.

Q: What happens if my batch does not finish in 24 hours?
A: Any request still in flight at the 24-hour mark is marked expired and returned in the results file with an error, not a completion, and you are not charged for expired requests. Anthropic notes that processing can slow down under high demand, so a batch that fits comfortably today can bump into the limit next week. If a deadline matters, size the batch well under your daily throughput and start it early.

Q: Does the batch discount stack with prompt caching?
A: Yes. A cache read inside a batched request is billed at the cache-hit input rate and then discounted for batch usage; a cache write is billed at the write rate and then discounted. Because a batch can run for longer than the default five-minute cache lifetime, Anthropic recommends the one-hour cache duration when many requests in the batch share the same prefix.

## Sources

- Anthropic docs: Message Batches API: https://docs.anthropic.com/en/docs/build-with-claude/batch-processing
  the /v1/messages/batches endpoint, the 100,000 request and 256 MB caps, the 24-hour expiry, the 29-day results window, the 50% batch price and the recommendation to use the 1-hour cache duration inside a batch
- OpenAI docs: Batch API: https://platform.openai.com/docs/guides/batch
  the JSONL input format with custom_id, method, url and body; the 50,000 request and 200 MB per-batch limits; the 24h completion window; the supported endpoints; and the note that output order is not guaranteed to match input order
- Google AI for Developers: Batch Mode: https://ai.google.dev/gemini-api/docs/batch-mode
  50% pricing versus interactive, 24-hour target turnaround, 20 MB inline and 2 GB JSONL file limits, six-week result retention and support for generateContent, embeddings and image generation with context caching

## Related terms

- model: https://ai.developertools.io/model/index.md
- prompt caching: https://ai.developertools.io/prompt-caching/index.md
- streaming: https://ai.developertools.io/streaming/index.md
- eval: https://ai.developertools.io/eval/index.md
- embedding: https://ai.developertools.io/embedding/index.md
- token: https://ai.developertools.io/token/index.md

Canonical page: https://ai.developertools.io/batch-inference/
Edited by Derek Wonk. Last updated 2026-09-12.