What is a rate limit?

What does a rate limit count?

Every major provider counts more than one thing at once, and you are limited by whichever counter runs out first. Anthropic's Messages API counts requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM). OpenAI counts RPM, requests per day, tokens per minute (TPM), tokens per day and, for image models, images per minute. Gemini counts RPM, input TPM and requests per day, and its daily quota resets at midnight Pacific time.

In OpenAI's example, a 20 RPM limit is full after 20 requests of 100 tokens each, while a 150k TPM limit is barely touched. Many small calls exhaust the request counter first; long prompts with big documents exhaust the token counter.

The scope is wider than one API key. Anthropic sets limits at the organization level and lets you set lower limits per workspace. OpenAI sets them per organization and per project. Gemini applies them per project, not per API key, so a second key on the same project gives you no extra headroom. Limits also differ per model, and some model families share one bucket.

People use "rate limit" loosely for any usage cap. The API docs keep two things apart: rate limits, which refill over minutes, and spend limits, which cap money per month. Your error handling needs to tell them apart.

Why do you get a 429 when you are under your limit?

Providers enforce the per-minute number over shorter intervals. Anthropic uses a token bucket algorithm, so capacity refills continuously instead of resetting on the minute, and its docs warn that 60 RPM might be enforced as 1 request per second. A burst of 30 calls in the first second can fail while your average for the minute looks fine.

Providers also limit how fast your traffic grows. Anthropic returns 429s from acceleration limits when an organization's usage rises sharply. OpenAI returns 429 with the code slow_down for the same reason, even when you are inside your RPM and TPM limits, and suggests that once you reach 1 million input TPM you grow by no more than 50% every 15 minutes. Gemini adds spend-based rate limits measured over a rolling 10-minute window, which return 429 RESOURCE_EXHAUSTED.

A 429 is about your quota. Anthropic's 529 overloaded_error and OpenAI's 503 server_is_overloaded mean the provider is short of capacity for everyone. Both are worth retrying, but raising your tier will not fix an overload.

What should your code do with a 429?

Read the headers before you read the body. Anthropic sends retry-after in seconds plus anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining and matching -limit and -reset headers on every response. OpenAI sends x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens and reset durations such as 6m0s. You can slow down before the 429 arrives by watching the remaining counts.

• Wait at least retry-after seconds. Anthropic says earlier retries will fail, and OpenAI says to treat the value as a minimum and add a small random delay so many clients do not retry at the same moment.
• When the header is missing, back off exponentially with jitter.
• Know what your SDK already does. Anthropic's SDKs retry rate limits and 5xx errors twice by default with exponential backoff and honor retry-after; set max_retries to change that.
• Do not retry a spend cap. Anthropic's tier spend-cap 429 has the same rate_limit_error type but no retry-after header, and carries error_code: enforced_spend_limit_reached. Retries keep failing until access resumes.

With streaming, the 429 arrives as an HTTP error before the stream starts. OpenAI warns that an error after output has begun can arrive as a stream event, and that you should not replay a request automatically once you have consumed output from it.

How do you get more throughput from the same limit?

Prompt caching raises effective throughput on Anthropic. For most Claude models, tokens read from the cache do not count toward ITPM; only uncached input and cache writes do. In Anthropic's worked example, a 2,000,000 ITPM limit with an 80% cache hit rate lets you process 10,000,000 input tokens a minute.

OTPM counts the output tokens actually generated, and Anthropic states that the max_tokens value you pass does not factor in, so a generous cap costs you no rate-limit headroom. Work that nobody is waiting on can move to batch inference: Anthropic's Message Batches API has its own limits, and OpenAI counts queued batch input tokens against a separate queue limit. After that, the remaining options are a higher usage tier, which all three providers raise automatically as your account builds usage or payment history, or asking the provider for more.

# 1. A rate limit: wait and retry
HTTP/1.1 429 Too Many Requests
retry-after: 12
anthropic-ratelimit-requests-limit: 50
anthropic-ratelimit-requests-remaining: 0
anthropic-ratelimit-requests-reset: 2026-10-09T14:02:12Z
anthropic-ratelimit-input-tokens-limit: 30000
anthropic-ratelimit-input-tokens-remaining: 4000

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "..."
  },
  "request_id": "req_011CSHoEeqs5C35K2UUqR7Fy"
}

# 2. The monthly spend cap: retrying will not help
HTTP/1.1 429 Too Many Requests

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "You have reached your API usage limits: your organization has crossed its monthly API usage threshold, set based on your organization's API tier. You will regain access on 2026-09-01 at 00:00 UTC.",
    "details": { "error_code": "enforced_spend_limit_reached" }
  },
  "request_id": "req_018EeWyXxfu5pfWkrYcMdjWG"
}