What is reasoning?

Where the idea came from

The 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models showed that asking a model to work through intermediate steps improved its results on arithmetic and multi-step problems, without changing the weights. The technique was a prompt: think step by step, then answer.

What happened next was that the technique moved inside the model. Reasoning models are trained to produce that intermediate work themselves, at length, before committing to an answer, so you no longer have to ask. The behaviour became a product tier with its own pricing rather than a phrase in your prompt.

What it costs, precisely

OpenAI's reasoning guide is unusually direct about the accounting. Reasoning tokens are not visible via the API, they still occupy space in the model's context window, and they are billed as output. The volume is not marginal either: the same documentation notes a model may generate anywhere from a few hundred to tens of thousands of reasoning tokens for a single response, and the exact count for a request appears in the usage object under output token details.

Three consequences follow, and they surprise people in this order. Your bill can rise sharply with no change in the visible answer length. Your effective context window shrinks, because hidden thinking is competing for the same space as your prompt and history. And latency grows with the amount of thinking, which is why a reasoning model is a poor fit for anything interactive that could be done by a cheaper one.

Anthropic exposes the same behaviour as extended thinking, where Claude thinks against a budget before producing its final answer. Read the current documentation before wiring the parameters in: the surface for controlling thinking budgets has changed across model generations, so a snippet from an older post may name a parameter that is now deprecated.

When more thinking helps, and when it does not

It helps where the task genuinely has intermediate state: multi-step arithmetic, constraint satisfaction, debugging from a stack trace, planning a change across several files, anything where a wrong early commitment ruins the answer. On those, the extra tokens buy accuracy you cannot get by rewording a prompt.

It does nothing for extraction, classification, formatting, summarising a passage you supplied, or any task where the answer is already present in the input and only has to be located. Paying reasoning prices for those is the most common avoidable line on an AI invoice. More thinking is also not monotonically better: past a point the extra steps add cost and latency without moving quality, which is why budgets exist at all.

What a chain of thought does not give you

It is not an audit trail. The visible or summarised steps are generated text, produced by the same process as the answer, so they can be a plausible narrative rather than a faithful record of how the answer was reached. A correct answer can arrive with flawed stated steps, and a wrong answer can arrive with steps that read impeccably.

Which means reasoning output is a debugging aid, not evidence. If you need to know that a conclusion is right, check the conclusion against something outside the model: run the code, resolve the link, look up the number. Reading the reasoning tells you what story the model told, and that is a different question.

# a short answer that was not a cheap request
{
  "output_text": "Yes, the retry loop is the cause.",   # ~9 visible tokens
  "usage": {
    "input_tokens": 2140,
    "output_tokens": 4312,
    "output_tokens_details": {
      "reasoning_tokens": 4297      <- hidden, billed as output,
    }                                  and taking context window space
  }
}

what you read : 15 tokens
what you paid : 4312 output tokens

rule of thumb: never estimate reasoning cost from answer length;
read output_tokens_details on real traffic instead.