What is a token?

What a tokenizer does

A model does not see characters or words. Before anything else happens, a tokenizer cuts the text into pieces drawn from a fixed vocabulary, learned during training by repeatedly merging the most frequent adjacent pairs. That is byte pair encoding, and it is why common words survive as single tokens while rarer ones are split into fragments, and why the pieces often start with a space.

The tokenizer belongs to the model. OpenAI's embedding models, for example, are documented as using the cl100k_base encoding, and a different model family has a different vocabulary and different splits. So identical text is a different number of tokens on different models, which quietly undermines every price comparison stated per million tokens without a matching count.

Why the count decides everything

Three of your hard constraints are denominated in tokens. The context window is the ceiling on prompt plus generated output for one request. Prices are quoted per million input and per million output tokens, with output normally the more expensive side. And a maximum output length caps a single reply, which is the usual reason a long answer stops mid sentence.

In agent runs the same unit becomes the budget you enforce, because a step cap alone does not stop a run whose steps grow. Counting tokens is therefore not an optimisation detail; it is how you know whether a design is affordable before you build it.

Why word-based estimates are wrong

The familiar guidance that a word is roughly one and a bit tokens holds only for ordinary English prose, which is the least representative content in most systems. Code fragments heavily, because identifiers, punctuation and indentation each cost. JSON is worse, since every brace, quote and key is billed on every request. Languages that do not use Latin script inflate substantially, so a translated interface can cost several times its English original for the same meaning.

Two consequences worth acting on. Estimate nothing that a call can measure, and measure on your real traffic rather than a sample paragraph. Then look at the shape of what you send: verbose JSON schemas, repeated tool definitions and pasted logs are usually a larger share of the bill than anything a user typed.

Counting them properly

Two routes. Locally, run the model family's own tokenizer: OpenAI publishes tiktoken for exactly this, and it is fast enough to sit in a build step or a test. Remotely, use the provider's token counting endpoint, which prices a full request including system prompt and tool definitions, and is the only way to be exact about the parts you did not write yourself.

Whichever you use, read the counts back from real responses too. The usage object on every reply reports what was actually consumed, and on reasoning models it is the only place the hidden thinking tokens show up, so a cost model built without it will understate the bill.

# local, OpenAI family: exact counts without a network call
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
len(enc.encode(open("payload.json").read()))

# remote: counts a FULL request, including the parts you forget
POST /v1/messages/count_tokens
{ "model": "...", "system": "...", "tools": [...], "messages": [...] }

# after the fact: what was really consumed
"usage": { "input_tokens": 2140, "output_tokens": 4312,
           "output_tokens_details": { "reasoning_tokens": 4297 } }

where the tokens usually are, in order of surprise:
  1. tool definitions, resent on every single request
  2. pasted logs and raw JSON tool results
  3. accumulated conversation history
  4. hidden reasoning tokens
  5. what the user typed