What is a hallucination?

Why do models hallucinate?

Because guessing scores better than admitting ignorance. The 2025 paper Why Language Models Hallucinate puts it directly: models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty, and the systems are optimised to be good test-takers. On a benchmark scored for correct answers, a confident guess has a positive expected value and I do not know scores zero. Anything trained against that scoreboard learns to answer.

That reframing matters, because it moves hallucination out of the category of bugs. There is no line of code producing it and no patch that removes it. It is an incentive built into how the field measures progress, which means the fix is partly evaluation design and partly, on your side, deciding what your system does when the model does not know.

Underneath the incentive sits the mechanism. The model predicts likely continuations of text. A plausible-looking citation, a plausible-looking function name and a real one are all high-probability strings, and nothing in the process distinguishes them. This is also why hallucinations cluster at the specific: names, numbers, dates, URLs and API signatures.

Two kinds worth telling apart

The 2023 survey A Survey on Hallucination in Large Language Models splits the phenomenon into factuality and faithfulness. A factuality hallucination contradicts the world: an invented statistic, a person who never held the role. A faithfulness hallucination contradicts what you supplied: a summary asserting something the source document does not say.

The distinction is operational, not academic. Faithfulness you can check yourself, because you have the source, so every claim can be traced back to a passage and flagged when it cannot. Factuality you cannot check without an external source of truth, which is why grounded systems attach citations and unground systems can only sound convincing.

What actually reduces hallucination, and what does not

Four things move the number. Ground the answer, by retrieving the relevant text and instructing the model to answer only from it and to cite. Give it an exit, by making the passages do not say an explicitly acceptable answer, since a model with no permitted way out will invent one. Verify outside the model, by checking that the file path exists, the link resolves and the number appears in the source. Narrow the surface, because a tool that returns structured data cannot be misquoted the way prose can.

What does not work is instructing the model not to hallucinate. It has no separate mode it is withholding from you and no reliable signal for its own uncertainty, so the instruction mostly buys hedged phrasing over the same wrong content. Nor does raising the model tier fix it, since a stronger model is wrong less often but no more visibly wrong when it is, which can make the failures harder to catch, not easier.

In agent runs, hallucination compounds

One wrong sentence in a chat is a nuisance a reader corrects. One wrong sentence inside an agent loop becomes an input to the next decision. The model invents a file path, the tool call fails, the model concludes the project is structured differently, and three steps later it is confidently working on the wrong thing, with a coherent chain of reasoning explaining why.

This is the argument for hard step budgets and for surfacing tool errors verbatim rather than summarising them. An agent that is told no such file can recover. An agent handed a tidy summary of a failure often builds on top of it instead.

# ungrounded: nothing constrains the answer, so a plausible one is produced
user: "what is the retention limit on the Team plan?"
model: "The Team plan retains logs for 30 days."      # sounds right, unverifiable

# grounded: passages supplied, and an exit is explicitly allowed
system: "Answer ONLY from the passages below. Quote the sentence you used.
         If they do not contain the answer, reply exactly: NOT IN SOURCES."
passages:
  [1] pricing.md: "Team includes 14 days of log retention."
  [2] pricing.md: "Enterprise retention is configurable."

model: "14 days. Quote: 'Team includes 14 days of log retention.' [1]"

# and when the passages are silent
model: "NOT IN SOURCES"     <- only reachable because you allowed it