What is an AI hallucination? Why models make things up, and what actually helps
- glossary
Fourth glossary entry, and the one that explains most AI horror stories. A model answers your question smoothly, cites a source, names a number, and none of it is true. Not garbled, not error-messaged: fluent, specific, confident, and false. That is a hallucination, it is a direct consequence of how an LLM works, and the good news is that the defenses against it are boring, mechanical, and effective.
- The plain definition. A hallucination is model output that is presented as fact and is not: an invented citation, a wrong number, a function that does not exist, an event that never happened.
- Why it happens. The model produces the most plausible continuation. When the truth is missing from its parameters and its context, plausible fills the gap.
- The tell that is not there. No change of tone. Fabricated output reads exactly like correct output, which is what makes it dangerous.
- The wrong frame. It is not a bug to be patched once. It is a property of generative prediction: manageable, reducible, not deletable.
- The defenses. Ground the model in real sources, validate outputs before they act, verify claims at the origin, and design for "I do not know".
The plain definition
A hallucination is any output a model states as fact that does not survive checking: quotes nobody said, papers that were never written, API parameters that do not exist, confident summaries of documents that say something else. The term covers the whole range from a wrong year in passing to a fully invented legal precedent, and the mechanism is the same at both ends.
It is worth separating from two neighbors. Outdated knowledge is the model correctly reporting a world that has since changed, a training cutoff problem. Disagreement about interpretation is just that. A hallucination is narrower: the claim was never true, and the model produced it anyway, because the shape of a good answer demanded something in that slot.
What it means in practice
The prediction engine always answers with the most plausible continuation. When the true answer is well represented in its training or sitting in its context, plausible and true coincide. When it is not, the machinery does not stop, it fills the slot with what a true answer would look like: a title in title case, a plausible-sounding function name, a number in the right range. Fluency is constant; accuracy is not. That asymmetry is the entire practical problem, because every reader's instinct, judging credibility by confidence and polish, is calibrated for humans and misleads here.
Where it bites hardest, in my experience, is not essays but specifics: numbers, names, links, citations, API signatures. Agents raise the stakes further, because an invented tool parameter does not just read wrong, it acts wrong. And it compounds in pipelines: a fabricated detail produced in step two becomes trusted input for step five. This is why serious agent setups treat model output as a proposal until something mechanical has checked it, the logic behind the validation gates in the production readiness checklist. The flip side is worth knowing too: narrow, constrained systems, a classifier choosing from a fixed list, tool calls validated against a schema, can be engineered down to practically zero fabrication. The open-ended prose is where the risk concentrates.
What people get wrong
| People say | What is true |
|---|---|
| Hallucinations are a bug they will fix | Rates fall with better models and grounding, but inventing plausible continuations is inherent to the approach. Plan for it |
| The model knows when it is unsure | By default it does not signal uncertainty reliably; fluency stays high while accuracy drops. Uncertainty has to be engineered in |
| Giving it documents eliminates the problem | Grounding in retrieved sources cuts hallucination a lot and can still fail: models misread, over-summarize and blend sources |
| A confident, detailed answer is a good sign | Detail and confidence are style properties of the generator, not evidence. Check specifics regardless of tone |
| It lied to me | Lying needs intent. The model completed a pattern. The distinction matters because the fix is engineering, not trust |
One example from my own work
The one that sharpened my process: while researching another vendor's developer documentation, I used a tool that fetches a page and returns a summary instead of the raw text. The summary came back neatly titled as documentation for a completely different product, the tool I happen to use daily. Same architecture described, wrong name on top: the summarizer had blended what it was reading with what it expected. I caught it only because identifiers inside the content, package names, namespaces, disagreed with the headline. The write-up that was eventually published was built from the verbatim files instead, and the rule that survived is the one I now apply everywhere: trust identifiers in the content over labels on the summary, and for any claim that carries weight, ask for the exact quote, not the gist. My drafts also pass an adversarial reviewer whose job is to attack every number and name, the judging pattern described in the agent monitoring post, and it catches something real in most drafts, including this series.
Where to go deeper
This entry closes the first loop of the glossary: What is AI drew the map, What is an LLM explained the engine, What is a token covered the unit it runs on, and this one covered its signature failure. The operator's toolkit against that failure lives in two earlier posts: the production gates for stopping fabrications before they act, and the monitoring loop for catching the ones that got through.