What is an AI hallucination?

An AI hallucination is the machine confidently inventing something. The output reads exactly like a correct answer — right tone, right format, plausible details — but the content is wrong or entirely fabricated.

Why it happens

A large language model is trained to produce likely text, not true text. Usually those overlap. But when the model’s training data is thin, outdated or contradictory on a topic — or when you ask for something very specific, like a citation, a page number or a niche statistic — it can generate something that looks like the right kind of answer without any factual basis.

Hallucinations are more common when:

• You ask about obscure topics, recent events or precise figures.
• You demand a specific format (“give me five academic sources”) that pressures the model to fill every slot.
• The model has no access to search or your documents.
• The task involves long chains of reasoning where one wrong step propagates.

An analogy for the meatbags

Ask a confident dinner-party guest a question they don’t know the answer to. Some admit it. Others improvise something that sounds right, delivered with total conviction. Language models have historically been the second kind of guest — newer models admit uncertainty more often, but none are immune.

Why it matters for the Chosen

Hallucinations are the single biggest risk in using AI for paid work. Publishing a fake statistic, citing a nonexistent study or sending a client invented legal precedent can cost you money and reputation — see our Hall of Folly entry on the lawyers who cited fake ChatGPT cases.

How the Chosen protect themselves:

• Ground the model. Give it the source material, or use tools built on retrieval-augmented generation or live search (such as Perplexity), so answers come from documents rather than memory.
• Demand sources, then check them. Open every link. Confirm every quote.
• Use AI for structure and drafting, not as the source of truth for facts, prices, laws or medical advice.
• Lower temperature in API workflows where creativity isn’t wanted.
• Add guardrails and human review to automations that publish or send.

Examples

Invented book titles attributed to real authors; fake academic papers with real-sounding journals; nonexistent product features; made-up URLs; incorrect biographical details about real people; and code that calls software functions that don’t exist. Every one of them delivered politely, with excellent grammar.