Hallucination

Definition

Hallucination in artificial intelligence refers to the generation of output that is factually incorrect, fabricated, or nonsensical, yet presented with the same confidence and linguistic fluency as accurate information. In large language models (LLMs), hallucinations manifest as invented citations, non-existent entities, false historical claims, or plausible-sounding but entirely fabricated technical details. Hallucination is a structural property of current generative models, arising from their statistical nature rather than from any retrieval or verification mechanism. It occurs across all major LLM architectures and persists despite improvements in model scale and training methodology. The phenomenon poses particular risks when AI-generated content is consumed without independent verification.

How It Relates to AI Threats

Hallucination intersects with threats in the Information Integrity and Human-AI Control domains. Within information integrity, hallucinated content directly contributes to misinformation when AI-generated text containing false claims is published, cited, or otherwise disseminated as fact. Within human-AI control, hallucination compounds the risk of overreliance and automation bias, where users place unwarranted trust in AI-generated output due to its confident presentation. The combination of plausible formatting, authoritative tone, and factual inaccuracy makes hallucinated content particularly difficult for non-expert users to identify without external verification.

Why It Occurs

• Large language models generate text based on statistical patterns rather than factual verification or retrieval
• Training data contains inconsistencies and errors that models cannot reliably distinguish from accurate information
• Models optimise for fluency and coherence, which can produce convincingly structured but false statements
• There is no internal mechanism in standard LLM architectures to flag uncertainty or distinguish known from unknown
• Reinforcement learning from human feedback (RLHF) may inadvertently reward confident-sounding responses over accurate ones

Real-World Context

The case of Mata v. Avianca (INC-23-0005) brought hallucination to public attention when a New York attorney submitted a legal brief containing fabricated case citations generated by ChatGPT. The invented cases, complete with plausible docket numbers and legal reasoning, were presented to the court as precedent. The incident resulted in sanctions against the attorney and highlighted the risks of incorporating AI-generated content into professional and legal contexts without verification. Research institutions and standards bodies have since emphasised the importance of human oversight when using LLM outputs in consequential decision-making.