Confabulation

Definition

Confabulation describes the tendency of generative AI systems, particularly large language models, to produce outputs that are coherent and linguistically fluent yet factually incorrect or entirely fabricated. Unlike deliberate deception, confabulation arises from the statistical nature of language generation, where models predict plausible token sequences without verifying factual accuracy. The term is borrowed from neuropsychology, where it describes patients filling memory gaps with fabricated narratives they believe to be true. In AI contexts, confabulation is especially problematic because outputs often carry a tone of authority that makes false claims difficult for users to detect.

How It Relates to AI Threats

Confabulation is a central failure mode within Information Integrity threats, directly contributing to the spread of hallucinated content. When AI systems generate false citations, fabricated statistics, or invented historical events, they undermine the reliability of information ecosystems. This is particularly dangerous when AI-generated text is used in educational, legal, medical, or journalistic contexts where accuracy is essential. Confabulation also erodes trust in AI systems more broadly, as users who encounter confident but incorrect outputs may lose confidence in legitimate AI applications.

Why It Occurs

• Language models optimise for plausibility rather than factual accuracy
• Training data contains contradictions that models cannot reliably resolve
• Models lack grounded knowledge verification mechanisms
• Autoregressive generation commits early to false premises then elaborates
• Users often prompt models beyond the boundaries of their training data

Real-World Context

Confabulation has manifested in legal proceedings where AI-generated briefs cited entirely fabricated case law, leading to judicial sanctions against attorneys who failed to verify outputs. In academic contexts, researchers have documented AI systems generating plausible-sounding references to non-existent journal articles. These incidents illustrate how confabulation can propagate misinformation through trusted institutional channels when human oversight is insufficient.