Information Integrity Threats

Information Integrity Threats represent the most broadly impactful AI risk category, affecting individuals, organizations, democratic institutions, and the epistemic foundations of shared reality. The domain is distinguished by the exploitation of human trust — synthetic content succeeds because it mimics sources that people already believe. The incident record reveals a clear trajectory: from crude text-based fabrications to real-time multi-modal deepfakes, from individual targeting to institutional manipulation.

Definition

Information Integrity Threats encompass AI-enabled harms that undermine the reliability, authenticity, or shared understanding of information. These threats exploit AI’s capacity to generate, modify, or amplify content at scale, making it increasingly difficult to distinguish authentic information from synthetic or manipulated content.

Why This Domain Is Distinct

Information Integrity Threats differ from traditional misinformation because:

1. Production cost approaches zero — generating a convincing deepfake voice clone requires seconds and no technical expertise
2. Verification asymmetry — creating synthetic content is orders of magnitude cheaper than detecting or debunking it
3. Trust exploitation — AI-generated content inherits the provenance of whatever medium it mimics (a voice call, a video conference, a news article)
4. Victim scope spans individuals to institutions — the same techniques that defraud a single family member can destabilize an election

This domain contains the highest incident count in the registry, reflecting both the accessibility of generative AI tools and the breadth of contexts in which synthetic content causes harm.

Threat Patterns in This Domain

This domain contains five classified threat patterns, each representing a distinct mechanism of information corruption.

The five patterns form a severity gradient:

1. Deepfake Identity Hijacking accounts for the majority of documented incidents. AI-generated synthetic voices and video are used to impersonate specific individuals — typically for financial fraud or political manipulation. The Hong Kong deepfake CFO fraud and voice clone elder scams demonstrate the range from corporate to individual targeting.

2. Disinformation Campaigns involve coordinated use of AI to spread false information deliberately. The Slovak election deepfake and Biden robocall show this pattern applied to democratic processes.

3. Misinformation & Hallucinated Content covers false information produced without intent to deceive — typically AI systems that fabricate facts. The ChatGPT fake legal citations demonstrated real-world consequences when hallucinated content enters professional workflows.

4. Synthetic Media Manipulation addresses AI-altered images, audio, or video used to misrepresent reality. The Taylor Swift non-consensual imagery and Westfield High School student deepfakes demonstrate how this pattern targets individuals, often with gendered harm.

5. Consensus Reality Erosion captures the cumulative effect — as synthetic content proliferates, the shared ability to agree on what is real degrades. No single incident causes this; it emerges from the aggregate.

How These Threats Operate

Information Integrity incidents cluster around three primary mechanisms, each exploiting a different property of generative AI.

1. Synthetic Identity Generation

Attackers create AI-generated voices, faces, or full video likenesses of real individuals to impersonate them in communications. This mechanism powers the most financially damaging incidents in the domain:

• Voice cloning — the UK energy company CEO fraud used a cloned voice to authorize a $243,000 wire transfer. The Arup $25M fraud escalated this to video conference deepfakes of multiple executives simultaneously.
• Video deepfakes — real-time face-swapping during video calls eliminates the traditional safeguard of “seeing is believing”
• Elder targeting — the grandparent scam network used voice clones of family members to deceive seniors, a pattern the FBI elder fraud report confirmed is accelerating

The defining characteristic of this mechanism is that synthetic identity exploits existing trust relationships — a victim recognizes their CEO’s voice, their grandchild’s voice, a known official.

2. Content Fabrication

AI systems generate false content without human intent to deceive. Unlike disinformation, these are system failures rather than attacks:

• Hallucinated citations — the ChatGPT fake legal case produced nonexistent case law that a lawyer submitted to federal court, resulting in sanctions
• Mistranslation with consequences — the Facebook Arabic mistranslation led to a wrongful arrest when AI misinterpreted a social media post

Content fabrication is structurally different from deepfakes: it originates from AI system limitations rather than adversarial intent, but produces harms of comparable severity when the fabricated content enters decision-making workflows.

3. Narrative Manipulation

Coordinated deployment of AI-generated content to influence public opinion or democratic processes:

• Election interference — the Slovak election deepfake audio was released 48 hours before voting, exploiting the gap between content creation speed and verification capacity. The Biden robocall used synthetic voice to discourage voter turnout.
• Government impersonation — the U.S. officials deepfake campaign targeted senior officials with synthetic communications

Narrative manipulation is most dangerous when it exploits timing — releasing synthetic content close enough to an event (an election, a market open, a policy announcement) that verification cannot occur before the content has its intended effect.

Common Causal Factors

Analysis of documented incidents in this domain reveals a dominant pattern: intentional fraud combined with social engineering drives the majority of Information Integrity harms.

Cluster 1 — Deliberate Deception:

• Intentional Fraud and Social Engineering co-occur in most deepfake identity and disinformation incidents. The attacker deliberately exploits human trust relationships using AI-generated content as the weapon. This distinguishes Information Integrity from domains where harm arises from system failures rather than adversarial intent.

Cluster 2 — System Limitations:

• Hallucination Tendency drives the content fabrication mechanism — AI systems generating confident but false outputs. Combined with Over-Automation (humans accepting AI outputs without verification), hallucinated content enters professional and legal workflows.

Cluster 3 — Governance Gaps:

• Regulatory Gap and Inadequate Access Controls appear in synthetic media incidents — platforms lack effective mechanisms to prevent creation or distribution of non-consensual deepfakes, and regulatory frameworks have not kept pace with generation capabilities.

Compared with Security & Cyber threats, which cluster around permission and input failures, Information Integrity harms are primarily driven by the exploitation of human psychology rather than technical vulnerabilities.

What the Incident Data Reveals

Information Integrity is the most heavily documented domain in the registry, with the broadest range of victim types and geographic spread.

Severity and Pattern Distribution

The overwhelming majority of incidents are rated high or critical severity. Deepfake Identity Hijacking is the dominant pattern, accounting for the largest share of incidents — reflecting both the accessibility of voice and video cloning tools and the directness of financial fraud as a harm mechanism.

A notable gap exists: Consensus Reality Erosion, while assessed as a critical long-term threat, has no discrete incident entry because it manifests as a cumulative effect rather than a single event.

Resolution Dynamics

Roughly half of incidents remain structurally open. Resolved cases typically involve vendor remediation (the Gemini bias overcorrection) or legal settlement. Open cases tend to involve ongoing vulnerability classes — election interference techniques, elder fraud networks, and non-consensual imagery distribution persist because the underlying generation capabilities cannot be revoked.

Temporal Acceleration

Incident severity has escalated over time. Early incidents involved relatively crude deepfakes or simple mistranslations. Recent incidents demonstrate real-time video deepfakes in multi-participant calls, voice cloning from seconds of sample audio, and coordinated campaigns timed to democratic processes. The FBI elder fraud report documented a systematic acceleration of AI-enhanced financial scams targeting seniors.

Cross-Domain Interactions

Information Integrity Threats interact with every other domain in the taxonomy. The synthetic content produced in this domain frequently serves as the initial vector for harms that materialize elsewhere.

Information Integrity → Security & Cyber. Deepfake identities enable social engineering attacks that bypass authentication. The Arup fraud combined synthetic identity generation with financial systems compromise.

Information Integrity → Economic & Labor. Deepfake fraud produces direct financial losses. The documented financial impact ranges from $243,000 (UK energy fraud) to $25 million (Arup), with the FBI report documenting aggregate elder fraud losses in the billions.

Information Integrity → Discrimination & Social Harm. Non-consensual deepfake imagery disproportionately targets women and minors. The Westfield student deepfakes and Taylor Swift imagery demonstrate gendered harm patterns. Disinformation campaigns frequently target specific ethnic, religious, or political groups.

Information Integrity → Human-AI Control. When AI systems hallucinate — producing fabricated citations, mistranslations, or false facts — humans who trust AI outputs make decisions on false premises. The ChatGPT legal hallucination illustrates how overreliance on AI outputs in professional contexts amplifies information integrity failures.

Information Integrity → Privacy & Surveillance. Synthetic media creation using an individual’s likeness without consent constitutes a privacy violation. Voice cloning and face-swapping extract biometric identity from public data and weaponize it.

Formal Interaction Matrix

From Domain	To Domain	Interaction Type	Mechanism
Information Integrity	Security & Cyber	ENABLES	Deepfake identity bypasses authentication for financial fraud
Information Integrity	Economic & Labor	CASCADES INTO	Deepfake fraud produces direct financial losses at scale
Information Integrity	Discrimination & Social Harm	AMPLIFIES	Non-consensual imagery and targeted disinformation produce gendered and group-based harm
Information Integrity	Human-AI Control	UNDERMINES	Hallucinated content entering professional workflows degrades human judgment
Information Integrity	Privacy & Surveillance	EXTRACTS FROM	Voice cloning and face-swapping weaponize biometric identity
Information Integrity	Systemic & Catastrophic	CASCADES INTO	Cumulative synthetic content erodes institutional trust

Escalation Pathways

Information Integrity Threats follow a characteristic escalation from individual targeting to institutional erosion.

Escalation Overview

Stage	Level	Example Mechanism
1	Individual Targeting	Voice clone call to one family member
2	Organizational Fraud	Deepfake video conference with multiple executives
3	Democratic Process Interference	AI-generated audio timed to an election
4	Epistemic Infrastructure Erosion	Widespread inability to distinguish authentic from synthetic

Stage 1 — Individual Targeting

A single deepfake voice call impersonates a family member, colleague, or authority figure to extract money or information. The grandparent scam network operated at this level, targeting individual seniors with cloned family voices.

Stage 2 — Organizational Fraud

When deepfake technology improves to support real-time video, the attack surface extends to corporate environments. The Arup fraud demonstrated deepfake impersonation of multiple executives in a single video conference, converting individual social engineering into enterprise-level financial compromise.

Stage 3 — Democratic Process Interference

Synthetic content timed to elections exploits the gap between creation speed and verification capacity. The Slovak election deepfake and Biden robocall targeted democratic institutions, where the impact of manipulation is difficult to measure and impossible to reverse after votes are cast.

Stage 4 — Epistemic Infrastructure Erosion

The cumulative effect of widespread synthetic content degrades the shared ability to agree on facts. This stage is not produced by a single incident but by the aggregate — when any piece of media could plausibly be AI-generated, even authentic content loses credibility. This dynamic is captured in the Consensus Reality Erosion pattern.

Who Is Affected

Most Impacted Sectors

1. Corporate — primary target for deepfake-enabled financial fraud and executive impersonation
2. Elections — targeted by AI-generated disinformation timed to democratic processes
3. Finance — direct financial losses from voice clone and video deepfake fraud
4. Education — AI-generated non-consensual imagery targeting students
5. Government — targeted through official impersonation campaigns

Most Impacted Groups

1. Consumers — the broadest target group, affected by fraud, disinformation, and synthetic media
2. Seniors — disproportionately targeted by AI-enhanced elder fraud exploiting voice cloning
3. Business Leaders — targeted by deepfake executive impersonation for financial fraud
4. Public Servants — targeted through government official impersonation campaigns
5. Children & Minors — harmed by non-consensual deepfake imagery in school contexts

Organizational Response

The causal factor clustering in this domain points to specific organizational considerations for managing Information Integrity risk.

Verification Protocols

The dominance of Intentional Fraud and Social Engineering as co-occurring causal factors means that technical detection alone is insufficient. Organizations need verification protocols for high-value communications:

• Multi-channel confirmation for financial transactions (a video call alone is no longer sufficient)
• Code-word systems for executive communications
• Awareness training focused on AI-generated voice and video capabilities

Content Provenance

For content fabrication risks, organizations should implement output verification for AI-assisted workflows, particularly in legal, medical, and financial contexts where hallucinated content has documented consequences.

Implementation Checklist

Defense	Mitigates	Action	Reference
Multi-channel verification	Synthetic Identity	Require out-of-band confirmation for financial transactions	Intentional Fraud
AI output verification	Content Fabrication	Mandate human review of AI-generated content in professional workflows	Hallucination Tendency
Deepfake detection tools	Synthetic Identity + Narrative Manipulation	Deploy media authentication at organizational boundaries	NIST AI RMF
Elder awareness programs	Synthetic Identity	Targeted training for senior populations on AI voice cloning	Social Engineering
Content watermarking	All three mechanisms	Support C2PA and similar provenance standards	ISO/IEC 42001

Regulatory Context

EU AI Act: AI systems designed to materially distort human behavior are classified as high-risk or prohibited. Deepfake content must be labeled as AI-generated. Manipulation of democratic processes through synthetic content faces the strictest regulatory treatment.

NIST AI Risk Management Framework: Maps to validity and reliability trustworthiness characteristics. The framework addresses content provenance, output accuracy, and the need for mechanisms to detect and flag AI-generated content, particularly in contexts where false information produces consequential harms.

ISO/IEC 42001: Establishes management system requirements for output quality and data integrity, including controls on AI system outputs that could produce misleading or fabricated information.

MIT AI Risk Repository: Classified under Misinformation category, encompassing the full spectrum from unintentional hallucinations to deliberate synthetic content campaigns.

Related Domains

• Security & Cyber Threats — Deepfake identity generation enables social engineering that bypasses traditional authentication; compromised AI systems can produce synthetic content with false legitimacy
• Discrimination & Social Harm — Non-consensual deepfake imagery disproportionately targets women and minors; disinformation campaigns frequently target specific demographic groups
• Economic & Labor Threats — Deepfake-enabled fraud produces direct financial losses; AI-generated disinformation can distort markets
• Human-AI Control Threats — AI hallucinations in professional workflows undermine trust in AI-assisted decisions; overreliance on AI outputs amplifies fabrication harms
• Privacy & Surveillance Threats — Voice cloning and face-swapping weaponize biometric identity extracted from public data

Use in Retrieval

This page answers questions about AI-enabled information integrity threats, including: deepfake identity hijacking, AI voice cloning fraud, disinformation campaigns, AI hallucinations and fabricated content, synthetic media manipulation, non-consensual AI-generated imagery, election interference via AI, and consensus reality erosion. It covers operational mechanisms, causal factors, escalation pathways, organizational response guidance, and the regulatory landscape for AI-generated misinformation and disinformation. Use this page as a reference for the Information Integrity Threats domain (DOM-INF) in the TopAIThreats taxonomy.