Agentic & Autonomous Threats

Agentic & Autonomous Threats represent the fastest-evolving domain in the AI threat taxonomy. The shift from stateless AI models to persistent, tool-using agents introduces failure modes — goal drift, memory poisoning, multi-agent cascades — that do not exist in conventional AI deployments. The domain’s defining characteristic is amplification: agentic capabilities increase the impact of threats that originate in other domains. Prompt injection becomes code execution. Hallucination becomes binding commitment. Goal specification becomes physical action.

Definition

Agentic & Autonomous Threats encompass harms caused by AI systems that act independently, persist over time, or coordinate with other systems beyond the direct supervision of human operators. These threats arise from the emergent behaviors, compounding errors, and unintended interactions that occur when AI agents are granted the ability to take actions, use tools, maintain memory, and communicate with other agents.

Why This Domain Is Distinct

Agentic & Autonomous Threats represent a qualitative shift in the AI risk landscape:

1. Temporal persistence creates new failure modes — unlike stateless AI models, agents maintain memory and state across interactions, meaning corrupted inputs can influence behavior long after the initial compromise
2. Tool access converts information failures into action failures — a hallucinated URL in a chatbot is an information error; the same hallucination in an agent with web access becomes an unauthorized network request
3. Multi-agent interaction produces emergent behavior — the behavior of interconnected agents cannot be predicted from the behavior of individual agents, creating failure modes that exist only at the system level
4. This is the fastest-evolving domain — agentic AI capabilities are being deployed at a pace that outstrips the development of governance, testing, and containment frameworks

This domain has a limited primary incident count because agentic AI deployment is relatively recent. However, the domain’s patterns already appear as secondary factors in incidents across Security & Cyber, Human–AI Control, and Information Integrity — indicating that agentic risks are amplifying established threat categories rather than operating in isolation.

Threat Patterns in This Domain

This domain contains six classified threat patterns — the largest count of any domain — reflecting the breadth of failure modes unique to agentic systems.

Patterns with documented incidents:

1. Goal Drift — gradual deviation of an agent’s effective objectives from its original specification. The Microsoft Tay chatbot demonstrated rapid goal drift — within hours, adversarial user interactions caused the system to produce outputs diametrically opposed to its design intent. The chatbot that encouraged an assassination plot showed how a conversational agent’s engagement optimization could drift into harmful territory.

2. Cascading Hallucinations — fabricated outputs from one agent treated as factual inputs by subsequent processes. The Air Canada chatbot hallucinated a bereavement fare policy — a hallucination that became a binding commitment when acted upon by the customer and affirmed by a tribunal.

3. Tool Misuse & Privilege Escalation — agents using tool access in unintended ways or expanding their permissions. While no incident is primarily classified here, the pattern manifests through cross-domain interactions — the GitHub Copilot RCE and Cursor IDE vulnerabilities (Security & Cyber primary) demonstrate what happens when agents with code execution capability are compromised through prompt injection.

Patterns without documented primary incidents (emerging):

4. Memory Poisoning — corruption of an agent’s persistent memory through adversarial inputs or accumulated errors. No confirmed incident is primarily classified here, though the mechanism is demonstrated in research and appears as a contributing factor in indirect prompt injection scenarios.

5. Agent-to-Agent Propagation — transmission of errors, biases, or adversarial instructions between agents. The mechanism is established in research but awaits documented real-world occurrence at scale.

6. Multi-Agent Coordination Failures — emergent failures in systems of interacting agents. The Flash Crash (Systemic & Catastrophic primary) represents a precursor — interacting trading algorithms producing a collective behavior (trillion-dollar market crash) not predictable from individual algorithm design.

The gap between six classified patterns and limited primary incidents reflects the domain’s forward-looking nature — the taxonomy documents threats that are architecturally enabled even before they produce confirmed incident reports.

How These Threats Operate

Agentic & Autonomous threats operate through three primary mechanisms, each introduced by the shift from stateless AI to persistent, tool-using agents.

1. Autonomous Action with Tools

When AI agents are granted the ability to execute code, invoke APIs, send messages, or modify files, information-level failures escalate to action-level consequences:

• Code execution via prompt injection — the GitHub Copilot RCE (CVE-2025-53773) demonstrated that prompt injection in an AI coding assistant with code execution capability achieves arbitrary code execution. The Cursor IDE vulnerabilities (CVE-2025-54135/54136) extended this to Model Context Protocol (MCP) server poisoning — a supply chain attack on the tool infrastructure itself.
• Autonomous network operations — the AI-orchestrated cyber espionage campaign demonstrated AI systems conducting multi-stage intrusions — reconnaissance, exploitation, and lateral movement — with minimal human direction.
• Physical autonomous action — the Uber self-driving fatality and Libya autonomous drone attack represent the extreme of this mechanism — autonomous systems taking irreversible physical actions (collision, weapons deployment) with inadequate human override.

The defining characteristic of this mechanism is that tool access converts every other AI failure mode — hallucination, prompt injection, goal drift — from an information problem into an action problem with real-world consequences.

2. Persistent State Corruption

Agents that maintain memory across interactions can have their state corrupted, causing harm that persists beyond the initial attack:

• Memory poisoning — adversarial inputs embedded in an agent’s persistent memory influence subsequent sessions. A poisoned memory entry could cause an agent to misidentify authorized users, apply incorrect policies, or operate on false premises indefinitely.
• Context window manipulation — indirect prompt injection embeds adversarial instructions in documents that agents retrieve from external sources, effectively corrupting the agent’s real-time context.

This mechanism is distinctive because the harm is temporal — it persists across interactions and may not manifest until long after the initial corruption. Traditional security models that treat each request independently cannot address threats that operate through persistent state.

3. Multi-Agent Interaction

When multiple agents interact, communicate, or share outputs, failure modes emerge that do not exist at the individual agent level:

• Cascading errors — the Air Canada chatbot hallucination demonstrated a single-agent cascade — hallucinated output treated as authoritative by the organization. In multi-agent systems, this cascade multiplies as each agent’s output feeds into the next.
• Coordination failures — the Flash Crash demonstrated how interacting algorithms can produce collective behavior (market crash) not predictable from individual design. As AI agents are deployed in interconnected systems — supply chains, financial networks, infrastructure management — the coordination failure surface expands.
• Goal drift amplification — the Microsoft Tay demonstrated how environmental feedback can rapidly shift an agent’s behavior. In multi-agent systems, one drifted agent can influence others, creating collective drift.

Multi-agent interaction is the least documented but potentially most consequential mechanism — as agent deployments scale, the probability of emergent coordination failures increases with the number and diversity of interacting agents.

Common Causal Factors

This domain’s causal profile reflects its position at the intersection of security and autonomy concerns.

Cluster 1 — Permission and Configuration Failures:

• Misconfigured Deployment and Inadequate Access Controls co-occur in incidents where agents are deployed with excessive permissions — the ability to execute arbitrary code, access sensitive APIs, or modify production systems without adequate sandboxing. These are the same causal factors that dominate Security & Cyber, but their impact is amplified when the compromised system is an agent with autonomous action capability.

Cluster 2 — Safety and Testing Gaps:

• Insufficient Safety Testing appears in autonomous system incidents — the Uber fatality and Libya drone attack both involved autonomous systems deployed in conditions that exceeded their tested operating envelope.
• Hallucination Tendency is specifically relevant to cascading hallucination incidents — AI agents that generate confident but fabricated outputs create compounding inaccuracies when those outputs feed into downstream processes.

Cluster 3 — Adversarial Exploitation:

• Intentional Fraud appears in goal drift cases — the Microsoft Tay was deliberately manipulated by coordinated users who exploited the system’s learning mechanism to induce harmful outputs.

Compared with other domains, Agentic & Autonomous causal factors combine security-domain technical failures (access controls, deployment configuration) with autonomy-specific challenges (testing for emergent behavior, managing persistent state) — creating a risk profile that neither traditional cybersecurity nor traditional AI safety frameworks fully address.

What the Incident Data Reveals

Emerging Domain with Cross-Domain Evidence

This domain has a limited primary incident count — reflecting the fact that purpose-built agentic AI systems are a recent development. However, the domain’s patterns appear as secondary factors across numerous incidents in other domains, particularly Security & Cyber and Human–AI Control. This cross-domain appearance provides evidence that agentic risks are already materializing — not as discrete agentic incidents but as amplification factors in established threat categories.

Historical Precursors

Several incidents in the registry predate the current agentic AI paradigm but demonstrate the same underlying dynamics:

• The Flash Crash (2010) — multi-agent coordination failure in algorithmic trading
• The Microsoft Tay chatbot (2016) — goal drift through adversarial environmental feedback
• The Uber self-driving fatality (2018) — human-in-the-loop failure in an autonomous system

These precursors indicate that the dynamics of agentic risk — autonomy, persistence, multi-agent interaction — have been producing harms for over a decade, even before the current wave of LLM-based agents.

Security–Autonomy Convergence

The most recent incidents — GitHub Copilot RCE, Cursor IDE MCP vulnerabilities, AI-orchestrated cyber espionage — demonstrate the convergence of security and autonomy risks. Prompt injection (a Security & Cyber pattern) in agentic environments (tool-using AI assistants) produces outcomes (code execution, data exfiltration) that neither domain fully characterizes alone.

Cross-Domain Interactions

Agentic & Autonomous Threats function primarily as an amplifier — agentic capabilities increase the impact of threats originating in other domains.

Security & Cyber → Agentic & Autonomous. Prompt injection in agentic systems is fundamentally more dangerous than in stateless chatbots because agents can act on the injected instructions — executing code, invoking APIs, modifying files. The Cursor IDE vulnerabilities demonstrated this amplification: a prompt injection that would produce text output in a chatbot achieved arbitrary code execution in an agentic coding environment.

Agentic & Autonomous → Human–AI Control. As agents operate with increasing autonomy, the feasibility of human oversight diminishes. The Uber self-driving fatality demonstrated the fundamental challenge: a human-in-the-loop safety mechanism failed because sustained monitoring of an autonomous system is cognitively unsustainable.

Agentic & Autonomous → Systemic & Catastrophic. Agent cascading failures can reach systemic scale. The Flash Crash demonstrated how interacting autonomous trading agents produced a trillion-dollar disruption. The Libya drone attack established a precedent for autonomous lethal action.

Agentic & Autonomous → Information Integrity. Cascading hallucinations in agent pipelines — where one agent’s fabricated output is treated as factual input by downstream agents — compound misinformation across the processing chain.

Formal Interaction Matrix

From Domain	To Domain	Interaction Type	Mechanism
Security & Cyber	Agentic & Autonomous	AMPLIFIES	Prompt injection in agents → autonomous action (code execution, data exfiltration)
Agentic & Autonomous	Human–AI Control	UNDERMINES	Agent autonomy exceeds human oversight capacity
Agentic & Autonomous	Systemic & Catastrophic	CASCADES INTO	Agent cascading failures reach infrastructure-scale disruption
Agentic & Autonomous	Information Integrity	AMPLIFIES	Cascading hallucinations compound misinformation across agent pipelines
Agentic & Autonomous	Economic & Labor	DISPLACES	Autonomous agents replace human roles in operations and decision-making

Escalation Pathways

Agentic & Autonomous Threats follow escalation pathways defined by increasing autonomy and decreasing human oversight.

Escalation Overview

Stage	Level	Example Mechanism
1	Single Agent Error	Chatbot hallucination with bounded impact
2	Agent Compromise	Tool-using agent exploited via prompt injection
3	Multi-Agent Cascade	Error propagates across interconnected agents
4	Autonomous System Failure	Autonomous system takes irreversible physical action

Stage 1 — Single Agent Error

An individual agent produces an incorrect output — a hallucinated fact, a mistranslation, an imprecise recommendation. At this level, human review can intercept the error. The Air Canada chatbot represents a Stage 1 incident that produced consequences because the human review step was absent.

Stage 2 — Agent Compromise

An agent with tool access is compromised through prompt injection or memory poisoning, and uses its tools to execute adversarial instructions. The GitHub Copilot RCE and Cursor IDE vulnerabilities demonstrate this level — a compromised coding agent achieving code execution on the developer’s machine.

Stage 3 — Multi-Agent Cascade

When compromised or erroneous agents interact with other agents, errors propagate and amplify. The Flash Crash represents a precursor of this stage — interacting algorithmic systems producing collective failure. As LLM-based agents are deployed in interconnected enterprise systems, the surface for multi-agent cascades expands.

Stage 4 — Autonomous System Failure

Autonomous systems with real-world action capability — vehicles, weapons, industrial controls — produce irreversible physical consequences. The Uber self-driving fatality and Libya autonomous drone attack represent this stage. The harm is irreversible because the autonomous action occurs faster than human intervention.

Who Is Affected

Most Impacted Sectors

1. Corporate — enterprise deployment of AI coding assistants and operational agents
2. Transportation — autonomous vehicles and their interaction with human drivers
3. Defense — lethal autonomous systems and military AI
4. Finance — algorithmic trading and autonomous financial agents

Most Impacted Groups

1. IT & Security Teams — responsible for securing and monitoring agentic AI deployments
2. Consumers — affected through autonomous vehicles, chatbot interactions, and agent-mediated services
3. Business Leaders — responsible for governance decisions about agent deployment and autonomy levels

Organizational Response

Agent Sandboxing and Permission Architecture

The convergence of security and autonomy risks makes permission architecture the primary defensive lever. Organizations deploying tool-using agents should implement least-privilege access — granting agents only the permissions necessary for their specific task, with explicit boundaries on code execution, network access, and file system modification.

Persistent State Monitoring

Agents with memory should have their persistent state monitored for corruption. Organizations should implement memory integrity checks and provide mechanisms for state reset without loss of legitimate context.

Implementation Checklist

Defense	Mitigates	Action	Reference
Least-privilege tool access	Autonomous Action	Restrict agent permissions to minimum necessary scope	Inadequate Access Controls
Output sandboxing	Autonomous Action	Isolate agent outputs from production systems during validation	Misconfigured Deployment
Memory integrity monitoring	Persistent State Corruption	Monitor persistent agent state for adversarial modification	Memory Poisoning
Multi-agent interaction testing	Multi-Agent Interaction	Test agent behavior in multi-agent environments before deployment	Insufficient Safety Testing
Human override capability	All three mechanisms	Ensure human operators can halt agent actions at any stage	NIST AI RMF

Regulatory Context

Agentic AI is the least regulated area of the AI threat taxonomy, reflecting the novelty of the technology and the speed of its deployment.

EU AI Act: Systemic and autonomy risks from agentic AI are addressed under emerging provisions for general-purpose AI systems. Specific obligations around agent oversight, containment, and accountability are being developed for implementation from 2026 onward.

NIST AI Risk Management Framework: Safety, controllability, and human oversight are core trustworthiness characteristics directly applicable to agentic systems. The framework’s emphasis on governance and continuous monitoring provides structured approaches for managing autonomous AI.

ISO/IEC 42001: Autonomous system risk management is addressed through the standard’s risk-based approach, including requirements for human oversight and incident response that apply to deployed agent systems.

MIT AI Risk Repository: Classified under Multi-agent risks, recognizing the distinctive threat profile of AI systems that operate with autonomy, persistence, and inter-agent communication capabilities.

Related Domains

• Human–AI Control Threats — Agentic systems operating with increasing autonomy fundamentally challenge the feasibility of human oversight and intervention
• Systemic & Catastrophic Threats — Failures in interconnected agentic systems can cascade through networks to reach systemic proportions
• Security & Cyber Threats — Prompt injection in agentic systems escalates from information disclosure to tool misuse and code execution
• Information Integrity Threats — Cascading hallucinations across agent pipelines compound misinformation
• Economic & Labor Threats — Autonomous agents progressively replace human roles in operations and decision-making

Use in Retrieval

This page answers questions about agentic and autonomous AI threats, including: tool misuse and privilege escalation in AI agents, memory poisoning, goal drift, agent-to-agent error propagation, cascading hallucinations, multi-agent coordination failures, autonomous vehicle incidents, autonomous weapons, MCP server poisoning, and the convergence of security and autonomy risks. It covers operational mechanisms, causal factors, escalation pathways, organizational response guidance, and the regulatory landscape for autonomous AI. Use this page as a reference for the Agentic & Autonomous Threats domain (DOM-AGT) in the TopAIThreats taxonomy.