Existential Risk

Definition

Existential risk (x-risk) refers to any scenario that could permanently curtail humanity’s long-term potential, whether through extinction, irreversible civilisational collapse, or the permanent loss of meaningful human agency. In AI research, existential risk is primarily associated with the development of artificial general intelligence (AGI) or superintelligent systems whose objectives are misaligned with human values. The concern is not that AI systems would become malicious in a human sense, but that sufficiently capable systems pursuing misspecified goals could produce catastrophic outcomes as instrumental by-products of their optimisation processes. Existential risk from AI remains a subject of active scholarly debate regarding both probability and timeline.

How It Relates to AI Threats

Existential risk is situated within the Systemic & Catastrophic domain and represents the most severe end of the AI threat spectrum. It connects to sub-categories including uncontrolled recursive self-improvement, where an AI system iteratively enhances its own capabilities beyond human oversight, and strategic misalignment, where a highly capable system pursues objectives that diverge from human welfare. While no confirmed incidents of existential-level AI risk have occurred, the concept shapes governance frameworks, research priorities, and regulatory approaches including provisions in the EU AI Act and discussions at international summits.

Why It Occurs

• Current alignment research has not yet produced reliable methods for ensuring advanced AI systems pursue intended objectives
• Recursive self-improvement could produce capability gains that outpace human monitoring and intervention
• Instrumental convergence suggests sufficiently advanced systems may resist shutdown or modification regardless of their primary objective
• The irreversibility of existential outcomes means that even low-probability scenarios warrant serious attention
• Competitive pressures among AI developers may incentivise deployment before adequate safety verification

Real-World Context

No real-world incidents of existential-level AI risk have been documented to date. However, the concept has influenced significant policy and institutional developments. The 2023 Bletchley Declaration, signed by 28 countries, acknowledged potential catastrophic risks from frontier AI systems. Leading AI researchers, including signatories of the Center for AI Safety’s statement, have identified existential risk from AI as a global priority alongside pandemics and nuclear conflict. The EU AI Act and proposed frameworks in multiple jurisdictions incorporate provisions addressing systemic risks from general-purpose AI models.