AI Threats to the Legal Sector

AI-enabled threats to the legal sector include hallucinated case citations in AI-generated briefs, deepfake evidence manipulation, algorithmic bias in judicial risk assessment tools, unauthorized practice of law by AI chatbots, and erosion of due process through automated legal decisions. These threats affect law firms, courts, regulatory bodies, legal aid organizations, and legal technology companies.

The legal sector faces distinctive AI risks because it depends on factual accuracy, evidentiary integrity, and professional standards of care, all of which are directly challenged by generative AI’s tendency toward hallucination, fabrication, and confident presentation of false information. The vast majority of documented legal sector incidents are classified high or critical severity, reflecting the irreversible consequences of AI failures affecting individual liberty and justice.

Use this page to brief leadership, inform legal risk assessments, and explore documented incidents affecting legal organizations.

Who this page is for

• Attorneys and law firm partners
• Judges and court administrators
• Legal technologists and innovation officers
• Compliance officers and ethics counsel
• Legal policy makers and bar association leaders

At a glance

• Severity profile: All documented incidents classified high or critical severity
• Primary threats: Hallucinated legal citations, AI-generated evidence manipulation, judicial AI bias, unauthorized practice of law, erosion of due process through automated legal decisions
• Key domains: Information Integrity, Human-AI Control, Discrimination & Social Harm
• Regulatory exposure: Professional conduct rules, court-specific AI disclosure requirements, EU AI Act (high-risk for judicial systems), bar association AI guidance

---

How AI Threats Appear in the Legal Sector

Legal sector AI risks cluster around five recurring threat patterns, each documented through real-world incidents in the TopAIThreats database.

Recurring AI threat patterns in the legal sector

Threat Pattern	Primary Domain	Key Indicator
Hallucinated legal citations	Information Integrity	AI-generated briefs citing nonexistent cases or statutes
Evidence manipulation	Information Integrity	AI-generated or altered documents, audio, or video introduced as evidence
Judicial AI bias	Discrimination & Social Harm	Risk assessment or sentencing tools producing disparate outcomes
Unauthorized practice of law	Human-AI Control	AI systems providing legal advice to consumers without attorney supervision
Automated legal decisions	Human-AI Control	Court or regulatory determinations delegated to AI without meaningful human review

• Hallucinated legal citations — Large language models generating fabricated case citations, statutory references, and legal arguments that appear authoritative but reference nonexistent authorities. The Mata v. Avianca ChatGPT hallucinated citations case resulted in sanctions against attorneys who submitted an AI-generated brief containing six fictitious cases.
• Evidence manipulation — AI-generated synthetic media — including deepfake video, audio, and documents — that challenges the evidentiary foundations of legal proceedings. The ability to create convincing fabricated evidence undermines both prosecution and defense when the authenticity of digital evidence can no longer be presumed.
• Judicial AI bias — Risk assessment tools used in bail, sentencing, and parole decisions that produce allocational harm through racially disparate predictions, often because training data encodes historical criminal justice disparities.
• Unauthorized practice of law — AI legal assistants and chatbots providing consumers with specific legal advice, document preparation, and case strategy without attorney oversight, creating risks of over-automation in a domain where incorrect guidance can cause irreversible harm.
• Automated legal decisions — AI systems making or heavily influencing judicial and regulatory determinations (immigration cases, benefits appeals, regulatory enforcement) with insufficient human review, creating loss of human agency in decisions affecting individual rights. The UnitedHealth AI claim denial court order demonstrates judicial pushback against AI-automated determinations affecting individual rights.

Emerging challenges for legal practice

AI is reshaping legal practice while creating new professional and ethical hazards:

• Duty of competence and AI — Bar associations increasingly expect attorneys to understand AI tools they use, yet the opacity of large language models makes it difficult to assess their reliability for specific legal tasks
• Discovery and e-discovery manipulation — AI tools used to process and review documents in discovery can be exploited through adversarial attacks designed to cause relevant documents to be classified as non-responsive
• Confidentiality risks from AI tools — Attorneys using cloud-based AI tools for legal research and drafting risk exposing privileged information through AI systems that retain and learn from input data

---

Relevant AI Threat Domains

Accuracy & evidence risks

• Information Integrity — Hallucinated content in legal research, synthetic media manipulation of evidence, and disinformation targeting judicial proceedings

Decision automation risks

• Human-AI Control — Automation bias in legal research and drafting, loss of human agency in judicial decisions, and implicit authority transfer to AI legal tools

Fairness & justice risks

• Discrimination & Social Harm — Allocational harm through biased risk assessment tools, proxy discrimination in predictive justice systems

---

What to Watch For

These are the most critical warning signs that legal organizations should monitor for AI-related risks, with actionable guidance for each.

• Attorneys relying on AI-generated legal research without verifying citations against primary sources — What law firm partners can do: Establish mandatory verification protocols for all AI-assisted legal research. Require attorneys to confirm every citation exists in an authoritative legal database before inclusion in any filing. Include AI use disclosure in work product review checklists.

• AI risk assessment tools influencing bail, sentencing, or parole decisions without bias auditing — What court administrators can do: Mandate regular bias and fairness auditing of judicial risk assessment tools. Require that judges receive training on AI tool limitations. Ensure defendants have the right to challenge AI-generated risk assessments.

• AI-generated evidence introduced without authentication safeguards — What judges and attorneys can do: Develop admissibility standards for AI-generated or AI-processed evidence. Require authentication frameworks that account for synthetic media capabilities. Consider the need for expert testimony on AI generation and detection.

• Legal AI chatbots providing specific advice to consumers without adequate disclaimers or attorney oversight — What regulators can do: Clarify the application of unauthorized practice of law rules to AI legal tools. Require conspicuous disclosure when consumers interact with AI rather than licensed attorneys. Ensure AI-generated legal guidance does not substitute for professional legal counsel.

---

Protective Measures

Detection & verification

• Detect AI-generated text — AI-generated text detection tools can flag AI-written legal documents for review. The guide to detecting AI-generated text covers capabilities and limitations relevant to legal practice.
• Detect deepfake evidence — Deepfake detection and content provenance and watermarking tools help authenticate digital evidence. See the guide to detecting deepfakes for forensic applications.
• Detect voice cloning — Voice cloning detection identifies synthetic voice recordings that may be presented as evidence. See the guide to detecting voice cloning.

Fairness & governance

• Audit for bias — Bias and fairness auditing tools test judicial risk assessment and sentencing recommendation systems. The guide to detecting AI bias covers criminal justice applications.
• Establish governance — Model governance controls provide oversight frameworks for AI tools used in legal practice. The AI deployment checklist covers pre-adoption evaluation.

Questions legal professionals should ask

• “What verification procedures exist for AI-generated legal research before it is included in filings?”
• “How are we protecting client confidentiality when using cloud-based AI tools for legal work?”
• “What risk assessment or predictive tools are used in our jurisdiction’s courts, and when were they last audited for bias?”
• “What is our firm’s policy on disclosing AI use to clients and courts?”

---

Regulatory Context

• EU AI Act (entered into force August 2024, high-risk provisions apply from August 2026) — Classifies AI systems used by judicial authorities for legal interpretation and case outcome analysis as high-risk (Annex III), with obligations for transparency, human oversight, and accuracy
• NIST AI RMF (version 1.0, January 2023) — Provides risk management guidance applicable to AI in legal and judicial contexts
• ISO/IEC 42001 (published December 2023) — Offers an AI management system framework for legal organizations deploying AI tools

Legal AI governance is emerging through multiple channels: court rules requiring AI disclosure in filings, bar association guidance on duty of competence for AI use, proposed legislation on algorithmic accountability in judicial decisions, and evolving evidence authentication standards. The ABA’s resolution on AI and the practice of law provides baseline ethical guidance for US practitioners. Rules vary significantly by jurisdiction and are changing rapidly.

---

Documented Incidents

Based on incident analysis, the legal sector is most frequently affected by threats in the Information Integrity domain (hallucinated citations and fabricated legal authorities) and Discrimination & Social Harm domain (biased risk assessment tools in criminal justice).