Disparate Impact

Definition

Disparate impact is a legal and analytical framework for identifying discrimination that occurs when a facially neutral policy or system produces disproportionately adverse outcomes for members of a protected group, regardless of whether discriminatory intent exists. In AI systems, disparate impact arises when algorithms trained on historical data or designed with seemingly neutral criteria produce systematically different outcomes across demographic groups defined by race, gender, age, disability, or other protected characteristics. The concept is significant because it shifts the focus from intent to measurable outcome differences.

How It Relates to AI Threats

Disparate impact is a core harm mechanism within Discrimination & Social Harm threats, particularly in allocational harm where AI systems distribute resources, opportunities, or penalties unevenly across demographic groups. Automated hiring systems, credit scoring models, predictive policing algorithms, and healthcare triage tools have all been found to produce disparate impacts. Because AI systems can embed and scale discriminatory patterns without explicit discriminatory rules, disparate impact analysis provides an essential framework for identifying and measuring AI-driven discrimination that might otherwise remain hidden within opaque algorithmic processes.

Why It Occurs

• Training data reflects historical patterns of unequal treatment
• Proxy variables correlate with protected characteristics indirectly
• Optimisation for aggregate accuracy masks group-level performance disparities
• System designers do not test for differential outcomes across subgroups
• Feedback loops compound initial disparities through repeated application

Real-World Context

Disparate impact has been documented in AI systems used for criminal risk assessment, where algorithms produced higher risk scores for Black defendants compared to white defendants with similar profiles. In lending, automated underwriting systems have been found to charge higher interest rates to minority borrowers. The U.S. Equal Employment Opportunity Commission has issued guidance on the application of disparate impact analysis to AI-driven hiring tools, reflecting growing regulatory attention to algorithmic discrimination.