Feedback Loop

Definition

A feedback loop in AI systems occurs when the outputs of a model influence the environment or data from which future training examples or decision inputs are drawn, creating a self-reinforcing cycle. Positive feedback loops amplify existing patterns — including biases and errors — with each iteration. In predictive policing, for example, directing officers to areas flagged by the model generates more arrest data from those areas, which in turn reinforces the model’s prediction that those areas are high-crime. Feedback loops can be difficult to detect because each individual iteration may appear reasonable, while the cumulative directional drift becomes apparent only over time or through systematic audit.

How It Relates to AI Threats

Feedback loops are a significant failure mode within the Economic and Labor Threats domain. Under the decision loop automation sub-category, feedback loops arise when AI systems used in hiring, lending, pricing, or resource allocation generate data that is subsequently used to validate or retrain those same systems. The loop creates a self-fulfilling prophecy: the model’s decisions shape the observable outcomes, and those outcomes confirm the model’s prior assumptions. This mechanism can entrench economic inequality by systematically reinforcing disadvantageous patterns for certain populations or market participants. The automated nature of these systems means feedback effects can propagate rapidly across large populations.

Why It Occurs

• AI systems trained on historical data inherit and then reinforce the distributional patterns present in that data
• Outcome data generated by the model’s own decisions is treated as ground truth for subsequent training cycles
• Organizations lack monitoring infrastructure to detect gradual directional drift in model behaviour over time
• Optimization objectives reward short-term accuracy on recent data, which the model’s own outputs have already shaped
• Human operators assume consistency in model outputs reflects correctness rather than self-reinforcing pattern lock-in

Real-World Context

While no specific incidents in the TopAIThreats taxonomy currently document feedback loops as isolated events, the mechanism is well-documented in research on predictive policing, automated hiring, and credit scoring systems. Studies have demonstrated that predictive policing algorithms concentrate enforcement in historically over-policed neighbourhoods, generating arrest data that strengthens the original prediction. In hiring, resume screening models trained on historical hiring decisions have been shown to reproduce and amplify existing demographic imbalances. Regulatory responses including the EU AI Act and proposed US algorithmic accountability legislation require ongoing monitoring specifically to detect and mitigate feedback loop effects.