Data Bias

Definition

Data bias refers to systematic distortions in training datasets that cause AI models to produce outputs reflecting and perpetuating historical inequities, demographic underrepresentation, or flawed measurement practices. Data bias can manifest as selection bias (non-representative sampling), measurement bias (inconsistent data collection across groups), label bias (prejudiced annotations), or historical bias (data encoding past discrimination as normative patterns). Because machine learning models learn statistical patterns from training data, biased datasets produce models that systematically disadvantage certain populations.

How It Relates to AI Threats

Data bias is a fundamental harm mechanism within Discrimination & Social Harm threats, serving as the upstream cause of many discriminatory AI outcomes. When training datasets underrepresent certain demographic groups, overrepresent stereotypical associations, or encode historical patterns of discrimination, the resulting models perpetuate and often amplify these biases at scale. Data bias is particularly insidious because it can produce discriminatory outcomes even when model designers have no discriminatory intent, making it a structural rather than intentional form of harm.

Why It Occurs

• Historical data encodes societal inequities as statistical patterns
• Data collection processes systematically exclude marginalised populations
• Annotation and labelling reflect the biases of human labellers
• Convenience sampling overrepresents easily accessible populations
• Feedback loops amplify initial dataset imbalances over deployment cycles

Real-World Context

Data bias has been documented across numerous AI applications. Facial recognition systems trained predominantly on lighter-skinned subjects demonstrated significantly higher error rates for darker-skinned individuals. Hiring algorithms trained on historical employment data systematically disadvantaged female candidates because the training data reflected decades of gender-biased hiring patterns. These cases illustrate how data bias translates historical inequity into automated discrimination.