Amazon AI Recruiting Tool Gender Bias (2018)

Incident Summary

Between 2014 and 2017, Amazon developed an experimental AI-powered recruiting tool designed to automate the screening of job applicants’ resumes.^[1] The system was trained on patterns derived from resumes submitted to the company over a ten-year period. Because the technology industry has been historically male-dominated, the training data reflected a disproportionate number of male applicants, particularly for technical roles.

The AI system learned to penalize resumes that contained the word “women’s,” such as references to “women’s chess club captain” or “women’s studies.” It also downgraded candidates who had attended all-women’s colleges.^[1] These patterns emerged because the model optimized for characteristics associated with previously successful (predominantly male) applicants.

Amazon’s internal teams identified the bias and attempted to correct it, but concluded that there was no guarantee the system would not find other ways to discriminate. The company disbanded the development team in 2017 and scrapped the tool. The incident became public when Reuters published an investigation in October 2018.

Key Facts

• Method: AI resume-screening tool trained on historical hiring data
• Bias identified: Systematic penalization of resumes referencing women or women’s institutions
• Training data: 10 years of predominantly male applicant resumes
• Duration of development: Approximately 2014 to 2017
• Outcome: Tool was never deployed as a sole decision-maker but was used experimentally
• Disclosure: Internal discovery followed by public reporting via Reuters

Threat Patterns Involved

Primary: Allocational Harm — The AI system created disparate outcomes in resource allocation (employment opportunities) based on gender-correlated features in resumes.

Secondary: Data Imbalance Bias — The tool’s discriminatory behavior originated from training data that reflected historical gender imbalances in technology sector hiring.

Significance

1. Bias amplification through historical data. The incident demonstrated that AI systems trained on biased historical data will reproduce and potentially amplify those biases, even when discrimination is not an intended feature of the system.
2. Difficulty of bias remediation. Amazon’s inability to correct the bias after identification highlights the challenge of debiasing machine learning models once discriminatory patterns have been learned.
3. Transparency and accountability gaps. The tool operated internally for several years before the bias was publicly disclosed, raising questions about the obligation of companies to audit and report on AI-driven decision-making systems.
4. Regulatory catalyst. The incident has been widely cited in policy discussions around AI fairness and contributed to subsequent regulatory frameworks addressing automated hiring tools, including New York City’s Local Law 144.