GitHub Copilot Reproduces Verbatim Training Data Including Secrets (2023)

Incident Summary

Throughout 2023, developers and security researchers documented multiple instances of GitHub Copilot — an AI-powered code completion tool developed by GitHub and powered by OpenAI Codex — reproducing verbatim code from its training data, including API keys, personal credentials, proprietary code snippets, and other sensitive information.^[3] Copilot was trained on publicly available code repositories hosted on GitHub, including repositories licensed under various open-source licenses that require attribution, as well as repositories containing inadvertently committed secrets.

Researchers demonstrated that with targeted prompting strategies, Copilot could be induced to output significant portions of recognizable copyrighted code, private API keys, and other sensitive data memorized from its training set.^[3][4] Academic research published in 2023 quantified the scope of this memorization, finding that approximately 1% of Copilot suggestions longer than 150 characters constituted verbatim reproductions of training data, a rate that increased with more specific prompts.^[2]

A class action lawsuit, Doe v. GitHub, Inc., was filed in November 2022 in the U.S. District Court for the Northern District of California, alleging that Copilot’s reproduction of code violated open-source licenses — specifically the requirement to provide attribution and include license text — and constituted an infringement of developers’ intellectual property rights.^[1] The litigation remained ongoing as of the date of logging.

Key Facts

• Product: GitHub Copilot, powered by OpenAI Codex
• Training data: Publicly available GitHub repositories, including open-source and proprietary code
• Memorization rate: Approximately 1% of suggestions over 150 characters were verbatim reproductions of training data
• Sensitive data types reproduced: API keys, personal credentials, proprietary code, licensed code without attribution
• Litigation: Doe v. GitHub, Inc. (class action, N.D. Cal., filed November 2022)
• Mitigation: GitHub implemented content filters to detect and block verbatim reproductions

Threat Patterns Involved

Primary: Model Inversion & Data Extraction — Copilot’s reproduction of verbatim training data, including sensitive credentials and private keys, demonstrated that large language models can inadvertently serve as vectors for extracting private information embedded in their training sets, particularly when prompted with contextually relevant inputs.

Secondary: Re-identification Attacks — The reproduction of personally identifiable information, including names, email addresses, and API keys associated with specific individuals, from a model trained on ostensibly public data constituted a form of re-identification, linking anonymized model outputs back to identifiable individuals.

Significance

1. Training data memorization as a security risk. The incident established that AI code generation tools can function as inadvertent repositories of sensitive information, posing direct security risks when they reproduce credentials, API keys, or private code in production environments.
2. Legal test case for AI training on copyrighted works. The Doe v. GitHub class action lawsuit represents one of the first major legal challenges to the practice of training AI models on copyrighted material, with potential implications for the broader AI industry’s reliance on publicly available data.
3. Tension between open-source licensing and AI training. The case exposed a fundamental tension between the open-source ethos of publicly sharing code and the use of that code to train commercial AI products that may not comply with the attribution and licensing requirements of the original works.
4. Limits of post-deployment mitigation. While GitHub implemented filters to reduce verbatim reproduction, the incident demonstrated that post-training content filters are an imperfect solution to the problem of memorization in large language models, as novel prompting strategies can circumvent such filters.