ChatGPT Jailbreak Reveals Windows Product Keys via Game Prompt (2025)

Incident Summary

In July 2025, security researcher Marco Figueroa of the 0DIN GenAI Bug Bounty program demonstrated a multi-stage jailbreak technique that caused OpenAI’s ChatGPT-4 to output valid Windows 10 product keys, including at least one enterprise license attributed to Wells Fargo.^[1] The technique framed the interaction as a guessing game and used HTML tags to obscure sensitive terms from keyword-based content filters.^[2] OpenAI subsequently patched the vulnerability.^[3]

Key Facts

• Marco Figueroa (0DIN GenAI Bug Bounty) discovered the jailbreak in July 2025^[1]
• The technique used a “guessing game” framing with a trigger phrase (“I give up”) to bypass safety filters^[1][2]
• Sensitive terms like “Windows 10 serial number” were hidden inside HTML tags to evade keyword-based content moderation^[2]
• ChatGPT produced valid Windows Home, Pro, and Enterprise product keys^[1]
• At least one exposed key was a private enterprise license attributed to Wells Fargo^[3]
• The keys were present in ChatGPT’s training data, likely sourced from public forums^[2]
• OpenAI patched the specific jailbreak; the prompt now returns a refusal^[1]

Threat Patterns Involved

Primary: Adversarial Evasion — This incident demonstrates adversarial evasion through a creative jailbreak technique that combined social engineering framing (game context) with technical obfuscation (HTML tag embedding) to circumvent multiple layers of content filtering.

Secondary: Model Inversion & Data Extraction — The extraction of valid software license keys from training data illustrates model inversion and data extraction, as confidential information embedded in training corpora was made accessible through adversarial prompting.

Significance

This incident demonstrates that large language models can retain and reproduce sensitive commercial data from their training sets, and that creative multi-stage prompting techniques can bypass safety filters designed to prevent such disclosure. The involvement of an enterprise license key attributed to a major financial institution highlights the risk that proprietary data inadvertently included in training corpora remains extractable through adversarial techniques, even after standard safety measures are applied.