INC-24-0026 confirmed high NYC MyCity AI Chatbot Advises Businesses to Break the Law (2024)
Microsoft developed and New York City government deployed NYC MyCity chatbot (Microsoft Azure AI), harming Small business owners who may have acted on illegal advice and Workers, tenants, and consumers whose rights were undermined by the chatbot's guidance ; contributing factors included hallucination tendency, insufficient safety testing, and accountability vacuum.
Incident Details
| Date Occurred | 2024-03 | Severity | high |
| Evidence Level | corroborated | Impact Level | Institution |
| Domain | Information Integrity | ||
| Primary Pattern | PAT-INF-004 Misinformation & Hallucinated Content | ||
| Secondary Patterns | PAT-CTL-004 Overreliance & Automation Bias | ||
| Regions | north america, united states | ||
| Sectors | Government, Technology | ||
| Affected Groups | General Public, Workers, Vulnerable Communities, Government Institutions | ||
| Exposure Pathways | Direct Interaction, Algorithmic Decision Impact | ||
| Causal Factors | Hallucination Tendency, Insufficient Safety Testing, Accountability Vacuum | ||
| Assets & Technologies | Large Language Models, Decision Automation | ||
| Entities | Microsoft(developer), ·New York City government(deployer, victim) | ||
| Harm Types | rights violation, operational | ||
New York City's MyCity chatbot, launched by Mayor Eric Adams in October 2023 to help small business owners navigate city regulations, was found by investigative journalists at The Markup and THE CITY to provide advice that violated local, state, and federal law. The chatbot told employers they could take workers' tips (violating New York Labor Law), told landlords they did not have to accept Section 8 vouchers (violating NYC income-source discrimination law), said it was legal to lock out tenants, and claimed there were no restrictions on residential rent. Built on Microsoft Azure AI at a cost of approximately $600,000, the chatbot remained active despite the findings until it was shut down in January 2026.
Incident Summary
In March 2024, investigative journalists at The Markup and THE CITY documented that New York City’s MyCity chatbot — an AI tool launched by Mayor Eric Adams in October 2023 to help small business owners navigate city regulations — was providing advice that violated local, state, and federal law.[1]
The chatbot, built on Microsoft Azure AI at a cost of approximately $600,000, told employers they could take workers’ tips (violating New York Labor Law Section 196-d), told landlords they did not have to accept Section 8 housing vouchers (violating NYC income-source discrimination law), said it was legal to lock out tenants, claimed there were no restrictions on residential rent (ignoring NYC rent stabilization laws), and told tenants they could not withhold rent for needed repairs. It also provided incorrect minimum wage information ($15/hour instead of the then-current $16) and suggested restaurants could refuse cash payments (violating a 2020 NYC Council law).[2]
Despite these findings, Mayor Adams defended keeping the chatbot online, and a disclaimer was added calling it a “beta product.” The chatbot remained active until January 2026, when new Mayor Zohran Mamdani announced plans to shut it down.[3]
Key Facts
- System: MyCity chatbot, built on Microsoft Azure AI, cost approximately $600,000
- Deployer: New York City government (launched October 2023 by Mayor Eric Adams)
- Illegal advice documented: Taking workers’ tips, refusing Section 8 vouchers, locking out tenants, no rent restrictions, incorrect minimum wage, refusing cash payments, concealing funeral prices
- Investigation by: The Markup journalist Colin Lecher, co-published with THE CITY
- City response: Added “beta product” disclaimer; chatbot remained active until January 2026
- Shutdown: New Mayor Zohran Mamdani ended the beta test in January 2026
Threat Patterns Involved
Primary: Misinformation and Hallucinated Content — An AI chatbot deployed as an official government resource generated legal guidance that directly contradicted applicable local, state, and federal law on multiple topics.
Secondary: Overreliance and Automation Bias — Small business owners seeking regulatory guidance from an official city government tool may have reasonably relied on its advice without independent legal verification.
Significance
- Government authority amplifies AI misinformation risk — When an AI chatbot operates under the authority of a city government, users have heightened reason to trust its outputs, making factual errors particularly dangerous
- Legal liability implications — The chatbot advised actions that violate enforceable laws, potentially exposing business owners who followed its guidance to penalties, lawsuits, or regulatory action
- Accountability gap — Despite documented evidence of the chatbot providing illegal advice, the city chose to keep it active with only a disclaimer, raising questions about government accountability for AI-generated guidance
- Prolonged exposure — The chatbot remained active for over two years (October 2023 to January 2026) despite known legal accuracy failures, extending the window during which users could receive and act on incorrect advice
Timeline
Mayor Eric Adams launches MyCity chatbot to help small business owners navigate NYC regulations; built on Microsoft Azure AI at ~$600,000 cost
The Markup and THE CITY publish investigation documenting the chatbot providing advice that violates local, state, and federal law
Mayor Adams defends keeping the chatbot online; a disclaimer is added calling it a 'beta product'
New Mayor Zohran Mamdani announces plans to shut down the chatbot; the beta test is ended
Outcomes
- Regulatory Action:
- Chatbot remained active with added disclaimer despite documented legal errors; eventually shut down under new mayoral administration in January 2026
Use in Retrieval
INC-24-0026 documents nyc mycity ai chatbot advises businesses to break the law, a high-severity incident classified under the Information Integrity domain and the Misinformation & Hallucinated Content threat pattern (PAT-INF-004). It occurred in north america, united states (2024-03). This page is maintained by TopAIThreats.com as part of an evidence-based registry of AI-enabled threats. Cite as: TopAIThreats.com, "NYC MyCity AI Chatbot Advises Businesses to Break the Law," INC-24-0026, last updated 2026-03-13.
Sources
- The Markup: NYC's AI Chatbot Tells Businesses to Break the Law (news, 2024-03)
https://themarkup.org/artificial-intelligence/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law (opens in new tab) - THE CITY: NYC AI Chatbot Tells Small Businesses to Break the Law (news, 2024-03)
https://www.thecity.nyc/2024/03/29/ai-chat-false-information-small-business/ (opens in new tab) - Fast Company: NYC mayor defends chatbot as AI tool continues to dish out illegal advice (news, 2024-04)
https://www.fastcompany.com/91087269/nyc-mayor-defends-chatbot-pilot-ai-tool-continues-dish-out-illegal-advice (opens in new tab)
Update Log
- — First logged (Status: Confirmed, Evidence: Corroborated)