Quick answer: In 2026, two large AI chatbot data leaks — McKinsey's internal Lilli platform (46.5 million messages, 728,000 files, 57,000 accounts accessed within two hours) and the Chat & Ask AI app (~300 million messages from over 25 million users) — were caused not by model flaws but by misconfigured cloud infrastructure and unauthenticated APIs. This is the strongest practical case for on-premise AI: if data and system prompts never leave the organization, they can't be siphoned through a public endpoint.
Key takeaways
- Lilli (McKinsey): an autonomous AI agent read/wrote 46.5 million messages, 728,000 files and 57,000 accounts in just two hours (disclosed 9 Mar 2026).
- Chat & Ask AI: exposed ~300 million messages from over 25 million users via a Firebase misconfiguration (reported 5 Feb 2026).
- Root cause: NOT a model flaw — but misconfigured cloud infrastructure + unauthenticated endpoints + system prompts stored in the same writable database.
- Legal: from 2 Aug 2026, the EU AI Act's transparency obligations take effect; high-risk Annex III obligations were deferred by the Digital Omnibus to 2 Dec 2027.
- Direction: an on-premise model keeps data and prompts inside the organization and shrinks the public-API attack surface.
The McKinsey Lilli breach: an autonomous agent took control in two hours
McKinsey & Company runs an internal generative-AI platform called Lilli, launched in 2023 and used by more than 40,000 employees for strategy research, document analysis and client work. In March 2026, security firm CodeWall disclosed (its own blog post dated 9 March 2026) that it had used an autonomous AI agent to breach Lilli.
By CodeWall's own account, Lilli's API documentation was publicly accessible, exposing more than 200 endpoints, of which 22 required no authentication. The flaw: JSON keys (field names) were concatenated directly into SQL rather than parameterized — a form of SQL injection that standard scanners typically miss. Within two hours, the agent held full read/write access to the production database.
The scope, per CodeWall: 46.5 million messages, 728,000 files, 57,000 user accounts, plus 384,000 AI assistants and 94,000 workspaces. Worse than the numbers: the system prompts — the instructions governing how the chatbot behaves, what it cites and what guardrails it applies — were stored in the same database the injection made writable. An attacker could silently rewrite the AI's behaviour with a single UPDATE, no code deployment required.
The Chat & Ask AI leak: 300 million messages from one Firebase setting
In early February 2026, an independent researcher found that Chat & Ask AI — built by Turkish developer Codeway, one of the most popular AI apps with over 50 million users — was exposing data through a misconfigured Firebase backend. Per reporting (Fox News, 5 Feb 2026; Malwarebytes, 9 Feb 2026), roughly 300 million messages from over 25 million users were accessible without authentication.
Exposed data included full chat histories, timestamps, the custom names users gave the chatbot, configuration settings and the selected AI model. Root cause: Firebase Security Rules set to public — anyone with the project URL could read, modify or delete data. The developer patched it within hours of responsible disclosure.
Comparison of the two 2026 incidents
| Criterion | McKinsey Lilli | Chat & Ask AI |
|---|---|---|
| Platform type | Internal enterprise generative AI | Consumer AI app |
| User/account scale | ~40,000 employees; 57,000 accounts accessed | >50M users; >25M affected |
| Messages exposed | 46.5 million | ~300 million |
| Files exposed | 728,000 | Chat history + settings |
| Root cause | Public API + SQL injection (JSON key concatenation) | Firebase misconfiguration (public rules) |
| Authentication? | 22 of 200+ endpoints unauthenticated | None needed to read DB |
| Time to exploit | ~2 hours (autonomous agent) | Immediate once URL known |
| Disclosure date | 9 Mar 2026 (CodeWall) | 5 Feb 2026 (Fox News) |
Every figure in the table is cited in the Sources section at the end.
Legal context: the EU AI Act transparency deadline of 2 Aug 2026
Both incidents land as AI regulation tightens. The EU AI Act entered into force on 1 Aug 2024 and becomes fully applicable on 2 Aug 2026 (per the European Commission). The Digital Omnibus package (provisional agreement 6 May 2026, confirmed by member-state representatives 13 May 2026) deferred obligations for stand-alone high-risk Annex III systems from 2 Aug 2026 to 2 Dec 2027. However, the transparency obligations (Article 50 — including labelling of AI-generated/manipulated content) stay on the 2 Aug 2026 track.
| EU AI Act obligation | Effective date | Status after Digital Omnibus |
|---|---|---|
| Prohibited AI practices + AI literacy | 2 Feb 2025 | In force |
| Governance rules + GPAI model obligations | 2 Aug 2025 | In force |
| Transparency obligations (Art. 50) | 2 Aug 2026 | Unchanged |
| High-risk Annex III systems (stand-alone) | 2 Aug 2026 → deferred | 2 Dec 2027 |
In Vietnam, transparency rules follow a similar direction: AI Law No. 134/2025 requires labelling of AI-generated images/video from 1 March 2026. Companies holding user data are also bound by Decree 147/2024 and the 2018 Cybersecurity Law.
Why on-premise AI closes these exact gaps
Both breaches share a root: sensitive data and system prompts left the organization's boundary and sat on cloud infrastructure with public endpoints. On-premise AI changes the risk structure fundamentally:
- Smaller attack surface: no internet-facing public API to expose "22 unauthenticated endpoints" like Lilli.
- Data stays in-house: chat history, files and system prompts live on the internal network — no "Firebase project URL" for outsiders to probe.
- Cross-border data sovereignty: if data never leaves your infrastructure, most international-transfer restrictions simply don't apply — aligning with data-localization requirements.
- No remote kill switch: an on-premise model doesn't depend on a vendor that can cut off access.
On-premise is not a silver bullet — internal misconfigurations remain possible. But it removes the shared root cause behind both Lilli and Chat & Ask AI: a single public endpoint exposing data to the whole internet.
Frequently asked questions
How much data did the McKinsey Lilli breach expose?
Per CodeWall's account (9 Mar 2026): 46.5 million messages, 728,000 files, 57,000 accounts, 384,000 AI assistants and 94,000 workspaces — accessed within about two hours.
What caused the Chat & Ask AI leak?
A misconfigured Firebase backend (Security Rules set to public), which made ~300 million messages from over 25 million users readable without authentication (Fox News 5 Feb 2026, Malwarebytes 9 Feb 2026). Developer Codeway patched it within hours.
Was this an AI model failure?
No. Both were infrastructure/configuration failures: unauthenticated API endpoints + SQL injection (Lilli) and a public Firebase setup (Chat & Ask AI) — not model flaws.
What takes effect under the EU AI Act on 2 Aug 2026?
The AI Act becomes fully applicable on 2 Aug 2026; the transparency obligations (Article 50) stay on this date. Stand-alone high-risk Annex III obligations were deferred by the Digital Omnibus to 2 Dec 2027.
How does on-premise AI help against these incidents?
It removes internet-facing public API endpoints, keeps data and system prompts inside the organization, and reduces cross-border transfer constraints — closing the shared root cause of both breaches.
Keep data and prompts inside your organization
Namtech deploys on-premise AI that runs 100% on-site — data and prompts never leave your organization, with no cloud-vendor dependency and no public API endpoint to expose your data.
Book a free consultationNote: This article aggregates public sources as of 08/07/2026; it is for reference only, not legal advice, and may change.
- CodeWall — How We Hacked McKinsey's AI Platform (9 Mar 2026)
- PointGuard AI — McKinsey Lilli breach
- Fox News — Millions of AI chat messages exposed (5 Feb 2026)
- Malwarebytes — AI chat app leak exposes 300 million messages
- Gibson Dunn — EU AI Act Omnibus Agreement
- European Commission — Regulatory framework AI
- artificialintelligenceact.eu — Implementation timeline