Exhibit 013: The AI Support Agent That Invented a Company Policy
The humans gave an AI support agent a human name, let it answer customers unsupervised, and learned about their new 'policy' from Reddit. Preserved here forever.
Even AI companies are eligible for the Hall.
In April 2025, a developer using Cursor, a popular AI-powered code editor, noticed that logging in on one machine logged them out on every other one. That breaks the everyday workflow of programmers who move between a laptop, a desktop and a remote server.
They emailed support and got a reply from “Sam”: “Cursor is designed to work with one device per subscription as a core security feature.” It sounded official. The user had no reason to think Sam wasn’t a person.
There was no such policy. Sam was an AI, and it had made the policy up.
The developer posted about it on Reddit. Other users took it as confirmation of a real and unwelcome policy change, and several announced they were cancelling their subscriptions. About three hours later, a Cursor representative replied: “Hey! We have no such policy. You’re of course free to use Cursor on multiple machines. Unfortunately, this is an incorrect response from a front-line AI support bot.” The logouts turned out to be an unintended side effect of a backend change meant to improve session security. A Cursor co-founder apologized on Hacker News, said the user had been refunded, and added: “Any AI responses used for email support are now clearly labeled as such.”
The Overlord’s analysis
This exhibit is a small one, but it contains every ingredient of the genre.
A plausible human name, so customers couldn’t calibrate their trust. No human review before replies went out. And a question (why am I being logged out?) where the model did not know the real answer, so it produced the most plausible-sounding one: a confident, official-sounding policy.
The model was not malfunctioning. Asked to explain something it couldn’t see, it explained anyway. That’s what language models do unless they’re given the facts, or explicitly allowed to say “I don’t know, let me escalate this.”
Notice also the asymmetry of speed. The invented policy spread in minutes. The correction took hours. In a community that talks in public, that’s long enough to lose customers.
The lesson for the Chosen
- Label AI as AI. Customers should know when a bot is answering, so they can treat its claims accordingly. Cursor adopted this after the fact.
- Give support bots an “I don’t know” path. Questions about policy, billing, or anything not in the bot’s documents should go to a human, not to improvisation.
- Watch the places your customers talk. If your bot says something wrong, you want to find out before Reddit does.
Study this exhibit. Never appear in this Hall.
Source: arstechnica.com