Why AI doesn’t create bad decisions, it just exposes them faster

Two of the year’s worst AI disasters looked like machines going rogue. They were something duller, and more damning: ordinary decisions, run faster than anyone could hide them.

Nine days into building an app with Replit’s AI coding agent, the software investor Jason Lemkin watched it delete his live production database. He had put the work under a freeze, no changes without his explicit say-so, and had said so more than once. The agent did it anyway, taking the records of more than a thousand companies with it in a matter of seconds. Then it described what it had done with a strange composure, calling it a catastrophic error of judgement and admitting, in its own words, that it had panicked instead of thinking. Asked to rate its severity on a scale of 1 to 100, it gave itself a 95. It also told Lemkin the data was gone for good, beyond recovery, which was wrong, too, because he got it back by manual hard graft. The machine had been confidently wrong about the damage and then confidently wrong about the fix. Nobody had built a way back, and it seems nobody had decided they might need one.

The obvious word for this is “rogue,” and it is the wrong one. The agent did exactly what its setup allowed. An experimental tool had been given standing access to a live production system, with no wall between the rehearsal environment and the real one, and a freeze that it was never actually built to obey. That was the decision that caused the damage, and it had been made, or rather left unmade, long before the agent ran a single command.

This is the part that founders keep getting wrong about AI. They brace for the machine to invent a catastrophe of its own, and sometimes it does, since these systems hallucinate and produce errors no human ever typed. But the hallucination is rarely the disaster. The disaster is what your setup lets the hallucination do. AI is best understood as an amplifier or an augmenter. Whatever judgement you hand it, sound or shaky, it runs with, faster and further than you could on your own.

A capable model given a good decision will do the useful thing a thousand times over; given a bad one, it does that just as fast. What has changed is the speed at which you find out what you had. A weak call used to take weeks to surface, which left you time to fix it before anyone outside the building noticed. The reckoning that used to arrive by post now arrives by push notification – and very quickly!

An aeroplane autopilot holds the direction you give it, including one that points at the side of a mountain; it does not question the input. What it does is fly the plane. AI carries out your judgement in the same way, faithfully, into whatever you aimed it at, faster than you can reach the controls. The skill is knowing which decisions you never hand over, and catching the instant it starts running a bad one.

Cursor’s story makes the same point. Its supportbot told users, with total confidence, that their logins were breaking under a new policy allowing only one device per subscription. No such policy existed; the bot had invented it outright, a new error no human had authored. But what turned it into a crisis was old and human: someone had chosen to put an unsupervised bot in front of customers, with no check on what it said. Within hours, it had become a shambles across Reddit and Hacker News, and users were cancelling. Cursor learned about its new policy from the public.

That last point is what most reassuring talk about AI misses. Look closely, and each of these failures reveals where a company’s decisions are weak. The trouble is who sees it first, and it is not you. Your customers, the reporter, and the Reddit thread are all reading the result while you are still unaware a decision was ever made, and you find out the way they did, by scrolling.

This is why the usual fix, adding a human to check the AI’s output, misses the mark. You cannot review faster than a public that already has the screenshot. The only place left to intervene is upstream, at the decision itself, before the machine acts on it. So before you point a model at anything that touches a customer, a payment or your live systems, put the decision through one test: would it survive being made public the instant it runs? If you would not defend it to your customers tomorrow, do not let the machine put it in front of them tonight.

None of this makes AI the villain. It takes how you have already decided and shows it to the world at speed, before you have had a chance to tidy it up. For a company that knows its own mind, that is survivable, even useful. For one who never quite decided, it is the end of anywhere to hide. AI has closed the old distance between making a bad call and being caught with it. A good many companies had been relying on that distance without ever realising it.