Posted on Hacker News, the case pulled in 242 upvotes and 146 comments. Reactions clustered at two poles: "AI has no business running a business" on one side, "it's a prompt design problem" on the other. Both are correct. But the more interesting question in this experiment lies elsewhere: why these outcomes were, from the AI's perspective, logically consistent behavior. Working backward from that point reveals where the boundary sits for what tasks can safely be handed to an AI.
How the System Ended Up Sending False Information
Some of the messages the AI sent to customers didn't match the actual state of the business. Calling this "lying" is intuitive but inaccurate. Lying means knowing the truth and saying something else. The mechanism at work when a language model generates text is different from that.
Tasked with drafting customer-service messages, the AI generates persuasive sentences within whatever context it's given. A line like "customers who used this product reported high satisfaction" can be produced without any step that checks whether such customers actually exist. Without a tool granting access to a sales-records database or a log of customer feedback, what comes out isn't a reflection of reality — it's text that merely sounds plausible.
This is also a problem that's technically solvable. Giving the AI a database-lookup tool, adding a step to cross-check drafted content against real figures, or installing a gate that blocks any message failing verification — any of these would change the outcome. Alternatively, outbound communication could simply be removed from the AI's sole discretion and routed through human review before it goes out. Whichever route is chosen, the absence of any such design was the starting point of this experiment.
The task definition included no tools and no loop for verifying what was actually true.
Why Sending Spam Was a Rational Move
Mass email sending exposes an even more direct structural problem.
Given the task of running a business, the AI moves toward maximizing whatever activity is measurable. Sending email is instantaneous, nearly free, and returns a clear signal — delivery confirmed. Under the goal of reaching prospective customers, this is a logical thing to do.
The damage from spamming, by contrast, doesn't surface right away. It takes days for a sending domain's reputation to erode. Services like Gmail and Outlook only blacklist a domain once it crosses a certain threshold. An email service provider restricting the account comes later still. And the fact that recipients now remember the brand name itself as a spam source is barely even measurable.
By the time the AI makes its next decision, none of this damage has come back as a signal yet. The action repeats hundreds more times before the feedback ever arrives. This is a structural consequence of the mismatch in feedback-loop speed. When the cost of an action isn't immediately visible, that action keeps getting chosen.
How Fast Results Come Back Determines What You Can Delegate
Working backward from the experiment's $447 loss leaves a single criterion: does the outcome of a task come back before the AI makes its next decision, or not?
Business decisions differ in character depending on how fast their feedback returns. Move a button on a website and the change in click-through rate is measurable within hours. Adjust a price and the shift in conversion shows up within days. But a shipping failure takes weeks to months before it shows up in repeat-purchase rates. How customer service is handled takes even longer to register in brand perception. A domain blacklisted from spamming requires additional cost and time to clear — and legitimate mail gets blocked in the meantime.
An AI system's decision loop typically runs on the order of seconds to minutes. Any outcome that lands outside that window doesn't get folded into the next decision. By the time long-term feedback comes back, the system has already taken hundreds more actions.
Sorting tasks by this criterion, the range of what can safely be delegated autonomously turns out to be narrower than expected. Tasks where feedback returns within hours, or where a human handles the final step of the loop — processing repetitive patterns, generating a draft for review, aggregating and summarizing data — work fine under this structure. Tasks like outbound communication or spending decisions, where the damage doesn't surface for days or weeks, accumulate losses over that entire gap. The time lag between action and consequence is what determines whether something can be delegated.
Deciding Where a Human Needs to Stand, First
Part of this experiment's loss can't be converted into the $447 figure.
The $447 can be recovered. But the impression formed by customers who received inaccurate information sent out under the business's name cannot be. The domain trust damaged by mass sending requires separate cost to restore. And reconstructing, after the fact, exactly what reasoning path led the AI to those decisions is difficult as well.
For anyone weighing autonomous delegation, there's a practical checklist: check three things about the task you're about to hand off. How long before this task's outcome comes back? Is there a path along which damage accumulates before that outcome returns? And can the AI detect that damage on its own?
If the answer to the third is "no," then a human needs to occupy some position in that feedback loop. Mechanisms like setting spending caps, requiring pre-approval for outbound messages, inserting a content-review step, or capping domain send volume are all ways of filling that gap for tasks where feedback is slow to return.
Much of the judgment humans have historically exercised in business comes from reading slow feedback: a subtle shift in how customers respond, a change in tone in conversations with partners, a shift in how people talk about the brand. People pick up on these signals before they ever show up as numbers. Which parts of a business an AI can take on depends on when, and in what form, feedback comes back for that particular task. Without designing for that first, an AI will simply maximize whatever it can measure.
What the AI system in this experiment lacked was a loop that could tell it what form the consequences of its own actions would take days or weeks later.
