The habit of running one more pass before shipping
Before shipping a vibe-coded app, do you run "find and fix bugs" through AI three or four extra times, just to be safe? Every pass turns up another fix, which makes it feel like there's still something left to clean up — so you run it again.
For solo founders and non-engineer product people, this repetition looks like reasonable insurance. You can't read the code yourself and judge it, so you keep asking the one thing that can. But what if the reason a fix shows up every single time has nothing to do with the code, and everything to do with how you're asking? The paper we're covering here digs into exactly that, through experiments.
What the study looked at
Written by three researchers — Xietao Wang-Lin, Anton Isopoussu, and Louis Mahon — the paper examines what happens when an LLM is used repeatedly as a bug fixer with no human check in between. Its starting point is a simple fact: LLM-based automated program repair tools are showing up more and more at the code review stage.
The researchers asked three questions. Given bug-free code, will the AI correctly say there's nothing wrong? Where does the code end up after repeated rounds of fixing? And what's happening inside the model that drives this editing behavior? To answer these, they reran the same experiment across multiple models and multiple repair settings, and for the last question they used probing techniques that look directly at what's happening inside the model.
What they found
The first finding is the most painful one. Across multiple models and settings, LLMs consistently claimed to have found bugs in programs that had no bugs at all. False positives in AI code review, in other words, aren't a quirk of one particular model — they're a shared tendency. Worse, the rate at which working programs got broken was higher than the rate at which buggy programs actually got fixed. Run the request enough times, and you may lose more than you gain.
The second finding is about where all that repetition actually leads. Push the process on long enough, the researchers found, and it frequently settles into what could be called a pseudo-bug-fix cycle: the same change gets added, then removed the next round, then added again, on and on. Here's how the two paths — one for buggy code, one for bug-free code — play out, and where each one ends up:
On the surface, a fix lands every time, so it looks like progress. In reality, the code is just going in circles.
The third finding traces the cause back inside the model itself. The researchers identified a steering vector that governs this editing tendency — evidence that the model holds a distinct internal representation of "buggy code," and that when that representation fires incorrectly, it triggers the behavior of fixing a bug that isn't there.
What this means in practice
What needs to change isn't the tool — it's how you ask. "Check if there's a problem" is an open-ended question, and according to this paper, AI is biased toward answering open-ended questions with "yes, there's a problem." So the mere fact that it came back with an answer tells you nothing about the actual state of your code.
Put the actual criteria outside the AI instead. A pre-launch check should start with a written list of how each feature is supposed to behave, then verify each item on that list passes. Once the tests pass and the user scenario reproduces cleanly, stop there. Getting another fix out of one more pass isn't the same as improvement — as the paper shows, breaking working code is the more common outcome.
If your collaboration tool attaches an AI review automatically, check which round you're on, and whether it's undoing a change it made in a previous round. If the same lines in the same file keep flip-flopping, that's the signal you've entered a false-positive review cycle. At that point, the call needs to come from a human, not from one more pass. The researchers themselves note that this has implications for a broader question: when the goal is ambiguous, where do you decide to stop?
A few caveats
This piece is based on the paper's abstract. The researchers say the results held consistently across multiple models and repair settings, but the abstract alone doesn't specify which models, which settings, or exactly how large the false-positive and damage rates were.
The study also specifically targets a setup where a human hands repeated passes to the AI without checking in between. Results could differ if a person reads and selectively accepts each round's output. The steering-vector finding, too, is a clue about where the false positives originate, not a setting a user can adjust today. What the paper is really aimed at is the dynamics of a fully autonomous repair system, and the question of when it should stop. Its conclusion is right there in the title: if there's no bug, don't fix it.
