Why What Worked Yesterday Fails Today
You've automated a repetitive task with n8n or Claude, only to have something that worked perfectly yesterday fail in some bizarre way today, forcing you to check the results by hand after all. You ask the agent to gather and summarize competitor news every morning, and some days it comes back with a tidy digest — other days it reads the wrong page entirely and rambles about something else.
When this keeps happening, you end up opening every result to check it yourself. You've automated the task, but all you've really done is add a new checking task on top of it. For a solo founder, the very tool built to save time ends up eating it instead.
A new paper tackles this problem head-on. Posted to arXiv by a research team led by Evelyn Duesterwald, it puts a name to the phenomenon of AI agents that look accurate on average yet remain hard to trust in day-to-day operation — and proposes a way to close that gap.
About the Study
The question the researchers asked is simple: how different is an agent's success rate on a single attempt from its success rate when the same task is run repeatedly and has to succeed every single time? Using the AppWorld benchmark, they had a GPT-4.1-based ReAct agent run the same task five times and separately measured the share of tasks that succeeded on all five runs. They call the difference between the average success rate and this "all-five-succeed" rate the consistency gap.
To close that gap, the researchers propose a self-evolving agent framework built around two components. The first is a consistency analyzer, which pinpoints which steps in an agent's execution trace tend to flip between runs — and why. The second is an instruction generator, which turns that diagnosis into a concrete instruction. The instruction is stored in episodic memory and then injected back into the agent's execution the next time it encounters a similar task. Here's the sequence, step by step, of how a wobbly step becomes an instruction stored in memory.
It's a loop: diagnose the point of failure, turn it into an instruction, and let the next run inherit that instruction.
What They Found
The first thing the researchers confirmed was the size of the gap. The GPT-4.1-based ReAct agent's average single-run success rate was 77%. But when the same task was run five times, only 53% of tasks succeeded on all five runs. The 24-point difference between those two numbers is the consistency gap. In other words, an agent that looks quite reliable on average can only be fully trusted a little over half the time when you hand it the same task five times in a row.
The researchers note that this gap has been observed before but never properly addressed, and argue that closing it is a precondition for deploying agents that can actually be trusted.
Applying the proposed framework raised the share of tasks that succeeded all five times by 16 points on the same tasks, and by 13 points when generalized to similar-but-different tasks. That doesn't close the gap entirely, but simply remembering which steps wobble and why closed a substantial part of it.
Putting This to Work
The biggest lesson here for solo founders and planners is to change the metric. Average success rate isn't a reliable way to measure how trustworthy an AI agent automation really is. The bar that actually matters for handing off work is whether the same task succeeds all five times you run it.
Before you attach a new automation, run it five times on the same input. Deploy it after seeing it work once, and you'll pay for that later in manual checking. If even one of those five runs goes wrong, the automation still needs human oversight.
When something fails, don't just look at the outcome — write down exactly where it went off track. That's you doing by hand what the paper's consistency analyzer does automatically. Once you can see which page it misread or under what conditions it took a different path, turn that into an explicit instruction in your prompt or workflow. This is what the paper's episodic memory looks like in practice. Build up the habit of turning wobbly points into instructions, and your agent automation's reliability climbs with every run.
Caveats
These results come from a single benchmark (AppWorld), a single agent architecture (ReAct), and a single model (GPT-4.1). Whether the same size of gap — or the same size of improvement — shows up with other models or other kinds of work can't be determined from the abstract alone.
The improvement is also limited. Of the 24-point gap, the framework closed 16 points on the same tasks, and that shrinks to 13 points on similar-but-different tasks. A gap still remains, so this isn't grounds for leaving automation completely unsupervised. The abstract also doesn't say what accumulating instructions costs, or what it does to execution speed.
Even so, the question this paper raises is clear: does your agent succeed all five times you run it? Simply asking that question before deployment means fewer mornings spent double-checking things by hand.
