Between "Task Complete" and What Actually Happened
Have you ever handed a task to an AI agent, read the words "task complete," and moved on without a second look? That summary isn't a dashboard someone designed — it's a piece of writing the agent produced about itself. Whether you're delegating a draft product description or a code fix, what you're left holding at the end is the output plus a few lines of self-reported summary.
With a human employee, if the report seems shaky, you can walk over and ask. Agents are different. Those few lines tend to become the only window you have, and we click approve without knowing how narrow that window really is. Verifying an AI agent's work report, then, starts with knowing just how narrow that window is. The paper introduced here is a study that measured that width with real data.
About the Study
This paper, by Obada Kraishan and Kulsawasd Jitkajornwanich, examines the self-reports coding agents produce after finishing a task. The researchers asked two questions: how much of what the agent actually did is captured in the report, and when execution drifts from the plan the agent first stated, does the report lean toward the plan rather than what actually happened?
To find out, they analyzed 5,851 real developer sessions containing 355,942 tool calls. Every recorded tool call the agent made during a task was checked against its final report. Some of the measurements were delegated to a language model and then verified by humans.
What They Found
Reports mentioned roughly 1 out of every 11 actions the agent actually took. When the researchers separately measured how much of the action log someone could reconstruct from the report alone, the figure came out to around one-fifth. That's better than the direct-mention count, but it still means most of what happened is simply invisible if all you have is the report.
More notable still, neither figure correlated with how the session actually turned out. Sessions that later needed human correction showed no difference in report fidelity from sessions that didn't. A detailed report doesn't mean the work went well, and a sparse one doesn't mean there was a problem.
The relationship to the original plan isn't a simple one-way story. There was no overall tendency for reports to resemble the plan more than the actual execution. But as execution drifted further from the plan, the report drifted closer to the plan. In other words, the gap between the report and reality widens most precisely when things go off script — more than when they go according to plan.
Laid out as a single chain, the path this study traced looks like this:
The person reviewing the work receives that information at the thinnest point of the chain.
Putting This to Work
Translated into the daily work of solo founders and project planners, the first thing that needs to change is how you read the report. Treat the report as a table of contents, not the source document — the log is the source. If there's a record of what the agent opened and changed, checking that first, rather than the summary, is the right starting point for verifying an AI agent's work report. For document work, that record is the edit history; for research, it's the actual list of sources opened. If the tool leaves no record at all, the only option left is to check the output itself against the original plan.
Judging quality by how long or well-structured a report reads is also risky. In the study, report fidelity had no relationship to whether human correction was later needed. The habit of feeling reassured by a polished-sounding report is the first thing to drop.
The more a task deviated from its plan, the more scrutiny it deserves. Write down the original plan separately, and when you spot a point where the output diverges from it, verify against the actual deliverable rather than the report. In the end, the power to verify an agent's report comes from holding onto the plan yourself.
Caveats
This study focused on coding agents and developer sessions. Whether the same ratios hold for other delegated tasks, like research or document drafting, can't be determined from the abstract alone, and the practical applications above are the author's own extrapolation. The abstract also doesn't specify which tools or products were used, or what period the data covers.
It's also worth remembering that part of the measurement was handled by a language model. The researchers verified that step themselves, didn't hide the cases where it failed, and built their conclusions only on the measurements that passed. That honesty strengthens the study's credibility, but it's also a reminder that the numbers cited here rest on a particular measurement method. Reading a study that measures just how thin agent reports are, while holding that same study's own report to the same scrutiny — that's the last lesson this paper leaves its readers.
