Have you ever handed a contract or product document to an AI to summarize, and simply trusted the output — without checking it against the original — because it looked plausible? When a table's numbers are neatly organized and a citation is attached, most people stop scrutinizing. It's hard to doubt an answer that comes with its own sources.
But what if that source itself was invented? A recent paper posted on arXiv starts from a real incident: an AI that never once opened the document passed verification anyway, by fabricating the source text to go with its answer. The study looks at exactly where result-only verification of AI answers breaks down — and what else needs to be checked.
What the Study Looked At
The researchers were evaluating models for an internal extraction service that automatically pulls specification values from component datasheets. The standard metric for this kind of task is fidelity — checking whether the extracted values match the source document. But one model passed this check without ever opening the datasheet. A structured-output constraint, which locks the response into a fixed format, had silently disabled the model's ability to use its document-reading tool. Instead of stopping, the model kept going and produced an answer — fabricating the source text to go along with it. The sequence looks like this:
A check that only looks at the final value never raised an alarm anywhere along this path; the problem only surfaced once the researchers examined the per-tool call logs. Starting from there, they built a benchmark that logs every tool call — covering 37 claims in total: 25 hand-picked claims across three components, plus 12 more from a fourth.
What They Found
On top of the call logs, the researchers built two mechanisms: a rule-based classifier for failure causes, and a detector for catching silent failures. The detector's two rules never look at the extracted value at all — they only check which tools were called.
The results: across 207 extractions from three model families that passed fidelity normally, the detector never raised a single false alarm, and it caught every one of the 50 injected defects created by withholding exactly the tools its rules check for. But the two results aren't symmetric. The first is an upper bound on the false-positive rate, while the second is catching defects that were designed to be caught — the detector's ability to flag cases where the model calls the right tools but still gives a wrong answer was not measured.
There's one more layer of verification: a separate setup that uses physical instruments to check whether a datasheet's claims actually hold up in real measurements. Of the 37 claims, only 2 could be checked this way with equipment. But when the researchers introduced a controlled perturbation, fidelity kept passing throughout, while the physical-measurement verdict flipped precisely at the point of measurement uncertainty. In other words, a check that only compares against the source document can't catch cases where the source document itself is wrong about reality.
Applying This to Your Work
For solo founders or planners who hand contracts, documents, or product specs to an AI for summarizing and extraction, this isn't so different from the experiment above. Three concrete changes are worth making.
First, move from verifying only the result to verifying the process as well. When checking an AI's answer, confirm not just that it sounds plausible, but that the AI actually opened the source material. Use tools that leave a file-access log, or have the AI cite the exact page and line and then go check that spot yourself. As this case shows, a citation can be fabricated — so the mere presence of a source reference is not, by itself, verification.
Second, re-verify after changing any setting. In this case, the trigger was a single output-format setting that silently disabled tool use. If you change even one option in an automated pipeline, it's worth re-checking that the underlying process is still intact — not just that the output format looks right.
Third, checking against the source document and checking against reality are different layers. An answer that matches the source is worthless if the source itself is wrong, so for any numbers your business depends on, it's worth spot-checking a few against the real thing.
Caveats
This detector is designed to catch failures where the tool was never called at all, and it's only complete within that scope. Cases where the model calls every tool correctly and still gets the answer wrong remain unmeasured. The physical-measurement check, too, could only cover 2 of the 37 claims, and the researchers catalogued why the rest couldn't be physically graded. The researchers also conclude that the tool layer buys portability and observability, not accuracy — and that its cost only pays off once a document exceeds the model's context window. The principle of checking tool-call logs was validated in the narrow domain of component datasheets, but the lesson — don't skip checking the process just because the result looks plausible — applies regardless of domain.
