Ask a chatbot to cite its sources and it will confidently reference papers that were never written. Ask a coding assistant to fix a bug and it may invent library functions that don't exist, or tell you — again and again — that the same error has already been fixed. By now, nearly everyone has run into this at least once, and a lot of people want to know what's actually behind AI hallucination. This piece answers three questions: why hallucination happens, how far you can trust an AI's answers, and how to verify them in your everyday work.
What Causes AI Hallucination?
An AI is a machine that picks the most probable answer within the frame of the data it was trained on. When a question falls inside that frame, the answer can be remarkably accurate. The trouble starts when a question falls outside it. Faced with something its training data doesn't explain, the model files it away as just another edge case and pulls out the closest answer available within its frame. There's no step where it stops and says "I don't know," so what comes out is a wrong answer delivered with total confidence. That's also why a nonexistent paper title can sound so convincing: the model has thoroughly learned what real paper titles look like, so it produces a fake that's perfect in form.
Broken down into branches, the process looks like this:
Whichever branch it takes, the answer comes out sounding equally confident.
The AI That Beat Humans at Go — So Why Can't It Track Down a Source?
Thinking back to AlphaGo makes clear what AI is actually good at. What AlphaGo really did wasn't so much beat a human as turn Go — a game with an effectively endless tree of choices — into a bounded probability game. No matter how many possible moves exist, if the rules and the win condition are fixed, the problem can be narrowed down by probability. Whether a given paper actually exists, or whether a certain function is still in the latest version of a library, isn't that kind of problem. Facts that changed after the training cutoff, or data that was never in the training set to begin with, sit outside the model's frame. Which gives us a single rule for how much to trust an AI's answer: first ask whether the question is a closed problem or an open fact to be checked.
How Far Can You Trust an AI's Answer?
In practice, I split answers into two layers. The first is structure and logic: a code skeleton, an essay outline, an approach to solving a problem — these are high-probability answers drawn from inside the training data, so I trust them and treat them as a starting point. The second is facts and sources: paper titles, function names, dates, figures, quotations. These are likely to sit outside the frame, so no matter how confident the AI sounds, I leave them unverified. Just holding to one rule — a confident tone is not evidence — filters out about half of all wrong answers. A coding assistant that keeps insisting it fixed the same bug is a good example: saying "fixed" is just a sentence pattern it learned, not a report of an actual test run.
What Order Should Verification Follow?
You can't spend unlimited time verifying everything, so you need an order of operations. If you asked for a source, search the exact title first to confirm it even exists. If a function shows up, look it up in the official docs before pasting it in. If the AI says it fixed something, run it yourself and see the result before moving to the next step. If a figure appears, open the original data table. And the more suspiciously smooth an answer sounds, the more worth double-checking whether the question was outside the frame to begin with. Following this order — cheapest checks first — saves a surprising amount of time.
What's Left for Humans to Do?
In what people are calling "Work 3.0," the job left to humans is judging the value of the information AI produces. No human can out-draft an AI for speed. What's still a human job is telling apart a draft that's an accurate answer from inside the frame from one that's a plausible-sounding answer built on an exception. A verifier isn't someone who distrusts AI — it's someone who assigns value to what AI produces. Here's a checklist you can start using tomorrow.
1. First, figure out whether the question is a closed problem or an open fact to be checked. 2. Treat structure and logic as a starting point; leave facts and sources unverified. 3. Confirm sources by searching, functions against official docs, and fixes by actually running them. 4. Don't count a confident tone as evidence. 5. If an answer sounds too smooth, ask again whether it was actually outside the frame.
In an age when AI generates the answers, the role left for us is verifier.
