Ask ChatGPT or any generative AI anything, and it answers with confidence every time. The trouble is that some of those answers are plausible-sounding fabrications — what we call hallucinations. It's no longer rare to get burned by copying an unverified AI answer straight into a report or a work document. This piece lays out exactly what AI can't do, why hallucination isn't a random glitch but a structural outcome, and what practical ground humans need to hold in the face of that limit.
What AI Can't Do Isn't a Bug — It's the Architecture
Treat hallucination as a bug born of insufficient performance, and your response will miss the mark. Generative AI is a machine that picks the most plausible combination from within an existing paradigm — the data it was trained on. That's why it's remarkably good at recombining what's already inside that paradigm, and remarkably bad at producing anything genuinely new, anything outside its training. Ask it about something it doesn't know, and it still assembles a plausible-sounding answer within its existing frame. Hallucination isn't an edge case; it's the shadow cast by the very way the system works.
AlphaGo Never Actually Beat a Human
No example illustrates AI's limits better than AlphaGo. AlphaGo didn't win by competing on human terms — it simply reframed Go, a game of theoretically infinite choices, as a game of finite, calculable probabilities. Pulling a problem into a computable frame is both the full extent of what AI does well and the boundary of its limits. Feed it a question that doesn't fit inside that frame, and AI forces the question into the frame anyway and manufactures an answer. Most of the plausible-sounding wrong answers we run into come from exactly this.
You Have to Recognize the "Compulsion to Answer" Before You Can Spot the Wrong Answers
AI never simply stops and says it doesn't know. Call this condition — being forced to produce an answer to any question whatsoever — the "answer compulsion." What's worth noting is that this compulsion isn't unique to AI; it's a broader condition of modern society. In an environment where producing a quick answer is what gets rewarded as achievement, we too prioritize answering over verifying, which is exactly why we end up accepting AI's confident wrong answers without filtering them out. The first step in identifying what AI can't do is to question both AI's compulsion and our own at the same time.
The Human's Job Is on the Question Side
AI that searches for the answer to a given question cannot make art. The work of setting a question anew — the way art does — of stepping outside an existing paradigm's frame to redefine the problem, is what's left for humans. In practice, that means this: hand the work of producing answers to AI, but keep for yourself the work of designing what to ask, verifying whether the answer that comes back is just an assembly within the existing frame, and changing the frame of the question itself when the answers aren't good enough.
A Checklist You Can Apply to Tomorrow's Work
- Any answer containing proper nouns, figures, dates, or sources must be checked against the original source before you use it. - When you get an answer, work out whether it's a verifiable fact or a plausible recombination of training data. - If an answer feels off, don't push the AI for a better one — redesign the question instead. - In planning or creative stages where you need genuine novelty, use AI to check the existing paradigm rather than as a first-draft generator. What AI produces is, after all, a map of what already exists.
To sum up: AI is a machine that assembles answers within a given frame, and building — and breaking — that frame is still a human's job. You can hand the answering over to AI, but don't give up your seat as the one who asks the questions.
