Generative AI like ChatGPT answers every question with confidence. The trouble is that some of those answers are convincing fabrications — what we call hallucinations. It's no longer rare to hear about someone who pasted an unverified AI answer into a report or a work document and paid for it later. This piece lays out exactly what AI can't do, why hallucination isn't a random slip but a structural outcome, and what humans need to keep hold of in the face of that limit — in terms you can apply at work.
What AI Can't Do Isn't a Bug. It's the Architecture.
Treat hallucination as a bug caused by insufficient performance, and your response will miss the mark. Generative AI is a machine that picks the most plausible combination from within the paradigm it already learned — the existing data it was trained on. That's why it's remarkably good at recombining what's already inside that paradigm, and unable to create anything genuinely new outside its training scope. Ask it about something it doesn't know, and it will still assemble a plausible-sounding answer within its existing framework. Hallucination isn't an exception to how the system works — it's the shadow cast by the mechanism itself.
AlphaGo Didn't Beat Humans
No example illustrates AI's limits better than AlphaGo. It didn't win by competing with humans on human terms — it simply reframed Go, a game of theoretically infinite choices, as a game of bounded probabilistic choices. Pulling a problem into a computable frame is the whole of what AI is good at, and also the boundary of what it can do. When AI is handed a question that doesn't fit that frame, it forces the question into the frame anyway and manufactures an answer. Most of the plausible-sounding wrong answers we run into come from exactly this move.
Spot the "Compulsion to Answer" and the Wrong Answers Become Visible
AI never stops to say "I don't know." Call this the compulsion to produce an answer no matter the question — an "answer compulsion." What's worth noting is that this compulsion isn't unique to AI; it's a broader problem of modern society. In an environment where being quick with an answer counts as performance, we too tend to lead with a response rather than verification, which is exactly why we end up accepting AI's confident wrong answers without filtering them out. The first step in identifying what AI can't do is to question both AI's compulsion and our own at the same time.
The Human's Job Is on the Question Side
An AI that finds answers to questions it's given cannot make art. The work that remains for humans is art-like work: setting up the question itself from scratch, stepping outside the existing paradigm to redefine the problem. In practice, that looks like this — hand the work of producing answers to AI, but keep the work of designing what to ask, verifying whether the answer that comes back is genuine or just an assembly within the frame, and reframing the question itself when the answer falls short, firmly in human hands.
A Checklist to Apply at Work Tomorrow
- Always verify against the original source before using any answer that includes proper nouns, figures, dates, or citations. - When you get an answer, sort out whether it's a verifiable fact or a plausible recombination of training data. - If an answer seems off, don't push the AI for a better one — redesign the question instead. - In planning or creative stages that call for genuine novelty, use AI to check against existing paradigms rather than as a first-draft generator. What AI produces is, after all, a map of what already exists.
To sum up: AI is a machine that assembles answers within a given frame, and building — or breaking — that frame is still a human's job. You can hand the answering over to AI. Just don't give up your seat as the one who asks the question.
