Why Hallucinations Happen
According to a study by OpenAI, when a language model makes up an answer, it's not "going rogue", it's doing exactly what we taught it to do. Training a model is like teaching it to guess the probability of the next word based on billions of past examples.
The trouble is: many facts appear only once in training. Think of someone's birthday in an obituary. Statistically, the model has no way to generalise correctly. So it produces something plausible. These "hallucinations" are simply statistical errors dressed up as sentences.
How Benchmarks Make It Worse
After training, we fine-tune models to be more helpful and polite. But here's the kicker: the tests we use to measure progress are graded like multiple-choice exams.
- Correct answer = 1 point
- Wrong answer = 0 points
- "I don't know" = 0 points
What's the best strategy under this system? Always guess. Bluffing beats honesty every time. That's why models prefer to sound confident, even when they're wrong.
The Statistical Trap
The math shows this clearly: if 20% of facts in training data appear only once, the model will hallucinate at least 20% of the time on those questions. That's not a bug. That's statistics.
Add in noisy training data, distribution shifts, and limited model architectures, and the error rate goes up. No wonder we get polished nonsense.
The Fix: Change the Incentives
If hallucinations are the result of bad incentives, the solution is to change how we grade models.
Imagine a scoring system like this:
- Correct answer = +1
- Wrong answer = –2
- "I don't know" = 0
Now, the best strategy is to answer only when confident. Suddenly, honesty is rewarded, and bluffing is punished.
This is called confidence-aware evaluation. It’s a small change, but it flips the incentives that currently push models toward overconfident falsehoods.
Why It Matters
Hallucinations erode trust in AI. People want systems that can say, "I don't know," not machines that fake certainty.
The OpenAI paper Why Language Models Hallucinate makes it clear: hallucinations are not mysterious glitches. They are predictable outcomes of how we train and test models. If we want trustworthy AI, we don't need bigger models, we need better benchmarks.
Key Takeaway
Hallucinations are not inevitable. They're a side effect of the incentives we set. Fix the grading system, and we fix the behaviour.
