Most people think of AI accuracy as a matter of bias, systematic error. But Daniel Kahneman’s book Noise reminds us that error has two parents: bias and variance. Bias tilts your aim; noise scatters your shots. You can correct bias with calibration. Noise requires structure.
The Hidden Half of Error
If you ask three loan officers to score the same applicant, you expect similar results. You’re wrong. Kahneman and colleagues found wild differences, sometimes equivalent to shifting interest rates by entire percentage points. The average may look fine, but the inconsistency itself is costly. This is “noise”: unpredictable variation when conditions are supposedly identical.
Now replace the loan officer with a language model. Same prompt, same temperature, same model, different answer. The system looks rational but behaves like a committee of moods. Noise didn’t vanish; it migrated into silicon.
Observability: Seeing the Invisible
Human noise hides in judgment; AI noise hides in computation. Observability makes it visible. It’s the act of watching not just what a model outputs, but how that output came to be, its data lineage, feature drift, response entropy, and latent-space volatility.
In engineering terms, observability is the nervous system of a deployed model. It logs the pulse of predictions over time. When those predictions start oscillating for no good reason, the problem isn’t necessarily bias, it’s noise creeping into the circuit.
Kahneman’s insight reframes observability as a variance audit, not a moral audit. The question is no longer “Is the model fair?” but “Is the model consistent?” A fair system that rolls dice on every request is still unreliable.
Evals: The New Judgment Hygiene
Kahneman proposed “decision hygiene” for humans, procedures to clean judgment before it festers into bias or chaos. In AI, we have “evals.” They serve the same function: they disinfect decision-making by standardising how we test, compare, and score models.
Three lenses emerge:
- Level: Is the model systematically off? (Bias)
- Spread: How unstable are its answers? (Noise)
- Pattern: What triggers those fluctuations? (Causal attribution)
Run a thousand identical prompts through an AI model and record the entropy of its responses. That curve is your noise signature. Most teams track accuracy; few track volatility. Yet volatility is what erodes trust when AI is embedded in live decision systems.
Order Without Illusion
Kahneman’s deeper lesson wasn’t about psychology; it was about systems. When a domain mixes uncertainty with judgment, medicine, law, finance, AI, you cannot eliminate error, but you can partition it. Once partitioned, you can design controls. Noise is not moral weakness; it’s structural looseness. Tighten structure, noise falls.
For AI, that means:
- Standardised prompt formats.
- Version-controlled model checkpoints.
- Seeded randomness for reproducibility.
- Ensemble averaging to smooth variance.
- Continuous monitoring for output drift.
Each practice is a small act of decision hygiene for machines.
The Convergence of Minds and Models
Humans and models fail for the same reason: they don’t know what they don’t measure. Kahneman’s framework turns AI observability into more than a dashboard, it becomes an epistemic tool. It lets us quantify the fuzziness of machine judgment with the same clarity we apply to human judgment.
Bias tells you where your aim is off. Noise tells you whether you even have a stable aim. Observability and evals are how you tell the difference. When you can measure both, you can finally trust the system, not because it’s perfect, but because its imperfections are visible.
Summary: Kahneman studied human inconsistency. AI inherits the same disease at scale. Observability is the cure’s engineering form; evals are its hygiene routine. Intelligence, human or artificial, becomes reliable only when it learns to silence its own noise.
