Nondeterminism in LLM Inference - Why AI Gives Different Answers to the Same Question

Why does an AI give different answers to the same question? Imagine you ask a chatbot, “Tell me about Richard Feynman.” You get a sharp, well-phrased response. A few minutes later you ask the exact same thing, expecting the same text, but the wording changes. The facts are fine, yet the phrasing shifts.

“Maybe it is being creative,” you think. But you set the so-called creativity setting, the temperature, to 0. That should push the model to choose the single most likely next word each time. Identical input, identical output. In theory.

In practice, it often differs. Let us play detective and find out why.

Clue 1: Parallel thinking on silicon

Modern AI runs on GPUs, which are like thousands of tiny calculators working at once. Parallel work raises a classic numerical quirk: floating-point non-associativity. In computer arithmetic with limited precision, the grouping of additions matters. In symbols: (a + b) + c != a + (b + c) can hold because rounding occurs after each step.

So maybe parallelism itself makes runs differ. Test it. Fix two matrices and multiply them again and again on the same GPU:

# Pseudocode
A = create_random_matrix()
B = create_random_matrix()

reference = A * B

for _ in range(1000):
    result = A * B
    assert result == reference  # bit-for-bit equality

Result: it passes. For a fixed operation on the same hardware and library stack, you get repeatable results. Parallelism alone is not the culprit. Our suspect has an alibi.

Clue 2: The original sin of computer maths

Floating-point numbers trade perfect precision for a huge dynamic range. That trade is powerful, but it introduces order sensitivity. With limited digits, a small number can vanish when added to a very large one. Consider:

(0.1 + 1e20) - 1e20  => 0.0
0.1 + (1e20 - 1e20)  => 0.1

Same inputs, different grouping, different output. We now know if the order of operations changes, the result can change. The real question becomes: what is changing the order when you send the same prompt twice?

Clue 3: The real culprit — batching

AI services are busy kitchens. To serve many people efficiently, servers group incoming requests into batches before sending them through the model. Think of baking pizzas ten at a time instead of one by one. It is faster and cheaper.

Here is the catch. Inside the model’s maths, certain reductions and normalisations depend on the size and composition of the current batch. That affects the exact order and pattern of floating-point operations. Since the arithmetic is not perfectly associative, a different batch can lead to tiny numeric differences. Those tiny differences can slightly shift the probability distribution over the next token. Over a long answer, those shifts can accumulate until one word changes, then another, and you end up with a different wording that still respects the facts.

Clue 4: Watching divergence in the wild

Engineers have run repeated queries to a large model with temperature set to 0. One such trial sent the same “Tell me about Richard Feynman” prompt a thousand times. For the first ±100 words the answers were identical, then a fork appeared:

• About 99.2% of responses continued with a phrase like “Queens, New York”.
• About 0.8% used “New York City”.

Nothing magical happened. The model remained “deterministic” in configuration, yet batching shifted the exact numeric pathway just enough for the next-token tie to break the other way on a small fraction of runs.

What temperature = 0 really guarantees

Temperature = 0 tells the sampler to pick the highest-probability token at each step. It does not guarantee that the underlying probabilities are bit-identical across deployments or time. If any of the following vary, outputs can diverge even at temperature 0:

• Batch size or composition during inference.
• Kernel selection or fused ops chosen by the runtime for efficiency.
• Library versions, low-level flags, or hardware micro-differences.

How this cascades into different wording

1. A tiny numeric nudge shifts the top two tokens from a near-tie to a flip at one position.
2. That one token slightly changes the context for the next step.
3. Downstream probabilities adjust, making later flips more likely.
4. After a few hundred tokens, you still have the same facts, but the phrasing has drifted.

This is sensitivity to initial conditions, not creativity. It is the sensible outcome of efficient serving plus floating-point reality.

Implications for builders and teams

• Require bit-stable outputs? Use offline single-example inference with fixed seeds, fixed batch size of 1, pinned library versions, and deterministic kernels where available. Expect a throughput penalty.
• Need consistent tone rather than identical text? Constrain style with system prompts, templates, and controlled generation parameters. Small numeric changes will then stay within your brand lane.
• Comparing A/B prompts? Run both prompts under identical serving conditions, repeat multiple times, and judge on factuality and task success, not bit sameness.

Conclusion: Not magic, just maths and crowds

The shifting answers you notice are not a ghost in the machine. They are a predictable consequence of two facts:

• Floating-point non-associativity means the order of operations can change results by tiny amounts.
• Batching for efficiency changes that order in real services, since your request rides alongside others.

Put simply, the system is deterministic about its rules, not always about its bits. Once you see that, the black box looks less like magic and more like practical engineering at scale.

Tip for students and teams: if you need the same exact paragraph later, store it. If you just need the same facts, focus on guardrails for accuracy and style. Let the service do what it does best, serve many people quickly, and design your workflow with that in mind.