Temperature zero is not a promise: one setup repeats, another does not

Temperature zero is supposed to mean deterministic. We ran it twice and got two different answers about that.

Run one: the small model that would not repeat

On Oct 12 we took Qwen2.5-0.5B-Instruct (Q4_K_M) on llama.cpp, build b11565, on a 2-core CPU box, and asked it the same question 100 times at temperature zero. Same prompt, same seed (42), same machine.

One request at a time: 100 runs, 1 unique output. All identical.

Four requests at a time: 100 runs, 16 unique outputs. The most common appeared 20 times. The one-at-a-time answer appeared 5 times.

Same prompt, same seed, same machine, same temperature. The only thing that changed was concurrency. The usual explanation is that batching changes the order of the floating-point arithmetic, and order changes the result. We did not test that explanation, so it stays our reading, not our finding.

The footage in the reel is a replay drawn from the recorded log: hashes, token counts, the order the answers finished, paced for the screen and labeled with the real running time. It is not a live screen recording.

Run two: the hosted model that held

On Oct 14 we ran a second, separate test: 120 short support-ticket labels (20 hand-written templates, 6 each) on Claude Haiku 4.5 at temperature zero, one at a time and 32 at a time. Every row logged. No label changed. Zero flips.

An earlier Oct 13 run also showed zero flips, but its rows were not saved, so we re-ran it with full row logging rather than ship a claim we could not recompute.

What this says, and what it does not

On the small local model, temperature zero repeated exactly for sequential requests and did not for concurrent ones. On the hosted model, temperature zero held for short classification labels at both concurrency levels.

Two setups, not a general rule. We did not test other models, other prompts, or other hardware. The honest version of “temperature zero is deterministic” is “temperature zero was deterministic on this setup, for this workload, at this concurrency.”

The pipeline question

If your pipeline assumes temperature zero means one answer, the assumption to check is not the temperature. It is the serving path: batching, concurrency, hardware, and everything between your prompt and the weights. Determinism is a property of the whole setup, not the parameter.

Testing that assumption is an eval. The LLM Pipeline Observability Pack ($45) is the template for running one. Link in bio.

This post started as an Instagram post →