Deterministic LLMs: Claims and Challenges

In this episode, we investigate the growing conversation around deterministic large language models (LLMs), models designed to always return the same output for the same input. We contrast this with the more common stochastic LLMs that rely on random sampling and parallel computation, making their outputs variable even with identical prompts.

We explore the engineering efforts aimed at reducing this variability, including recent claims by Thinking Machines about “batch invariance.” While it’s a step forward, some analysts argue it’s overstated as a singular breakthrough. The episode dives into the multiple causes of non-determinism, from floating-point arithmetic and system-level batching to architectural features like Mixture-of-Experts.

We also weigh the pros and cons of determinism. On the plus side: improved debugging, reproducible benchmarks, and greater trust in high-stakes applications like finance or medicine. On the downside: reduced creative output, increased computational overhead, and significant engineering complexity.

Ultimately, we ask: Is true end-to-end determinism a worthwhile goal—or just an ideal that forces too many trade-offs?