# New arXiv paper fixes cross-GPU LLM nondeterminism with fused-upcast GEMM kernels

> An arXiv CS.LG preprint claims fixed-configuration fused-upcast GEMM kernels make LLM linear layers bitwise identical across NVIDIA Ampere, Ada, and Hopper GPUs.

- **Topic**: Research
- **Published**: 2026-09-24T05:31:04.767Z
- **Canonical URL**: https://highsignal.sh/stories/new-arxiv-paper-fixes-cross-gpu-llm-nondeterminism-with-fused-upcast-gemm-kernel-f67c00ce

## Why It Matters

LLM outputs from the same prompt can silently differ across GPUs because inference frameworks pick different matrix-multiplication kernels, whose parallel reduction orders and tensor-core rounding flip tokens; the paper frames this as reproducibility, not just performance.

## Key Findings & Analysis

### What happened

> An arXiv CS.LG preprint (arXiv:2609.25624v1) proposes fixing the floating-point reduction order of linear layers as a pure function of problem shape, so every GPU runs the same operation sequence. > The authors say this reduces cross-architecture reproducibility to correct IEEE-754 arithmetic rather than keeping rounding differences below a tie-flip threshold. > The paper attributes nondeterministic greedy decoding to floating-point non-associativity plus hardware-dependent kernel selection, and says prior solutions have imperfect cross-architecture reproducibility and a significant performance penalty.

### What the authors claim

> The authors report bitwise-identical linear-layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. > They claim 1.17x to 3.1x faster end-to-end performance than the state-of-the-art solution, and half the weight-memory traffic. > Method: fixed-configuration fused-upcast GEMM kernels load 16-bit weights, upcast to FP32 in registers, and accumulate under IEEE-754 in a device-independent reduction order. > These are the paper's own claims; no independent benchmark or third-party reproduction is in the supplied evidence.

### Why it matters

> The paper targets reproducibility of the linear layers specifically, so it does not by itself establish end-to-end bitwise-identical full-model output. > The claim is cross-architecture determinism for three NVIDIA generations; nothing in the supplied evidence covers other vendors' GPUs. > Practical relevance is reproducibility testing and debugging, where identical runs across hardware are otherwise not guaranteed.

## Primary Sources & Citations

- [Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures](https://arxiv.org/abs/2609.25624) — *ArXiv CS.LG (Machine Learning)* (Reporting)

---

[← Back to front page](https://highsignal.sh/) | [Daily Brief](https://highsignal.sh/brief) | [All stories](https://highsignal.sh/latest)